Payload

Payload Module to implement functions to process Terrarium payload.

This code processes a Terraform (HCL) or YAML payload file that includes various settings to auto-configure a OpenText Content Management environment, including Content Server (OTCS), OTDS; AppWorks, CoreShare, M365.

• WebHooks (URLs) to call (e.g. to start-up external services or applications)
• OTDS partitions and OAuth clients
• OTDS trusted sites and system attributes
• OTDS licenses
• Content Server users and groups
• Microsoft 365 user, groups, and teams
• Salesforce users and groups
• SuccessFactors users
• Core Share users and groups
• Content Server Admin Settings (LLConfig)
• Content Server External System Connections (SAP, SuccessFactors, ...)
• Content Server Transport Packages (scenarios and demo content)
• Content Server CS Applications (typically based on Web Reports)
• Content Server Web Reports to run
• Content Server Workspaces to create (incl. members, workspace relationships)
• Content Server User photos, user favorites and user settings
• Content Server Items to create and permissions to apply
• Content Server Items to rename
• Content Server Documents to generate (from templates)
• Content Server Assignments (used e.g. for Government scenario)
• Content Server Records Management settings, Security Clearance, Supplemental Markings, and Holds
• SAP RFCs (Remote Function Calls)
• Commands to execute in Kubernetes Pods
• Browser Automations (for things that cannot be automated via an API)

This code typically runs in a container as part of the cloud automation.

Payload

Class Payload is used to process Terrarium payload.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

class Payload:
    """Class Payload is used to process Terrarium payload."""

    logger: logging.Logger = default_logger

    # _debug controls whether or not transport processing is
    # stopped if one transport fails:
    _debug: bool = False
    _otcs: OTCS
    _otcs_backend: OTCS
    _otcs_frontend: OTCS
    _otac: OTAC | None
    _otds: OTDS
    _otiv: OTIV | None
    _otpd: OTPD | None
    _otmm: OTMM | None
    _pht: None
    _nhc: None
    _otcs_source: OTCS | None
    _k8s: K8s | None
    _http_object: HTTP | None
    _m365: M365 | None
    _core_share: CoreShare | None
    _sap: SAP | None
    _successfactors: SuccessFactors | None
    _salesforce: Salesforce | None
    _servicenow: ServiceNow | None
    _guidewire_policy_center: Guidewire | None
    _guidewire_claims_center: Guidewire | None
    _custom_settings_dir = ""
    _otawp: OTAWP | None
    _otca: OTCA | None
    _otkd: OTKD | None
    _avts: AVTS | None

    # _payload_source (string): This is either path + filename of the yaml payload
    # or an path + filename of the Terraform HCL payload
    _payload_source = ""

    # _payload is a dict of the complete payload file.
    # It is initialized by the init_payload() method:
    _payload = {}

    """
    _payload_sections is a list of dicts with these keys:
    - name (str)
    - enabled (bool)
    - restart (bool)
    See below and method init_payload() for details of existing payload sections
    """
    _payload_sections = []

    # Declare options dict to read payloadOptions
    _payload_options = {}

    #
    # Initialize payload section variables. They are all list of dicts:
    #

    """
    _webhooks and webhooks_post: List of webHooks.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - description (str, optional)
    - url (str, mandatory)
    - method (str) - either POST, PUT, GET
    - payload (dict, optional, default = {})
    - headers (dict, optional, default = {})
    """
    _webhooks = []
    _webhooks_post = []

    """
    _resources: List of OTDS resources.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - name (str, mandatory)
    - description (str, optional)
    - display_name (str, optional)
    - activate (bool, optional, default = True) - if a secret is provided the resource will automatically be activated
    - allow_impersonation (bool, optional, default = True)
    - resource_id (str, optional, default = None) - a predefined resource ID. If specified, also secrethas to be provided
    - secret (string, optional, default = None) - a predefined secret. Should be 24 characters long and should end with '=='
    - additional_payload (dict, optional)
    """
    _resources = []

    """
    _partitions and _synchronized_partitions: Lists of OTDS partitions (for users and groups).
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - name (str, mandatory)
    - description (str, optional)
    - access_role (str, optional)
    - licenses (list, optional)
    """
    _partitions = []
    _synchronized_partitions = []

    """
    _licenses: Lists of OTDS Licenses to be added to a resource.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - path (str, mandatory)
    - product_name (str, mandatory)
    - resource (str, mandatory)
    - description (str, optional)
    """
    _licenses = []

    """
    _oauth_clients: List of OTDS OAuth Clients. Each element
    is a dict with these keys:
    - enabled (bool, optional, default = True)
    - name (str, mandatory)
    - description (str, optional, default = "")
    - confidential (bool, optional, default = True)
    - partition (str, optional, default = "Global")
    - redirect_urls (list, optional, default = [])
    - permission_scopes (list)
    - default_scopes (list)
    - allow_impersonation (bool, optional, default = True)
    - secret (str, optional, default = "") - option to provide a predefined secret
    """
    _oauth_clients = []

    """
    _oauth_handlers: List of OTDS OAuth handler. Each element
    is a dict with these keys:
    - enabled (bool, optional, default = True)
    - name (str, mandatory)
    - description (str, optional)
    - scope (str, optional, default = None)
    - type (str, mandatory) - handler type, like SAML, SAP, OAUTH
    - priority (int, optional)
    - active_by_default (bool, optional, default = False)
    - provider_name (str, mandatory for type = SAML and type = SAP)
    - auth_principal_attributes (list)
    - nameid_format (str, optional)
    - saml_url (str, mandatory for type = SAML)
    - otds_sp_endpoint (str, mandatory for type = SAML)
    - certificate_file (str, mandatory for type = SAP)
    - certificate_password (str, mandatory for type = SAP)
    - client_id (str, mandatory for type = OAUTH)
    - client_secret (str, mandatory for type = OAUTH)
    - authorization_endpoint (str, mandatory for type = OAUTH)
    - token_endpoint (str, optional for type = OAUTH)
    - scope_string (str, optional)
    """
    _auth_handlers = []

    """
    _trusted_sites: List of OTDS trasted sites.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - url (str, mandatory)
    """
    _trusted_sites = []

    """
    _system_attributes: List of OTDS System Attributes.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - name (str, mandatory)
    - value (str, mandatory)
    - description (str, optional)
    """
    _system_attributes = []

    """
    _system_attributes: List of OTPD Settings.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - description (str, optional)
    - name (str, mandatory)
    - value (str, mandatory)
    - tenant (str, optional)
    """
    _docgen_settings = []

    """
    _groups: List of groups.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - name (str, mandatory),
    - parent_groups (list, optional),
    - enable_o365 (bool, optional, default = False)
    - enable_salesforce (bool, optional, default = False)
    - enable_core_share (bool, optional, default = False)
    """
    _groups = []

    """
    _users: List of users.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - name (str, mandatory) (= login)
    - password (str, optional, will be generated if not provided)
    - firstname (str, optional, default = "")
    - lastname (str, optional, default = "")
    - title (str, optional, default = "")
    - email (str, optional, default = "")
    - base_group (str, optional, default = "DefaultGroup")
    - user_type (str, optional, default = "User") - possible values are "User" and "ServiceUser"
    - company (str, optional, default = "Innovate") - currently used for Salesforce users only
    - privileges (list, optional, default = ["Login", "Public Access"])
    - groups (list, optional)
    - favorites (list of str, optional)
    - security_clearance (int, optional)
    - supplemental_markings (list of str)
    - location (str, optional, default = "US") - only relevant for M365 users
    - enable_sap (bool, optional, default = False)
    - enable_successfactors (bool, optional, default = False)
    - enable_salesforce (bool, optional, default = False)
    - enable_o365 (bool, optional, default = False)
    - enable_core_share (bool, optional, default = False)
    - m365_skus (list of str) - only relevant for M365 users
    - extra_attributes (list of dict)
    """
    _users = []
    _user_customization = True

    """
    _admin_settings: list of admin settings (XML file to import).
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - description (str, optional)
    - filename (str, mandatory) - without path
    """
    _admin_settings = []
    _admin_settings_post = []

    """
    _exec_pod_commands: List of commands to be executed in the pods.
    list elements need to be dicts with pod name, command, etc.
    - enabled (bool, optional, default = True)
    - command (str, mandatory)
    - pod_name (str, mandatory)
    - description (str, optional)
    - interactive (bool, optional, default = False)
    """
    _exec_pod_commands = []

    """
    _exec_commands: List of commands to be executed in the customizer pod (as local process).
    Each list element need to be a dict with these keys:
    - enabled (bool, optional, default = True)
    - command (str, mandatory)
    - description (str, optional)
    """
    _exec_commands = []

    """
    _exec_database_commands: list of database command sets to be executed.
    Each list is a dict with these keys:
    - enabled (bool, optional, default = True)
    - db_connection (dict, mandatory) - supported dictionary keys:
        - db_name (str, mandatory) - name of the database
        - db_hostname (str, mandatory) - hostname of the database server
        - db_port (int, optional) - port to communicate to the database server; default is 5432
        - db_username (str, mandatory) - username
        - db_password (str, mandatory) - password
    - db_commands (list, mandatory) - each list item is a dictionary with these keys:
        - command (str, mandatory) - needs to have %s paceholder for each parameter
        - params (list, optional) - parameter values to be inserted into the %s postions in the command
    """
    _exec_database_commands = []

    """
    external_systems (list): List of external systems.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - external_system_type (str, mandatory) - possible values: SuccessFactors, SAP, Salesforce, Appworks Platform, Business Scenario Sample
    - external_system_name (str, mandatory) - for SAP this is the System ID
    - external_system_number (str)
    - description (str)
    - as_url (str, mandatory)
    - base_url (str, optional, default = "")
    - client (str, optional - only relevant for SAP, default = 100)
    - destination (str, optional - only relevant for SAP, default = "")
    - group (str, optional - only relevant for SAP, default = "PUBLIC")
    - username (str, optional - depends on external_system_type)
    - password (str, optional - depends on external_system_type)
    - certificate_file (str, optional - only relevant for SAP, used for Auth Handler)
    - certificate_password (str, optional - only relevant for SAP, used for Auth Handler)
    - external_system_hostname (str, mandatory - only relevant for SAP)
    - archive_logical_name (str, optional - only relevant for SAP)
    - archive_certificate_file (str, optional - only relevant for SAP)
    - oauth_client_id (str, optional)
    - oauth_client_secret (str, optional)
    - skip_connection_test (bool, optional, default = False)
    """
    _external_systems = []

    """
    _transport_packages (list): List of transport packages systems.
    Each list element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - name (str, mandatory)
    - url (str, mandatory)
    - description (str, optional, default = "")
    - replacements (list, optional, default = None)
    - extractions (list, optional, default = None)
    """
    _transport_packages = []
    _content_transport_packages = []
    _transport_packages_post = []

    # _business_object_types (list): Business object types are not in payload
    # but retrieved from transport package:
    _business_object_types = []

    # _workspace_types (list): Workspace types are not in payload but imported with transport package:
    _workspace_types = []

    """
    _workspace_templates (list): actually these are also imported via transport
    but used if we want to define standard members on template basis.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - type_name (str, mandatory)
    - template_name (str, mandatory)
    - members (list, optional)
      * role (str, mandatory)
      * users (list, optional, default = [])
      * groups (list, optional, default = [])
    - categories (list, optional)
      * nickname (str, optional) - the nickname of the category
      * path (list, optional) - the path to the category object in the category volume
      * inheritance (bool, optional) - should category inheritance be turned on on the workspace template node?
      * apply_to_sub_items (bool, optional) - should the category be inherited to existing sub-items?
    """
    _workspace_templates = []

    """
    _workspaces (list): list of Content Server business workspaces.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - id (str, mandatory) - logical ID of the workspace - used for cross-referencing inside the payload. This is NOT this node ID!
    - name (str, mandatory)
    - description (str, optional, default = "")
    - type_name (str, mandatory)
    - template_name (str, optional, default = just take first template)
    - business_objects (list, optional, default = [])
    - parent_id (str, optional, default = None) - this is a LOGICAL ID used in the payload - not the node ID!
    - parent_path (list, optional, default = None)
    - categories (list, optional, default = None)
      * name (str, mandatory)
      * set (str, default = "")
      * row (int, optional)
      * attribute (str, mandatory)
      * value (str, mandatory)
    - nickname (str, optional, default = ignore)
    - photo_nickname (str, optional, default = ignore)
    - rm_classification_path (list, optional, default = [])
    - classification_pathes (list of lists, optional, default = [])
    - members (list, optional, default = [])
      * role (name)
      * users (list, optional, default = [])
      * groups (list, optional, default = [])
    - relationships (list, optional, default = []) - list of related workspaces.
      The elements of the list can be:
      * string or integer with logical workspace ID
      * string with nickname of the related workspace
      * dictionaries with keys "type" and "name" of the related workspace
      * list of strings with the top-down path in the Enterprise volume
    """
    _workspaces = []

    """
    _sap_rfcs (list) and _sap_rfcs_post (list) include distinct lists of SAP RFC calls.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - description (str)
    - parameters (dict)
    - call_options (dict)
    """
    _sap_rfcs = []
    _sap_rfcs_post = []

    """
    _web_reports (list) and _web_reports_post (list) include all web report payload definitions.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - nickname (str, mandatory)
    - description (str, optional, default = "")
    - restart (bool, optional, default = False)
    - parameters (dict, optional, default = {}) - the dict keys are the parameter names and the dict values are the actual parameter values
    """
    _web_reports = []
    _web_reports_post = []

    """
    _cs_applications (list): List of Content Server Applications to deploy.
    Each list element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - name (str, mandatory)
    - descriptions (str, optional, default = "")
    """
    _cs_applications = []

    """
    _additional_group_members: List of memberships to establish.
    Each element is a dict with these keys:
    - parent_group (string)
    - user_name (string)
    - group_name (string)
    """
    _additional_group_members = []

    """
    _additional_access_role_members: List of memberships to establish.
    Each element is a dict with these keys:
    - access_role (string)
    - user_name (string)
    - group_name (string)
    - partition_name (string)
    """
    _additional_access_role_members = []

    """
    _additional_application_role_assignments: List of assignments to establish.
    Each element is a dict with these keys:
    - role (string)
    - user_name (string)
    - group_name (string)
    """
    _additional_application_role_assignments = []

    """
    _renamings (list). List of items to be renamed.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - name (str, mandatory)
    - nodeid (int, mandatory if no volume is specified) - this is the technical OTCS ID - typically only known for some preinstalled items
    - volume (int, mandatory if no nodeid is specified)
    - path (list, optional) - can be combined with volume - to specify a top-down path in the volume to the item to be renamed
    - nickname (str, optional) - the nickname of the node to rename - alternative to to volume/path or nodeid
    """
    _renamings = []

    """
    _items: List of items to create in Content Server
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - parent_nickname (str)
    - parent_path (list)
    - name (str)
    - description (str, optional, default = "")
    - type (str)
    - url (str) - "" means not set
    - original_nickname
    - original_path (list)
    """
    _items = []
    _items_post = []

    """
    _permissions: List of permissions changes to apply
    Each element is a dict with these keys:
    - path (list, optional)
    - volume (int, optional)
    - nickname (str, optional)
    - owner_permissions (list, optional)
    - owner_group_permissions (list, optional)
    - public_permissions (list, optional)
    - groups (list, optional)
      + name (str)
      + permissions (list)
    - users (list, optional)
      + name (str)
      + permissions (list)
    - apply_to (int)
    """
    _permissions = []
    _permissions_post = []

    """
    _workspace_permissions: List of workspace permissions changes to apply.
    Each element is a dict with these keys:
    - workspace_folder (str, optional)
    - workspace_type (str)
    - regex (bool, optional)
    - owner_permissions (list, optional)
    - owner_group_permissions (list, optional)
    - public_permissions (list, optional)
    - groups (list, optional)
      + name (str)
      + permissions (list)
    - users (list, optional)
      + name (str)
      + permissions (list)
    - apply_to (int)
    """
    _workspace_permissions = []

    """
    _assignments: List of assignments. Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - subject (str, mandatory)
    - instruction (str, optional)
    - workspace (str, optional if nickname is specified)
    - nickname (str, optional if workspace is specified)
    - groups (list)
    - users (list)
    """
    _assignments = []

    """
    _doc_generators: List of document generators that use the Document Template capabilities of Content Server.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - workspace_type (str, mandatory)
    - template_path (list, mandatory)
    - classification_path (list, mandatory)
    - category_name (str, optional, default = "")
    - workspace_folder_path (list, optional, default = []) - default puts the document in the workspace root
    - exec_as_user (str, optional, default = "")
    """
    _doc_generators = []

    """
    _workflows: List of workflow initiations inside workspace instances of a workspace type
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - worklow_nickname (str, mandatory) - the nickname of the workflow
    - initiate_as_user (str, mandatory) - user that initiates the workflow
    - workspace_type (str, mandatory) - for each instance of the given workspace type a workflow is started
    - workspace_folder_path (list, optional) - the subfolder that contains the document the workflow is started with
    - attributes (list, optional) - the list of attributes (name, value) the workflow is started with
    - steps (list)
      * action (Initiate, )
      * exec_as_user
      * attributes
        - name
        - value
        - type
    """
    _workflows = []

    """
    _browser_automations: List of browser automation for things that can only be
    automated via the web user interface. Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - name (str, mandatory)
    - description (str, optional)
    - base_url (str, mandatory)
    - user_name (str, optional)
    - password (str, optional)
    - wait_time (float, optional, default = 30.0) - wait time in seconds
    - wait_until (str, optional) - the page load / navigation `wait until` strategy. Possible values: `load`, `networkidle`, `domcontentloaded`
    - debug (bool, optional, default = False) - if True take screenshots and save to container
    - automations (list, mandatory)
      * dependent (bool, optional) - decide if current automation step is dependent on the previous step. Default is True. If dependent = True and previous step failed this step is skipped.
      * type (str, optional, default = "") - possible types: `login`, `get_page`, `click_elem`, `set_elem`, `check_elem`
      * page (str, optional, default = "") - the page-specific part of the URL. Will be concatenated with the `base_url`
      * selector (str, optional, default = "")
      * selector_type (str, optional, default = "id") - the find strategy - either `id`, `name`, `css`, `xpath`, `role`, `text`
      * role_type (str, optional, default = "") - the ARIA role of an element. Only relevant for selector_type = `role`.
      * scroll_to_element (bool, optional, default = true) - scroll to the element before clicking it - for type `click_elem` only. Should actually not be necessary as locators should scroll automatically.
      * value (str, optional, default = "") - the new value of element. Relevant for type = `set_elem`.
      * user_field (str, optional, default = "") - the name of the HTML field holding the user name - only for type `login`.
      * password_field (str, optional, default = "") - the name of the HTML field holding the password - only for type `login`.
      * iframe: name of the iframe if the elem is inside an iframe
      * press_enter: simulate pressing the "Enter" key after setting the elem value - for type set_elem only
      * typing (bool, optional) - deciding if to simulate keyboard input - this is required for many type-ahead fields to work - for type `set_elem` only
      * exact_match (bool, optional) - deciding if the element should be identified with an exact match
      * hover_only (bool, optional) - deciding if instead of clicking the element only a mouse over hovering should be simulated - for type `click_elem` only
      * regex (bool, optional) - deciding if the selector should be treated as a regular expression - for type `set_elem`
      * wait_until (str, optional, default = "") - an automation-step specific value for `wait_until` (see above)
      * wait_time (int, optional) - number of seconds to explicitly wait for an element to appear - for type `click_elem` only - for edge cases only
      * volume (int, optional, default = 141 (Enterprise Volume)) - the OTCS volume ID. Only relevant for type = `get_page`.
      * path (list), optional, default = []) - a top-down list of folder / workspace names. Only relevant for type = `get_page`.
      * navigation (bool, optional, default = False) - whether or not the click issues a nvigation event. Relevant only for type = `click_elem`.
      * popup_window (bool, optional) - deciding if the click will popup a new browser window - for type `click_elem` only
      * close_window (bool, optional) - deciding if the click will close the current window - for type `click_elem` only
      * checkbox_state (bool, optional, default = None) - defines the required state of a checkbox element (True = checked, False = unchecked)
      * attribute (str, optional, default = "") - the attribute name of an HTML element. Relevant only for type = `check_elem`.
      * substring (bool, optional, default = False) - with or not a string comparison should consider substrings. Relevant only for type = `check_elem`.
      * min_count (int, optional, default = 1) - defines how many elements should be found at a minimum by type = `check_elem`.
      * want_exist (bool, optional, default = True) - defines for type = `check_elem` if the existence or non-existence should be checked.
      * show_error (bool, optional) - deciding if an an error is logged
    """
    _browser_automations = []
    _browser_automations_post = []
    _test_automations = []

    """
    _security_clearances: List of Security Clearances. Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - name (str, mandatory)
    - level (str, mandatory)
    - description (str, optional, default = "")
    """
    _security_clearances = []

    """
    _supplemental_markings: List of supplemental markings. Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - code (str, mandatory)
    - description (str, optional, default = "")
    """
    _supplemental_markings = []

    _records_management_settings = []

    """
    _holds: List of Records Management holds. Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - name (str, mandatory)
    - type (str, mandatory)
    - group (str, optional)
    - comment (str, optional, default = "")
    - alternate_id (str, optional)
    - date_applied (str, optional, default = "")
    - date_to_remove (str, optional, default = "")
    """
    _holds = []

    """
    _bulk_datasources: list of bulk datasources.
    Each element is a dict with these keys:
    - data (Data object), this is not in payload but automatically filled
    - type (str, mandatory) - either excel, servicenow, otmm, otcs, pht, json, xml, nhc
    - csv_files (list, mandatory if type = csv)
    - csv_delimiter (str, optional) - value delimiter in the file - default is a comma
    - csv_header_index (int, optional) - if the file has a header line this parameter specifies the index
      (0 = first line, this is the default)
    - csv_column_names (list, optional) - if the file has no header line the column name can be
      specified as a list of strings
    - csv_use_columns (list, optional) - this list can either include integers (index of columns to keep), strings
      (names of columns to keep), or a list of boolean values (True = column is kept, False = column is dropped)
    - json_files (list, mandatory if type = json)
    - xml_files (list, optional, default = []) - only relevant for type = xml
    - xml_directories (list, optional, default = []) - only relevant for type = xml
    - xml_xpath (str, optional, default = None) - only relevant if xml_directories is set
    - xlsx_files (list, optional, default = [])
    - xlsx_sheets (list, optional, default = 0)
    - xlsx_columns (list, optional, default = None)
    - xlsx_skip_rows (int, optional, default = 0) - number of rows to skip on top of sheet
    - xlsx_na_values (list, optional, default = [])
    - pht_base_url (str, mandatory if type = pht)
    - pht_username (str, mandatory if type = pht)
    - pht_password (str, mandatory if type = pht)
    - pht_business_unit_exclusions (list, optional, default = [])
    - pht_business_unit_inclusions (list, optional, default = [])
    - pht_product_exclusions (list, optional, default = [])
    - pht_product_inclusions (list, optional, default = [])
    - pht_product_category_exclusions (list, optional, default = [])
    - pht_product_category_inclusions (list, optional, default = [])
    - pht_product_status_exclusions (list, optional, default = [])
    - pht_product_status_inclusions (list, optional, default = [])
    - pht_product_attributes (list, optional, default = []) - a list of attribute names that should be extracted and added as columns to the data frame
    - otmm_username (str, optional, default = "")
    - otmm_password (str, optional, default = "")
    - otmm_client_id (str, optional, default = None)
    - otmm_client_secret (str, optional, default = None)
    - otmm_thread_number (int, optional, default = BULK_THREAD_NUMBER)
    - otmm_download_dir (str, optional, default = "/data/mediaassets")
    - otmm_business_unit_exclusions (list, optional, default = [])
    - otmm_business_unit_inclusions (list, optional, default = [])
    - otmm_product_exclusions (list, optional, default = [])
    - otmm_product_inclusions (list, optional, default = [])
    - sn_base_url (str, mandatory if type = servicenow)
    - sn_auth_type (str, optional, default = "basic")
    - sn_username (str, optional, default = "")
    - sn_password (str, optional, default = "")
    - sn_client_id (str, optional, default = None)
    - sn_client_secret (str, optional, default = None)
    - sn_queries (list, mandatory if type = servicenow)
      * sn_table_name (str, mandatory) - name of the ServiceNow database table for the query
      * sn_query (str, mandatory) - query string
    - sn_thread_number (int, optional, default = BULK_THREAD_NUMBER)
    - sn_download_dir (str, optional, default = "/data/knowledgebase")
    - sn_skip_existing_downloads (bool, optional, default = True)
    - otcs_hostname (str, mandatory if type = otcs)
    - otcs_protocol (str, optional, default = "https")
    - otcs_port (str, optional, default = "443")
    - otcs_basepath (str, optional, default = "/cs/cs")
    - otcs_username (str, mandatory if type = otcs)
    - otcs_password (str, mandatory if type = otcs)
    - otcs_thread_number (int, optional, default = BULK_THREAD_NUMBER)
    - otcs_download_dir (str, optional, default = "/data/contentserver")
    - otcs_root_node_ids (list | int, mandatory if type = otcs)
    - otcs_include_workspaces (bool, optional, default = True)
    - otcs_include_items (bool, optional, default = True)
    - otcs_include_workspace_metadata (bool, optional, default = True)
    - otcs_include_item_metadata (bool, optional, default = True)
    - otcs_filter_workspace_depth (int, optional, default = 0)
    - otcs_filter_workspace_subtypes (int, optional, default = 0)
    - otcs_filter_workspace_category (str, optional, default = None) - name of the category the workspace needs to have
    - otcs_filter_workspace_attributes (dict | list, optional, default = None)
      * set (str, optional, default = None) - name of the attribute set
      * row (int, optional, default = None) - row number (starting with 1) - only required for multi-value sets
      * attribute (str, mandatory) - name of the attribute
      * value (str, mandatory) - value the attribute should have to pass the filter
    - otcs_filter_item_depth (int, optional, default = None)
    - otcs_filter_item_subtypes (int, optional, default = 0)
    - otcs_filter_item_category (str, optional, default = None) - name of the category the workspace needs to have
    - otcs_filter_item_attributes (dict | list, optional, default = None)
      * set (str, optional, default = None) - name of the attribute set
      * row (int, optional, default = None) - row number (starting with 1) - only required for multi-value sets
      * attribute (str, mandatory) - name of the attribute
      * value (str, mandatory) - value the attribute should have to pass the filter
    - cleansings (dict, optional, default = {}) - the keys of this dict are the field names! The values of the dict
      are sub-dicts with these keys:
      * upper (bool, optional, default = False)
      * lower (bool, optional, default = False)
      * capitalize (bool, optional, default = false) - first character upper case, rest lower-case
      * title (bool, optional, default = false) - first character of each word upper case
      * length (int, optional, default = None)
      * replacements (dict, optional, default = {}) - the keys are regular expressions and the values are
        replacement values
    - columns_to_drop (list, optional, default = [])
    - columns_to_keep (list, optional, default = [])
    - columns_to_add (list, optional, default = []) - elements are dicts with these keys:
      * source_column (str, mandatory)
      * name (str, mandatory)
      * reg_exp (str, optional, default = None)
      * prefix (str, optional, default = "")
      * suffix (str, optional, default = "")
      * length (int, optional, default = None)
      * group_chars (str, optional, default = None)
      * group_separator (str, optional, default =".")
    - columns_to_add_list (list, optional, default = []): add a new column with list values.
      Each payload item is a dictionary with these keys:
      * source_columns (str, mandatory) - names of the columns from which row values are taken from to create
        the list of string values
      * name (str, mandatory) - name of the new column
    - columns_to_add_concat (list, optional, default = []): add a new column with concatenated values.
      Each payload item is a dictionary with these keys:
      * source_columns (str, mandatory) - names of the columns from which row values are taken from to create
        the list of string values
      * name (str, mandatory) - name of the new column
      * concat_char (str, optional) - concatenation char e.g. "-", ".". Default is None = empty.
      * columns_to_add_concat_lower (bool,optional) - convert result to lower case
      * columns_to_add_concat_upper (bool,optional) - convert result to upper case
      * columns_to_add_concat_capitalize (bool,optional) - capitalize result
      * columns_to_add_concat_title (bool,optional) - convert result to title case
    - columns_to_add_table (list, optional, default = []): add a new column with table values.
      Each payload item is a dictionary with these keys:
      * source_columns (str, mandatory) - names of the columns from which row values are taken from to create a list
        of dictionary values. It is expected that the source columns already have list items or are strings with
        delimiter-separated values.
      * name (str, mandatory) - name of the new column
      * list_splitter (str, optional, default = ",")
    - conditions (list, optional, default = []) - each list item is a dict with these keys:
      * field (str, mandatory)
      * value (str | bool | list, optional, default = None)
      * equal (bool, optional): if True test for equality, if False test for non-equality
    - explosions (list, optional, default = []) - each list item is a dict with these keys:
      * explode_fields (str | list, mandatory)
      * flatten_fields (list, optional, default = [])
      * split_string_to_list (bool, optional, default = False)
      * list_splitter (str, optional, default = ",;") - string with characters that are used to split
        a string into list items.
    - name_column (str, optional, default = None)
    - synonyms_column (str, optional, default = None)
    """
    _bulk_datasources = []

    """
    _bulk_workspaces: List of bulk workspace definitions.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - type_name (str, mandatory) - type of the workspace
    - data_source (str, mandatory)
    - force_reload (bool, optional, default = True) - enforce a reload of the data source, e.g. useful if data source
      has been modified before by column operations or explosions
    - copy_data_source (bool, optional, default = False) - to avoid side-effects for repeative usage of the data source
    - operations (list, optional, default = ["create"]) - possible values: "create", "update", "delete", "recreate"
      (delete existing + create new)
    - update_operations (list, optional, default = ["name", "description", "categories", "nickname"]) - possible values:
      "name", "description", "categories", "nickname"
    - unique (list, optional, default = []) - list of fields (columns) that should be unique -> deduplication
    - sort (list, optional, default = []) - list of fields to sort the data frame by
    - name (str, mandatory) - name of the workspace - can include placeholder surrounded by {...}
    - name_alt (str, optional, default = None) - alternative name - can include placeholder surrounded by {...}
    - description (str, optional, default = "")
    - description_alt (str, optional, default = "") - alternative description (using different placeholders)
    - template_name (str, optional, default = take first template)
    - categories (list, optional, default = []) - each list item is a dict that may have these keys:
      * name (str, mandatory)
      * set (str, default = "")
      * row (int, optional)
      * attribute (str, mandatory)
      * value (str, optional if value_field is specified, default = None)
      * value_field (str, optional if value is specified, default = None) - can include placeholder surrounded by {...}
      * value_type (str, optional, default = "string") - possible values: "string", "date", "list" and "table".
        If list then string with comma-separated values will be converted to a list.
      * attribute_mapping (dict, optional, default = None) - only relevant for value_type = "table" -
        defines a mapping from the data frame column names to the category attribute names
      * value_mapping (dict, optional, default = None) - dictionary keys are the original values and dictionary values
        are the mapped values. This makes most sense for values with a limited / fixed domain of possible values
      * list_splitter (str, optional, default = ";,")
      * lookup_data_source (str, optional, default = None)
      * lookup_data_failure_drop (bool, optional, default = False) -
        should we clear / drop values that cannot be looked up?
      * is_key (bool,optional, default = False) - find document with old name. For this we expect a "key" value to be
        defined in the bulk workspace and one of the category / attribute item to be marked with "is_key" = True
    - workspaces (dict, dynamically bult up, default = {}) - list of already generated workspaces
    - external_create_date (str, optional, default = "")
    - external_modify_date (str, optional, default = "")
    - key (str, optional, default = None) - lookup key for workspaces other then the name
    - replacements (dict, optional, default = {}) - Each dictionary item has the field name as the dictionary key and a
      list of regular expressions as dictionary value
    - nickname (str, optional, default = None)
    - nickname_alt (str, optional, default = None) - alternative nickname
    - conditions (list, optional, default = [])
      * field (str, mandatory)
      * value (str | bool | list, optional, default = None)
    """
    _bulk_workspaces = []

    """
    _bulk_workspace_relationships: List of bulk workspace relationships.
    Each element is a dict with these keys:
    - enabled (bool, optional, default = True)
    - from_workspace (str, mandatory)
    - from_workspace_type (str, optional, default = None)
    - from_workspace_name (str, optional, default = None)
    - from_workspace_data_source (str, optional, default = None)
    - from_sub_workspace_name (str, optional, default = None) - if the related workspace is a sub-workspace
    - from_sub_workspace_path (list, optional, default = None) - the folder path under the main workspace where
      the sub-workspaces are located
    - to_workspace (str, mandatory)
    - to_workspace_type (str, optional, default = None)
    - to_workspace_name (str, optional, default = None)
    - to_workspace_data_source (str, optional, default = None)
    - to_sub_workspace_name (str, optional, default = None) - if the related workspace is a sub-workspace
    - to_sub_workspace_path (list, optional, default = None) - the folder path under the main workspace
      where the sub-workspaces are located
    - type (str, optional, default = "child") - type of the relationship (defines if the _from_ workspace
      is the parent or the child)
    - data_source (str, mandatory)
    - force_reload (bool, optional, default = True) - enforce a reload of the data source, e.g. useful if data source
      has been modified before by column operations or explosions
    - copy_data_source (bool, optional, default = False) - to avoid side-effects for repeative usage of the data source
    - explosions (list, optional, default = []) - each list item is a dict with these keys:
      * explode_fields (str | list, mandatory)
      * flatten_fields (list, optional, default = [])
      * split_string_to_list (bool, optional, default = False)
      * list_splitter (str, optional, default = ",;") - string with characters that are used to split
        a string into list items.
    - unique (list, optional, default = [])
    - sort (list, optional, default = [])
    - thread_number (int, optional, default = BULK_THREAD_NUMBER)
    - replacements (list, optional, default = None)
    - conditions (list, optional, default = None)
      * field (str, mandatory)
      * value (str | bool | list, optional, default = None)
    - from_workspace_lookup_error (bool, optional, default = True) - whether or not an error should be logged
      if a relationship endpoint cannot be looked up
    - to_workspace_lookup_error (bool, optional, default = True) - whether or not an error should be logged
      if a relationship endpoint cannot be looked up
    """
    _bulk_workspace_relationships = []

    """
    _bulk_documents (list): List of bulk document payload. Each element
    is a dict with these keys:
    - enabled (bool, optional, default = True)
    - data_source (str, mandatory)
    - force_reload (bool, optional, default = True) - enforce a reload of the data source, e.g. useful if data source
      has been modified before by column operations or explosions
    - copy_data_source (bool, optional, default = False) - to avoid side-effects for repeative usage of the data source
    - explosions (list of dicts, optional, default = [])
      * explode_fields (str | list, mandatory)
      * flatten_fields (list, optional, default = [])
      * split_string_to_list (bool, optional, default = False)
      * list_splitter (str, optional, default = ",;") - string with characters that are used to split a
        string into list items.
    - unique (list, optional, default = []) - list of column names which values should be unique -> deduplication
    - sort (list, optional, default = []) - list of fields to sort the data frame by
    - operations (list, optional, default = ["create"])
    - update_operations (list, optional, default = ["name", "description", "categories", "nickname", "version"]) - possible values:
      "name", "description", "categories", "nickname", "version", "purge"
    - name (str, mandatory) - can include placeholder surrounded by {...}
    - name_alt (str, optional, default = None) - can include placeholder surrounded by {...}
    - name_regex (str, optional, default = r"") - regex replacement for document names. The pattern and replacement are separated by pipe character |
    - description (str, optional, default = None) - can include placeholder surrounded by {...}
    - download_name (str, optional, default = name) - can include placeholder surrounded by {...}
    - download_name_wildcards (bool, optional, default = False) - defines if the download name includes wildcards,
      e.g. "*.pdf"
    - nickname (str, optional, default = None) - can include placeholder surrounded by {...}
    - download_url (str, optional, default = None)
    - download_url_alt (str, optional, default = None)
    - download_dir (str, optional, default = BULK_DOCUMENT_PATH)
    - delete_download (bool, optional, default = True)
    - file_extension (str, optional, default = "")
    - file_extension_alt (str, optional, default = "html")
    - mime_type (str, optional, default = "application/pdf")
    - mime_type_alt (str, optional, default = "text/html")
    - nickname (str, optional, default = None)
    - categories (list, optional, default = [])
      * name (str, mandatory)
      * set (str, default = "")
      * row (int, optional)
      * attribute (str, mandatory)
      * value (str, optional if value_field is specified, default = None)
      * value_field (str, optional if value is specified, default = None) - can include placeholder surrounded by {...}
      * value_type (str, optional, default = "string") - possible values: "string", "date", "list" and "table".
        If list then string with comma-separated values will be converted to a list.
      * attribute_mapping (dict, optional, default = None) - only relevant for value_type = "table" - defines a mapping
        from the data frame column names to the category attribute names
      * list_splitter (str, optional, default = ";,")
      * lookup_data_source (str, optional, default = None)
      * lookup_data_failure_drop (bool, optional, default = False) -
        should we clear / drop values that cannot be looked up?
      * is_key (bool, optional, default = False) - find document is old name. For this we expect a "key" value
        to be defined for the bulk document and one of the category / attribute item to be marked with "is_key" = True
    - thread_number (int, optional, default = BULK_THREAD_NUMBER)
    - external_create_date (str, optional, default = "")
    - external_modify_date (str, optional, default = "")
    - key (str, optional, default = None) - lookup key for documents other then the name
    - download_wait_time (int, optional, default = 30)
    - download_retries (int, optional, default = 2)
    - replacements (list, optional, default = [])
    - conditions (list, optional, default = []) - all conditions must evaluate to true
      * field (str, mandatory)
      * value (str | bool | list, optional, default = None)
    - workspaces (list, optional, default = [])
      * workspace_name (str, mandatory)
      * conditions (list, optional, default = [])
        + field (str, mandatory)
        + value (str | bool | list, optional, default = None)
      * workspace_type (str, mandatory)
      * data_source (str, optional, default = None)
      * workspace_folder (str, optional, default = "")
      * workspace_path (list, optional, default = [])
      * sub_workspace_type (str, optional, default = "")
      * sub_workspace_name (str, optional, default = "")
      * sub_workspace_template (str, optional, default = "")
      * sub_workspace_folder (str, optional, default = "")
      * sub_workspace_path (list, optional, default = [])
    """
    _bulk_documents = []

    """
    _bulk_items (list): List of bulk items payload. Each element
    is a dict with these keys:
    - enabled (bool, optional, default = True)
    - data_source (str, mandatory)
    - force_reload (bool, optional, default = True) - enforce a reload of the data source, e.g. useful if data source
      has been modified before by column operations or explosions
    - copy_data_source (bool, optional, default = False) - to avoid side-effects for repeative usage of the data source
    - explosions (list of dicts, optional, default = [])
      * explode_fields (str | list, mandatory)
      * flatten_fields (list, optional, default = [])
      * split_string_to_list (bool, optional, default = False)
      * list_splitter (str, optional, default = ",;") - string with characters that are used to split a
        string into list items.
    - unique (list, optional, default = []) - list of column names which values should be unique -> deduplication
    - sort (list, optional, default = []) - list of fields to sort the data frame by
    - operations (list, optional, default = ["create"])
    - update_operations (list, optional, default = ["name", "description", "categories", "nickname"]) - possible values:
      "name", "description", "categories", "nickname", "url"
    - name (str, mandatory) - can include placeholder surrounded by {...}
    - name_alt (str, optional, default = None) - can include placeholder surrounded by {...}
    - name_regex (str, optional, default = r"") - regex replacement for document names. The pattern and replacement are separated by pipe character |
    - description (str, optional, default = None) - can include placeholder surrounded by {...}
    - nickname (str, optional, default = None) - can include placeholder surrounded by {...}
    - nickname (str, optional, default = None)
    - categories (list, optional, default = [])
      * name (str, mandatory)
      * set (str, default = "")
      * row (int, optional)
      * attribute (str, mandatory)
      * value (str, optional if value_field is specified, default = None)
      * value_field (str, optional if value is specified, default = None) - can include placeholder surrounded by {...}
      * value_type (str, optional, default = "string") - possible values: "string", "date", "list" and "table".
        If list then string with comma-separated values will be converted to a list.
      * attribute_mapping (dict, optional, default = None) - only relevant for value_type = "table" - defines a mapping
        from the data frame column names to the category attribute names
      * list_splitter (str, optional, default = ";,")
      * lookup_data_source (str, optional, default = None)
      * lookup_data_failure_drop (bool, optional, default = False) -
        should we clear / drop values that cannot be looked up?
      * is_key (bool, optional, default = False) - find document is old name. For this we expect a "key" value
        to be defined for the bulk document and one of the category / attribute item to be marked with "is_key" = True
    - thread_number (int, optional, default = BULK_THREAD_NUMBER)
    - external_create_date (str, optional, default = "")
    - external_modify_date (str, optional, default = "")
    - key (str, optional, default = None) - lookup key for documents other then the name
    - replacements (list, optional, default = [])
    - conditions (list, optional, default = []) - all conditions must evaluate to true
      * field (str, mandatory)
      * value (str | bool | list, optional, default = None)
    - workspaces (list, optional, default = [])
      * workspace_name (str, mandatory)
      * conditions (list, optional, default = [])
        + field (str, mandatory)
        + value (str | bool | list, optional, default = None)
      * workspace_type (str, mandatory)
      * data_source (str, optional, default = None)
      * workspace_folder (str, optional, default = "")
      * workspace_path (list, optional, default = [])
      * sub_workspace_type (str, optional, default = "")
      * sub_workspace_name (str, optional, default = "")
      * sub_workspace_template (str, optional, default = "")
      * sub_workspace_folder (str, optional, default = "")
      * sub_workspace_path (list, optional, default = [])
    """
    _bulk_items = []

    _bulk_classifications = []

    _nifi_flows = []

    _placeholder_values = {}

    # Link to the method in customizer.py to restart the Content Server pods.
    _otcs_restart_callback: Callable

    # Link to the method in customizer.py to print a log header (separator line)
    _log_header_callback: Callable

    _aviator_enabled = False

    _transport_extractions: list = []
    _transport_replacements: list = []
    _appworks_configurations = []

    _avts_repositories: list = []

    _embeddings: list = []

    # Disable Status files
    upload_status_files: bool = True

    def __init__(
        self,
        payload_source: str,
        custom_settings_dir: str,
        k8s_object: K8s | None,
        otds_object: OTDS,
        otac_object: OTAC | None,
        otcs_backend_object: OTCS,
        otcs_frontend_object: OTCS,
        otcs_restart_callback: Callable,
        otiv_object: OTIV | None,
        otpd_object: OTPD | None,
        m365_object: M365 | None,
        core_share_object: CoreShare | None,
        placeholder_values: dict,
        log_header_callback: Callable,
        browser_headless: bool = True,
        stop_on_error: bool = False,
        aviator_enabled: bool = False,
        upload_status_files: bool = True,
        otawp_object: OTAWP | None = None,
        otca_object: OTCA | None = None,
        otkd_object: OTKD | None = None,
        avts_object: AVTS | None = None,
        logger: logging.Logger = default_logger,
    ) -> None:
        """Initialize the Payload object.

        Args:
            payload_source (str):
                The path or URL to payload source file.
            custom_settings_dir (str):
                Provide a custom settings directory.
            k8s_object (K8s | None):
                The Kubernetes object.
                Pass None if deployment is not running in Kubernetes.
            otds_object (OTDS):
                The OTDS object. This is mandatory.
            otac_object (OTAC | None):
                The optional OTAC object (Archive Center).
                Pass None if Archive Center is not part of the deployment.
            otcs_backend_object (OTCS):
                The OTCS backend object.
            otcs_frontend_object (OTCS):
                The OTCS frontend object.
            otcs_restart_callback (Callable):
                A function to call if OTCS service needs a restart.
            otiv_object (OTIV | None):
                The OTIV object (Intelligent Viewing).
                Pass None if Intelligent Viewing is not part of the deployment.
            otpd_object (OTPD | None):
                The OTPD object (PowerDocs).
                Pass None if PowerDocs is not part of the deployment.
            m365_object (object):
                The M365 object to talk to Microsoft Graph API.
            core_share_object (CoreShare | None):
                The Core Share object.
            placeholder_values (dict):
                A dictionary of placeholder values to be replaced in admin settings.
            log_header_callback:
                Method to print a section break / header line into the log.
            browser_headless (bool):
                If true, the Browser for the Automation will be started in Headless mode (default)
            stop_on_error (bool):
                This flag controls if transport deployment should stop
                if a transport deployment in OTCS fails.
            aviator_enabled (bool):
                Flag that indicates whether or not the Content Aviator is enabled.
            upload_status_files (bool, optional):
                Whether or not status file should be uploaded to the peronal workspace
                of the admin user in Content Server.
            otawp_object (OTAWP):
                An optional AppWorks Platform object.
            otca_object (OTCA):
                An optional Content Aviator object.
            otkd_object (OTKD):
                An optional Knowledge Discovery object.
            avts_object (AVTS):
                An optional Aviator Search object.
            logger (logging.Logger, optional):
                The logging object to use for all log messages. Defaults to default_logger.

        """

        self.logger = logger.getChild("payload")
        for logfilter in logger.filters:
            self.logger.addFilter(logfilter)

        self._stop_on_error = stop_on_error
        self._payload_source = payload_source
        self._k8s = k8s_object
        self._otds = otds_object
        self._otac = otac_object
        self._otcs = otcs_backend_object
        self._otcs_backend = otcs_backend_object
        self._otcs_frontend = otcs_frontend_object
        self._otiv = otiv_object
        self._otpd = otpd_object
        self._m365 = m365_object
        self._core_share = core_share_object
        # The SAP, SuccessFactors and Salesforce objects only exists after external systems have been processed
        self._sap = None
        self._successfactors = None
        self._salesforce = None
        self._servicenow = None
        self._guidewire_policy_center = None
        self._guidewire_claims_center = None
        self._otmm = None
        self._otcs_source = None
        self._pht = None  # the OpenText prodcut hierarchy
        self._nhc = None  # National Hurricane Center
        self._otca = otca_object  # Content Aviator
        self._otkd = otkd_object  # Knowledge Discovery
        self._avts = avts_object  # Aviator Search
        self._browser_headless = browser_headless
        self._custom_settings_dir = custom_settings_dir
        self._placeholder_values = placeholder_values
        self._otcs_restart_callback = otcs_restart_callback
        self._log_header_callback = log_header_callback
        self._aviator_enabled = aviator_enabled
        self._http_object = HTTP(logger=self.logger)
        self.upload_status_files = upload_status_files
        self._otawp = otawp_object

    # end method definition

    def thread_wrapper(self, target: Callable, *args: tuple, **kwargs: dict) -> None:
        """Wrap around threads to catch exceptions during exection.

        Args:
            target (callable):
                The method (callable) the Thread should run.
            args (tuple):
                The arguments for the method.
            kwargs (dict):
                Keyword arguments for the method.

        """

        try:
            target(*args, **kwargs)
        except Exception:
            thread_name = threading.current_thread().name
            self.logger.error(
                "Thread '%s' failed!",
                thread_name,
            )
            self.logger.error(traceback.format_exc())

    # end method definition

    def replace_placeholders(self, content: str) -> str:
        """Replace placeholders in file content.

        The content of the file is provided via a parameter.
        The replacements are defined in a object variable
        _placeholder_values (type = dictionary)
        The placeholder values are supposed to be surrounded by
        double % signs like %%OTAWP_RESOURCE_ID%%

        Args:
            content (str):
                The file content to replace placeholders in.

        Returns:
            str:
                Updated content with all defined replacements.

        """
        # https://stackoverflow.com/questions/63502218/replacing-placeholders-in-a-text-file-with-python

        # if no placeholders are defined we can return the
        # initial value:
        if not self._placeholder_values:
            return content

        try:
            # We do a dynamic replacement here. The replacement is calculated
            # by the lambda function that is basically a lookup of the replacement
            # key found in the settings file with the value defined in the list
            # of replacement values in self._placeholder_values
            return re.sub(
                r"%%(\w+?)%%",
                lambda match: self._placeholder_values[match.group(1)],
                content,
            )
        except KeyError:
            self.logger.error(
                "Found placeholder in settings file without a defined value!",
            )
            return content
        except re.error:
            self.logger.error("Regex substitution error!")
            return content

        # end method definition

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="init_payload")
    def init_payload(self) -> dict | None:
        """Read the YAML or Terraform HCL payload file.

        Args:
            None

        Returns:
            dict | None:
                The payload as a Python dict. Elements are the different payload sections.
                None in case the file couldn't be found or read.

        """

        self._payload = load_payload(self._payload_source, self.logger)
        if not self._payload:
            return None

        self._payload_sections = self._payload.get("payloadSections", None)
        self._payload_options = self._payload.get("payloadOptions", None)

        # Retrieve all the payload sections and store them in lists
        # (these are object variables that we want to initialize
        # even if payload is empty):
        self._webhooks = self.get_payload_section("webHooks")
        self._webhooks_post = self.get_payload_section("webHooksPost")
        self._resources = self.get_payload_section("resources")
        self._partitions = self.get_payload_section("partitions")
        self._licenses = self.get_payload_section("licenses")
        self._synchronized_partitions = self.get_payload_section(
            "synchronizedPartitions",
        )
        self._oauth_clients = self.get_payload_section("oauthClients")
        self._application_roles = self.get_payload_section("applicationRoles")
        self._auth_handlers = self.get_payload_section("authHandlers")
        self._trusted_sites = self.get_payload_section("trustedSites")
        self._system_attributes = self.get_payload_section("systemAttributes")
        self._docgen_settings = self.get_payload_section("docgenSettings")
        self._groups = self.get_payload_section("groups")
        self._users = self.get_payload_section("users")
        if self._users:
            # Check if multiple user instances should be created
            self.init_payload_user_instances()
        self._admin_settings = self.get_payload_section("adminSettings")
        self._admin_settings_post = self.get_payload_section("adminSettingsPost")
        self._exec_pod_commands = self.get_payload_section("execPodCommands")
        self._exec_commands = self.get_payload_section("execCommands")
        self._kubernetes = self.get_payload_section("kubernetes")
        self._exec_database_commands = self.get_payload_section("execDatabaseCommands")
        self._external_systems = self.get_payload_section("externalSystems")
        self._transport_packages = self.get_payload_section("transportPackages")
        self._content_transport_packages = self.get_payload_section(
            "contentTransportPackages",
        )
        self._transport_packages_post = self.get_payload_section(
            "transportPackagesPost",
        )
        self._workspace_templates = self.get_payload_section("workspaceTemplates")
        self._workspaces = self.get_payload_section("workspaces")
        self._bulk_datasources = self.get_payload_section("bulkDatasources")
        self._bulk_workspaces = self.get_payload_section("bulkWorkspaces")
        self._bulk_workspace_relationships = self.get_payload_section(
            "bulkWorkspaceRelationships",
        )
        self._bulk_documents = self.get_payload_section("bulkDocuments")
        self._bulk_items = self.get_payload_section("bulkItems")
        self._bulk_classifications = self.get_payload_section("bulkClassifications")
        self._sap_rfcs = self.get_payload_section("sapRFCs")
        self._sap_rfcs_post = self.get_payload_section("sapRFCsPost")
        self._web_reports = self.get_payload_section("webReports")
        self._web_reports_post = self.get_payload_section("webReportsPost")
        self._cs_applications = self.get_payload_section("csApplications")
        self._additional_group_members = self.get_payload_section(
            "additionalGroupMemberships",
        )
        self._additional_access_role_members = self.get_payload_section(
            "additionalAccessRoleMemberships",
        )
        self._additional_application_role_assignments = self.get_payload_section(
            "additionalApplicationRoleAssignments",
        )
        self._renamings = self.get_payload_section("renamings")
        self._items = self.get_payload_section("items")
        self._items_post = self.get_payload_section("itemsPost")
        self._permissions = self.get_payload_section("permissions")
        self._permissions_post = self.get_payload_section("permissionsPost")
        self._workspace_permissions = self.get_payload_section("workspacePermissions")
        self._assignments = self.get_payload_section("assignments")
        self._security_clearances = self.get_payload_section("securityClearances")
        self._supplemental_markings = self.get_payload_section("supplementalMarkings")
        self._records_management_settings = self.get_payload_section(
            "recordsManagementSettings",
        )
        self._holds = self.get_payload_section("holds")
        self._doc_generators = self.get_payload_section("documentGenerators")
        self._workflows = self.get_payload_section("workflows")
        self._browser_automations = self.get_payload_section("browserAutomations")
        self._browser_automations_post = self.get_payload_section(
            "browserAutomationsPost",
        )
        self._test_automations = self.get_payload_section("testAutomations")
        self._appworks_configurations = self.get_payload_section("appworks")
        self._avts_repositories = self.get_payload_section("avtsRepositories")
        self._avts_questions = self.get_payload_section("avtsQuestions")
        self._embeddings = self.get_payload_section("embeddings")
        self._nifi_flows = self.get_payload_section("nifi")

        return self._payload

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="init_payload_user_instances")
    def init_payload_user_instances(self) -> None:
        """Read setting for Multiple User instances."""

        for dic in self._payload_sections:
            if dic.get("name") == "users":
                users_payload = dic
                break
        user_instances = users_payload.get("additional_instances", 0)

        if user_instances == 0:
            self.logger.info("No additional user instances configured (default = 0).")
            return

        i = 0

        original_users = copy.deepcopy(self._users)
        while i <= user_instances:
            for user in copy.deepcopy(original_users):
                user["name"] = user["name"] + "-" + str(i).zfill(2)
                user["lastname"] = user["lastname"] + " " + str(i).zfill(2)
                user["enable_sap"] = False
                user["enable_o365"] = False
                user["enable_core_share"] = False
                user["enable_salesforce"] = False
                user["enable_successfactors"] = False

                self.logger.info(
                    "Creating additional user instance -> '%s'",
                    user["name"],
                )
                self.logger.debug("Create user instance -> %s", user)
                self._users.append(user)

            i = i + 1

        return

    # end method definition

    def get_payload_section(self, payload_section_name: str) -> list:
        """Get a specific section of the payload based on its name.

        The section is delivered as a list of settings.
        It deliveres an empty list if this payload section is disabled by the corresponding
        payload switch (this is read from the payloadSections dictionary of the payload)

        Args:
            payload_section_name (str):
                The name of the dict element in the payload structure.

        Returns:
            list:
                The section of the payload as a Python list. Empty list if section does not exist
                or section is disabled by the corresponding payload switch.

        """

        if not isinstance(self._payload, dict):
            return []

        # if the secton is not in the payload we return an empty list:
        if not self._payload.get(payload_section_name):
            return []

        # Return an empty list if the payload section does not exist or is disabled:
        sections = self._payload.get("payloadSections")
        if sections:
            section = next(
                (item for item in sections if item["name"] == payload_section_name),
                None,
            )
            if not section or not section.get("enabled", False):
                return []

        return self._payload[payload_section_name]

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_appworks_configurations")
    def process_appworks_configurations(self, section_name: str = "appworks") -> bool:
        """Process the configurations for AppWorks projects.

        This method is responsible for setting up the necessary configurations for AppWorks projects.
        If the payload contains a `appworks` section, it will execute the corresponding actions
        to process and apply the custom configuration.

        This includes:
        * Resource configuration
            - add the OTCS user partition to the OTDS access role for the AppWorks organization
            - add the OTCS admin partition to the OTDS access role for the AppWorks organization
            - add an OTAWP license
        * Solution configuration
            - create the AppWorks artifacts

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._appworks_configurations:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        #
        # 1: Loop to create the required OTDS resources for the AppWorks organization:
        #
        for appworks_configuration in self._appworks_configurations:
            organization = appworks_configuration.get("organization", None)
            if not organization:
                self.logger.error("AppWorks configuration is missing the organization name! Skipping...")
                success = False
                continue

            self._log_header_callback(
                text="Process AppWorks resource configuration for '{}'".format(organization), char="-"
            )

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not appworks_configuration.get("enabled", True):
                self.logger.info(
                    "Payload for AppWorks configuration -> '%s' is disabled. Skipping...",
                    organization,
                )
                continue

            if not appworks_configuration.get("resource_config", False):
                self.logger.info("AppWorks resource configuration is disabled for -> '%s'. Skipping...", organization)
                continue

            access_role_name = "Access to " + organization

            # make sure code is idempotent and only try to add ressource if it doesn't exist already:
            awp_resource = self._otds.get_resource(organization)
            if not awp_resource:
                self.logger.info(
                    "OTDS resource -> '%s' for AppWorks Platform does not yet exist. Creating...",
                    organization,
                )
                awp_resource = self._otds.add_resource(
                    name=organization,
                    description="AppWorks Platform - {}".format(organization),
                    display_name="AppWorks Platform - {}".format(organization),
                    additional_payload=OTAWP.resource_payload(
                        org_name=organization,
                        username=self._otawp.username(),
                        password=self._otawp.password(),
                    ),
                )
            else:
                self.logger.info(
                    "OTDS resource -> '%s' for AppWorks Platform does already exist.",
                    organization,
                )

            awp_resource_id = awp_resource["resourceID"]

            self.logger.info(
                "AppWorks Platform organization -> '%s' got OTDS resource ID -> %s",
                organization,
                awp_resource_id,
            )

            # Loop to wait for OTCS to create its OTDS user partition:
            otcs_partition = self._otds.get_partition(
                self._otcs.partition_name(),
                show_error=False,
            )
            while otcs_partition is None:
                self.logger.warning(
                    "OTDS user partition -> '%s' for Content Server does not exist yet. Waiting...",
                    self._otcs.partition_name(),
                )

                time.sleep(30)
                otcs_partition = self._otds.get_partition(
                    name=self._otcs.partition_name(),
                    show_error=False,
                )

            # Add the OTDS user partition for OTCS to the AppWorks Platform Access Role in OTDS.
            # This will effectvely sync all OTCS users with AppWorks Platform:
            self._otds.add_partition_to_access_role(
                access_role=access_role_name,
                partition=self._otcs.partition_name(),
            )

            # Add the OTDS admin partition to the AppWorks Platform Access Role in OTDS.
            self._otds.add_partition_to_access_role(
                access_role=access_role_name,
                partition=self._otds.admin_partition_name(),
            )

            # Set Group inclusion for Access Role for OTAWP to "True":
            self._otds.update_access_role_attributes(
                name=access_role_name,
                attribute_list=[{"name": "pushAllGroups", "values": ["True"]}],
            )

            # Add ResourceID User to 'otdsadmins' group to allow push
            self._otds.add_user_to_group(
                user=str(awp_resource_id) + "@otds.admin",
                group="otdsadmins@otds.admin",
            )

            # Allow impersonation for all users:
            self._otds.impersonate_resource(resource_name=organization)

            # Editing configmap
            config_map = self._k8s.get_config_map(config_map_name=self._otawp.config_map_name())
            if not config_map:
                self.logger.error(
                    "Failed to retrieve AppWorks Kubernetes config map -> '%s'!",
                    self._otawp.config_map_name(),
                )
                success = False
            else:
                self.logger.info(
                    "Update Kubernetes config map for AppWorks organization -> '%s' with OTDS resource IDs...",
                    organization,
                )
                solution = yaml.safe_load(
                    config_map.data[organization + ".yaml"],
                )

                solution["platform"]["organizations"][organization]["otds"]["resourceId"] = awp_resource_id
                solution["platform"]["organizations"][organization]["database"]["connections"]["sysConnection"][
                    "connectionString"
                ] = "jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_NAME}"

                solution["platform"]["organizations"][organization]["database"]["connections"]["sysConnection"][
                    "database"
                ] = "${DATABASE_NAME}"

                solution["platform"]["organizations"][organization]["database"]["connections"]["sysConnection"][
                    "password"
                ] = "${DATABASE_PASSWORD}"

                solution["platform"]["organizations"][organization]["content"]["ContentServer"]["contentServerUrl"] = (
                    self._otcs.cs_url()
                )
                solution["platform"]["organizations"][organization]["content"]["ContentServer"][
                    "contentServerSupportDirectoryUrl"
                ] = self._otcs.cs_support_url()
                solution["platform"]["organizations"][organization]["content"]["ContentServer"]["otdsResourceId"] = (
                    self._otcs.resource_id()
                )
                solution["platform"]["organizations"][organization]["authenticators"]["OTDSAuth"]["publicLoginUrl"] = (
                    self._otds.base_url() + "/otdsws/login"
                )

                config_map.data[organization + ".yaml"] = yaml.dump(solution)
                result = self._k8s.replace_config_map(
                    config_map_name=self._otawp.config_map_name(),
                    config_map_data=config_map.data,
                )
                if result:
                    self.logger.info(
                        "Successfully updated AppWorks solution YAML for organization -> '%s'.",
                        organization,
                    )
                else:
                    self.logger.error(
                        "Failed to update AppWorks solution YAML for organization -> '%s'!",
                        organization,
                    )
                self.logger.debug(
                    "Solution YAML for Appworks organization -> '%s': %s",
                    organization,
                    solution,
                )
            # Add SPS license for OTAWP
            license_name = self._otawp.product_name()
            product_name = self._otawp.product_name() + "_" + organization.upper()
            product_description = self._otawp.product_name() + organization
            if os.path.isfile(self._otawp.license_file()):
                self.logger.info(
                    "Found OTAWP license file -> '%s', assiging it to OTDS resource -> '%s'...",
                    self._otawp.license_file(),
                    organization,
                )

                otawp_license = self._otds.add_license_to_resource(
                    path_to_license_file=self._otawp.license_file(),
                    product_name=product_name,
                    product_description=product_description,
                    resource_id=awp_resource["resourceID"],
                )
                if not otawp_license:
                    self.logger.error(
                        "Couldn't apply license -> '%s' for product -> '%s' to OTDS resource -> '%s'!",
                        self._otawp.license_file(),
                        product_name,
                        awp_resource["resourceID"],
                    )
                else:
                    self.logger.info(
                        "Successfully applied license -> '%s' for product -> '%s' to OTDS resource -> '%s'.",
                        self._otawp.license_file(),
                        product_name,
                        awp_resource["resourceID"],
                    )

                # Assign AppWorks license to Content Server Members Partiton and otds.admin:
                for partition_name in ["otds.admin", self._otcs.partition_name()]:
                    if self._otds.is_partition_licensed(
                        partition_name=partition_name,
                        resource_id=awp_resource["resourceID"],
                        license_feature="USERS",
                        license_name=license_name,
                    ):
                        self.logger.info(
                            "Partition -> '%s' is already licensed for -> '%s' (%s).",
                            partition_name,
                            product_name,
                            "USERS",
                        )
                    else:
                        assigned_license = self._otds.assign_partition_to_license(
                            partition_name=partition_name,
                            resource_id=awp_resource["resourceID"],
                            license_feature="USERS",
                            license_name=license_name,
                        )
                        if not assigned_license:
                            self.logger.error(
                                "Partition -> '%s' could not be assigned to license -> '%s' (%s)!",
                                partition_name,
                                product_name,
                                "USERS",
                            )
                            success = False
                        else:
                            self.logger.info(
                                "Partition -> '%s' successfully assigned to license -> '%s' (%s).",
                                partition_name,
                                product_name,
                                "USERS",
                            )
                # end for partition_name in ["otds.admin", self._otcs.partition_name()]:
            # end if os.path.isfile(self._otawp.license_file()):

            self.logger.info("Restart AppWorks Kubernetes stateful set -> '%s'...", self._otawp.hostname())

            self._k8s.restart_stateful_set(sts_name=self._otawp.hostname(), force=True, wait=True)

            self._otawp.set_organization(organization)
            otawp_cookie = self._otawp.authenticate(revalidate=True)
            if not otawp_cookie:
                self.logger.error(
                    "Authentication at AppWorks failed! Cannot proceed with processing of AppWorks configuration -> '%s'!",
                    organization,
                )
                success = False
                continue
            self._otawp.create_cws_config(
                partition=self._otcs.partition_name(), resource_name=organization, otcs_url=self._otcs.cs_url()
            )
            self._otawp.assign_role_to_user(
                organization=organization, user_name=self._otawp.username(), role_name="Developer"
            )
        # end for appworks_configuration in self._appworks_configurations:

        #
        # 2: Loop to create the AppWorks workspaces, projects, and entities:
        #
        for appworks_configuration in self._appworks_configurations:
            organization = appworks_configuration.get("organization", None)
            if not organization:
                self.logger.error("AppWorks configuration is missing the organization name! Skipping...")
                success = False
                continue

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not appworks_configuration.get("enabled", True):
                self.logger.info(
                    "Payload for AppWorks configuration -> '%s' is disabled. Skipping...",
                    organization,
                )
                continue

            self._log_header_callback(
                text="Process AppWorks workspaces, project, entities for '{}'".format(organization), char="-"
            )

            self._otawp.set_organization(organization)
            otawp_cookie = self._otawp.authenticate(revalidate=True)
            if not otawp_cookie:
                self.logger.error(
                    "Authentication at AppWorks failed. Cannot proceed with processing of AppWorks configutation -> '%s'!",
                    organization,
                )
                success = False
                continue

            if "workspaces" not in appworks_configuration:
                self.logger.warning(
                    "No workspace information in AppWorks configuration -> '%s'. Skipping...", organization
                )
                continue

            for workspace in appworks_configuration["workspaces"]:
                workspace_id = workspace.get("workspace_id", None)
                workspace_name = workspace.get("name", None)
                workspace_path = workspace.get("path", None)
                if not workspace_id or not workspace_name or not workspace_path:
                    self.logger.error(
                        "AppWorks workspace configuration for -> '%s'%s requires 'workspace_id', 'name', and 'path' settings! Skipping...",
                        organization,
                        " (workspace name -> {})".format(workspace_name) if workspace_name else "",
                    )
                    success = False
                    continue

                response, created = self._otawp.create_workspace(
                    workspace_name=workspace_name, workspace_id=workspace_id
                )
                if not response:
                    self.logger.info("Failed to create workspace -> '%s' (%s)!", workspace_name, workspace_id)
                    success = False
                    continue

                if created:
                    self.logger.info("Setup new workspace -> '%s' (%s)...", workspace_name, workspace_id)
                    response = self._otawp.sync_workspace(
                        workspace_name=workspace_name,
                        workspace_id=workspace_id,
                    )
                    if not response:
                        self.logger.error("Failed to synchronize workspace -> '%s' (%s)!", workspace_name, workspace_id)
                        success = False
                        continue
                    self.logger.info(
                        "Copy projects artifacts to workspace -> '%s' (%s) in AppWorks pod -> '%s'...",
                        workspace_name,
                        workspace_id,
                        "appworks-0",
                    )
                    self._k8s.exec_pod_command(
                        pod_name="appworks-0",
                        command=[
                            "/bin/sh",
                            "-c",
                            f'cp -r "{workspace_path}/"* "/opt/appworks/cluster/shared/cws/sync/{organization}/{workspace_name}"',
                        ],
                        timeout=600,
                    )
                    self.logger.info(
                        "Copying of projects artifacts to workspace -> '%s' (%s) completed.",
                        workspace_name,
                        workspace_id,
                    )
                self.logger.info("Re-sync existing workspace -> '%s' (%s)...", workspace_name, workspace_id)
                response = self._otawp.sync_workspace(
                    workspace_name=workspace_name,
                    workspace_id=workspace_id,
                )
                if not response:
                    self.logger.error("Failed to synchronize workspace -> '%s' (%s)!", workspace_name, workspace_id)
                    success = False
                    continue

                if "projects" in workspace:
                    for project in workspace["projects"]:
                        if project.get("name") and project.get("documentId"):
                            if not self._otawp.publish_project(
                                workspace_name=workspace_name,
                                workspace_id=workspace_id,
                                project_name=project.get("name"),
                                project_id=project.get("documentId"),
                            ):
                                success = False
                                continue
                        else:
                            self.logger.error(
                                "Skipping project -> '%s' due to missing required project fields 'name' or 'documentId'!",
                                project.get("name"),
                            )
                            success = False
                            continue
                    # end for project in workspace["projects"]:
                # end if "projects" in workspace
            # end for workspace in value["cws"]["workspaces"]

            # Process the entities in the payload:
            entities = appworks_configuration.get("entities", [])
            if entities:
                self._log_header_callback(
                    text="Process AppWorks entities for organization -> '{}'".format(organization), char="-"
                )
                for entity in entities:
                    if not self.process_appworks_entity(entity=entity):
                        success = False
                self.logger.info("Entity processing completed for organization -> '%s'.", organization)
        # end for appworks_configuration in self._appworks_configurations:

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._appworks_configurations,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_appworks_entity")
    def process_appworks_entity(self, entity: dict) -> bool:
        """Process a single AppWorks entity payload.

        Args:
            entity (dict):
                Entity payload.
                Should have a selection of the following keys:
                * type
                * name
                * description
                * status
                * prefix
                * category
                * priority
                * customer
                * case_type
                * ...

        Returns:
            bool: True = success, False = failure.

        """

        entity_type = entity.get("type")
        if not entity_type:
            return False

        match entity_type:
            case "category":
                cat = self._otawp.get_category_by_name(name=entity.get("name"))
                if cat:
                    cat_id = self._otawp.get_entity_value(entity=cat, key="id")
                    self.logger.info(
                        "Category -> '%s' (%s) does already exist. Skipping...", entity.get("name"), str(cat_id)
                    )
                    self.logger.debug("Category -> %s", str(cat))
                else:
                    response = self._otawp.create_category(
                        case_prefix=entity.get("prefix"),
                        name=entity.get("name"),
                        description=entity.get("description", ""),
                        status=entity.get("status", 1),
                    )
                    if not response or "Identity" not in response:
                        self.logger.error("Failed to create category -> '%s'!", entity.get("name"))
                        return False
                    cat_id = response["Identity"].get("Id")
                    self.logger.info(
                        "Successfully created category -> '%s' (%s).",
                        entity.get("name"),
                        cat_id,
                    )
                    self.logger.debug("Response -> %s", str(response))
                if "sub_entities" in entity:
                    for sub_entity in entity["sub_entities"]:
                        if sub_entity["type"] != "subCategory":
                            self.logger.warning(
                                "Found a category sub-entities with wrong type -> '%s'!", sub_entity["type"]
                            )
                            continue
                        response = self._otawp.create_sub_category(
                            parent_id=cat_id,
                            name=sub_entity.get("name"),
                            description=sub_entity.get("description", ""),
                            status=sub_entity.get("status", 1),
                        )
                        if not response or "Identity" not in response:
                            self.logger.error("Failed to create sub-category -> '%s'!", sub_entity.get("name"))
                            return False
                        self.logger.info(
                            "Successfully created sub-category -> '%s' (%s) in parent category -> '%s' (%s).",
                            sub_entity.get("name"),
                            response["Identity"].get("Id"),
                            entity.get("name"),
                            cat_id,
                        )
                        self.logger.debug("Response -> %s", str(response))
                return True
            case "priority":
                priority = self._otawp.get_priority_by_name(name=entity.get("name"))
                if priority:
                    priority_id = self._otawp.get_entity_value(entity=priority, key="id")
                    (
                        self.logger.info(
                            "Priority -> '%s' (%s) does already exist. Skipping...",
                            entity.get("name"),
                            str(priority_id),
                        )
                    )
                    return True
                response = self._otawp.create_priority(
                    name=entity.get("name"),
                    description=entity.get("description", ""),
                    status=entity.get("status", 1),
                )
                if not response or "Identity" not in response:
                    self.logger.error("Failed to create priority -> '%s'!", entity.get("name"))
                    return False
                self.logger.info(
                    "Successfully created priority -> '%s' (%s).", entity.get("name"), response["Identity"].get("Id")
                )
                self.logger.debug("Response -> %s", str(response))
                return True
            case "caseType":
                case_type = self._otawp.get_case_type_by_name(name=entity.get("name"))
                if case_type:
                    case_type_id = self._otawp.get_entity_value(entity=case_type, key="id")
                    self.logger.info(
                        "Case type -> '%s' (%s) does already exist. Skipping...", entity.get("name"), str(case_type_id)
                    )
                    return True
                response = self._otawp.create_case_type(
                    name=entity.get("name"),
                    description=entity.get("description", ""),
                    status=entity.get("status", 1),
                )
                if not response or "Identity" not in response:
                    self.logger.error("Failed to case type -> '%s'!", entity.get("name"))
                    return False
                self.logger.info(
                    "Successfully created case type -> '%s' (%s).", entity.get("name"), response["Identity"].get("Id")
                )
                self.logger.debug("Response -> %s", str(response))
                return True
            case "customer":
                customer = self._otawp.get_customer_by_name(name=entity.get("name"))
                if customer:
                    customer_id = self._otawp.get_entity_value(entity=customer, key="id")
                    self.logger.info(
                        "Customer -> '%s' (%s) does already exist. Skipping...", entity.get("name"), str(customer_id)
                    )
                    return True
                response = self._otawp.create_customer(
                    customer_name=entity.get("name"),
                    legal_business_name=entity.get("legal_business_name", ""),
                    trading_name=entity.get("trading_name", ""),
                )
                if not response or "Identity" not in response:
                    self.logger.error("Failed to create customer -> '%s'!", entity.get("name"))
                    return False
                self.logger.info(
                    "Successfully created customer -> '%s' (%s).", entity.get("name"), response["Identity"].get("Id")
                )
                self.logger.debug("Response -> %s", str(response))
                return True
            case "case":
                if "subject" not in entity:
                    self.logger.error("Cannot create a case without a subject!")
                    return False

                category_name = entity.get("category")
                if category_name:
                    category = self._otawp.get_category_by_name(name=category_name)
                    category_id = self._otawp.get_entity_value(entity=category, key="id")
                    if not category_id:
                        self.logger.error(
                            "Cannot find case category -> '%s' to create case -> '%s'!",
                            category_name,
                            entity["subject"],
                        )
                        return False
                else:
                    self.logger.warning(
                        "Case entity -> '%s' does not have a category specified in its payload!", entity["subject"]
                    )
                    category_id = None

                sub_category_name = entity.get("sub_category")
                if category_id and sub_category_name:
                    sub_category_id = self._otawp.get_sub_category_id(name=sub_category_name, parent_id=category_id)
                else:
                    sub_category_id = None

                priority_name = entity.get("priority")
                if priority_name:
                    priority = self._otawp.get_priority_by_name(name=priority_name)
                    priority_id = self._otawp.get_entity_value(entity=priority, key="id")
                    if not priority_id:
                        self.logger.error(
                            "Cannot find case priority -> '%s' to create case -> '%s'!",
                            priority_name,
                            entity["subject"],
                        )
                        return False
                else:
                    self.logger.warning(
                        "Case entity -> '%s' does not have a priority specified in its payload!", entity["subject"]
                    )
                    priority_id = None

                case_type_name = entity.get("case_type")
                if case_type_name:
                    case_type = self._otawp.get_case_type_by_name(name=case_type_name)
                    case_type_id = self._otawp.get_entity_value(entity=case_type, key="id")
                    if not case_type_id:
                        self.logger.error(
                            "Cannot find case type -> '%s' to create case -> '%s'!",
                            case_type_name,
                            entity["subject"],
                        )
                        return False
                else:
                    self.logger.warning(
                        "Case entity -> '%s' does not have a case type specified in its payload!", entity["subject"]
                    )
                    case_type_id = None

                customer_name = entity.get("customer")
                if customer_name:
                    customer = self._otawp.get_customer_by_name(name=customer_name)
                    customer_id = self._otawp.get_entity_value(entity=customer, key="id")
                    if not customer_id:
                        self.logger.error(
                            "Cannot find customer -> '%s' to create case -> '%s'!",
                            customer_name,
                            entity["subject"],
                        )
                        return False
                else:
                    self.logger.warning(
                        "Case entity -> '%s' does not have a customer specified in its payload!", entity["subject"]
                    )
                    customer_id = None

                response = self._otawp.create_case(
                    subject=entity.get("subject"),
                    description=entity.get("description", ""),
                    loan_amount=entity.get("loan_amount", 1),
                    loan_duration_in_months=entity.get("loan_duration_in_month", 2),
                    category_id=category_id,
                    sub_category_id=sub_category_id,
                    priority_id=priority_id,
                    case_type_id=case_type_id,
                    customer_id=customer_id,
                )
                if not response:
                    self.logger.error("Failed to create case with subject -> '%s'!", entity.get("subject"))
                    return False
                self.logger.info(
                    "Successfully created case with subject -> '%s'%s.",
                    entity.get("subject"),
                    " for customer with ID -> '{}'".format(customer_id) if customer_id else "",
                )
                self.logger.debug("Response -> %s", str(response))
                return True
            case _:
                self.logger.error("Illegal entity type -> '%s'!", entity_type)
                return False

        return False

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_all_group_names")
    def get_all_group_names(self) -> list:
        """Construct a list of all group name.

        Returns:
            list:
                A list of all group names.

        """

        return [group.get("name") for group in self._groups]

    # end method definition
    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_status_file_name")
    def get_status_file_name(
        self,
        payload_section_name: str,
        payload_specific: bool = True,
        prefix: str = "success_",
    ) -> str:
        """Construct the name of the status file.

        Args:
            payload_section_name (str):
                The name of the payload section. This
                is used to construct the file name
            payload_specific (bool, optional):
                Whether or not the success should be specific for
                each payload file or if success is "global" - like for the deletion
                of the existing M365 teams (which we don't want to execute per
                payload file)
            prefix (str, optional):
                The prefix of the file name. Typically, either "success_" or "failure_".

        Returns:
            str:
                The constructed name of the payload section file.

        """

        # Some sections are actually not payload specific like teamsM365Cleanup
        # we don't want external payload runs to re-apply this processing:
        if payload_specific:
            file_name = os.path.basename(self._payload_source)  # remove directories
            # Split once at the first occurance of a dot
            # as the _payload_source may have multiple suffixes
            # such as .yml.gz.b64:
            file_name = file_name.split(".", 1)[0]
            file_name = prefix + file_name + "_" + payload_section_name + ".json"
        else:
            file_name = prefix + payload_section_name + ".json"

        return file_name

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="check_status_file")
    def check_status_file(
        self,
        payload_section_name: str,
        payload_specific: bool = True,
        prefix: str = "success_",
    ) -> bool:
        """Check if the payload section has been processed before.

        This is done by checking the existance of a text file in the Admin Personal
        workspace in Content Server with the name of the payload section.

        Args:
            payload_section_name (str):
                The name of the payload section. This
                is used to construct the file name
            payload_specific (bool, optional):
                Whether or not the success should be specific for
                each payload file or if success is "global" - like for the deletion
                of the existing M365 teams (which we don't want to execute per
                payload file)
            prefix (str, optional):
                The prefix of the file. Typically, either "success_" or "failure_".

        Returns:
            bool:
                True if the payload has been processed successfully before, False otherwise

        """

        message = "successfully" if prefix.startswith("success_") else "with failures"

        self.logger.info(
            "Check if payload section -> '%s' has been processed %s before...",
            payload_section_name,
            message,
        )

        while not self._otcs.is_ready():
            self.logger.info(
                "OTCS is not ready. Cannot check status file for -> '%s' in OTCS. Waiting 30 seconds and retry...",
                payload_section_name,
            )
            time.sleep(30)

        response = self._otcs.get_node_by_volume_and_path(
            volume_type=self._otcs.VOLUME_TYPE_PERSONAL_WORKSPACE,
        )  # write to Personal Workspace of Admin
        target_folder_id = self._otcs.get_result_value(response=response, key="id")
        if not target_folder_id:
            target_folder_id = 2004  # use Personal Workspace of Admin as fallback

        file_name = self.get_status_file_name(
            payload_section_name=payload_section_name,
            payload_specific=payload_specific,
            prefix=prefix,
        )

        status_document = self._otcs.get_node_by_parent_and_name(
            parent_id=int(target_folder_id),
            name=file_name,
            show_error=False,
        )
        if status_document and status_document.get("results", []):
            name = self._otcs.get_result_value(response=status_document, key="name")
            if name == file_name:
                self.logger.info(
                    "Payload section -> '%s' has been processed %s before. %s...",
                    payload_section_name,
                    message,
                    "Skipping" if prefix.startswith("success_") else "Retrying",
                )
                return True
        self.logger.info(
            "Payload section -> '%s' has not been processed %s before. Processing...",
            payload_section_name,
            message,
        )
        return False

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="write_status_file")
    def write_status_file(
        self,
        success: bool,
        payload_section_name: str,
        payload_section: list,
        payload_specific: bool = True,
    ) -> bool:
        """Write a status file into the Admin Personal Workspace in Content Server.

        This is to indicate that the payload section has been deployed successfully.
        This speeds up the customizing process in case the customizer pod
        is restarted.

        Args:
            success (bool):
                True, if the section was processed successful, False otherwise.
            payload_section_name (str):
                The name of the payload section.
            payload_section (list):
                The payload section content - this is written as JSon into the file.
            payload_specific (bool, optional):
                Whether or not the success should be specific for
                each payload file or if success is "global" - like for the deletion
                of the existing M365 teams (which we don't want to execute per
                payload file)

        Returns:
            bool:
                True if the status file as been upladed to Content Server successfully, False otherwise

        """

        # Do we want status files to be uploaded?
        if not self.upload_status_files:
            return True

        if success:
            self.logger.info(
                "Payload section -> '%s' has been completed successfully.",
                payload_section_name,
            )
            prefix = "success_"
        else:
            self.logger.error(
                "Payload section -> '%s' had failures!",
                payload_section_name,
            )
            prefix = "failure_"

        while not self._otcs.is_ready():
            self.logger.info(
                "OTCS is not ready. Cannot write status file for -> '%s' to OTCS. Waiting 30 seconds and retry...",
                payload_section_name,
            )
            time.sleep(30)

        response = self._otcs.get_node_by_volume_and_path(
            volume_type=self._otcs.VOLUME_TYPE_PERSONAL_WORKSPACE,
        )  # write to Personal Workspace of Admin (with Volume Type ID = 142)
        target_folder_id = self._otcs.get_result_value(response=response, key="id")
        if not target_folder_id:
            target_folder_id = 2004  # use Personal Workspace of Admin as fallback

        file_name = self.get_status_file_name(
            payload_section_name=payload_section_name,
            payload_specific=payload_specific,
            prefix=prefix,
        )

        full_path = os.path.join(tempfile.gettempdir(), file_name)

        with open(full_path, mode="w", encoding="utf-8") as localfile:
            localfile.write(json.dumps(payload_section, indent=2))

        # Check if the status file has been uploaded before.
        # This can happen if we re-run the python container.
        # In this case we add a version to the existing document:
        response = self._otcs.get_node_by_parent_and_name(
            parent_id=int(target_folder_id),
            name=file_name,
            show_error=False,
        )
        target_document_id = self._otcs.get_result_value(response=response, key="id")
        if target_document_id:
            response = self._otcs.add_document_version(
                node_id=int(target_document_id),
                file_url=full_path,
                file_name=file_name,
                mime_type="text/plain",
                description="Updated status file after re-run of customization",
            )
        else:
            response = self._otcs.upload_file_to_parent(
                file_url=full_path,
                file_name=file_name,
                mime_type="text/plain",
                parent_id=int(target_folder_id),
            )

        if response:
            self.logger.info(
                "Status file -> '%s' has been written to Personal Workspace of admin user.",
                file_name,
            )
            return True

        self.logger.error(
            "Failed to write status file -> '%s' to Personal Workspace of admin user!",
            file_name,
        )

        return False

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_status_file")
    def get_status_file(
        self,
        payload_section_name: str,
        payload_specific: bool = True,
        prefix: str = "success_",
    ) -> list | None:
        """Get the status file and read it into a list of dictionaries.

        Args:
            payload_section_name (str):
                The name of the payload section. This
                is used to construct the file name.
            payload_specific (bool, optional):
                Whether or not the success should be specific for
                each payload file or if success is "global" - like for the deletion
                of the existing M365 teams (which we don't want to execute per
                payload file)
            prefix (str, optional):
                The prefix of the file. Typically, either "success_" or "failure_".

        Returns:
            list:
                Content of the status file as a list of dictionaries or None in case of an error.

        """

        self.logger.info(
            "Get the status file of the payload section -> '%s'...",
            payload_section_name,
        )

        while not self._otcs.is_ready():
            self.logger.info(
                "OTCS is not ready. Cannot read status file for -> '%s' from OTCS. Waiting 30 seconds and retry...",
                payload_section_name,
            )
            time.sleep(30)

        response = self._otcs.get_node_by_volume_and_path(
            volume_type=self._otcs.VOLUME_TYPE_PERSONAL_WORKSPACE,
        )  # read from Personal Workspace of Admin
        source_folder_id = self._otcs.get_result_value(response=response, key="id")
        if not source_folder_id:
            source_folder_id = 2004  # use Personal Workspace ID of Admin as fallback

        file_name = self.get_status_file_name(
            payload_section_name=payload_section_name,
            payload_specific=payload_specific,
            prefix=prefix,
        )

        status_document = self._otcs.get_node_by_parent_and_name(
            parent_id=int(source_folder_id),
            name=file_name,
            show_error=True,
        )
        status_file_id = self._otcs.get_result_value(response=status_document, key="id")
        if not status_file_id:
            self.logger.error("Cannot find status file -> '%s'!", file_name)
            return None

        return self._otcs.get_json_document(node_id=status_file_id)

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_payload")
    def get_payload(self, drop_bulk_datasources_data: bool = False) -> dict:
        """Get the Payload as reference.

        Optional a copy of the payload can be delivered the does not include the
        "data" value of "bulkDatasource" (its content can be HUGE and many times we don't
        want it).

        Args:
            drop_bulk_datasources_data (bool, optional):
                If True, returns a !copy! of the payload without the bulkDatasources
                "data" in it. Defaults to False.

        Returns:
            dict: Returns the payload

        """

        if drop_bulk_datasources_data and "bulkDatasources" in self._payload:
            # Create a Copy of the payload, but without the bulkDatasources
            # using deepcopy would require significantly more memory, just to drop it directly after
            payload = {k: v for k, v in self._payload.items() if k != "bulkDatasources"}

            payload["bulkDatasources"] = [
                {k: v for k, v in data.items() if k != "data"} for data in self._payload.get("bulkDatasources", {})
            ]
            return payload

        return self._payload

    # end method definition

    def get_users(self) -> list:
        """Get all users in payload."""
        return self._users

    def get_groups(self) -> list:
        """Get all groups in payload."""
        return self._groups

    def get_workspaces(self) -> list:
        """Get all workspaces in payload."""
        return self._workspaces

    def get_otcs_frontend(self) -> object:
        """Get OTCS Frontend oject."""
        return self._otcs_frontend

    def get_otcs_backend(self) -> object:
        """Get OTCS Backend object."""
        return self._otcs_backend

    def get_otds(self) -> object:
        """Get the OTDS object."""
        return self._otds

    def get_k8s(self) -> object:
        """Get the K8s object."""
        return self._k8s

    def get_m365(self) -> object:
        """Get the M365 object."""
        return self._m365

    def generate_password(
        self,
        length: int,
        use_special_chars: bool = False,
        min_special: int = 1,
        min_numerical: int = 1,
        min_upper: int = 1,
        min_lower: int = 1,
        override_special: str | None = None,
    ) -> str | None:
        """Generate random passwords with a given specification.

        Args:
            length (int):
                Define password length.
            use_special_chars (bool, optional):
                Define if special characters should be used. Defaults to False.
            min_special (int, optional):
                Define min amount of special characters. Defaults to 1.
            min_numerical (int, optional):
                Define if numbers should be used. Defaults to 1.
            min_upper (int, optional):
                Define mininum number of upper case letters. Defaults to 1.
            min_lower (int, optional):
                Define minimum number of lower case letters. Defaults to 1.
            override_special (string | None, optional):
                Define special characters to be used, if not set: !@#$%^&*()_-+=<>?/{}[]. Defaults to None.

        Returns:
            str | None:
                The generated password. None in case of an error.

        """
        # Define character sets
        lowercase_letters = string.ascii_lowercase
        uppercase_letters = string.ascii_uppercase
        numerical_digits = string.digits
        special_characters = "!@#$%^&*()_-+=<>?/{}[]"

        if override_special:
            special_characters = override_special
        # Ensure minimum requirements are met

        if min_special + min_numerical + min_upper + min_lower > length:
            self.logger.error("Minimum requirements exceed password length! Cannot generate password.")
            return None

        # Initialize the password
        password = []

        # Add required characters
        password.extend(random.sample(lowercase_letters, min_lower))
        password.extend(random.sample(uppercase_letters, min_upper))
        password.extend(random.sample(numerical_digits, min_numerical))

        if use_special_chars:
            password.extend(random.sample(special_characters, min_special))

        # Fill the rest of the password with random characters
        remaining_length = length - len(password)
        all_chars = lowercase_letters + uppercase_letters + numerical_digits

        if use_special_chars:
            all_chars += special_characters

        password.extend(random.choices(all_chars, k=remaining_length))

        # Shuffle the password to ensure randomness
        random.shuffle(password)

        # Convert the password list to a string
        final_password = "".join(password)

        return final_password

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_group_id")
    def determine_group_id(self, group: dict) -> int:
        """Determine the id of a group - either from payload or from OTCS.

        If the group is found in OTCS write back the ID into the payload.

        Args:
            group (dict):
                The group payload element.

        Returns:
            int: group ID

        Side Effects:
            the group items are modified by adding an "id" dict element that
            includes the technical ID of the group in Content Server.

        """

        # Is the ID already known in payload? (if determined before)
        if "id" in group:
            return group["id"]

        if "name" not in group:
            self.logger.error("Group needs a name to lookup the ID!")
            return 0
        group_name = group["name"]

        existing_groups = self._otcs.get_group(name=group_name)
        # We use the lookup method here as get_group() could deliver more
        # then 1 result element (in edge cases):
        existing_group_id = self._otcs.lookup_result_value(
            response=existing_groups,
            key="name",
            value=group_name,
            return_key="id",
        )

        # Have we found an exact match?
        if existing_group_id:
            self.logger.debug(
                "Found existing group -> '%s' with ID -> %s. Update ID in payload...",
                group_name,
                existing_group_id,
            )
            # Write ID back into the payload:
            group["id"] = existing_group_id
            return group["id"]
        else:
            self.logger.debug(
                "Cannot find an existing group -> '%s'",
                group_name,
            )
            return 0

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_group_id_m365")
    def determine_group_id_m365(self, group: dict) -> str | None:
        """Determine the id of a M365 group - either from payload or from M365 via Graph API.

        If the group is found in M365 write back the M365 group ID into the payload.

        Args:
            group (dict):
                The group payload element.

        Returns:
            str | None:
                M365 group ID or None if the group is not found.

        Side Effects:
            The group items are modified by adding an "m365_id" dict item that
            includes the technical ID of the group in Microsoft 365.

        """

        # is the payload already updated with the M365 group ID?
        if "m365_id" in group:
            return group["m365_id"]

        if "name" not in group:
            self.logger.error("Group needs a name to lookup the M365 group ID!")
            return None
        group_name = group["name"]

        existing_group = self._m365.get_group(group_name=group_name)
        existing_group_id = self._m365.get_result_value(
            response=existing_group,
            key="id",
        )
        if existing_group_id:
            self.logger.debug(
                "Found existing Microsoft 365 group -> '%s' with ID -> %s. Update m365_id in payload...",
                group_name,
                existing_group_id,
            )
            # write back the M365 user ID into the payload
            group["m365_id"] = existing_group_id
            return group["m365_id"]
        else:
            self.logger.debug(
                "Cannot find an existing M365 group -> '%s'",
                group_name,
            )
            return None

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_group_id_core_share")
    def determine_group_id_core_share(self, group: dict) -> str | None:
        """Determine the id of a Core Share group.

        This can either be derived from payload or from Core Share directly.

        Args:
            group (dict):
                The payload dictionary of the group.

        Returns:
            str | None:
                Core Share Group ID or None.

        """

        # Is the ID already known in payload? (if determined before)
        if "core_share_id" in group:
            return group["core_share_id"]

        if "name" not in group:
            self.logger.error("Group needs a name to lookup the Core Share ID!")
            return None

        if not isinstance(self._core_share, CoreShare):
            self.logger.error(
                "Core Share connection not setup properly!",
            )
            return None

        core_share_group = self._core_share.get_group_by_name(name=group["name"])
        core_share_group_id = self._core_share.get_result_value(
            response=core_share_group,
            key="id",
        )

        # Have we found the group?
        if core_share_group_id:
            self.logger.debug(
                "Found existing Core Share group -> '%s' with ID -> %s. Update m365_id in payload...",
                group["name"],
                core_share_group_id,
            )
            # Write ID back into the payload:
            group["core_share_id"] = core_share_group_id
            return group["core_share_id"]
        else:
            self.logger.debug(
                "Cannot find an existing Core Share group -> '%s'",
                group["name"],
            )
            return None

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_user_id")
    def determine_user_id(self, user: dict) -> int:
        """Determine the id of a user - either from payload or from OTCS.

        If the user is found in OTCS write back the ID into the payload.

        Args:
            user (dict):
                The user payload element.

        Returns:
            int:
                The user ID in OTCS.

        Side Effects:
            The user items are modified by adding an "id" dict element that
            includes the technical ID of the user in Content Server

        """

        # Is the ID already known in payload? (if determined before)
        if "id" in user:
            return user["id"]

        user_name = user.get("name")
        if not user_name:
            self.logger.error("User needs a login name to lookup the ID!")
            return 0

        user_type = 17 if user.get("type", "User") == "ServiceUser" else 0

        response = self._otcs.get_user(name=user_name, user_type=user_type)

        # We use the lookup method here as get_user() could deliver more
        # then 1 result element (in edge cases):
        user_id = self._otcs.lookup_result_value(
            response=response,
            key="name",
            value=user_name,
            return_key="id",
        )

        # Have we found an exact match?
        if user_id:
            # Write ID back into the payload
            user["id"] = user_id
            return user["id"]
        else:
            self.logger.debug(
                "Cannot find an existing user -> '%s'!",
                user_name,
            )
            return 0

        # end method definition

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_user_id_m365")
    def determine_user_id_m365(self, user: dict) -> str | None:
        """Determine the id of a M365 user - either from payload or from M365 via Graph API.

        If the user is found in M365 write back the M365 user ID into the payload.

        Args:
            user (dict):
                The user payload element.

        Returns:
            str | None:
                The M365 user ID or None if the user is not found.

        Side Effects:
            the user items are modified by adding an "m365_id" dict element that
            includes the technical ID of the user in Microsoft 365

        """

        # is the payload already updated with the M365 user ID?
        if "m365_id" in user:
            return user["m365_id"]

        if "name" not in user:
            self.logger.error("User needs a login name to lookup the M365 user ID!")
            return None
        user_name = user["name"]

        m365_user_name = user_name + "@" + self._m365.config()["domain"]
        existing_user = self._m365.get_user(user_email=m365_user_name)
        if existing_user:
            self.logger.debug(
                "Found existing Microsoft 365 user -> '%s' (%s) with ID -> %s. Update m365_id in payload...",
                existing_user["displayName"],
                existing_user["userPrincipalName"],
                existing_user["id"],
            )
            # write back the M365 user ID into the payload
            user["m365_id"] = existing_user["id"]
            return user["m365_id"]
        else:
            self.logger.debug(
                "Did not find an existing M365 user -> '%s'",
                user_name,
            )
            return None

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_user_id_core_share")
    def determine_user_id_core_share(self, user: dict) -> str | None:
        """Determine the ID of a Core Share user.

        The ID is either taken from payload or from Core Share directly.

        Args:
            user (dict):
                The payload dictionary of the user.

        Returns:
            str | None:
                Core Share User ID or None.

        """

        # Is the ID already known in payload? (if determined before)
        if "core_share_id" in user:
            return user["core_share_id"]

        if not isinstance(self._core_share, CoreShare):
            self.logger.error(
                "Core Share connection not setup properly!",
            )
            return False

        core_share_user_id = None

        # Next try to lookup ID via the email address:
        if "email" in user:
            core_share_user = self._core_share.get_user_by_email(email=user["email"])
            core_share_user_id = self._core_share.get_result_value(
                response=core_share_user,
                key="id",
            )

        # Last resort is to lookup the ID via firstname + lastname.
        # This is handy in case the Email has changed:
        if not core_share_user_id and "lastname" in user and "firstname" in user:
            core_share_user = self._core_share.get_user_by_name(
                first_name=user["firstname"],
                last_name=user["lastname"],
            )
            core_share_user_id = self._core_share.get_result_value(
                response=core_share_user,
                key="id",
            )

        # Have we found the user?
        if core_share_user_id:
            # Write ID back into the payload:
            user["core_share_id"] = core_share_user_id
            return user["core_share_id"]
        else:
            if "email" in user:
                self.logger.debug(
                    "Did not find an existing Core Share user with email -> '%s'",
                    user["email"],
                )
            else:
                self.logger.debug(
                    "Cannot find an existing Core Share user -> '%s %s'",
                    user.get("firstname"),
                    user.get("lastname"),
                )
            return None

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_workspace_id")
    def determine_workspace_id(self, workspace: dict) -> int:
        """Determine the node ID of a workspace - either from payload or from OTCS.

        Args:
            workspace (dict):
                The workspace payload element.

        Returns:
            int:
                The workspace Node ID.

        Side Effects:
            The workspace items are modified by adding an "nodeId" dict element that
            includes the node ID of the workspace in Content Server.

        """

        if "nodeId" in workspace:
            return workspace["nodeId"]

        response = self._otcs.get_workspace_by_type_and_name(
            type_name=workspace["type_name"],
            name=workspace["name"],
        )
        workspace_id = self._otcs.get_result_value(response=response, key="id")
        if workspace_id:
            if not isinstance(workspace_id, int):
                self.logger.warning("Converting workspace ID -> %s to integer...", str(workspace_id))
                workspace_id = int(workspace_id)
            # Write nodeID back into the payload
            workspace["nodeId"] = workspace_id
            return workspace_id
        else:
            self.logger.info(
                "Workspace of type -> '%s' and name -> '%s' does not yet exist.",
                workspace["type_name"],
                workspace["name"],
            )
            return 0

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_workspace_type_and_template_id")
    def determine_workspace_type_and_template_id(
        self,
        workspace_type_name: str,
        workspace_template_name: str = "",
    ) -> tuple[int | None, int | None]:
        """Determine the IDs of type and template based on the provided names.

        This depends on the self._workspace_types list to be up to date
        (see process_workspace_types()).

        Args:
            workspace_type_name (str):
                Name of the workspace type.
            workspace_template_name (str, optional):
                Name of the workspace template. Defaults to "".

        Returns:
            tuple[int, int]:
                IDs of the workspace type (first) and workspace template (second).

        """

        # Check if the customizer has initialized the workspace type list
        if not self._workspace_types:
            self.logger.error(
                "Workspace type list is not initialized! This should never happen!",
            )
            return (None, None)

        # Find the workspace type with the name given in the payload:
        workspace_type = next(
            (item for item in self._workspace_types if item["name"] == workspace_type_name),
            None,
        )
        if workspace_type is None:
            self.logger.error(
                "Workspace type -> '%s' not found!",
                workspace_type_name,
            )
            return (None, None)

        workspace_type_id = workspace_type["id"]

        if not workspace_type.get("templates", []):
            self.logger.warning(
                "Workspace type -> '%s' does not have templates!",
                workspace_type_name,
            )
            return (workspace_type_id, None)

        if workspace_template_name:
            workspace_template = next(
                (item for item in workspace_type["templates"] if item["name"] == workspace_template_name),
                None,
            )
            if workspace_template:  # does this template exist?
                self.logger.info(
                    "Workspace template -> '%s' has been specified in payload and it does exist.",
                    workspace_template_name,
                )
            else:
                self.logger.error(
                    "Workspace template -> '%s' has been specified in payload but it doesn't exist!",
                    workspace_template_name,
                )
                self.logger.error(
                    "Workspace type -> '%s' has only these templates -> %s",
                    workspace_type_name,
                    workspace_type["templates"],
                )
                return (workspace_type_id, None)

        # template to be used is NOT specified in the payload - then we just take the first one:
        else:
            workspace_template = workspace_type["templates"][0]
            self.logger.info(
                "Workspace template has not been specified in payload - taking the first one (%s)...",
                workspace_template,
            )

        workspace_template_id = workspace_template["id"]

        return (workspace_type_id, workspace_template_id)

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="add_transport_extractions")
    def add_transport_extractions(self, extractions: list) -> int:
        """Determine the number of extrations.

        Safe them in a global list self._transport_extractions.

        Args:
            extractions (list):
                A list of extractions from a single transport package.

        Returns:
            int:
                THE number of extractions that have actually extracted data.

        """

        counter = 0
        for extraction in extractions:
            if extraction.get("enabled", True) and "data" in extraction:
                self._transport_extractions.append(extraction)
                counter += 1
        self.logger.info("Added -> %s transport extractions.", str(counter))

        return counter

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_payload")
    def process_payload(self) -> None:
        """Process a payload file."""

        if not self._payload_sections:
            self.logger.warning("No payload sections defined!")
            return

        for payload_section in self._payload_sections:
            with tracer.start_as_current_span("process_payload-section") as t:
                t.set_attributes({"payload_section": payload_section["name"]})
                self.logger.debug("processing payload_section -> %s", payload_section)
                if not payload_section.get("enabled", True):
                    self.logger.info(
                        "Payload section -> '%s' is disabled. Skipping...",
                        payload_section.get("name", "Name not defined"),
                    )
                    continue

                match payload_section["name"]:
                    case "embeddings" | "feme":
                        self._log_header_callback(
                            text="Process additional Embeddings for Content Aviator",
                        )
                        self.process_embeddings()
                    case "avtsRepositories":
                        self._log_header_callback(
                            text="Process Aviator Search repositories",
                        )
                        self.process_avts_repositories()
                    case "avtsQuestions":
                        self._log_header_callback(
                            text="Process Aviator Search Sample Questions",
                        )
                        self.process_avts_questions()
                    case "appworks":
                        self._log_header_callback(text="Process AppWorks Configurations")
                        self.process_appworks_configurations()
                    case "webHooks":
                        self._log_header_callback(text="Process Web Hooks")
                        self.process_web_hooks(webhooks=self._webhooks)
                    case "webHooksPost":
                        self._log_header_callback(text="Process Web Hooks (post)")
                        self.process_web_hooks(
                            webhooks=self._webhooks_post,
                            section_name="webHooksPost",
                        )
                    case "resources":
                        self._log_header_callback(text="Process OTDS Resources")
                        self.process_resources()
                    case "partitions":
                        self._log_header_callback(text="Process OTDS Partitions")
                        self.process_partitions()
                    case "licenses":
                        self._log_header_callback(text="Process OTDS Licenses")
                        self.process_licenses()
                    case "synchronizedPartitions":
                        self._log_header_callback(
                            text="Process OTDS Synchronized Partitions",
                        )
                        self.process_synchronized_partitions()
                    case "oauthClients":
                        self._log_header_callback(text="Process OTDS OAuth Clients")
                        self.process_oauth_clients()
                    case "applicationRoles":
                        self._log_header_callback(text="Process OTDS Application Roles")
                        self.process_application_roles()
                    case "authHandlers":
                        self._log_header_callback(text="Process OTDS Auth Handlers")
                        self.process_auth_handlers()
                    case "trustedSites":
                        self._log_header_callback(text="Process OTDS Trusted Sites")
                        self.process_trusted_sites()
                    case "systemAttributes":
                        self._log_header_callback(text="Process OTDS System Attributes")
                        self.process_system_attributes()
                    case "docgenSettings":
                        if self._otpd and isinstance(self._otpd, OTPD):
                            self._log_header_callback(text="Process OTPD Settings")
                            self.process_docgen_settings()
                    case "groups":
                        self._log_header_callback(text="Process Groups")
                        self.process_groups()
                        # Add all groups with ID the a lookup dict for placeholder replacements
                        # in adminSetting. This also updates the payload with group IDs from OTCS
                        # if the group already exists in Content Server. This is important especially
                        # if the customizer pod is restarted / run multiple times:
                        self.process_group_placeholders()
                        if self._core_share and isinstance(self._core_share, CoreShare):
                            self._log_header_callback(text="Process Core Share Groups", char="-")
                            self.process_groups_core_share()
                        if self._m365 and isinstance(self._m365, M365):
                            self._log_header_callback(text="Cleanup existing M365 Teams", char="-")
                            self.cleanup_all_teams_m365()
                            self._log_header_callback(text="Process M365 Groups", char="-")
                            self.process_groups_m365()
                    case "users":
                        self._log_header_callback(text="Process Users")
                        self._user_customization = bool(
                            payload_section.get("user_customization", "True"),
                        )
                        self.process_users()
                        # Add all users with ID the a lookup dict for placeholder replacements
                        # in adminSetting. This also updates the payload with user IDs from OTCS
                        # if the user already exists in Content Server. This is important especially
                        # if the cutomizer pod is restarted / run multiple times:
                        self.process_user_placeholders()
                        self._log_header_callback(
                            text="Assign OTCS licenses to users",
                            char="-",
                        )
                        self.process_user_licenses(
                            resource_name=self._otcs.config()["resource"],
                            license_feature=self._otcs.config()["license"],
                            user_specific_payload_field="licenses",
                        )
                        self._log_header_callback(
                            text="Process OTDS user settings",
                            char="-",
                        )
                        self.process_user_settings()
                        if self._core_share and isinstance(self._core_share, CoreShare):
                            self._log_header_callback(text="Process Core Share users", char="-")
                            self.process_users_core_share()
                        if self._m365 and isinstance(self._m365, M365):
                            self._log_header_callback(text="Process M365 users", char="-")
                            self.process_users_m365()
                            # We need to do the MS Teams creation after the creation of
                            # the M365 users as we require Group Owners to create teams.
                            # Note: this is just for the teams of the top-level OTCS groups
                            # (departments), not the MS Teams for the Workspaces. These
                            # are created via the scheduled bots!
                            self._log_header_callback(text="Process M365 Teams for departmental groups", char="-")
                            self.process_teams_m365()
                    case "adminSettings":
                        self._log_header_callback(
                            text="Process OTCS administration settings",
                        )
                        restart_required = self.process_admin_settings(
                            admin_settings=self._admin_settings,
                        )
                        if restart_required:
                            self.logger.info(
                                "Admin Settings require a restart of OTCS services...",
                            )
                            # Restart OTCS frontend and backend pods:
                            self._otcs_restart_callback(
                                backend=self._otcs_backend,
                                frontend=self._otcs_frontend,
                            )
                    case "adminSettingsPost":
                        self._log_header_callback(
                            text="Process OTCS administration settings (post)",
                        )
                        restart_required = self.process_admin_settings(
                            self._admin_settings_post,
                            "adminSettingsPost",
                        )
                        if restart_required:
                            self.logger.info(
                                "Admin settings (Post) require a restart of OTCS services...",
                            )
                            # Restart OTCS frontend and backend pods:
                            self._otcs_restart_callback(
                                backend=self._otcs_backend,
                                frontend=self._otcs_frontend,
                            )
                    case "execPodCommands":
                        self._log_header_callback(text="Process Pod Commands")
                        self.process_exec_pod_commands()
                    case "kubernetes":
                        self._log_header_callback(text="Process Kubernetes Commands")
                        self.process_kubernetes()
                    case "execCommands":
                        self._log_header_callback(text="Process Commands in Customizer Pod")
                        self.process_exec_commands()
                    case "execDatabaseCommands":
                        self._log_header_callback(text="Process Database Commands")
                        self.process_exec_database_commands()
                    case "csApplications":
                        self._log_header_callback(text="Process CS Apps (backend)")
                        self.process_cs_applications(
                            otcs_object=self._otcs_backend,
                            section_name="csApplicationsBackend",
                        )
                        self._log_header_callback("Process CS Apps (frontend)")
                        self.process_cs_applications(
                            otcs_object=self._otcs_frontend,
                            section_name="csApplicationsFrontend",
                        )
                    case "externalSystems":
                        self._log_header_callback(
                            text="Process External System Connections",
                        )
                        self.process_external_systems()
                        # Now the SAP, SuccessFactors and Salesforce objects
                        # should be initialized and we can process users and groups
                        # in these external systems:
                        if self._sap and isinstance(self._sap, SAP):
                            self._log_header_callback(text="Process SAP Users")
                            self.process_users_sap()
                        if self._successfactors and isinstance(
                            self._successfactors,
                            SuccessFactors,
                        ):
                            self._log_header_callback(text="Process SuccessFactors Users")
                            self.process_users_successfactors()
                        if self._salesforce and isinstance(self._salesforce, Salesforce):
                            self._log_header_callback(text="Process Salesforce Groups")
                            self.process_groups_salesforce()
                            self._log_header_callback(text="Process Salesforce Users")
                            self.process_users_salesforce()
                    case "transportPackages":
                        self._log_header_callback(text="Process Transport Packages")
                        self.process_transport_packages(self._transport_packages)
                        # Right after the transport that create the business object types
                        # and the workspace types we extract them and put them into
                        # generated payload lists:
                        self._log_header_callback(text="Process Business Object Types")
                        self.process_business_object_types()
                        self._log_header_callback(text="Process Workspace Types")
                        self.process_workspace_types()
                        if self._m365 and isinstance(self._m365, M365):
                            # Right after the transport that creates the top level folders
                            # we can add the M365 Teams apps for Extended ECM as its own tab:
                            self._log_header_callback(text="Process M365 Teams Apps")
                            self.process_teams_m365_apps()
                            # The SharePoint sites require the top-level departmental folder.
                            # So we can do this only after the transport packages have been
                            # processed:
                            self._log_header_callback(text="Process M365 SharePoint sites for departmental groups")
                            self.process_sites_m365()
                    case "contentTransportPackages":
                        self._log_header_callback("Process Content Transport Packages")
                        self.process_transport_packages(
                            transport_packages=self._content_transport_packages,
                            section_name="contentTransportPackages",
                        )
                        # Process workspace permissions after content has been added:
                        # (this is a workaround for a flaw in transport warehouse that don't
                        # set workspace role permissions for content transported into workspaces)
                        self._log_header_callback(
                            text="Process Workspace Member Permissions",
                            char="-",
                        )
                        self.process_workspace_member_permissions()
                    case "transportPackagesPost":
                        self._log_header_callback("Process Transport Packages (post)")
                        self.process_transport_packages(
                            transport_packages=self._transport_packages_post,
                            section_name="transportPackagesPost",
                        )
                    case "workspaceTemplates":
                        # If a payload file (e.g. additional ones) does not have
                        # transportPackages then it can happen that the
                        # self._business_object_types and self._workspace_types are
                        # not yet initialized. As we need these structures for
                        # workspaceTemplates we initialize them here:
                        if not self._business_object_types:
                            self._log_header_callback("Process Business Object Types")
                            self.process_business_object_types()
                        if not self._workspace_types:
                            self._log_header_callback("Process Workspace Types")
                            self.process_workspace_types()

                        self._log_header_callback(
                            "Process Workspace Templates (Template Role Assignments)",
                        )
                        self.process_workspace_templates()
                    case "workspaces":
                        # If a payload file (e.g. additional ones) does not have
                        # transportPackages then it can happen that the self._business_object_types and
                        # self._workspace_types are not yet initialized. As we need
                        # these structures for workspaces we initialize it here:
                        if not self._business_object_types:
                            self._log_header_callback("Process Business Object Types")
                            self.process_business_object_types()
                        if not self._workspace_types:
                            self._log_header_callback("Process Workspace Types")
                            self.process_workspace_types()

                        self._log_header_callback("Process Workspaces")
                        self.process_workspaces()
                        self._log_header_callback("Process Workspace Relationships")
                        self.process_workspace_relationships()
                        self._log_header_callback("Process Workspace Memberships")
                        self.process_workspace_members()

                        # This has to run after the processing of webReports that are
                        # used to enable Content Aviator in KINI database table:
                        if self._aviator_enabled:
                            self._log_header_callback("Process Workspace Aviators")
                            self.process_workspace_aviators()
                    case "bulkDatasources":
                        # this is here just to avoid an error in catch all below
                        # the bulkDatasources dictionary will be processed in
                        # the other bulk* sections
                        pass
                    case "bulkWorkspaces":
                        if not self._workspace_types:
                            self._log_header_callback("Process Workspace Types")
                            self.process_workspace_types()
                        self._log_header_callback("Process Bulk Workspaces")
                        self.process_bulk_workspaces()
                    case "bulkWorkspaceRelationships":
                        if not self._workspace_types:
                            self._log_header_callback("Process Workspace Types")
                            self.process_workspace_types()
                        self._log_header_callback("Process Bulk Workspace Relationships")
                        self.process_bulk_workspace_relationships()
                    case "bulkDocuments":
                        if not self._workspace_types:
                            self._log_header_callback("Process Workspace Types")
                            self.process_workspace_types()
                        self._log_header_callback("Process Bulk Documents")
                        self.process_bulk_documents()
                    case "bulkItems":
                        if not self._workspace_types:
                            self._log_header_callback("Process Workspace Types")
                            self.process_workspace_types()
                        self._log_header_callback("Process Bulk Items")
                        self.process_bulk_items()
                    case "bulkClassifications":
                        self._log_header_callback("Process Bulk Classifications")
                        self.process_bulk_classifications()
                    case "sapRFCs":
                        if self._sap and isinstance(self._sap, SAP):
                            self._log_header_callback("Process SAP RFCs")
                            self.process_sap_rfcs(sap_rfcs=self._sap_rfcs)
                        else:
                            self.logger.warning(
                                "SAP RFC in payload but SAP external system is not configured or not enabled! RFCs will not be processed.",
                            )
                    case "sapRFCsPost":
                        if self._sap and isinstance(self._sap, SAP):
                            self._log_header_callback("Process SAP RFCs (post)")
                            self.process_sap_rfcs(
                                sap_rfcs=self._sap_rfcs_post,
                                section_name="sapRFCsPost",
                            )
                        else:
                            self.logger.warning(
                                "SAP RFC in payload but SAP external system is not configured or not enabled! RFCs will not be processed.",
                            )
                    case "webReports":
                        self._log_header_callback(text="Process Web Reports")
                        restart_required = self.process_web_reports(
                            web_reports=self._web_reports,
                        )
                        if restart_required:
                            self.logger.info(
                                "Web Reports require a restart of OTCS services...",
                            )
                            # Restart OTCS frontend and backend pods:
                            self._otcs_restart_callback(
                                backend=self._otcs_backend,
                                frontend=self._otcs_frontend,
                            )
                    case "webReportsPost":
                        self._log_header_callback(text="Process Web Reports (post)")
                        restart_required = self.process_web_reports(
                            web_reports=self._web_reports_post,
                            section_name="webReportsPost",
                        )
                        if restart_required:
                            self.logger.info(
                                "WebReports (Post) require a restart of OTCS services...",
                            )
                            # Restart OTCS frontend and backend pods:
                            self._otcs_restart_callback(
                                backend=self._otcs_backend,
                                frontend=self._otcs_frontend,
                            )
                    case "additionalGroupMemberships":
                        self._log_header_callback(
                            text="Process additional group members for OTDS",
                        )
                        self.process_additional_group_members()
                    case "additionalAccessRoleMemberships":
                        self._log_header_callback(
                            text="Process additional access role members for OTDS",
                        )
                        self.process_additional_access_role_members()
                    case "additionalApplicationRoleAssignments":
                        self._log_header_callback(
                            text="Process additional application role assignments for OTDS",
                        )
                        self.process_additional_application_role_assignments()
                    case "renamings":
                        self._log_header_callback(text="Process Node Renamings")
                        self.process_renamings()
                    case "items":
                        self._log_header_callback(text="Process Items")
                        self.process_items(items=self._items)
                    case "itemsPost":
                        self._log_header_callback(text="Process Items (post)")
                        self.process_items(items=self._items_post, section_name="itemsPost")
                    case "permissions":
                        self._log_header_callback(text="Process Permissions")
                        self.process_permissions(permissions=self._permissions)
                    case "permissionsPost":
                        self._log_header_callback(text="Process Permissions (post)")
                        self.process_permissions(
                            permissions=self._permissions_post,
                            section_name="permissionsPost",
                        )
                    case "workspacePermissions":
                        self._log_header_callback(text="Process Workspace Permissions")
                        self.process_workspace_permissions()
                    case "assignments":
                        self._log_header_callback(text="Process Assignments")
                        self.process_assignments()
                    case "securityClearances":
                        self._log_header_callback(text="Process Security Clearances")
                        self.process_security_clearances()
                    case "supplementalMarkings":
                        self._log_header_callback(text="Process Supplemental Markings")
                        self.process_supplemental_markings()
                    case "recordsManagementSettings":
                        self._log_header_callback(
                            text="Process Records Management Settings",
                        )
                        self.process_records_management_settings()
                    case "holds":
                        self._log_header_callback(text="Process Records Management Holds")
                        self.process_holds()
                    case "documentGenerators":
                        # If a payload file (e.g. additional ones) does not have
                        # transportPackages then it can happen that the
                        # self._workspace_types is not yet initialized. As we need
                        # this structure for documentGenerators we initialize it here:
                        if not self._workspace_types:
                            self._log_header_callback(text="Process Workspace Types")
                            self.process_workspace_types()

                        self._log_header_callback(text="Process Document Generators")
                        self.process_document_generators()
                    case "workflowInitiations":
                        # If a payload file (e.g. additional ones) does not have
                        # transportPackages then it can happen that the
                        # self._workspace_types is not yet initialized. As we need
                        # this structure for workflowInitiations we initialize it here:
                        if not self._workspace_types:
                            self._log_header_callback(text="Process Workspace Types")
                            self.process_workspace_types()

                        self._log_header_callback(text="Process Workflow Initiations")
                        self.process_workflows()
                    case "browserAutomations":
                        self._log_header_callback(text="Process Browser Automations")
                        self.process_browser_automations(
                            browser_automations=self._browser_automations,
                        )
                    case "browserAutomationsPost":
                        self._log_header_callback(text="Process Browser Automations (post)")
                        self.process_browser_automations(
                            browser_automations=self._browser_automations_post,
                            section_name="browserAutomationsPost",
                        )
                    case "testAutomations":
                        self._log_header_callback(text="Process Test Automations")
                        self.process_browser_automations(
                            browser_automations=self._test_automations, section_name="testAutomations"
                        )
                    case "workspaceTypes":
                        if not self._workspace_types:
                            self._log_header_callback(text="Process Workspace Types")
                            self.process_workspace_types()
                    case "businessObjectTypes":
                        if not self._business_object_types:
                            self._log_header_callback(text="Process Business Object Types")
                            self.process_business_object_types()
                    case "nifi":
                        self._log_header_callback("Process Knowledge Discovery Nifi Flows")
                        self.process_nifi_flows()
                    case _:
                        self.logger.error(
                            "Illegal payload section name -> '%s' in payloadSections!",
                            payload_section["name"],
                        )
                payload_section_restart = payload_section.get("restart", False)
                if payload_section_restart:
                    self.logger.info(
                        "Payload section -> '%s' requests a restart of OTCS services...",
                        payload_section["name"],
                    )
                    # Restart OTCS frontend and backend pods:
                    self._otcs_restart_callback(
                        backend=self._otcs_backend,
                        frontend=self._otcs_frontend,
                    )
                # Avoid out of cycle message for bulkDatasources if it is
                # passed in the payload:
                elif payload_section["name"] != "bulkDatasources":
                    self.logger.info(
                        "Payload section -> '%s' does not require a restart of OTCS services.",
                        payload_section["name"],
                    )

        with tracer.start_as_current_span("process_payload_users"):
            if self._users and self._user_customization:
                self._log_header_callback("Process User Profile Photos")
                self.process_user_photos()
                if self._m365 and isinstance(self._m365, M365):
                    self._log_header_callback("Process M365 User Profile Photos")
                    self.process_user_photos_m365()
                if self._salesforce and isinstance(self._salesforce, Salesforce):
                    self._log_header_callback("Process Salesforce User Profile Photos")
                    self.process_user_photos_salesforce()
                if self._core_share and isinstance(self._core_share, CoreShare):
                    self._log_header_callback("Process Core Share User Profile Photos")
                    self.process_user_photos_core_share()
                self._log_header_callback("Process User Favorites and Profiles")
                self.process_user_favorites_and_profiles()
                self._log_header_callback("Process User Security")
                self.process_user_security()

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_web_hooks")
    def process_web_hooks(self, webhooks: list, section_name: str = "webHooks") -> bool:
        """Process Web Hooks in payload and do HTTP requests.

        Args:
            webhooks (list):
                The list of web hook payload settings.
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise.

        """

        if not webhooks:
            self.logger.info(
                "Payload section -> %s is empty. Skipping...",
                section_name,
            )
            return True

        # Even if this payload section has been processed successfully before we
        # want to call the webhook.

        success: bool = True

        for webhook in webhooks:
            url = webhook.get("url")

            # Check if element has been disabled in payload (enabled = false).
            # In this case we skip the element:
            enabled = webhook.get("enabled", True)

            if not enabled and not url:
                self.logger.info("Payload for Web Hook is disabled. Skipping...")
                continue
            if not url:
                self.logger.info("Web Hook does not have a url. Skipping...")
                success = False
                continue
            if not enabled:
                self.logger.info(
                    "Payload for Web Hook -> '%s' is disabled. Skipping...",
                    url,
                )
                continue

            description = webhook.get("description")

            method = webhook.get("method", "POST")

            payload = webhook.get("payload", {})

            headers = webhook.get("headers", {})

            if description:
                self.logger.info(
                    "Calling Web Hook -> %s: %s (%s)...",
                    method,
                    url,
                    description,
                )
            else:
                self.logger.info("Calling Web Hook -> %s: %s...", method, url)

            response = self._http_object.http_request(
                url=url,
                method=method,
                payload=payload,
                headers=headers,
                retries=webhook.get("retries", 0),
                wait_time=webhook.get("wait_time", 0),
            )
            if not response or not response.ok:
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=webhooks,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_resources")
    def process_resources(self, section_name: str = "resources") -> bool:
        """Process OTDS resources in payload and create them in OTDS.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._resources:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for resource in self._resources:
            resource_name = resource.get("name")
            if not resource_name:
                self.logger.error("OTDS resource payload does not have a name! Skipping...")
                success = False
                continue

            # Check if element has been disabled in payload (enabled = false).
            # In this case we skip the element (default is enabled):
            if not resource.get("enabled", True):
                self.logger.info(
                    "Payload for OTDS Resource -> '%s' is disabled. Skipping...",
                    resource_name,
                )
                continue

            resource_description = resource.get("description", "")
            display_name = resource.get("display_name", "")
            additional_payload = resource.get("additional_payload", {})
            activate_resource = resource.get("activate", True)
            resource_id = resource.get("resource_id", None)
            allow_impersonation = resource.get("allow_impersonation", True)
            secret = resource.get("secret", None)

            # Check if Partition does already exist
            # (in an attempt to make the code idem-potent)
            self.logger.info(
                "Check if OTDS resource -> '%s' does already exist...",
                resource_name,
            )
            response = self._otds.get_resource(name=resource_name, show_error=False)
            if response:
                self.logger.info(
                    "OTDS Resource -> '%s' does already exist. Skipping...",
                    resource_name,
                )
                continue

            # Only continue if Partition does not exist already
            self.logger.info(
                "Resource -> '%s' does not exist. Creating...",
                resource_name,
            )

            response = self._otds.add_resource(
                name=resource_name,
                description=resource_description,
                display_name=display_name,
                allow_impersonation=allow_impersonation,
                resource_id=resource_id,
                secret=secret,
                additional_payload=additional_payload,
            )
            if response:
                self.logger.info("Added OTDS resource -> '%s'.", resource_name)
            else:
                self.logger.error("Failed to add OTDS resource -> '%s'!", resource_name)
                success = False
                continue

            # If resource_id and secret are provided then the resource will
            # automatically be activated.
            if activate_resource and not secret:
                resource_id = response["resourceID"]
                self.logger.info(
                    "Activate OTDS resource -> '%s' with ID -> %s...",
                    resource_name,
                    resource_id,
                )
                response = self._otds.activate_resource(resource_id=resource_id)

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._resources,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_synchronized_partitions")
    def process_synchronized_partitions(
        self,
        section_name: str = "synchronizedPartitions",
    ) -> bool:
        """Process OTDS synchronized partitions in payload and create them in OTDS.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise.

        """

        # check if section present, if not return True
        if not self._synchronized_partitions:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True
        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success = True

        for partition in self._synchronized_partitions:
            partition_name = partition["spec"].get("profileName") if "spec" in partition else None
            if not partition_name:
                self.logger.error(
                    "Synchronized partition does not have a profile name! Skipping...",
                )
                success = False
                continue

            # Check if partition has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not partition.get("enabled", True):
                self.logger.info(
                    "Payload for synchronized partition -> '%s' is disabled. Skipping...",
                    partition_name,
                )
                continue

            partition_description = partition["spec"].get("description", "")

            # Check if Partition does already exist
            # (in an attempt to make the code idem-potent)
            self.logger.info(
                "Check if OTDS synchronized partition -> '%s' does already exist...",
                partition_name,
            )
            response = self._otds.get_partition(name=partition_name, show_error=False)
            if response:
                self.logger.info(
                    "Synchronized partition -> '%s' does already exist.",
                    partition_name,
                )
            else:
                # Only continue if synchronized Partition does not exist already
                self.logger.info(
                    "Synchronized partition -> '%s' does not yet exist. Creating...",
                    partition_name,
                )

                response = self._otds.add_synchronized_partition(
                    name=partition_name,
                    description=partition_description,
                    data=partition["spec"],
                )
                if response:
                    self.logger.info(
                        "Added synchronized partition -> '%s' to OTDS.",
                        partition_name,
                    )
                else:
                    self.logger.error(
                        "Failed to add synchronized partition -> '%s' to OTDS!",
                        partition_name,
                    )
                    success = False
                    continue

            result = self._otds.import_synchronized_partition_members(
                name=partition_name,
            )
            if result:
                self.logger.info(
                    "Successfully imported members to OTDS synchronized partition -> '%s'.",
                    partition_name,
                )
            else:
                self.logger.error(
                    "Failed to import members to OTDS synchronized partition -> '%s'!",
                    partition_name,
                )
                success = False

            access_role = partition.get("access_role")
            if access_role:
                response = self._otds.add_partition_to_access_role(
                    access_role=access_role,
                    partition=partition_name,
                )
                if response:
                    self.logger.info(
                        "Added OTDS synchronized partition -> '%s' to access role -> '%s'.",
                        partition_name,
                        access_role,
                    )
                else:
                    self.logger.error(
                        "Failed to add OTDS synchronized partition -> '%s' to access role -> '%s'!",
                        partition_name,
                        access_role,
                    )
                    success = False
            # end if access_role

            # Partions may have an optional list of licenses in
            # the payload. Assign the partition to all these licenses:
            partition_specific_licenses = partition.get("licenses")
            if partition_specific_licenses:
                # We assume these licenses are Extended ECM licenses!
                otcs_resource_name = self._otcs.config()["resource"]
                otcs_resource = self._otds.get_resource(name=otcs_resource_name)
                if not otcs_resource:
                    self.logger.error(
                        "Cannot find OTCS resource -> '%s'! Skipping...",
                        otcs_resource_name,
                    )
                    success = False
                    continue
                otcs_resource_id = otcs_resource["resourceID"]
                otcs_license_name = "EXTENDED_ECM"
                for license_item in partition_specific_licenses:
                    if isinstance(license_item, dict):
                        license_feature = license_item.get("feature")
                        license_name = license_item.get("name", "EXTENDED_ECM")
                        if "enabled" in license_item and not license_item["enabled"]:
                            self.logger.info(
                                "Payload for License '%s' -> '%s' is disabled. Skipping...",
                                license_name,
                                license_feature,
                            )
                            continue
                        if "resource" in license_item:
                            try:
                                resource_id = self._otds.get_resource(name=license_item["resource"])["resourceID"]
                            except Exception:
                                self.logger.error(
                                    "Error getting resource ID from resource -> %s", license_item["resource"]
                                )
                        else:
                            resource_id = otcs_resource_id

                    elif isinstance(license_item, str):
                        license_feature = license_item
                        resource_id = otcs_resource_id
                        license_name = otcs_license_name
                    else:
                        self.logger.error("Invalid License feature specified -> %s!", license_item)
                        success = False
                        continue

                    if self._otds.is_partition_licensed(
                        partition_name=partition_name,
                        resource_id=resource_id,
                        license_feature=license_feature,
                        license_name=license_name,
                    ):
                        self.logger.info(
                            "Partition -> '%s' is already licensed for -> '%s' (%s).",
                            partition_name,
                            license_name,
                            license_feature,
                        )
                        continue

                    assigned_license = self._otds.assign_partition_to_license(
                        partition_name=partition_name,
                        resource_id=resource_id,
                        license_feature=license_feature,
                        license_name=license_name,
                    )

                    if not assigned_license:
                        self.logger.error(
                            "Failed to assign synchronized partition -> '%s' to license feature -> '%s' of license -> '%s'!",
                            partition_name,
                            license_feature,
                            license_name,
                        )
                        success = False
                    else:
                        self.logger.info(
                            "Successfully assigned synchronized partition -> '%s' to license feature -> '%s' of license -> '%s'.",
                            partition_name,
                            license_feature,
                            license_name,
                        )
            # end if partition_specific_licenses

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._synchronized_partitions,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_partitions")
    def process_partitions(self, section_name: str = "partitions") -> bool:
        """Process OTDS partitions in payload and create them in OTDS.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise

        """

        if not self._partitions:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for partition in self._partitions:
            partition_name = partition.get("name")
            if not partition_name:
                self.logger.error("Partition does not have a name in payload! Skipping...")
                success = False
                continue

            # Check if partition has explicitly been disabled in payload
            # (enabled = false). In this case we skip the element:
            if not partition.get("enabled", True):
                self.logger.info(
                    "Payload for partition -> '%s' is disabled. Skipping...",
                    partition_name,
                )
                continue

            partition_description = partition.get("description", "")

            # Check if Partition does already exist (to make the code idem-potent):
            self.logger.info(
                "Check if OTDS partition -> '%s' does already exist...",
                partition_name,
            )
            response = self._otds.get_partition(name=partition_name, show_error=False)
            if response:
                self.logger.info(
                    "Partition -> '%s' does already exist.",
                    partition_name,
                )
            else:
                # Only continue if Partition does not exist already
                self.logger.info(
                    "Partition -> '%s' does not yet exist. Creating...",
                    partition_name,
                )

                response = self._otds.add_partition(
                    name=partition_name,
                    description=partition_description,
                )
                if response:
                    self.logger.info(
                        "Added OTDS partition -> '%s'%s.",
                        partition_name,
                        " ({})".format(partition_description) if partition_description else "",
                    )
                else:
                    self.logger.error(
                        "Failed to add OTDS partition -> '%s'!",
                        partition_name,
                    )
                    success = False
                    continue

            access_role = partition.get("access_role")
            if access_role:
                response = self._otds.add_partition_to_access_role(
                    access_role=access_role,
                    partition=partition_name,
                )
                if response:
                    self.logger.info(
                        "Added OTDS partition -> '%s' to access role -> '%s'.",
                        partition_name,
                        access_role,
                    )
                else:
                    self.logger.error(
                        "Failed to add OTDS partition -> '%s' to access role -> '%s'!",
                        partition_name,
                        access_role,
                    )
                    success = False
            # end if access_role

            # Partions may have an optional list of licenses in
            # the payload. Assign the partition to all these licenses:
            partition_specific_licenses = partition.get("licenses")
            if partition_specific_licenses:
                # We assume these licenses are Extended ECM licenses!
                otcs_resource_name = self._otcs.config()["resource"]
                otcs_resource = self._otds.get_resource(name=otcs_resource_name)
                if not otcs_resource:
                    self.logger.error(
                        "Cannot find OTCS resource -> '%s'!",
                        otcs_resource_name,
                    )
                    success = False
                    continue
                otcs_resource_id = otcs_resource["resourceID"]
                otcs_license_name = "EXTENDED_ECM"
                for license_item in partition_specific_licenses:
                    if isinstance(license_item, dict):
                        license_feature = license_item.get("feature")
                        license_name = license_item.get("name", "EXTENDED_ECM")
                        if "enabled" in license_item and not license_item["enabled"]:
                            self.logger.info(
                                "Payload for License '%s' -> '%s' is disabled. Skipping...",
                                license_name,
                                license_feature,
                            )
                            continue

                        if "resource" in license_item:
                            try:
                                resource_id = self._otds.get_resource(name=license_item["resource"])["resourceID"]
                            except Exception:
                                self.logger.error(
                                    "Error getting resource ID from resource -> '%s'!", license_item["resource"]
                                )
                        else:
                            resource_id = otcs_resource_id

                    elif isinstance(license_item, str):
                        license_feature = license_item
                        resource_id = otcs_resource_id
                        license_name = otcs_license_name
                    else:
                        self.logger.error("Invalid License feature specified -> %s!", license_item)
                        success = False
                        continue

                    if self._otds.is_partition_licensed(
                        partition_name=partition_name,
                        resource_id=resource_id,
                        license_feature=license_feature,
                        license_name=license_name,
                    ):
                        self.logger.info(
                            "Partition -> '%s' is already licensed for -> '%s' (%s).",
                            partition_name,
                            license_name,
                            license_feature,
                        )
                        continue

                    assigned_license = self._otds.assign_partition_to_license(
                        partition_name=partition_name,
                        resource_id=resource_id,
                        license_feature=license_feature,
                        license_name=license_name,
                    )

                    if not assigned_license:
                        self.logger.error(
                            "Failed to assign partition -> '%s' to license feature -> '%s' of license -> '%s'!",
                            partition_name,
                            license_feature,
                            license_name,
                        )
                        success = False
                    else:
                        self.logger.info(
                            "Successfully assigned partition -> '%s' to license feature -> '%s' of license -> '%s'.",
                            partition_name,
                            license_feature,
                            license_name,
                        )
            # end if partition_specific_licenses:
        # end for partition in self._partitions:

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._partitions,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_licenses")
    def process_licenses(self, section_name: str = "licenses") -> bool:
        """Process OTDS licenses in payload and create them in OTDS.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise

        """

        if not self._licenses:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for lic in self._licenses:
            self.logger.debug("Start processing License -> '%s'", lic)
            if not lic.get("enabled", True):
                self.logger.info(
                    "Payload for License -> '%s' is disabled. Skipping...",
                    lic,
                )
                continue

            path = lic.get("path")
            if not path:
                self.logger.error("Required attribute path not specified (%s)! Skipping ...", lic)
                success = False
                continue

            product_name = lic.get("product_name")
            if not product_name:
                self.logger.error("Required attribute 'product_name' not specified (%s)! Skipping ...", lic)
                success = False
                continue

            if "resource" in lic:
                try:
                    resource_id = self._otds.get_resource(name=lic["resource"])["resourceID"]
                except Exception:
                    self.logger.error("Error getting resource ID from resource -> '%s'!", lic["resource"])
                    success = False
                    continue
            else:
                self.logger.error("Required attribute 'resource' not specified (%s)! Skipping ...", lic)
                success = False
                continue

            self.logger.info(
                "Adding license -> '%s' for product -> '%s' to resource -> '%s'...", path, product_name, resource_id
            )

            add_license = self._otds.add_license_to_resource(
                path_to_license_file=path,
                product_name=product_name,
                product_description=lic.get("description", ""),
                resource_id=resource_id,
                update=lic.get("update", True),
            )

            if not add_license:
                success = False
                continue

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._licenses,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_oauth_clients")
    def process_oauth_clients(self, section_name: str = "oauthClients") -> bool:
        """Process OTDS OAuth clients in payload and create them in OTDS.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._oauth_clients:
            self.logger.info("Payload section -> %s is empty. Skipping...", section_name)
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for oauth_client in self._oauth_clients:
            client_name = oauth_client.get("name")
            if not client_name:
                self.logger.error("OAuth client does not have a name in payload. Skipping...")
                success = False
                continue

            # Check if oauth client has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not oauth_client.get("enabled", True):
                self.logger.info(
                    "Payload for OAuth client -> '%s' is disabled in payload. Skipping...",
                    client_name,
                )
                continue

            client_description = oauth_client.get("description", "")
            client_confidential = oauth_client.get("confidential", True)
            client_partition = oauth_client.get("partition", "Global")
            if client_partition == "Global":
                client_partition = []
            client_redirect_urls = oauth_client.get("redirect_urls", [])
            client_permission_scopes = oauth_client.get("permission_scopes")
            client_default_scopes = oauth_client.get("default_scopes")
            client_allow_impersonation = oauth_client.get("allow_impersonation", True)
            client_secret = oauth_client.get("secret", "")

            # Check if OAuth client does already exist
            # (in an attempt to make the code idem-potent)
            self.logger.info(
                "Check if OTDS OAuth client -> '%s' does already exist...",
                client_name,
            )
            response = self._otds.get_oauth_client(
                client_id=client_name,
                show_error=False,
            )
            if response:
                self.logger.info(
                    "OAuth client -> '%s' does already exist. Skipping...",
                    client_name,
                )
                continue
            self.logger.info(
                "OAuth client -> '%s' does not exist. Creating...",
                client_name,
            )

            response = self._otds.add_oauth_client(
                client_id=client_name,
                description=client_description,
                redirect_urls=client_redirect_urls,
                allow_impersonation=client_allow_impersonation,
                confidential=client_confidential,
                auth_scopes=client_partition,
                allowed_scopes=client_permission_scopes,
                default_scopes=client_default_scopes,
                secret=client_secret,
            )
            if response:
                self.logger.info("Added OTDS OAuth client -> '%s'.", client_name)
            else:
                self.logger.error(
                    "Failed to add OTDS OAuth client -> '%s'!",
                    client_name,
                )
                success = False
                continue

            # in case the secret has not been provided in the payload we retrieve
            # the automatically created secret:
            client_secret = response.get("secret")
            if not client_secret:
                self.logger.error(
                    "OAuth client -> '%s' does not have a secret!",
                    client_name,
                )
                continue

            client_description += " Client Secret: " + str(client_secret)
            response = self._otds.update_oauth_client(
                client_id=client_name,
                updates={"description": client_description},
            )
            # Write the secret back into the payload
            oauth_client["secret"] = client_secret

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._oauth_clients,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_application_roles")
    def process_application_roles(self, section_name: str = "applicationRoles") -> bool:
        """Process OTDS application rolesin payload and create them in OTDS.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._application_roles:
            self.logger.info("Payload section -> %s is empty. Skipping...", section_name)
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for role in self._application_roles:
            role_name = role.get("name")
            if not role_name:
                self.logger.error("Application role does not have a name in payload! Skipping...")
                success = False
                continue

            # Check if application role has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not role.get("enabled", True):
                self.logger.info(
                    "Payload for application roles -> '%s' is disabled. Skipping...",
                    role_name,
                )
                continue

            role_description = role.get("description")
            role_partition = role.get("partition", "OAuthClients")

            # Check if application role does already exist
            # (in an attempt to make the code idem-potent)
            self.logger.info(
                "Check if application role -> '%s' does already exist in partition -> '%s'...",
                role_name,
                role_partition,
            )
            response = self._otds.get_application_role(
                name=role,
                partition=role_partition,
                show_error=False,
            )
            if response:
                self.logger.info(
                    "Application role -> '%s' does already exist in partition -> '%s'. Skipping...",
                    role_name,
                    role_partition,
                )
                continue
            self.logger.info(
                "Application role -> '%s' does not exist in partition -> '%s'. Creating...", role_name, role_partition
            )

            response = self._otds.add_application_role(
                name=role_name,
                partition_id=role_partition,
                description=role_description,
                values=role.get("values", None),
                custom_attributes=role.get("custom_attributes", None),
            )
            if response:
                self.logger.info(
                    "Successfully added OTDS Application role -> '%s' to partition -> '%s'.", role_name, role_partition
                )
            else:
                self.logger.error(
                    "Failed to add OTDS Application role -> '%s' to partition -> '%s'!", role_name, role_partition
                )
                success = False
                continue

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._application_roles,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_auth_handlers")
    def process_auth_handlers(self, section_name: str = "authHandlers") -> bool:
        """Process OTDS authorization handlers in payload and create them in OTDS.

        An authorization handler defined the connection to an Identity Provider (IdP).

        The payload section is a list of dicts with these items:
        {
            enabled (bool):
                True or False to enable or disable the payload item
            name (str):
                Name of the authorization handler. This is shown in the first
                column of the Auth Handler list in OTDS.
            description (str):
                Description of the handler. This is shown in the second
                column of the Auth Handler
            type (str):
                Type of the handler. Possible values are SALM, SAP, OAUTH
            priority (int):
                A numeric value to order different handlers in OTDS by priority
            active_by_default (bool):
                Whether to activate this handler for any request to the
                OTDS login page. If True, any login request to the OTDS
                login page will be redirected to the IdP. If false, the
                user has to select the provider on the login page.
            provider_name:
                The name of the identity provider. This should be a single word
                since it will be part of the metadata URL. This is what is
                shown as a button on the OTDS login page.
            auth_principal_attributes:
                Authentication principal attributes (list)
            nameid_format:
                Specifies which NameID format supported by the identity provider
                contains the desired user identifier. The value in this identifier
                must correspond to the value of the user attribute specified for the
                authentication principal attribute.
            saml_url:
                Required for SAML Authentication Handler. The URL for the IdP's federation metadata.
            otds_sp_endpoint:
                Used for SAML Authentication Handler. Specifies the service provider URL that will
                be used to identify OTDS to the identity provider.
            certificate_file:
                Required for SAP Authentication Handler (SAPSSOEXT).
                Fully qualified file name (with path) to the certificate file (URI)
            certificate_password:
                Required for SAP Authentication Handler (SAPSSOEXT).
                Password of the certificate file.
            client_id:
                Client ID. Required for OAUTH authentication handler.
            client_secret:
                Client Secret. Required for OAUTH authentication handler.
            authorization_endpoint:
                Required for OAUTH authentication handler.
                The URL to redirect the browser to for authentication.
                It is used to retrieve the authorization code or an OIDC id_token.
            token_endpoint:
                Used for OAUTH authentication handler. The URL from which to retrieve the access token.
                Not strictly required with OpenID Connect if using the implicit flow.
            scope_string:
                Used for OAUTH authentication handler. Space delimited scope values to send.
                Include 'openid' to use OpenID Connect.
        }

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._auth_handlers:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for auth_handler in self._auth_handlers:
            handler_name = auth_handler.get("name")

            if not handler_name:
                self.logger.error("Auth handler does not have a name in payload. Skipping...")
                success = False
                continue

            # Check if Auth Handler does already exist (e.g. after a restart of
            # the customizer pod):
            if self._otds.get_auth_handler(name=handler_name, show_error=False):
                self.logger.info(
                    "Auth handler -> '%s' does already exist. Skipping...",
                    handler_name,
                )
                continue

            handler_description = auth_handler.get("description")

            # Check if auth handler has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not auth_handler.get("enabled", True):
                self.logger.info(
                    "Payload for OTDS authorization handler -> '%s' is disabled. Skipping...",
                    handler_name,
                )
                continue

            handler_scope = auth_handler.get("scope")
            if not handler_scope:
                # Make sure to pass None also if scope is empty string
                handler_scope = None

            handler_type = auth_handler.get("type")
            if not handler_type:
                self.logger.error(
                    "OTDS authorization handler does not have a type in payload! Skipping...",
                )
                success = False
                continue

            priority = auth_handler.get("priority")
            active_by_default = auth_handler.get("active_by_default")
            if not active_by_default:
                active_by_default = False

            match handler_type:
                case "SAML":
                    provider_name = auth_handler.get("provider_name")
                    if not provider_name:
                        self.logger.error(
                            "SAML authorization handler needs a provider name in payload! Skipping...",
                        )
                        success = False
                        continue
                    saml_url = auth_handler.get("saml_url")
                    if not saml_url:
                        self.logger.error(
                            "SAML authorization handler needs a SAML URL in payload! Skipping...",
                        )
                        success = False
                        continue
                    otds_sp_endpoint = auth_handler.get("otds_sp_endpoint")
                    if not otds_sp_endpoint:
                        self.logger.error(
                            "SAML authorization handler needs a OTDS SP endpoint in payload! Skipping...",
                        )
                        success = False
                        continue
                    auth_principal_attributes = auth_handler.get(
                        "auth_principal_attributes",
                    )
                    nameid_format = auth_handler.get("nameid_format")
                    response = self._otds.add_auth_handler_saml(
                        name=handler_name,
                        description=handler_description,
                        scope=handler_scope,
                        provider_name=provider_name,
                        saml_url=saml_url,
                        otds_sp_endpoint=otds_sp_endpoint,
                        priority=priority,
                        active_by_default=active_by_default,
                        auth_principal_attributes=auth_principal_attributes,
                        nameid_format=nameid_format,
                    )
                case "SAP":
                    certificate_file = auth_handler.get("certificate_file")
                    if not certificate_file:
                        self.logger.error(
                            "SAP Authorization handler -> '%s' (%s) requires a certificate file name in payload! Skipping...",
                            handler_name,
                            handler_type,
                        )
                        success = False
                        continue
                    certificate_password = auth_handler.get("certificate_password")
                    if not certificate_password:
                        # This is not an error - we canhave this key with empty string!
                        self.logger.info(
                            "SAP Authorization handler -> '%s' (%s) does not have a certificate password - this can be OK.",
                            handler_name,
                            handler_type,
                        )
                    response = self._otds.add_auth_handler_sap(
                        name=handler_name,
                        description=handler_description,
                        scope=handler_scope,
                        certificate_file=certificate_file,
                        certificate_password=certificate_password,
                        priority=priority,
                    )
                case "OAUTH":
                    provider_name = auth_handler.get("provider_name")
                    if not provider_name:
                        self.logger.error(
                            "OAUTH Authorization handler -> '%s' (%s) requires a provider name in payload! Skipping...",
                            handler_name,
                            handler_type,
                        )
                        success = False
                        continue
                    client_id = auth_handler.get("client_id")
                    if not client_id:
                        self.logger.error(
                            "OAUTH Authorization handler -> '%s' (%s) requires a client ID in payload! Skipping...",
                            handler_name,
                            handler_type,
                        )
                        success = False
                        continue
                    client_secret = auth_handler.get("client_secret")
                    if not client_secret:
                        self.logger.error(
                            "OAUTH Authorization handler -> '%s' (%s) requires a client secret in payload! Skipping...",
                            handler_name,
                            handler_type,
                        )
                        success = False
                        continue
                    authorization_endpoint = auth_handler.get("authorization_endpoint")
                    if not authorization_endpoint:
                        self.logger.error(
                            "OAUTH Authorization handler -> '%s' (%s) requires a authorization endpoint in payload! Skipping...",
                            handler_name,
                            handler_type,
                        )
                        success = False
                        continue
                    token_endpoint = auth_handler.get("token_endpoint")
                    if not token_endpoint:
                        self.logger.warning(
                            "OAUTH Authorization handler -> '%s' (%s) does not have a token endpoint!",
                            handler_name,
                            handler_type,
                        )
                    scope_string = auth_handler.get("scope_string")
                    response = self._otds.add_auth_handler_oauth(
                        name=handler_name,
                        description=handler_description,
                        scope=handler_scope,
                        provider_name=provider_name,
                        client_id=client_id,
                        client_secret=client_secret,
                        priority=priority,
                        active_by_default=active_by_default,
                        authorization_endpoint=authorization_endpoint,
                        token_endpoint=token_endpoint,
                        scope_string=scope_string,
                    )
                case _:
                    self.logger.error(
                        "Unsupported authorization handler type -> '%s'!",
                        handler_type,
                    )
                    return False

            if response:
                self.logger.info(
                    "Successfully added OTDS authorization handler -> '%s' (%s).",
                    handler_name,
                    handler_type,
                )
            else:
                self.logger.error(
                    "Failed to add OTDS authorization handler -> '%s' (%s)!",
                    handler_name,
                    handler_type,
                )
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._auth_handlers,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_trusted_sites")
    def process_trusted_sites(self, section_name: str = "trustedSites") -> bool:
        """Process OTDS trusted sites in payload and create them in OTDS.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise

        """

        if not self._trusted_sites:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for trusted_site in self._trusted_sites:
            # old payload may still have trusted sites as list of string
            # we changed also the trusted sites to dict with 23.3
            # We want to be backwards compatible so we handle both cases:
            url = None
            if isinstance(trusted_site, dict):
                url = trusted_site.get("url")
            elif isinstance(trusted_site, str):
                url = trusted_site

            # Check if element has been disabled in payload (enabled = false).
            # In this case we skip the element:
            if isinstance(trusted_site, dict) and "enabled" in trusted_site and not trusted_site["enabled"]:
                self.logger.info(
                    "Payload for OTDS Trusted Site -> '%s' is disabled. Skipping...",
                    url if url else "<undefined>",
                )
                continue

            if not url:
                self.logger.error("OTDS Trusted site does not have a URL! Skipping...")
                success = False
                continue

            response = self._otds.add_trusted_site(trusted_site=url)
            if response:
                self.logger.info("Added OTDS trusted site -> '%s'.", url)
            else:
                self.logger.error("Failed to add trusted site -> '%s'!", url)
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._trusted_sites,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_system_attributes")
    def process_system_attributes(self, section_name: str = "systemAttributes") -> bool:
        """Process OTDS system attributes in payload and create them in OTDS.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._system_attributes:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for system_attribute in self._system_attributes:
            attribute_name = system_attribute.get("name")
            if not attribute_name:
                self.logger.error("OTDS system attribute needs a name in payload! Skipping...")
                success = False
                continue

            # Check if system attribute has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not system_attribute.get("enabled", True):
                self.logger.info(
                    "Payload for OTDS system attribute -> '%s' is disabled. Skipping...",
                    attribute_name,
                )
                continue

            attribute_value = system_attribute.get("value")
            if not attribute_value:
                self.logger.error(
                    "OTDS system attribute -> '%s' needs a value in payload! Skipping...",
                    attribute_name,
                )
                continue

            attribute_description = system_attribute.get("description", "")

            # Add the system attribute to OTDS:
            response = self._otds.add_system_attribute(
                name=attribute_name,
                value=attribute_value,
                description=attribute_description,
            )
            if response:
                self.logger.info(
                    "Added OTDS system attribute -> '%s' with value -> '%s'.",
                    attribute_name,
                    attribute_value,
                )
            else:
                self.logger.error(
                    "Failed to add OTDS system attribute -> '%s' with value -> '%s'!",
                    attribute_name,
                    attribute_value,
                )
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._system_attributes,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_docgen_settings")
    def process_docgen_settings(self, section_name: str = "docgenSettings") -> bool:
        """Process OTPD settings in payload and configure them in OTPD.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._docgen_settings:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for docgen_setting in self._docgen_settings:
            # Check if the setting has a proper name in the payload:
            setting_name = docgen_setting.get("name")
            if not setting_name:
                self.logger.error(
                    "OTPD document generation setting needs a name in payload! Skipping...",
                )
                success = False
                continue

            # Check if the document geneneration has been explicitly disabled in payload
            # (enabled = false). In this case we skip the payload element:
            if not docgen_setting.get("enabled", True):
                self.logger.info(
                    "Payload for OTPD setting -> '%s' is disabled. Skipping...",
                    setting_name,
                )
                continue

            setting_value = docgen_setting.get("value")
            if not setting_value:
                self.logger.error(
                    "OTPD setting -> '%s' needs a value in payload! Skipping...",
                    setting_name,
                )
                continue

            tenant_name = docgen_setting.get("tenant", None)
            description = docgen_setting.get("description", "")

            # Apply the setting to PowerDocs (OTPD):
            response = self._otpd.apply_setting(
                setting_name=setting_name,
                setting_value=setting_value,
                tenant_name=tenant_name,
            )
            if response:
                self.logger.info(
                    "Added OTPD setting -> '%s' with value -> '%s'%s%s.",
                    setting_name,
                    setting_value,
                    " for tenant -> '{}'".format(tenant_name) if tenant_name else "",
                    " ({})".format(description) if description else "",
                )
            else:
                self.logger.error(
                    "Failed to configure OTPD setting -> '%s' with value -> '%s'%s%s!",
                    setting_name,
                    setting_value,
                    " for tenant -> '{}'".format(tenant_name) if tenant_name else "",
                    " ({})".format(description) if description else "",
                )
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._docgen_settings,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_group_placeholders")
    def process_group_placeholders(self) -> None:
        """Replace a group placeholder (sourrounded by %%...%%) with the actual ID of the Content Server group.

        For this we prepare a lookup dict. The dict self._placeholder_values already includes
        lookups for the OTCS and OTAWP OTDS resource IDs (see main.py)
        """

        for group in self._groups:
            group_name = group.get("name")
            if not group_name:
                self.logger.error(
                    "Group needs a name for placeholder definition in payload! Skipping...",
                )
                continue

            # Check if group has been explicitly disabled in payload
            # (enabled = false). In this case we skip the payload element:
            if not group.get("enabled", True):
                self.logger.info(
                    "Payload for group -> '%s' is disabled. Skipping...",
                    group_name,
                )
                continue

            # Now we determine the ID. Either it is in the payload section from
            # the current customizer run or we try to look it up in the system.
            # The latter case may happen if the customizer pod got restarted.
            group_id = self.determine_group_id(group=group)
            if not group_id:
                self.logger.warning(
                    "Group needs an ID for placeholder definition. Skipping...",
                )
                continue

            # Add Group with its ID to the dict self._placeholder_values:
            self._placeholder_values["OTCS_GROUP_ID_" + group_name.upper().replace(" & ", "_").replace(" ", "_")] = str(
                group_id,
            )

        self.logger.debug(
            "Placeholder values after group processing = %s",
            self._placeholder_values,
        )

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_placeholders")
    def process_user_placeholders(self) -> None:
        """Replace a user placeholder (sourrounded by %%...%%) with the ID of the Content Server user.

        For this we prepare a lookup dict. The dict self._placeholder_values already includes
        lookups for the OTCS and OTAWP OTDS resource IDs (see customizer.py).
        """

        for user in self._users:
            user_name = user.get("name")
            if not user_name:
                self.logger.error(
                    "User needs a name for placeholder definition in payload! Skipping...",
                )
                continue

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the payload element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_name,
                )
                continue

            # Now we determine the ID. Either it is in the payload section from
            # the current customizer run or we try to look it up in the system.
            # The latter case may happen if the customizer pod got restarted.
            user_id = self.determine_user_id(user=user)
            if not user_id:
                self.logger.warning(
                    "User needs an ID for placeholder definition. Skipping...",
                )
                continue

            # Add user with its ID to the dict self._placeholder_values:
            self._placeholder_values["OTCS_USER_ID_%s", user_name.upper()] = str(
                user_id,
            )

        self.logger.debug(
            "Placeholder values after user processing = %s",
            str(self._placeholder_values),
        )

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_groups")
    def process_groups(self, section_name: str = "groups") -> bool:
        """Process groups in payload and create them in Content Server.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        Side Effects:
            The group items are modified by adding an "id" dict element that
            includes the technical ID of the group in Content Server

        """

        if not self._groups:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # First run through groups: create all groups in payload
        # and store the IDs of the created groups:
        for group in self._groups:
            group_name = group.get("name")
            if not group_name:
                self.logger.error("Group needs a name inb payload! Skipping...")
                success = False
                continue

            # Check if group has been explicitly disabled in payload
            # (enabled = false). In this case we skip the payload element:
            if not group.get("enabled", True):
                self.logger.info(
                    "Payload for group -> '%s' is disabled. Skipping...",
                    group_name,
                )
                continue

            # Check if the group does already exist (e.g. if job is restarted)
            group_id = self.determine_group_id(group=group)
            if group_id:
                self.logger.info(
                    "Found existing group -> '%s' (%s). Skipping to next group...",
                    group_name,
                    group_id,
                )
                continue

            # Now we know it is a new group...
            new_group = self._otcs.add_group(name=group_name)
            if new_group:
                new_group_id = self._otcs.get_result_value(response=new_group, key="id")
                self.logger.info(
                    "Successfully created group -> '%s' (%s).",
                    group_name,
                    new_group_id,
                )
                # Remember the OTCS group ID in the payload structure:
                group["id"] = new_group_id
            else:
                self.logger.error("Failed to create group -> '%s'!", group_name)
                success = False
                continue

            # Assign usage privileges to the new group:
            usage_privileges = group.get("usage_privileges", [])
            for usage_privilege in usage_privileges:
                response = self._otcs.assign_usage_privilege(
                    usage_privilege=usage_privilege,
                    member_id=new_group_id,
                )
                if response:
                    self.logger.info(
                        "Successfully assigned usage privilege -> '%s' to group -> '%s' (%s).",
                        usage_privilege,
                        group_name,
                        new_group_id,
                    )
                else:
                    self.logger.info(
                        "Failed to assign usage privilege -> '%s' to group -> '%s' (%s)!",
                        usage_privilege,
                        group_name,
                        new_group_id,
                    )

            # Assign usage privileges to the new group:
            object_privileges = group.get("object_privileges", [])
            for object_type in object_privileges:
                response = self._otcs.assign_object_privilege(
                    object_type=object_type,
                    member_id=new_group_id,
                )
                if response:
                    self.logger.info(
                        "Successfully assigned object privilege -> '%s' to group -> '%s' (%s).",
                        object_type,
                        group_name,
                        new_group_id,
                    )
                else:
                    self.logger.info(
                        "Failed to assign object privilege -> '%s' to group -> '%s' (%s)!",
                        object_type,
                        group_name,
                        new_group_id,
                    )

        # end for group in self._groups: (first run)

        # Second run through groups: create all group memberships
        # (nested groups) based on the IDs created in first run:
        for group in self._groups:
            group_name = group.get("name")
            group_id = group.get("id")  # this should have been set in the first loop
            if not group_id:
                self.logger.error(
                    "Group -> '%s' does not have an ID! Creation may have failed before. Skipping...", group_name
                )
                success = False
                continue
            parent_group_names = group.get("parent_groups", [])
            for parent_group_name in parent_group_names:
                # First, try to find parent group in payload by parent group name:
                parent_group = next(
                    (item for item in self._groups if item["name"] == parent_group_name),
                    None,
                )
                if parent_group is None:
                    # If this didn't work, try to get the parent group from OTCS. This covers
                    # cases where the parent group is system-generated or part
                    # of a former payload processing run (or part of another payload):
                    parent_group = self._otcs.get_group(name=parent_group_name)
                    parent_group_id = self._otcs.get_result_value(
                        response=parent_group,
                        key="id",
                    )
                    if not parent_group_id:
                        self.logger.error(
                            "Parent group -> '%s' not found! Skipping...",
                            parent_group_name,
                        )
                        success = False
                        continue
                elif "id" not in parent_group:
                    self.logger.error(
                        "Parent group -> '%s' does not have an ID! Cannot establish group nesting. Skipping...",
                        parent_group["name"],
                    )
                    success = False
                    continue
                else:  # we can read the ID from the
                    parent_group_id = parent_group["id"]

                # retrieve all members of the parent group (1 = get only groups)
                # to check if the current group is already a member in the parent group:
                members = self._otcs.get_group_members(
                    group=parent_group_id,
                    member_type=1,
                )
                if self._otcs.exist_result_item(
                    response=members,
                    key="id",
                    value=group["id"],
                ):
                    self.logger.info(
                        "Group -> '%s' (%s) is already a member of parent group -> '%s' (%s). Skipping to next parent group...",
                        group_name,
                        group_id,
                        parent_group_name,
                        parent_group_id,
                    )
                else:
                    self.logger.info(
                        "Add group -> '%s' (%s) to parent group -> '%s' (%s)...",
                        group_name,
                        group_id,
                        parent_group_name,
                        parent_group_id,
                    )
                    self._otcs.add_group_member(
                        member_id=group["id"],
                        group_id=parent_group_id,
                    )
            # end for parent_group_name in parent_group_names:

            # Assign application roles to the new group:
            application_roles = group.get("application_roles", [])
            if application_roles:
                self.logger.info(
                    "Group -> '%s' has application roles -> %s. Assigning...", group_name, str(application_roles)
                )
            for role in application_roles:
                group_partition = self._otcs.config()["partition"]
                if not group_partition:
                    self.logger.error("Group partition not found! Skipping application role -> %s...", str(role))
                    success = False
                    continue

                # Split role at the @ sign to identify Partitions
                role_parts = role.split("@", 1)
                role_name = role_parts[0]
                role_partition = role_parts[1] if len(role_parts) > 1 else "OAuthClients"

                # This is on OTDS method (not OTCS) thuis the group ID is the group name!
                response = self._otds.assign_group_to_application_role(
                    group_id=group_name,
                    group_partition=group_partition,
                    role_name=role_name,
                    role_partition=role_partition,
                )

                if response:
                    self.logger.info(
                        "Successfully assigned application role -> '%s' in partition -> '%s' to group -> '%s' (%s) in partition -> '%s'.",
                        role_name,
                        role_partition,
                        group_name,
                        group_id,
                        group_partition,
                    )
                else:
                    self.logger.info(
                        "Failed to assign application role -> '%s' in partition -> %s to group -> '%s' (%s) in partition -> '%s'!",
                        role_name,
                        role_partition,
                        group_name,
                        group_id,
                        group_partition,
                    )
            # end for role in application_roles:
        # end for group in self._groups: (second run)

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._groups,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_groups_m365")
    def process_groups_m365(self, section_name: str = "groupsM365") -> bool:
        """Process groups in payload and create them in Microsoft 365.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not isinstance(self._m365, M365):
            self.logger.error(
                "Microsoft 365 connection not setup properly. Skipping payload section '%s'...",
                section_name,
            )
            return False

        if not self._groups:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # First run through groups: create all groups in payload
        # and store the IDs of the created groups:
        for group in self._groups:
            group_name = group.get("name")
            if not group_name:
                self.logger.error("Group needs a name. Skipping...")
                success = False
                continue

            # Check if group has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not group.get("enabled", True):
                self.logger.info(
                    "Payload for group -> '%s' is disabled. Skipping...",
                    group_name,
                )
                continue
            # M365 is disabled per default. There needs to be "enable_o365" in payload
            # and it needs to be True:
            if not group.get("enable_o365", False):
                self.logger.info(
                    "Microsoft 365 is not enabled in payload for group -> '%s'. Skipping...",
                    group_name,
                )
                continue

            # Check if the group does already exist (e.g. if job is restarted)
            # as this is a pattern search it could return multiple groups:
            existing_groups = self._m365.get_group(group_name=group_name)

            if existing_groups and existing_groups["value"]:
                self.logger.debug(
                    "Found existing Microsoft 365 groups -> %s",
                    existing_groups["value"],
                )
                # Get list of all matching groups:
                existing_groups_list = existing_groups["value"]
                # Find the group with the exact match of the name:
                existing_group = next(
                    (item for item in existing_groups_list if item["displayName"] == group_name),
                    None,
                )
                # Have we found an exact match?
                if existing_group is not None:
                    self.logger.info(
                        "Found existing Microsoft 365 group -> '%s' (%s) - skip creation of group...",
                        existing_group["displayName"],
                        existing_group["id"],
                    )
                    # Write M365 group ID back into the payload (for the success file)
                    group["m365_id"] = existing_group["id"]
                    continue

            self.logger.info(
                "Creating a new Microsoft 365 group -> '%s'...",
                group_name,
            )

            # Now we know it is a new group...
            new_group = self._m365.add_group(name=group_name)
            if new_group is not None:
                # Store the Microsoft 365 group ID in payload:
                group["m365_id"] = new_group["id"]
                self.logger.info(
                    "New Microsoft 365 group -> '%s' with ID -> %s has been created",
                    group_name,
                    group["m365_id"],
                )
            else:
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._groups,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_groups_salesforce")
    def process_groups_salesforce(self, section_name: str = "groupsSalesforce") -> bool:
        """Process groups in payload and create them in Salesforce.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not isinstance(self._salesforce, Salesforce):
            self.logger.error(
                "Salesforce connection not setup properly. Skipping payload section '%s'...",
                section_name,
            )
            return False

        if not self._groups:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # First run through groups: create all groups in payload
        # and store the IDs of the created groups:
        for group in self._groups:
            group_name = group.get("name")
            if not group_name:
                self.logger.error("Group needs a name. Skipping...")
                success = False
                continue

            # Check if group has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not group.get("enabled", True):
                self.logger.info(
                    "Payload for group -> '%s' is disabled. Skipping...",
                    group_name,
                )
                continue

            # Salesforce is disabled per default. There needs to be "enable_salesforce" in payload
            # and it needs to be True:
            if not group.get("enable_salesforce", False):
                self.logger.info(
                    "Salesforce is not enabled in payload for group -> '%s'. Skipping...",
                    group_name,
                )
                continue

            # Check if the group does already exist (e.g. if job is restarted)
            existing_group_id = self._salesforce.get_group_id(group_name=group_name)
            if existing_group_id:
                self.logger.info(
                    "Found existing Salesforce group -> '%s' (%s). Skipping...",
                    group_name,
                    existing_group_id,
                )
                # Write M365 group ID back into the payload (for the success file)
                group["salesforce_id"] = existing_group_id
                continue

            self.logger.info(
                "Creating a new Salesforce group -> '%s'...",
                group_name,
            )

            # Now we know it is a new group...
            new_group = self._salesforce.add_group(group_name=group_name)
            new_group_id = self._salesforce.get_result_value(
                response=new_group,
                key="id",
            )
            if new_group_id:
                # Store the Microsoft 365 group ID in payload:
                group["salesforce_id"] = new_group_id
                self.logger.info(
                    "Successfully created Salesforce group -> '%s' with ID -> %s.",
                    group_name,
                    new_group_id,
                )
            else:
                self.logger.error(
                    "Failed to create Salesforce group -> '%s'!",
                    group_name,
                )
                success = False

        # Second run through groups: create all group memberships
        # (nested groups) based on the IDs created in first run:
        for group in self._groups:
            if "salesforce_id" not in group:
                self.logger.info(
                    "Group -> '%s' does not have an Salesforce ID. Skipping...",
                    group["name"],
                )
                # Not all groups may be enabled for Salesforce. This is not an error.
                continue
            group_id = group["salesforce_id"]
            parent_group_names = group["parent_groups"]
            for parent_group_name in parent_group_names:
                # First, try to find parent group in payload by parent group name:
                parent_group = next(
                    (item for item in self._groups if item["name"] == parent_group_name),
                    None,
                )
                if not parent_group:
                    self.logger.error(
                        "Parent group -> '%s' not found. Cannot establish group nesting. Skipping...",
                        parent_group["name"],
                    )
                    success = False
                    continue
                if "salesforce_id" not in parent_group:
                    self.logger.info(
                        "Parent group -> '%s' does not have a Salesforce ID. Cannot establish group nesting. Parent group may not be enabled for Salesforce. Skipping...",
                        parent_group["name"],
                    )
                    # We don't treat this as an error - there may be payload groups which are not enabled for Salesforce!
                    continue

                parent_group_id = parent_group["salesforce_id"]

                # retrieve all members of the parent group
                # to check if the current group is already a member in the parent group:
                members = self._salesforce.get_group_members(group_id=parent_group_id)
                if self._salesforce.exist_result_item(
                    members,
                    "UserOrGroupId",
                    group_id,
                ):
                    self.logger.info(
                        "Salesforce group -> '%s' (%s) is already a member of parent Salesforce group -> '%s' (%s). Skipping to next parent group...",
                        group["name"],
                        group["id"],
                        parent_group_name,
                        parent_group_id,
                    )
                    continue
                self.logger.info(
                    "Add Salesforce group -> '%s' (%s) to parent Salesforce group -> '%s' (%s)...",
                    group["name"],
                    group_id,
                    parent_group_name,
                    parent_group_id,
                )
                self._salesforce.add_group_member(
                    group_id=parent_group_id,
                    member_id=group_id,
                )

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._groups,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_groups_core_share")
    def process_groups_core_share(self, section_name: str = "groupsCoreShare") -> bool:
        """Process groups in payload and create them in Core Share.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not isinstance(self._core_share, CoreShare):
            self.logger.error(
                "Core Share connection not setup properly. Skipping payload section '%s'...",
                section_name,
            )
            return False

        if not self._groups:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # Create all groups specified in payload
        # and store the IDs of the created Core Share groups:
        for group in self._groups:
            group_name = group.get("name")
            if not group_name:
                self.logger.error("Group needs a name. Skipping...")
                success = False
                continue

            # Check if group has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not group.get("enabled", True):
                self.logger.info(
                    "Payload for group -> '%s' is disabled. Skipping...",
                    group_name,
                )
                continue

            # Core Share is disabled per default. There needs to be "enable_core_share" in payload
            # and it needs to be True:
            if not group.get("enable_core_share", False):
                self.logger.info(
                    "Group -> '%s' is not enabled for Core Share. Skipping...",
                    group_name,
                )
                continue

            # Check if the group does already exist (e.g. if job is restarted)
            core_share_group = self._core_share.get_group_by_name(name=group_name)
            core_share_group_id = self._core_share.get_result_value(
                response=core_share_group,
                key="id",
            )
            if core_share_group_id:
                self.logger.info(
                    "Found existing Core Share group -> '%s' (%s). Just do cleanup of potential left-overs...",
                    group_name,
                    core_share_group_id,
                )
                # Write Core Share group ID back into the payload (for the success file)
                group["core_share_id"] = core_share_group_id

                # For existing users we want to cleanup possible left-overs form old deployments
                self.logger.info(
                    "Cleanup existing file shares of Core Share group -> '%s' (%s)...",
                    group_name,
                    core_share_group_id,
                )
                response = self._core_share.cleanup_group_shares(
                    group_id=core_share_group_id,
                )
                if not response:
                    self.logger.error("Failed to cleanup group shares!")

                continue

            self.logger.info(
                "Creating a new Core Share group -> '%s'...",
                group_name,
            )

            # Now we know it is a new group...
            new_group = self._core_share.add_group(group_name=group_name)
            new_group_id = self._core_share.get_result_value(
                response=new_group,
                key="id",
            )
            if new_group_id:
                # Store the Microsoft 365 group ID in payload:
                group["core_share_id"] = new_group_id
                self.logger.info(
                    "Successfully created Core Share group -> '%s' with ID -> %s.",
                    group_name,
                    new_group_id,
                )
            else:
                self.logger.error(
                    "Failed to create Core Share group -> %s!",
                    group_name,
                )
                success = False

        # Core Share groups cannot be nested. So we are fone here.

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._groups,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_users")
    def process_users(self, section_name: str = "users") -> bool:
        """Process users in payload and create them in Content Server.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        Side Effects:
            The user items are modified by adding an "id" dict element that
            includes the technical ID of the user in Content Server

        """

        if not self._users:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # Even if this payload section has been processed successfully before we
        # to process it once more. So we are NOT checking the status file.

        success: bool = True

        # Add all users in payload and establish membership in
        # specified groups:
        for user in self._users:
            user_name = user.get("name")
            if not user_name:
                self.logger.error("User is missing a login. Skipping to next user...")
                success = False
                continue

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_name,
                )
                continue

            # Check if the user does already exist (e.g. if job is restarted)
            # determine_user_id() also writes back the user ID into the payload
            # if it has gathered it from OTCS.
            user_id = self.determine_user_id(user=user)
            if user_id:
                self.logger.info(
                    "Found existing user -> '%s' (%s). Skipping to next user...",
                    user_name,
                    user_id,
                )
                continue
            self.logger.info(
                "Cannot find an existing user -> '%s' - creating a new user...",
                user_name,
            )

            # Sanity checks:
            if "password" not in user or user["password"] is None or user["password"] == "":
                self.logger.info(
                    "User -> '%s' no password defined in payload, generating random password...",
                    user_name,
                )
                user["password"] = self.generate_password(
                    length=10,
                    use_special_chars=True,
                )

                description_attribue = {
                    "name": "description",
                    "value": "initial password: " + user["password"],
                }

                try:
                    user["extra_attributes"].append(description_attribue)
                except KeyError:
                    user["extra_attributes"] = [description_attribue]

            # Sanity checks:
            if "base_group" not in user or not user["base_group"]:
                self.logger.warning(
                    "User -> '%s' is missing a base group - setting to default group.",
                    user_name,
                )
                user["base_group"] = "DefaultGroup"

            # Find the base group of the user. Assume 'Default Group' (= 1001) if not found:
            base_group = next(
                (item["id"] for item in self._groups if item["name"] == user["base_group"] and item.get("id")),
                1001,
            )

            # Get type to be able to create ServiceUsers, default to User -> 0
            user_type = 17 if user.get("type", "User") == "ServiceUser" else 0

            # Now we know it is a new user...
            new_user = self._otcs.add_user(
                name=user_name,
                password=user["password"],
                first_name=user.get("firstname", ""),  # be careful - can be empty
                last_name=user.get("lastname", ""),  # be careful - can be empty
                email=user.get("email", ""),  # be careful - can be empty
                title=user.get("title", ""),  # be careful - can be empty
                base_group=base_group,
                phone=user.get("phone", ""),
                privileges=user.get("privileges", ["Login", "Public Access"]),
                user_type=user_type,
            )

            new_user_id = self._otcs.get_result_value(response=new_user, key="id")
            if not new_user_id:
                self.logger.error(
                    "Failed to create new user -> '%s'!",
                    user_name,
                )
                success = False
                continue

            self.logger.info(
                "Successfully created user -> '%s' with ID -> %s.",
                user_name,
                new_user_id,
            )
            # Write back user ID into payload
            user["id"] = new_user_id

            # Assign usage privileges to the new user:
            usage_privileges = user.get("usage_privileges", [])
            for usage_privilege in usage_privileges:
                response = self._otcs.assign_usage_privilege(
                    usage_privilege=usage_privilege,
                    member_id=new_user_id,
                )
                if response:
                    self.logger.info(
                        "Assigned usage privilege -> '%s' to user -> '%s' (%s).",
                        usage_privilege,
                        user_name,
                        new_user_id,
                    )
                else:
                    self.logger.info(
                        "Failed to assign usage privilege -> '%s' to user -> '%s' (%s)!",
                        usage_privilege,
                        user_name,
                        new_user_id,
                    )

            # Assign usage privileges to the new user:
            object_privileges = user.get("object_privileges", [])
            for object_type in object_privileges:
                response = self._otcs.assign_object_privilege(
                    object_type=object_type,
                    member_id=new_user_id,
                )
                if response:
                    self.logger.info(
                        "Assigned object privilege -> '%s' to user -> '%s' (%s).",
                        object_type,
                        user_name,
                        new_user_id,
                    )
                else:
                    self.logger.info(
                        "Failed to assign object privilege -> '%s' to user -> '%s' (%s)!",
                        object_type,
                        user_name,
                        new_user_id,
                    )

            # Process group memberships of new user:
            user_groups = user.get("groups", [])  # list of groups the user is in
            for user_group in user_groups:
                # Try to find the group dictionary item in the payload
                # for user group name:
                group = next(
                    (item for item in self._groups if item["name"] == user_group),
                    None,
                )
                if group:
                    group_id = group.get("id")  # Careful ID may not exist
                    group_name = group["name"]
                else:
                    # if group is not in payload try to find group in OTCS
                    # in case it is a pre-existing group:
                    group = self._otcs.get_group(name=user_group)
                    group_id = self._otcs.get_result_value(response=group, key="id")
                    if group_id is None:
                        self.logger.error(
                            "Group -> '%s' not found. Skipping...",
                            user_group,
                        )
                        success = False
                        continue
                    group_name = self._otcs.get_result_value(
                        response=group,
                        key="name",
                    )

                if group_id is None:
                    self.logger.error(
                        "Group -> '%s' does not have an ID. Cannot add user -> '%s' to this group. Skipping...",
                        group_name,
                        user["name"],
                    )
                    success = False
                    continue

                self.logger.info(
                    "Add user -> '%s' (%s) to group -> '%s' (%s)...",
                    user["name"],
                    user["id"],
                    group_name,
                    group_id,
                )
                response = self._otcs.add_group_member(
                    member_id=user["id"],
                    group_id=group_id,
                )
                if not response:
                    success = False
            # end for user_group in user_groups:

            # For some unclear reason the user is not added to its base group in OTDS
            # so we do this explicitly:
            self.logger.info(
                "Add user -> '%s' to its base group -> '%s'...",
                user["name"],
                user["base_group"],
            )
            response = self._otds.add_user_to_group(
                user["name"],
                user["base_group"],
            )
            if not response:
                success = False

            # Extra OTDS attributes for the user can be provided in "extra_attributes"
            # as part of the user payload.
            if "extra_attributes" in user:
                extra_attributes = user["extra_attributes"]
                for extra_attribute in extra_attributes:
                    attribute_name = extra_attribute.get("name")
                    attribute_value = extra_attribute.get("value")
                    if not attribute_name or not attribute_value:
                        self.logger.error(
                            "User attribute is missing a name or value. Skipping...",
                        )
                        success = False
                        continue
                    if "password" in attribute_value:
                        self.logger.info(
                            "Set user attribute -> '%s' to -> '*******' (sensitive password)",
                            attribute_name,
                        )
                    else:
                        self.logger.info(
                            "Set user attribute -> '%s' to -> '%s'.",
                            attribute_name,
                            attribute_value,
                        )
                    user_partition = self._otcs.config()["partition"]
                    if not user_partition:
                        self.logger.error("User partition not found!")
                        success = False
                        continue
                    self._otds.update_user(
                        user_partition,
                        user["name"],
                        attribute_name,
                        attribute_value,
                    )
            # end if "extra_attributes" in user

            # Assign application roles to the new user:
            application_roles = user.get("application_roles", [])

            for role in application_roles:
                user_partition = self._otcs.config()["partition"]
                if not user_partition:
                    self.logger.error("User partition not found!")
                    success = False
                    continue

                # Split role at the @ sign to identify Partitions
                role_parts = role.split("@", 1)
                role_name = role_parts[0]
                role_partition = role_parts[1] if len(role_parts) > 1 else "OAuthClients"

                response = self._otds.assign_user_to_application_role(
                    user_id=user["name"],
                    user_partition=user_partition,
                    role_name=role_name,
                    role_partition=role_partition,
                )

                if response:
                    self.logger.info(
                        "Successfully assigned application role -> '%s' (%s) to user -> '%s' (%s).",
                        role_name,
                        role_partition,
                        user_name,
                        user_partition,
                    )
                else:
                    self.logger.info(
                        "Failed to assign application role -> '%s' (%s) to user -> '%s' (%s) in OTDS!",
                        role_name,
                        role_partition,
                        user_name,
                        user_partition,
                    )

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._users,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_users_sap")
    def process_users_sap(self, section_name: str = "usersSAP") -> bool:
        """Process users in payload and sync them with SAP (passwords only for now).

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool: True if payload has been processed without errors, False otherwise

        Side Effects:
            The user items are modified by adding an "sap_sync_result" dict element
            that documents if the user password was properly synced with SAP.

        """

        if not self._users:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        if not self._sap:
            self.logger.error("SAP connection is not initialized. Bailing out...")
            return False

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        rfc_name = "ZFM_TERRA_RFC_CHNG_USR_PW"
        rfc_description = "RFC to update the SAP user password"
        rfc_call_options = ()

        # Update SAP password for all users in payload:
        for user in self._users:
            user_name = user.get("name")
            if not user_name:
                self.logger.error("User is missing a login. Skipping to next user...")
                success = False
                continue

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_name,
                )
                continue

            # Check if the user is explicitly enabled for SAP:
            if not user.get("enable_sap", False):
                self.logger.info(
                    "User -> '%s' is not enabled for SAP. Skipping...",
                    user_name,
                )
                continue

            # Lookup mandatory password in payload:
            user_password = user.get("password")
            if not user_password:
                self.logger.error(
                    "User -> '%s' is missing a password. Cannot sync with SAP. Skipping to next user...",
                    user_name,
                )
                success = False
                continue

            rfc_params = {
                "USERNAME": user_name,
                "PASSWORD": user_password,
            }

            self.logger.info(
                "Updating password of user -> '%s' in SAP. Calling SAP RFC -> '%s' (%s) with parameters -> %s ...",
                user_name,
                rfc_name,
                rfc_description,
                str(rfc_params),
            )

            result = self._sap.call(
                rfc_name=rfc_name,
                options=rfc_call_options,
                rfc_parameters=rfc_params,
            )
            if result is None:
                self.logger.error(
                    "Failed to call SAP RFC -> '%s' to update password of user -> '%s'!",
                    rfc_name,
                    user_name,
                )
                success = False
            elif result.get("RESULT") != "OK":
                self.logger.error(
                    "Result of SAP RFC -> '%s' is not OK, it returned -> %s failed items in result -> %s",
                    rfc_name,
                    str(result.get("FAILED")),
                    str(result),
                )
                success = False
                # Save result for status file content
                user["sap_sync_result"] = result
            else:
                self.logger.info(
                    "Successfully called RFC -> '%s'. Result -> %s.",
                    rfc_name,
                    str(result),
                )
                # Save result for status file content
                user["sap_sync_result"] = result

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._users,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_users_successfactors")
    def process_users_successfactors(
        self,
        section_name: str = "usersSuccessFactors",
    ) -> bool:
        """Process users in payload and sync them with SuccessFactors (passwords and email).

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        Side Effects:
            The user items are modified by adding an "successfactors_user_id" dict element that
            includes the personIdExternal of the user in SuccessFactors

        """

        if not self._users:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        if not self._successfactors:
            self.logger.error(
                "SuccessFactors connection is not initialized. Bailing out...",
            )
            return False

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # traverse all users in payload:
        for user in self._users:
            user_name = user.get("name")
            if not user_name:
                self.logger.error("User is missing a login. Skipping to next user...")
                success = False
                continue

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_name,
                )
                continue

            # Check if the user is explicitly enabled for SuccessFactors:
            if not user.get("enable_successfactors", False):
                self.logger.info(
                    "User -> '%s' is not enabled for SuccessFactors. Skipping...",
                    user_name,
                )
                continue

            # Lookup password and email in payload:
            user_password = user.get("password", "")
            user_email = user.get("email", "")

            # first we need to get the SuccessFactors user account object
            # to determine the personIdExternal that we need to update the
            # SuccessFactors user.
            response = self._successfactors.get_user_account(username=user_name)
            user_id = self._successfactors.get_result_value(
                response=response,
                key="personIdExternal",
            )
            if user_id is None:
                self.logger.error(
                    "Failed to get personIDExternal of SuccessFactors user -> '%s'!",
                    user_name,
                )
                success = False
                continue
            self.logger.info(
                "SuccessFactors user -> '%s' has external user ID -> %s.",
                user_name,
                str(user_id),
            )

            # Now let's update the user password and email address:
            update_data = {}
            if user_password:
                self.logger.info(
                    "Updating password of SuccessFactors user -> '%s' (%s)...",
                    user_name,
                    str(user_id),
                )
                update_data["password"] = user_password
            if user_email:
                update_data["email"] = user_email

            response = self._successfactors.update_user(
                user_id=user_id,
                update_data=update_data,
            )
            if response:
                self.logger.info(
                    "Successfully updated SuccessFactors user -> '%s'.",
                    str(user_name),
                )
                # Save result for status file content
                user["successfactors_user_id"] = user_id
            else:
                self.logger.error(
                    "Failed to update SuccessFactors user -> '%s'! Skipping...",
                    user_name,
                )
                success = False
                continue

            if not user_email:
                continue

            self.logger.info(
                "Updating email of SuccessFactors user -> '%s' to -> %s...",
                user_name,
                user_email,
            )
            response = self._successfactors.update_user_email(
                user_id=user_id,
                email_address=user_email,
            )
            if response:
                self.logger.info(
                    "Successfully updated email address of SuccessFactors user -> '%s' to -> '%s'.",
                    user_name,
                    user_email,
                )
            else:
                self.logger.error(
                    "Failed to update email address of SuccessFactors user -> '%s' to -> '%s'!",
                    user_name,
                    user_email,
                )
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._users,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_users_salesforce")
    def process_users_salesforce(
        self,
        section_name: str = "usersSalesforce",
    ) -> bool:
        """Process users in payload and sync them with Salesforce (passwords and email).

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        Side Effects:
            The user items are modified by adding "salesforce_user_id", "salesforce_user_login"
            dict element that includes the ID of the user in Salesforce. This will be written
            into the status file.

        """

        if not self._users:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        if not self._salesforce:
            self.logger.error(
                "Salesforce connection is not initialized. Bailing out...",
            )
            return False

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # traverse all users in payload:
        for user in self._users:
            user_name = user.get("name")
            if not user_name:
                self.logger.error("User is missing a login. Skipping to next user...")
                success = False
                continue

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_name,
                )
                continue

            # Check if the user is explicitly enabled for Salesforce:
            if not user.get("enable_salesforce", False):
                self.logger.info(
                    "User -> '%s' is not enabled for Salesforce. Skipping...",
                    user_name,
                )
                continue

            extra_attributes = user.get("extra_attributes", None)
            if not extra_attributes or len(extra_attributes) == 0:
                self.logger.info(
                    "User -> '%s' does not have the extra attributes for Salesforce. Cannot determine the Salesforce login for user. Skipping...",
                    user_name,
                )
                continue
            user_login = extra_attributes[0].get("value", "")
            if not user_login:
                self.logger.info(
                    "User -> '%s' does not have the extra attributes value for Salesforce. Skipping...",
                    user_name,
                )
                continue

            user_email = user.get("email", "")
            need_email_verification = False

            #
            # 1. Check if user does already exist in Salesforce:
            #

            salesforce_user_id = self._salesforce.get_user_id(username=user_login)

            #
            # 2: Create or Update user in Salesforce:
            #

            if salesforce_user_id is None:
                self.logger.info(
                    "Salesforce user -> '%s' does not exist. Creating a new Salesforce user...",
                    user_name,
                )
                response = self._salesforce.add_user(
                    username=user_login,
                    email=user.get("email", ""),
                    firstname=user.get("firstname", ""),
                    lastname=user.get("lastname", ""),
                    department=user.get("base_group", ""),
                    title=user.get("title", ""),
                    company_name=user.get("company", "Innovate"),
                )
                salesforce_user_id = self._salesforce.get_result_value(
                    response=response,
                    key="id",
                )
                if not salesforce_user_id:
                    self.logger.error(
                        "Failed to create Salesforce user -> '%s'! Skipping...",
                        user_name,
                    )
                    success = False
                    continue
                self.logger.info(
                    "Successfully created Salesforce user -> '%s' with ID -> %s.",
                    user_name,
                    salesforce_user_id,
                )
                # We need email verification for new users (unclear if this is really the case...)
                need_email_verification = True
            # The user does already exist in Salesforce and we just update it:
            else:
                update_data = {
                    "FirstName": user.get("firstname", ""),
                    "LastName": user.get("lastname", ""),
                    "Department": user.get("base_group", ""),
                    "Title": user.get("title", ""),
                    "CompanyName": user.get("company", ""),
                }
                self.logger.info(
                    "Salesforce user -> '%s' does already exist. Updating Salesforce user with -> %s...",
                    user_name,
                    str(update_data),
                )

                # Check if the mail address has really changed. Otherwise we
                # don't need to set it again and can avoid email verification:
                salesforce_user = self._salesforce.get_user(user_id=salesforce_user_id)
                salesforce_user_email = self._salesforce.get_result_value(
                    response=salesforce_user,
                    key="Email",
                )
                if user_email != salesforce_user_email:
                    self.logger.info(
                        "Email for Salesforce user -> '%s' has changed from -> '%s' to -> '%s'.",
                        user_name,
                        salesforce_user_email,
                        user_email,
                    )
                    # Additional email payload for user update:
                    update_data["Email"] = user_email
                    # OK, email has changed - we need the email verification below...
                    need_email_verification = True

                # Update the existing Salesforce user with new / changed data:
                response = self._salesforce.update_user(
                    user_id=salesforce_user_id,
                    update_data=update_data,
                )
                if not response:
                    self.logger.error(
                        "Failed to update Salesforce user -> '%s'! Skipping...",
                        user_login,
                    )
                    success = False
                    continue
                self.logger.info(
                    "Successfully updated Salesforce user -> '%s'.",
                    user_login,
                )

            # Save result for status file content
            user["salesforce_user_id"] = salesforce_user_id
            user["salesforce_user_login"] = user_login

            #
            # 3: Update user password in Salesforce (we need to do this also for new users!):
            #

            # Lookup password in payload:
            user_password = user.get("password", "")

            if user_password:
                response = self._salesforce.update_user_password(
                    user_id=salesforce_user_id,
                    password=user_password,
                )
                if response:
                    self.logger.info(
                        "Successfully updated password of Salesforce user -> '%s' (%s).",
                        user_login,
                        salesforce_user_id,
                    )
                else:
                    self.logger.error(
                        "Failed to update Salesforce password for user -> '%s' (%s)! Skipping...",
                        user_login,
                        salesforce_user_id,
                    )
                    success = False
                    continue

            #
            # 4: Handle Email verification:
            #

            # We now need to wait for the verification mail from Salesforce,
            # get it from the M365 Outlook inbox of the user (or the admin
            # if the user does not have its own inbox) and click the
            # verification link...

            if need_email_verification and user.get("enable_o365", False):
                self.logger.info(
                    "Processing Email verification for user -> '%s' (%s). Wait a few seconds to make sure verification mail in user's inbox...",
                    user_name,
                    user_email,
                )
                time.sleep(20)

                # Process verification mail sent by Salesforce.
                # This has some hard-coded value. We may want to optimize it further in the future:
                result = self._m365.email_verification(
                    user_email=user_email,
                    sender="QA_SUPPORT@salesforce.com",
                    subject="Finish changing your Salesforce",
                    url_search_pattern="setup/emailverif",
                )
                if not result:
                    # Email verification was not successful
                    self.logger.warning(
                        "Salesforce email verification failed. No verification mail received in user's inbox using email address -> '%s'!",
                        user_email,
                    )
                    # don't treat as error nor do "continue" here - we still want to process the user groups...
                else:
                    self.logger.info(
                        "Successfully verified new email address -> '%s'.",
                        user_email,
                    )
            # end if need_email_verification

            #
            # 5: Add users into groups in Salesforce:
            #

            self.logger.info(
                "Processing group memberships of Salesforce user -> '%s' (%s)...",
                user_name,
                user_email,
            )
            user_groups = user.get("groups", [])
            base_group = user.get("base_group", None)
            if base_group and base_group not in user_groups:
                user_groups.append(base_group)  # list of groups the user is in

            for user_group in user_groups:
                # "Business Administrators" is a OTCS generated group that we won't find
                # in payload - skip this group.
                if user_group == "Business Administrators":
                    continue
                # Try to find the group dictionary item in the payload
                # for user group name:
                group = next(
                    (item for item in self._groups if item["name"] == user_group),
                    None,
                )
                if not group:
                    self.logger.error(
                        "Cannot find group -> '%s'. Cannot establish membership in Salesforce. Skipping to next group...",
                        user_group,
                    )
                    success = False
                    continue

                group_id = group.get("salesforce_id")  # Careful ID may not exist
                group_name = group["name"]
                if group_id is None:
                    self.logger.info(
                        "Group -> '%s' does not have a Salesforce ID. Cannot add user -> '%s' to this Salesforce group (group may not be enabled for Salesforce). Skipping...",
                        group_name,
                        user_name,
                    )
                    # We don't treat this as an error - there may be payload groups which are not enabled for Salesforce!
                    continue

                existing_members = self._salesforce.get_group_members(group_id=group_id)
                if not existing_members or not self._salesforce.exist_result_item(
                    response=existing_members,
                    key="UserOrGroupId",
                    value=salesforce_user_id,
                ):
                    self.logger.info(
                        "Add Salesforce user -> '%s' (%s) to Salesforce group -> '%s' (%s)...",
                        user_name,
                        salesforce_user_id,
                        group_name,
                        group_id,
                    )
                    response = self._salesforce.add_group_member(
                        group_id=group_id,
                        member_id=salesforce_user_id,
                    )
                    member_id = self._salesforce.get_result_value(
                        response=response,
                        key="id",
                    )
                    if not member_id:
                        self.logger.error(
                            "Failed to add Salesforce user -> '%s' (%s) as member to Salesforce group -> '%s' (%s)!",
                            user_name,
                            salesforce_user_id,
                            group_name,
                            group_id,
                        )
                        success = False
                        continue
                else:
                    self.logger.info(
                        "Salesforce user -> '%s' (%s) does already exist in Salesforce group -> '%s' (%s). Skipping...",
                        user_name,
                        salesforce_user_id,
                        group_name,
                        group_id,
                    )
            # end for loop user groups
        # end for loop users

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._users,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_users_core_share")
    def process_users_core_share(
        self,
        section_name: str = "usersCoreShare",
    ) -> bool:
        """Process users in payload and sync them with Core Share (passwords and email).

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        Side Effects:
            The user items are modified by adding "core_share_user_id"
            dict element that includes the ID of the user in Core Share. This will be written
            into the status file.

        """

        if not self._users:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        if not self._core_share:
            self.logger.error(
                "Core Share connection is not initialized. Bailing out...",
            )
            return False

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # traverse all users in payload:
        for user in self._users:
            user_last_name = user.get("lastname", "")  # Default is important here
            user_first_name = user.get("firstname", "")
            user_name = " ".join(filter(None, [user_first_name, user_last_name]))
            user_enabled = user.get("enabled", True) and user.get("enable_core_share", False)
            if not user_name and user_enabled:  # Avoid a warning if not enbaled
                self.logger.error(
                    "User is missing last name and first name. Skipping to next user...",
                )
                success = False
                continue

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_name,
                )
                continue

            # Check if the user is enabled for Core Share.
            # Core Share is disabled per default for the users.
            # There needs to be "enable_core_share" in user payload
            # and it needs to be True:
            if not user.get("enable_core_share", False):
                self.logger.info(
                    "User -> '%s' is not enabled for Core Share. Skipping...",
                    user_name,
                )
                continue

            user_email = user.get("email", "")
            user_password = user.get("password", "")

            # Initialize variables:
            need_email_verification = False
            subject = None
            url_search_pattern = None

            #
            # 1. Check if user does already exist in Core Share:
            #

            core_share_user_id = self.determine_user_id_core_share(user=user)

            #
            # 2: Create or Update user in Core Share:
            #

            # Check if we need to create a new Core Share user:
            if core_share_user_id is None:
                self.logger.info(
                    "Core Share user -> '%s' does not exist. Creating a new Core Share user...",
                    user_name,
                )
                response = self._core_share.add_user(
                    first_name=user_first_name,
                    last_name=user_last_name,
                    email=user_email,
                    title=user.get("title", None),
                    company=user.get("company", "Innovate"),
                    password=user.get("password", None),
                )
                core_share_user_id = self._core_share.get_result_value(
                    response=response,
                    key="id",
                )
                if not core_share_user_id:
                    self.logger.error(
                        "Failed to create Core Share user -> '%s'! Skipping...",
                        user_name,
                    )
                    success = False
                    continue
                self.logger.info(
                    "Successfully created Core Share user -> '%s' (%s).",
                    user_name,
                    core_share_user_id,
                )

                # Check if the user is still in pending state:
                is_confirmed = self._core_share.get_result_value(
                    response=response,
                    key="isConfirmed",
                )
                # we need to differentiate False an None here - don't simplify to "if not is_confirmed"!
                if is_confirmed is False:
                    self.logger.info(
                        "New Core Share user -> '%s' is not yet confirmed and in a 'pending' state!",
                        user_name,
                    )
                elif is_confirmed is True:
                    self.logger.info(
                        "New Core Share user -> '%s' is already confirmed!",
                        user_name,
                    )

                # We write the user password in addition to the "Other" Address field
                # to determine in a subsequent deployment the "old" password:
                update_data = {
                    "addresses": [
                        {"type": "other", "value": user.get("password", None)},
                    ],
                }
                response = self._core_share.update_user(
                    user_id=core_share_user_id,
                    update_data=update_data,
                )

                # We need email verification for new users:
                need_email_verification = True
                url_search_pattern = "verify-email"
                subject = "Welcome to OpenText Core Share"

                # For new users the old password is equal to the new password:
                old_password = user.get("password", None)
            # The user does already exist in Core Share:
            else:
                update_data = {
                    "firstName": user.get("firstname", ""),
                    "lastName": user.get("lastname", ""),
                    "title": user.get("title", ""),
                    "company": user.get("company", ""),
                }
                self.logger.info(
                    "Core Share user -> '%s' does already exist. Updating Core Share user with -> %s...",
                    user_name,
                    str(update_data),
                )

                # Fetch the existing user:
                core_share_user = self._core_share.get_user_by_id(
                    user_id=core_share_user_id,
                )

                # Check if the user is still in pending state:
                is_confirmed = self._core_share.get_result_value(
                    response=core_share_user,
                    key="isConfirmed",
                )
                # we need to differentiate False an None here - don't simplify to "if not is_confirmed"!
                if is_confirmed is False:
                    self.logger.warning(
                        "Core Share user -> '%s' has not yet confirmed the email invitation and is in 'pending' state! Resend invite...",
                        user_name,
                    )
                    # We try the email verification once more...
                    self._core_share.resend_user_invite(core_share_user_id)
                    need_email_verification = True
                    url_search_pattern = "confirm-account"
                    subject = "Invitation to OpenText™ Core Share"

                # Check if we have the old password of the user in the "Other" address field:
                core_share_user_addresses = self._core_share.get_result_value(
                    core_share_user,
                    "addresses",
                )
                if core_share_user_addresses and len(core_share_user_addresses) > 0:
                    old_password = core_share_user_addresses[0]["value"]
                    self.logger.info(
                        "Found old password for Core Share user -> '%s' (%s).",
                        user_name,
                        core_share_user_id,
                    )
                else:
                    self.logger.info(
                        "No old password found for Core Share user -> '%s'. Cannot set a new password.",
                        user_name,
                    )
                    old_password = ""

                # We store the current password into the address field (this adds to the update dictionary
                # defined above and used below): THIS IS CURRENTLY NOT WORKING!
                update_data["addresses"] = [{"type": "other", "value": user_password}]

                # Check if the mail address has really changed. Otherwise we
                # don't need to set it again and can avoid email verification:
                core_share_user_email = self._core_share.get_result_value(
                    core_share_user,
                    "email",
                )
                if user_email != core_share_user_email:
                    self.logger.info(
                        "Email for Core Share user -> '%s' has changed from -> '%s' to -> '%s'. We need to verify this via email.",
                        user_name,
                        core_share_user_email,
                        user_email,
                    )
                    # Additional email payload for user update:
                    update_data["email"] = user_email
                    # If email is changed this needs to be confirmed by passing
                    # the current (old) password:
                    update_data["password"] = old_password if old_password else user_password
                    # As email has changed - we need the email verification below...
                    need_email_verification = True
                    url_search_pattern = "verify-email"
                    subject = "OpenText Core Share: Email Updated"

                # Update the existing Core Share user with new / changed data:
                response = self._core_share.update_user(
                    user_id=core_share_user_id,
                    update_data=update_data,
                )
                if not response:
                    self.logger.error(
                        "Failed to update Core Share user -> '%s'! Skipping...",
                        user_name,
                    )
                    success = False
                    continue
                self.logger.info(
                    "Successfully updated Core Share user -> '%s'.",
                    user_name,
                )

                # Now update the password:
                if user_password and old_password and user_password != old_password:
                    response = self._core_share.update_user_password(
                        user_id=core_share_user_id,
                        password=old_password,
                        new_password=user_password,
                    )
                    if response:
                        self.logger.info(
                            "Successfully updated password of Core Share user -> '%s' (%s).",
                            user_name,
                            core_share_user_id,
                        )
                    else:
                        self.logger.error(
                            "Failed to update Core Share password for user -> '%s' (%s)! Skipping...",
                            user_name,
                            core_share_user_id,
                        )
                        success = False
                        continue
                elif not old_password:
                    self.logger.warning(
                        "Cannot change Core Share user password for -> '%s' (%s). Need both, old and new passwords.",
                        user_name,
                        core_share_user_id,
                    )
                else:
                    self.logger.info(
                        "Core Share user password for -> '%s' (%s) is unchanged.",
                        user_name,
                        core_share_user_id,
                    )

                # For existing users we want to cleanup possible left-overs form old deployments
                self.logger.info(
                    "Cleanup existing file shares of Core Share user -> '%s' (%s)...",
                    user_name,
                    core_share_user_id,
                )
                response = self._core_share.cleanup_user_files(
                    user_id=core_share_user_id,
                    user_login=core_share_user_email,
                    user_password=user_password,
                )
                if not response:
                    self.logger.error(
                        "Failed to cleanup user files for user -> '%s' (%s)!", user_name, core_share_user_id
                    )

            # Save result for status file content
            user["core_share_user_id"] = core_share_user_id

            #
            # 3: Handle Email verification:
            #

            # We now need to wait for the verification mail from Core Share,
            # get it from the M365 Outlook inbox of the user (or the admin
            # if the user does not have its own inbox) and click the
            # verification link...

            if need_email_verification and user.get("enable_o365", False):
                self.logger.info(
                    "Processing Email verification for user -> '%s' (%s). Wait 20 seconds to make sure verification mail in user's inbox...",
                    user_name,
                    user_email,
                )
                time.sleep(20)

                # Process verification mail sent by Core Share.
                # TODO: This has some hard-coded value. We may want to optimize it further in the future:
                result = self._m365.email_verification(
                    user_email=user_email,
                    sender="noreply@opentext.cloud",
                    subject=subject,
                    url_search_pattern=url_search_pattern,
                    line_end_marker="=",
                    multi_line=True,
                    multi_line_end_marker="%3D",
                    replacements=None,
                    max_retries=6,
                    use_browser_automation=True,
                    password=user_password,
                    password_field_id="passwordInput",
                    password_confirmation_field_id="confirmResetPassword",
                    password_submit_xpath="//button[@type='submit']",
                    terms_of_service_xpath="//div[@id = 'termsOfService']//button[@type='submit']",
                )
                if not result:
                    # Email verification was not successful
                    self.logger.warning(
                        "Core Share email verification failed. No verification mail received in user's inbox.",
                    )
                    # don't treat as error nor do "continue" here - we still want to process the user groups...
                else:
                    self.logger.info(
                        "Successfully verified new email address -> '%s'.",
                        user_email,
                    )
            # end if need_email_verification

            #
            # 4: Add users into groups in Core Share:
            #

            self.logger.info(
                "Processing group memberships of Core Share user -> '%s' (%s)...",
                user_name,
                user_email,
            )
            user_groups = user.get("groups", [])
            base_group = user.get("base_group", None)
            if base_group and base_group not in user_groups:
                user_groups.append(base_group)  # list of groups the user is in

            for user_group in user_groups:
                # "Business Administrators" is a OTCS generated group that we won't find
                # in payload - skip this group.
                if user_group == "Business Administrators":
                    # Users that are Business Administrators in Content Server
                    # become Content Manager (role = 5) in Core Share:
                    self.logger.info(
                        "User -> '%s' is a business administrator in Content Server and becomes a 'Content Manager' (access role 5) in Core Share",
                        user_name,
                    )
                    self._core_share.add_user_access_role(
                        user_id=core_share_user_id,
                        role_id=5,
                    )
                    continue
                # Try to find the group dictionary item in the payload
                # for user group name:
                group = next(
                    (item for item in self._groups if item["name"] == user_group),
                    None,
                )
                if not group:
                    self.logger.error(
                        "Cannot find group -> '%s'. Cannot establish membership in Core Share. Skipping to next group...",
                        user_group,
                    )
                    success = False
                    continue

                group_name = group["name"]
                group_id = self.determine_group_id_core_share(
                    group=group,
                )  # Careful ID may not exist
                if group_id is None:
                    self.logger.info(
                        "Group -> '%s' does not have a Core Share ID. Cannot add user -> '%s' to this Core Share group (group may not be enabled for Core Share). Skipping...",
                        group_name,
                        user_name,
                    )
                    # We don't treat this as an error - there may be payload groups which are not enabled for Core Share!
                    continue

                existing_members = self._core_share.get_group_members(group_id)

                # Only add user as new member if not yet a member or a 'pending' member:
                is_member = self._core_share.exist_result_item(
                    response=existing_members,
                    key="id",
                    value=core_share_user_id,
                    results_marker="groupMembers",
                )
                is_pending_member = self._core_share.exist_result_item(
                    response=existing_members,
                    key="email",
                    value=user_email,
                    results_marker="pending",
                )

                if not is_member and not is_pending_member:
                    self.logger.info(
                        "Add Core Share user -> '%s' (%s) to Core Share group -> '%s' (%s)...",
                        user_name,
                        core_share_user_id,
                        group_name,
                        group_id,
                    )
                    # We make users that have this group as base_group
                    # to Admins of the Core Share group:
                    is_group_admin = user_group == base_group
                    response = self._core_share.add_group_member(
                        group_id=group_id,
                        user_id=core_share_user_id,
                        is_group_admin=is_group_admin,
                    )
                    # the add_group_member() has a special return value
                    # which is a list (not a dict). It has mostly 1 element
                    # which is a dict with a "success" item. This (and not response.ok)
                    # determines if the call was successful!
                    success: bool = self._core_share.get_result_value(
                        response,
                        "success",
                    )
                    if not success:
                        errors = self._core_share.get_result_value(
                            response=response,
                            key="errors",
                        )
                        self.logger.error(
                            "Failed to add Core Share user -> '%s' (%s) as member to Core Share group -> '%s' (%s)! Error -> %s",
                            user_name,
                            core_share_user_id,
                            group_name,
                            group_id,
                            str(errors),
                        )
                        success = False
                        continue
                else:
                    self.logger.info(
                        "Core Share user -> '%s' (%s) is already a %s of Core Share group -> '%s' (%s). Skipping...",
                        user_name,
                        core_share_user_id,
                        "member" if is_member else "pending member",
                        group_name,
                        group_id,
                    )
            # end for loop user groups
        # end for loop users

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._users,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_users_m365")
    def process_users_m365(self, section_name: str = "usersM365") -> bool:
        """Process users in payload and create them in Microsoft 365 via MS Graph API.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not isinstance(self._m365, M365):
            self.logger.error(
                "Microsoft 365 connection not setup properly. Skipping payload section -> '%s'...",
                section_name,
            )
            return False

        if not self._users:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # Add all users in payload and establish membership in
        # specified groups:
        for user in self._users:
            user_name = user.get("name")
            if not user_name:
                self.logger.error("User is missing a login. Skipping to next user...")
                success = False
                continue

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_name,
                )
                continue

            # Check if the user is enabled for Microsoft 365.
            # M365 is disabled per default for the users.
            # There needs to be "enable_o365" in user payload
            # and it needs to be True:
            if not user.get("enable_o365", False):
                self.logger.info(
                    "Microsoft 365 is not enabled in payload for user -> '%s'. Skipping...",
                    user_name,
                )
                continue

            # Sanity checks:
            if "password" not in user:
                self.logger.error(
                    "User -> '%s' is missing a password. Skipping to next user...",
                    user_name,
                )
                success = False
                continue
            user_password = user["password"]
            # be careful with the following fields - they could be empty
            user_department = user.get("base_group", "")
            user_first_name = user.get("firstname", "")
            user_last_name = user.get("lastname", "")
            user_location = user.get("location", "US")
            user_email = user.get("email", user_name)

            # Check if the user does already exist in M365 (e.g. if job is restarted)
            m365_user_id = self.determine_user_id_m365(user=user)
            if not m365_user_id:
                self.logger.info(
                    "Did not find existing Microsoft 365 user - creating user -> '%s'...",
                    user_email,
                )

                # Now we know it is a new user...
                # We are not 1:1 using the email address from the
                # payload as this could by an alias address using the "+" syntax:
                m365_user_name = user_name + "@" + self._m365.config()["domain"]

                new_user = self._m365.add_user(
                    email=m365_user_name,
                    password=user_password,
                    first_name=user_first_name,
                    last_name=user_last_name,
                    location=user_location,
                    department=user_department,
                )
                if new_user is not None:
                    # Store the Microsoft 365 user ID in payload:
                    user["m365_id"] = new_user["id"]
                    m365_user_id = new_user["id"]
                    self.logger.info(
                        "New Microsoft 365 user -> '%s' with ID -> %s has been created.",
                        user_name,
                        m365_user_id,
                    )
                else:
                    self.logger.error(
                        "Failed to create new Microsoft 365 user -> '%s'! Skipping...",
                        user_name,
                    )
                    success = False
                    continue
            else:
                # if the user exists we just set the password according
                # the the payload definition to allow to bulk
                # update existing M365 users with new passwords:
                self.logger.info(
                    "Found existing Microsoft 365 user -> '%s' - updating password...",
                    user_name,
                )
                new_password_settings = {
                    "passwordProfile": {
                        "forceChangePasswordNextSignIn": False,
                        "password": user_password,
                    },
                }
                response = self._m365.update_user(
                    user_id=m365_user_id,
                    updated_settings=new_password_settings,
                )
                if not response:
                    self.logger.error(
                        "Failed to update password of M365 user -> '%s' (%s)!",
                        user_name,
                        m365_user_id,
                    )

            # Now we assign a license to the new M365 user.
            # First we see if there's a M365 SKU list in user
            # payload - if not we wrap the default SKU configured
            # for the m365 object into a single item list:
            existing_user_licenses = self._m365.get_user_licenses(user_id=m365_user_id)
            sku_list = user.get("m365_skus", [self._m365.config()["skuId"]])
            for sku_id in sku_list:
                # Check if the M365 user already has this license:
                if not self._m365.exist_result_item(
                    existing_user_licenses,
                    "skuId",
                    sku_id,
                ):
                    response = self._m365.assign_license_to_user(m365_user_id, sku_id)
                    if not response:
                        self.logger.error(
                            "Failed to assign license -> '%s' to Microsoft 365 user -> '%s'!",
                            sku_id,
                            user_name,
                        )
                        success = False
                    else:
                        if (
                            "m365_skus" not in user
                        ):  # this is only True if the default license from the m365 object is taken
                            user["m365_skus"] = [sku_id]
                        self.logger.info(
                            "License -> '%s' has been assigned to Microsoft 365 user -> '%s'.",
                            sku_id,
                            user_name,
                        )
                else:
                    self.logger.info(
                        "Microsoft 365 user -> '%s' already has the license -> '%s'.",
                        user_name,
                        sku_id,
                    )

            # Now we assign the Content Server Teams App to the new M365 user.
            # First we check if the app is already assigned to the user.
            # If not we install / assign the app. If the user already has
            # the Content Server app we try to upgrade it:
            app_name = self._m365.config()["teamsAppName"]
            app_external_id = self._m365.config()["teamsAppExternalId"]
            app_internal_id = self._m365.config().get("teamsAppInternalId", None)
            response = self._m365.get_teams_apps_of_user(
                user_id=m365_user_id,
                filter_expression="contains(teamsAppDefinition/displayName, '{}')".format(
                    app_name,
                ),
            )
            if self._m365.exist_result_item(
                response=response,
                key="displayName",
                value=app_name,
                sub_dict_name="teamsAppDefinition",
            ):
                self.logger.info(
                    "M365 Teams app -> '%s' is already assigned to M365 user -> '%s' (%s). Trying to upgrade app...",
                    app_name,
                    user_name,
                    m365_user_id,
                )
                response = self._m365.upgrade_teams_app_of_user(
                    user_id=m365_user_id,
                    app_name=app_name,
                )
            else:
                self.logger.info(
                    "Assign M365 Teams app -> '%s' (%s) to M365 user -> '%s' (%s)...",
                    app_name,
                    app_external_id,
                    user_name,
                    m365_user_id,
                )
                # This can produce errors because the app may be assigned organization-wide.
                # So we don't treat it as an error and just show a warning.
                # We also try to use the internal app id instead of the name:
                if app_internal_id:
                    response = self._m365.assign_teams_app_to_user(
                        user_id=m365_user_id,
                        app_name=app_name,
                        app_internal_id=app_internal_id,
                        show_error=False,
                    )
                else:
                    response = self._m365.assign_teams_app_to_user(
                        user_id=m365_user_id,
                        app_name=app_name,
                        show_error=False,
                    )

            # Process Microsoft 365 group memberships of new user:
            # don't forget the base group (department) if it is not yet in groups!
            group_names = user.get("groups", [])
            if user_department and user_department not in group_names:
                group_names.append(user_department)
            self.logger.info(
                "User -> '%s' has these groups in payload -> %s (including base group -> %s). Checking if they are Microsoft 365 groups...",
                user_name,
                group_names,
                user_department,
            )

            # Go through all group names:
            for group_name in group_names:
                # Find the group payload item to the parent group name:
                group = next(
                    (item for item in self._groups if item["name"] == group_name),
                    None,
                )
                if not group:
                    # if group is not in payload then this membership
                    # is not relevant for Microsoft 365. This could be system generated
                    # groups like "PageEdit" or "Business Administrators".
                    # In this case we do "continue" as we can't process parent groups
                    # either:
                    self.logger.debug(
                        "No payload found for group -> '%s'. Skipping...",
                        group_name,
                    )
                    continue
                if "enable_o365" not in group or not group["enable_o365"]:
                    # If Microsoft 365 is not enabled for this group in
                    # the payload we don't create a M365 but we do NOT continue
                    # as there may still be parent groups that are M365 enabled
                    # we want to put the user in (see below):
                    self.logger.info(
                        "Payload group -> '%s' is not enabled for M365.",
                        group_name,
                    )
                else:
                    response = self._m365.get_group(group_name=group_name)
                    if response is None or "value" not in response or not response["value"]:
                        self.logger.error(
                            "Microsoft 365 group -> '%s' not found. Skipping...",
                            group_name,
                        )
                        success = False
                    else:
                        group_id = response["value"][0]["id"]

                        # Check if user is already a member. We don't want
                        # to throw an error if the user is not found as a member
                        # so we pass show_error=False:
                        if self._m365.is_member(
                            group_id,
                            m365_user_id,
                            show_error=False,
                        ):
                            self.logger.info(
                                "Microsoft 365 user -> '%s' (%s) is already in Microsoft 365 group -> '%s' (%s).",
                                user["name"],
                                m365_user_id,
                                group_name,
                                group_id,
                            )
                        else:
                            self.logger.info(
                                "Add Microsoft 365 user -> '%s' (%s) to Microsoft 365 group -> '%s' (%s)...",
                                user["name"],
                                m365_user_id,
                                group_name,
                                group_id,
                            )
                            response = self._m365.add_group_member(
                                group_id=group_id,
                                member_id=m365_user_id,
                            )
                            if not response:
                                self.logger.error(
                                    "Failed to add Microsoft 365 user -> '%s' (%s) to Microsoft 365 group -> '%s' (%s)!",
                                    user["name"],
                                    m365_user_id,
                                    group_name,
                                    group_id,
                                )
                                success = False

                            # As each group should have at least one owner in M365
                            # we set all users also as owners for now. Later we
                            # may want to configure this via payload:
                            self.logger.info(
                                "Make Microsoft 365 user -> '%s' (%s) owner of Microsoft 365 group -> '%s' (%s)...",
                                user["name"],
                                m365_user_id,
                                group_name,
                                group_id,
                            )
                            response = self._m365.add_group_owner(
                                group_id,
                                m365_user_id,
                            )
                            if not response:
                                self.logger.error(
                                    "Failed to make Microsoft 365 user -> '%s' (%s) owner of Microsoft 365 group -> '%s' (%s)!",
                                    user["name"],
                                    m365_user_id,
                                    group_name,
                                    group_id,
                                )
                                success = False

                # As M365 groups are flat (not nested), we also add the
                # user as member to the parent groups of the current group
                # if the parent group is enabled for M365:
                parent_group_names = group.get("parent_groups")
                self.logger.info(
                    "Group -> '%s' has the following parent groups -> %s",
                    group_name,
                    parent_group_names,
                )
                for parent_group_name in parent_group_names:
                    # Find the group dictionary item to the parent group name:
                    parent_group = next(
                        (item for item in self._groups if item["name"] == parent_group_name),
                        None,
                    )
                    if parent_group is None or "enable_o365" not in parent_group or not parent_group["enable_o365"]:
                        # if parent group is not in payload then this membership
                        # is not relevant for Microsoft 365.
                        # If Microsoft 365 is not enabled for this parent group in
                        # the payload we can also skip:
                        self.logger.info(
                            "Parent group -> '%s' is not enabled for M365. Skipping...",
                            group_name,
                        )
                        continue

                    response = self._m365.get_group(group_name=parent_group_name)
                    if response is None or "value" not in response or not response["value"]:
                        self.logger.error(
                            "Microsoft 365 group -> '%s' not found. Skipping...",
                            group_name,
                        )
                        success = False
                        continue
                    parent_group_id = response["value"][0]["id"]

                    # Check if user is already a member. We don't want
                    # to throw an error if the user is not found as a member:
                    if self._m365.is_member(
                        group_id=parent_group_id,
                        member_id=m365_user_id,
                        show_error=False,
                    ):
                        self.logger.info(
                            "Microsoft 365 user -> '%s' (%s) is already in Microsoft 365 group -> '%s' (%s).",
                            user["name"],
                            m365_user_id,
                            parent_group_name,
                            parent_group_id,
                        )
                    else:
                        self.logger.info(
                            "Add Microsoft 365 user -> '%s' (%s) to Microsoft 365 group -> '%s' (%s)...",
                            user["name"],
                            m365_user_id,
                            parent_group_name,
                            parent_group_id,
                        )
                        self._m365.add_group_member(
                            group_id=parent_group_id,
                            member_id=m365_user_id,
                        )
                # end for parent_group_name

                # Make this user follow the SharePoint site of his/her department.
                # We only do this for users that have a valid M365 license (SKU):
                if group_name == user_department and user["m365_skus"]:
                    group = self._m365.get_group(group_name=group_name)
                    group_id = self._m365.get_result_value(response=group, key="id")
                    if group_id:
                        group_site = self._m365.get_sharepoint_site_for_group(group_id=group_id)
                        group_site_id = self._m365.get_result_value(response=group_site, key="id")
                        if group_site_id:
                            group_site_name = self._m365.get_result_value(response=group_site, key="name")
                            # Make sure the user's mysite (drive) has been provisioned.
                            # This is a prerequisite for a user being able to follow a
                            # SharePoint site. For this to work we need "Files.ReadWrite"
                            # as a delegated permission in the Azure AppRegistration!
                            self.logger.info(
                                "Make sure user -> '%s' has a drive (mySite) provisioned...",
                                user["email"],
                            )
                            # We need to have 'delegated' permissions for this so we authenticate
                            # as the user... The scope is important here - the user's drive can only
                            # be provisioned if "Files.ReadWrite" scope is provided:
                            response = self._m365.authenticate_user(
                                username=user["email"], password=user["password"], scope="Files.ReadWrite"
                            )
                            if not response:
                                self.logger.error(
                                    "Couldn't authenticate as M365 user -> '%s' to provision user's drive!",
                                    username=user["email"],
                                )
                                success = False
                                continue
                            # Retrieve the drive endpoint to trigger the drive provisioning. It is important
                            # to use the 'me=True' to make sure the request is done with the user credentials,
                            # not the app credentials (using purely client_id / client_secret):
                            response = self._m365.get_user_drive(user_id=user["email"], me=True)
                            if not response:
                                self.logger.error("Couldn't get M365 drive of user -> '%s'!", user["email"])
                                success = False
                                continue
                            self.logger.info(
                                "Make user -> '%s' follow departmental SharePoint site -> '%s'...",
                                user["email"],
                                group_site_name,
                            )
                            response = self._m365.follow_sharepoint_site(site_id=group_site_id, user_id=m365_user_id)
                            if not response:
                                self.logger.warning(
                                    "User -> '%s' cannot follow SharePoint site -> '%s'!",
                                    user["email"],
                                    group_site_name,
                                )
                                success = False
                        # end if group_site_id:
                    # end if group_id:
                # end if group_name == user_department and user["m365_skus"]:
            # end for group name in group_names:
        # end for user

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._users,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_teams_m365")
    def process_teams_m365(self, section_name: str = "teamsM365") -> bool:
        """Process groups in payload and create matching Teams in Microsoft 365.

        We need to do this after the creation of the M365 users as we require
        Group Owners to create teams. These are NOT the teams for OTCS
        workspaces! Those are created by Scheduled Bots (Jobs) from OTCS!

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not isinstance(self._m365, M365):
            self.logger.error(
                "Microsoft 365 connection not setup properly. Skipping payload section -> '%s'...",
                section_name,
            )
            return False

        if not self._groups:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for group in self._groups:
            if "name" not in group:
                self.logger.error("Team needs a name. Skipping...")
                success = False
                continue
            group_name = group["name"]

            # Check if group has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not group.get("enabled", True):
                self.logger.info(
                    "Payload for group -> '%s' is disabled. Skipping...",
                    group_name,
                )
                continue

            if "enable_o365" not in group or not group["enable_o365"]:
                self.logger.info(
                    "Microsoft 365 is not enabled in payload for group -> '%s'. Skipping...",
                    group_name,
                )
                continue

            # Check if the M365 group does not exist (this should actually never happen at this point)
            m365_group_id = self.determine_group_id_m365(group=group)
            if not m365_group_id:
                # The "m365_id" value is set by the method process_groups_m365()
                self.logger.error(
                    "No M365 group exist for group -> '%s' (M365 group creation may have failed). Skipping...",
                    group_name,
                )
                success = False
                continue

            if self._m365.has_team(group_name=group_name):
                self.logger.info(
                    "M365 group -> '%s' already has an MS Team connected. Skipping...",
                    group_name,
                )
                continue

            self.logger.info(
                "Create M365 team -> '%s' for existing M365 group -> '%s'...",
                group_name,
                group_name,
            )
            # Now "upgrading" this group to a MS Team:
            new_team = self._m365.add_team(name=group_name)
            if not new_team:
                success = False
                continue

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._groups,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_teams_m365_apps")
    def process_teams_m365_apps(
        self,
        section_name: str = "teamsM365Apps",
        tab_name: str = "Extended ECM",
    ) -> bool:
        """Process groups in payload and configure Extended ECM Teams Apps.

        The app is exposed as a tab called "Extended ECM" in the "General"
        channel of the M365 Team.

        We need to do this after the transports as we need top level folders
        we can point the Extended ECM teams app to.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.
            tab_name (str, optional):
                Name of the Extended ECM tab. Default is "Extended ECM".

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not isinstance(self._m365, M365):
            self.logger.error(
                "Microsoft 365 connection not setup properly. Skipping payload section -> '%s'...",
                section_name,
            )
            return False

        if not self._groups:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # Determine the ID of the Extended ECM App:
        app_name = self._m365.config()["teamsAppName"]
        app_internal_id = self._m365.config()["teamsAppInternalId"]
        if not app_internal_id:
            response = self._m365.get_teams_apps(
                filter_expression="contains(displayName, '{}')".format(app_name),
            )
            # Get the App catalog ID:
            app_internal_id = self._m365.get_result_value(
                response=response,
                key="id",
                index=0,
            )
        if not app_internal_id:
            self.logger.error("M365 Teams app -> '%s' not found in catalog!", app_name)
            return False

        for group in self._groups:
            if "name" not in group:
                self.logger.error("Team needs a name. Skipping...")
                success = False
                continue
            group_name = group["name"]

            # Check if group has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not group.get("enabled", True):
                self.logger.info(
                    "Payload for group -> '%s' is disabled. Skipping...",
                    group_name,
                )
                continue

            if "enable_o365" not in group or not group["enable_o365"]:
                self.logger.info(
                    "Microsoft 365 is not enabled in payload for group -> '%s'. Skipping...",
                    group_name,
                )
                continue

            #
            # Now we create a tab in the "General" channel for the Extended ECM Teams App
            #

            # 1. Check if the tab is already assigned to the General channel.
            # This determines if we need to create or update the tab / app:
            app_name = self._m365.config()["teamsAppName"]
            response = self._m365.get_team_channel_tabs(
                team_name=group_name,
                channel_name="General",
            )
            # Check if tab is already there:
            if self._m365.exist_result_item(
                response=response,
                key="displayName",
                value=tab_name,
            ):
                self.logger.info(
                    "M365 Teams app -> '%s' is already configured for M365 Team -> '%s' (tab -> '%s' does already exist). Updating it with new URLs and IDs...",
                    app_name,
                    group_name,
                    tab_name,
                )
                update = True  # update existing tab
            else:
                self.logger.info(
                    "Add tab -> '%s' with app -> '%s' to channel -> 'General' of M365 Team -> '%s' ",
                    tab_name,
                    app_name,
                    group_name,
                )
                update = False  # create new tab

            # 2. Determine the M365 Team ID. If the team is not found then skip:
            response = self._m365.get_team(name=group_name)
            team_id = self._m365.get_result_value(response=response, key="id", index=0)
            if not team_id:
                self.logger.error("M365 Team -> '%s' not found!", group_name)
                success = False
                continue

            # 3. Install the App for the particular M365 Team (if it is not yet installed):
            response = self._m365.get_teams_apps_of_team(
                team_id=team_id,
                filter_expression="contains(teamsAppDefinition/displayName, '{}')".format(
                    app_name,
                ),
            )
            if self._m365.exist_result_item(
                response=response,
                key="displayName",
                value=app_name,
                sub_dict_name="teamsAppDefinition",
            ):
                self.logger.info(
                    "M365 Teams app -> '%s' is already installed for M365 Team -> '%s' (%s). Trying to upgrade app...",
                    app_name,
                    group_name,
                    team_id,
                )
                response = self._m365.upgrade_teams_app_of_team(
                    team_id=team_id,
                    app_name=app_name,
                )
                if not response:
                    self.logger.error(
                        "Failed to upgrade the existing app -> '%s' for the M365 Team -> '%s'!",
                        app_name,
                        group_name,
                    )
                    success = False
                    continue
            else:
                self.logger.info(
                    "Install M365 Teams app -> '%s' for M365 team -> '%s'...",
                    app_name,
                    group_name,
                )
                response = self._m365.assign_teams_app_to_team(
                    team_id=team_id,
                    app_id=app_internal_id,
                )
                if not response:
                    self.logger.error(
                        "Failed to install app -> '%s' (%s) for M365 Team -> '%s'!",
                        app_name,
                        app_internal_id,
                        group_name,
                    )
                    success = False
                    continue

            # 4. Create a Tab in the "General" channel of the M365 Team:
            if group_name == "Innovate":
                # Use the Enterprise Workspace for the
                # top-level group "Innovate":
                node_id = 2000
            else:
                # We assume the departmental group names are identical to
                # top-level folders in the Enterprise volume
                node = self._otcs.get_node_by_parent_and_name(
                    parent_id=2000,
                    name=group_name,
                )
                node_id = self._otcs.get_result_value(response=node, key="id")
                if not node_id:
                    self.logger.info(
                        "Cannot find a top-level folder for group -> '%s'. Cannot configure M365 Teams app. Skipping...",
                        group_name,
                    )
                    continue

            app_url = self._otcs_frontend.cs_support_public_url()  # it is important to use the frontend pod URL here
            app_url += "/xecmoffice/teamsapp.html?nodeId="
            app_url += str(node_id) + "&type=container&parentId=2000&target=content&csurl="
            app_url += self._otcs_frontend.cs_public_url()
            app_url += "&appId=" + app_internal_id

            if update:
                # App / Tab exist but needs to be updated with new
                # IDs for the new deployment of Extended ECM
                # as the M365 Teams survive between Terrarium deployments:

                self.logger.info(
                    "Updating tab -> '%s' of M365 Team channel -> 'General' for app -> '%s' (%s) with new URLs and node IDs...",
                    tab_name,
                    app_name,
                    app_internal_id,
                )

                response = self._m365.update_teams_app_of_channel(
                    team_name=group_name,
                    channel_name="General",
                    tab_name=tab_name,
                    app_url=app_url,
                    cs_node_id=node_id,
                )
            else:
                # Tab does not exist in "General" channel so we
                # add / configure it freshly:

                self.logger.info(
                    "Adding tab -> '%s' with app -> '%s' (%s) in M365 Team channel -> 'General'...",
                    tab_name,
                    app_name,
                    app_internal_id,
                )

                response = self._m365.add_teams_app_to_channel(
                    team_name=group_name,
                    channel_name="General",
                    app_id=app_internal_id,
                    tab_name=tab_name,
                    app_url=app_url,
                    cs_node_id=node_id,
                )
                if not response:
                    self.logger.error(
                        "Failed to add tab -> '%s' with app -> '%s' (%s) to M365 Team channel -> 'General'!",
                        tab_name,
                        app_name,
                        app_internal_id,
                    )

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._groups,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="cleanup_stale_teams_m365")
    def cleanup_stale_teams_m365(self, workspace_types: list) -> bool:
        """Delete Microsoft Teams that are left-overs from former deployments.

        This method is currently not used.

        Args:
            workspace_types (list):
                The list of all workspace types.

        Returns:
            bool:
                True if successful, False otherwise.

        """

        if not isinstance(self._m365, M365):
            self.logger.error(
                "Microsoft 365 connection not setup properly. Skipping cleanup...",
            )
            return False

        if workspace_types == []:
            self.logger.error("Empty workspace type list!")
            return False

        for workspace_type in workspace_types:
            if "name" not in workspace_type:
                self.logger.error(
                    "Workspace type -> '%s' does not have a name. Skipping...",
                    workspace_type,
                )
                continue

            workspace_instances = self._otcs.get_workspace_instances_iterator(
                type_name=workspace_type["name"],
            )
            for workspace in workspace_instances:
                workspace = workspace.get("data").get("properties")

                workspace_name = workspace["name"]
                self.logger.info(
                    "Check if stale Microsoft 365 Teams -> '%s' exist...",
                    workspace_name,
                )
                self._m365.delete_teams(name=workspace_name)

        return True

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="cleanup_all_teams_m365")
    def cleanup_all_teams_m365(self, section_name: str = "teamsM365Cleanup") -> bool:
        """Delete Microsoft Teams that are left-overs from former deployments.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if teams have been deleted, False otherwise.

        """

        if not isinstance(self._m365, M365):
            self.logger.error(
                "Microsoft 365 connection not setup properly. Skipping payload section -> '%s'...",
                section_name,
            )
            return False

        # We want this cleanup to only run once even if we have
        # multiple payload files - so we pass payload_specific=False here:
        if self.check_status_file(
            payload_section_name=section_name,
            payload_specific=False,
        ):
            self.logger.info(
                "Payload section -> '%s' has been processed successfully before. Skip cleanup of M365 teams...",
                section_name,
            )
            return True

        self.logger.info("Processing payload section -> '%s'...", section_name)

        # We don't want to delete MS Teams that are matching the regular OTCS Group Names (like "Sales")
        exception_list = self.get_all_group_names()

        # These are the patterns that each MS Teams needs to match at least one of to be deleted
        # Pattern 1: all MS teams with a name that has a number in brackets, like "(1234)"
        # Pattern 2: all MS Teams with a name that starts with a number followed by a space,
        #            followed by a "-" and followed by another space
        # Pattern 3: all MS Teams with a name that starts with "WS" and a 1-4 digit number
        #            (these are the workspaces for Purchase Contracts generated for Intelligent Filing)
        # Pattern 4: all MS Teams with a name that ends with a 1-2 character + a number in brackets, like (US-1000)
        #            this is a specialization of pattern 1
        # Pattern 5: remove the teams that are created for the dummy copy&paste template for the
        #            Intelligent Filing workspaces
        pattern_list = [
            r"\(\d+\)",
            r"\d+\s-\s",
            r"^WS\d{1,4}$",
            r"^.+?\s\(.{1,2}-\d+\)$",
            r"Purchase\sContract\s\(Template\sfor\sIntelligent\sFiling\)",
            r"^OpenText.*$",
            r"^P-100.*$",
            r"^OILRIG.*$",
            r"^AGILUM.*$",
            r"^HD-102T.*$",
            r"^SG325A.*$",
            r"^[A-Za-z0-9]{18} - .*$",  # delete teams that start with the typical Salesforce IDs (e.g. opportunities)
            r".*\s\([A-Z]{3,4}\)$",  # delete stale Locations from NTSB scenario
        ]

        result = self._m365.delete_all_teams(
            exception_list=exception_list,
            pattern_list=pattern_list,
        )

        # We want this cleanup to only run once even if we have
        # multiple payload files - so we pass payload_specific=False here:
        self.write_status_file(
            success=True,
            payload_section_name=section_name,
            payload_section=exception_list + pattern_list,
            payload_specific=False,
        )

        return result

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_sites_m365")
    def process_sites_m365(self, section_name: str = "sitesM365") -> bool:
        """Process M365 groups in payload and configure SharePoint sites in Microsoft 365.

        These are NOT the SharePoint sites for Business Workspaces which are created
        by Scheduled Bots (Jobs) from OTCS via the creation of MS teams
        (each MS Team has a SharePoint site behind it)!

        These are the SharePoint sites for the departmental groups such as "Sales",
        "Procurement", "Enterprise Asset Management", ...
        Only departmental group that have a top-level folder with the exact same
        name as the Department are configured.

        For each departmental group:
        1. Determine a departmental folder in the Enterprise Workspace
        2. Determine the M365 Group
        3. Determine the SharePoint Site (based on the M365 group ID)
        4. Determine the Page in the SharePoint site
        5. Determine or create the SharePoint webpart for the OTCS browser
        6. Create URL object pointing to SharePoint site inside top level department folder

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not isinstance(self._m365, M365):
            self.logger.error(
                "Microsoft 365 connection not setup properly. Skipping payload section -> '%s'...",
                section_name,
            )
            return False

        if not self._groups:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for group in self._groups:
            if "name" not in group:
                self.logger.error("Group needs a name. Cannot configure SharePoint site for it. Skipping...")
                success = False
                continue
            group_name = group["name"]

            # Check if group has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not group.get("enabled", True):
                self.logger.info(
                    "Payload for group -> '%s' is disabled. Cannot configure SharePoint site. Skipping...",
                    group_name,
                )
                continue

            if "enable_o365" not in group or not group["enable_o365"]:
                self.logger.info(
                    "Microsoft 365 is not enabled in payload for group -> '%s'. Skipping...",
                    group_name,
                )
                continue

            #
            # 1. Determine a departmental folder in the Enterprise Workspace:
            #

            if group_name == "Innovate":
                folder_id = 2000
            else:
                folder = self._otcs.get_node_by_volume_and_path(
                    volume_type=self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE,
                    path=[group_name],
                    show_error=False,
                )
                folder_id = self._otcs.get_result_value(response=folder, key="id")
            if not folder_id:
                self.logger.warning(
                    "Group -> '%s' has no folder associated. Cannot configure SharePoint site. Skipping...",
                    group_name,
                )
                continue

            #
            # 2. Determine the M365 Group
            #

            # Check if the M365 group does not exist (this should actually never happen at this point)
            m365_group_id = self.determine_group_id_m365(group=group)
            if not m365_group_id:
                # The "m365_id" value is set by the method process_groups_m365()
                self.logger.error(
                    "No M365 group exist for group -> '%s' (M365 group creation may have failed). Skipping...",
                    group_name,
                )
                success = False
                continue

            #
            # 3. Determine the SharePoint Site:
            #

            site = self._m365.get_sharepoint_site_for_group(group_id=m365_group_id)
            site_id = self._m365.get_result_value(response=site, key="id")
            if not site_id:
                self.logger.info(
                    "M365 group -> '%s' has no M365 SharePoint site connected. Skipping...",
                    group_name,
                )
                success = False
                continue
            site_name = self._m365.get_result_value(response=site, key="name")

            # Store the SharePoint site ID in the payload.
            group["m365_site_id"] = site_id
            group["m365_site_name"] = site_name
            group["m365_folder_id"] = folder_id

            self.logger.info(
                "Configure M365 SharePoint site -> '%s' (%s) for M365 group -> '%s'...",
                site_name,
                site_id,
                group_name,
            )

            #
            # 4. Determine the Page in the SharePoint site:
            #

            site_page_name = group.get("o365_site_page_name", "OpenText Content Management")

            site_page_id = None
            site_pages = self._m365.get_sharepoint_pages(site_id=site_id)
            if site_pages:
                site_page_id = self._m365.lookup_result_value(
                    response=site_pages,
                    key="title",
                    value=site_page_name,
                    return_key="id",
                )
                if site_page_id:
                    self.logger.info(
                        "Found existing page -> '%s' (%s) for SharePoint site -> '%s' (%s).",
                        site_page_name,
                        site_page_id,
                        site_name,
                        site_id,
                    )
            if not site_page_id:
                site_page = self._m365.add_sharepoint_page(site_id=site_id, page_name=site_page_name)
                if site_page:
                    site_page_id = self._m365.get_result_value(response=site_page, key="id")
                    self.logger.info(
                        "Successfully created page -> '%s' (%s) for SharePoint site -> '%s' (%s).",
                        site_page_name,
                        site_page_id,
                        site_name,
                        site_id,
                    )
                else:
                    self.logger.error(
                        "Failed to create page -> '%s' for SharePoint site -> '%s' (%s)!",
                        site_page_name,
                        site_name,
                        site_id,
                    )
                    success = False
                    continue

            #
            # 5. Determine or create the SharePoint webpart for the OTCS browser:
            #

            # Get all webparts on the page:
            site_webparts = self._m365.get_sharepoint_webparts(site_id=site_id, page_id=site_page_id)
            # Check if there's already a webpart for the OTCS browser:
            site_webpart_id = self._m365.lookup_result_value(
                response=site_webparts,
                key="webPartType",
                value="cecfdba4-2e82-4538-9436-dbd1c4c01a80",  # OTCS Browser Type
                return_key="id",
            )
            if site_webpart_id:
                self.logger.info(
                    "Found existing OTCS Browser webpart -> '%s'. Updating with folder ID -> %s...",
                    site_webpart_id,
                    str(folder_id),
                )
                # Update the webpart with the new ID (which has changed after redeployment):
                update_data = {
                    "properties": {
                        "ContentServerFolderParentWP": "2000",
                        "ContentServerFolderSelectedWP": str(folder_id),
                    },
                }
                response = self._m365.update_sharepoint_webpart(
                    site_id=site_id,
                    page_id=site_page_id,
                    webpart_id=site_webpart_id,
                    update_data=update_data,
                )
                if response:
                    self.logger.info(
                        "Successfully updated OTCS Browser webpart -> '%s' with new folder ID -> %s.",
                        site_webpart_id,
                        str(folder_id),
                    )
                else:
                    self.logger.error(
                        "Failed to update OTCS Browser webpart -> '%s' with new folder ID -> %s!",
                        site_webpart_id,
                        str(folder_id),
                    )
                    success = False
                    continue
            else:
                # Check if the section we want for the webpart does already
                # exist. Otherwise create it. Per default we use horizontal
                # section 2. This allows to have a banner as first section
                # that is not affected by the customizer and can be configured
                # manually:
                site_page_section_id = group.get("o365_site_page_section_id", 1)
                site_page_section_type = group.get("o365_site_page_section_type", "horizontalSections")
                self.logger.info(
                    "Check if %s section -> %s on page -> '%s' (%s) of SharePoint site -> '%s' (%s) does already exist...",
                    "horizontal" if site_page_section_type == "horizontalSections" else "vertical",
                    site_page_section_id,
                    site_page_name,
                    site_page_id,
                    site_name,
                    site_id,
                )
                site_page_section = self._m365.get_sharepoint_section(
                    site_id=site_id,
                    page_id=site_page_id,
                    section_type=site_page_section_type,
                    section_id=site_page_section_id,
                    show_error=False,
                )
                if not site_page_section:
                    site_page_section = self._m365.add_sharepoint_section(
                        site_id=site_id,
                        page_id=site_page_id,
                        section_type=site_page_section_type,
                        section_id=site_page_section_id,
                    )
                    if not site_page_section:
                        success = False
                        continue
                    self.logger.info(
                        "Created %s section -> %s on page -> '%s' (%s) of SharePoint site -> '%s' (%s)",
                        "horizontal" if site_page_section_type == "horizontalSections" else "vertical",
                        site_page_section_id,
                        site_page_name,
                        site_page_id,
                        site_name,
                        site_id,
                    )
                else:
                    self.logger.info(
                        "Using existing %s section -> %s on page -> '%s' (%s) of SharePoint site -> '%s' (%s)",
                        "horizontal" if site_page_section_type == "horizontalSections" else "vertical",
                        site_page_section_id,
                        site_page_name,
                        site_page_id,
                        site_name,
                        site_id,
                    )

                # If the sharepointAppRootSite is not configured
                # we try to extract the site URL from the site ID
                # which typically has a format like this:
                # ideateqa.sharepoint.com,2c59000d-f3e7-44d1-9a8e-e5df82b8ab01,34b48533-af41-4743-8b41-185a21f0b80f
                site_url = (
                    self._m365.config()["sharepointAppRootSite"]
                    if self._m365.config()["sharepointAppRootSite"]
                    else "https://" + site_id.split(",", 1)[0]
                )
                # Build the web part create data:
                create_data = {
                    "@odata.type": "microsoft.graph.webPartData",
                    "audiences": [],
                    "dataVersion": "1.0",
                    "description": "Browse, access and work with documents from OpenText Content Server.",
                    "title": "Content Server Browser",
                    "properties": {
                        "AzureAppId": self._m365.config()["clientId"],
                        "ContentServerURLWP": "",
                        "URLPrefixWP": "",
                        "SSOEnabledWP": "",
                        "ShowPersonalWorkspaceWP": "",
                        "ShowNavigationBreadcrumbWP": "",
                        "PageSizeWP": "",
                        "ContentServerFolderParentWP": "2000",
                        "ContentServerFolderSelectedWP": str(folder_id),
                        "ContentServerFolderDisplayWP": group_name,
                        "SettingStorageURLSite": site_url,
                        "ContentServerURLSite": self._otcs.config()["csPublicUrl"],
                        "URLPrefixSite": "/cssupport",
                        "ShowPersonalWorkspaceSite": "",
                        "ShowNavigationBreadcrumbSite": "",
                        "PageSizeSite": "",
                        "ContentServerFolderParentSite": "",
                        "ContentServerFolderSelectedSite": "",
                        "ContentServerFolderDisplaySite": "",
                        "SSOEnabledSite": "",
                        "SettingStorageURLGL": "{}/sites/appcatalog".format(
                            site_url,
                        ),
                        "ContentServerURLGL": self._otcs.config()["csPublicUrl"],
                        "URLPrefixGL": "/cssupport",
                        "ShowPersonalWorkspaceGL": "No",
                        "ShowNavigationBreadcrumbGL": "No",
                        "PageSizeGL": "",
                        "ContentServerFolderParentGL": "",
                        "ContentServerFolderSelectedGL": "",
                        "ContentServerFolderDisplayGL": "",
                        "SSOEnabledGL": "No",
                        "TargetPlatform": "SPOnline",
                        "ConfigurationSiteUrl": "",
                        "WebPartVersion": "23.4.0.0",
                        "ContentServerURLAltGL": "",
                        "SavedQueryIdGL": "2344",
                        "SavedQueryParentIdGL": "2329",
                        "SavedQueryNameGL": "Full Text Business Workspaces Search",
                        "isCurrentUserSiteAdmin": True,
                        "isCurrentUserGlobalAdmin": True,
                        "isTeamsContext": False,
                        "ContentServerUrlLocal": self._otcs.config()["csPublicUrl"],
                    },
                }

                response = self._m365.add_sharepoint_webpart(
                    site_id=site_id,
                    page_id=site_page_id,
                    webpart_type_id="cecfdba4-2e82-4538-9436-dbd1c4c01a80",
                    section_id=site_page_section_id,
                    create_data=create_data,
                )
                if response:
                    self.logger.info(
                        "Successfully added OTCS browser webpart -> '%s' to page -> '%s' (%s) on site ->'%s' (%s).",
                        self._m365.get_result_value(response=response, key="id"),
                        site_page_name,
                        site_page_id,
                        site_name,
                        site_id,
                    )
                else:
                    self.logger.error(
                        "Failed to add OTCS browser webpart to page -> '%s' (%s) on site ->'%s' (%s)!",
                        site_page_name,
                        site_page_id,
                        site_name,
                        site_id,
                    )
                    success = False
                # end else
            # end else

            #
            # 6. Create URL object pointing to SharePoint site inside top level department folder
            #

            item_name = (
                "SharePoint site for {} department".format(group_name)
                if folder_id != 2000
                else "SharePoint site for Innovate"
            )
            response = self._otcs.get_node_by_parent_and_name(parent_id=folder_id, name=item_name)
            item_id = self._otcs.get_result_value(response=response, key="id")
            if not item_id:
                response = self._otcs.create_item(
                    parent_id=folder_id,
                    item_type=self._otcs.ITEM_TYPE_URL,
                    item_name=item_name,
                    url=site["webUrl"],
                )
                self.logger.info(
                    "Created URL item -> '%s' (%s).",
                    item_name,
                    site["webUrl"],
                )
            else:
                self.logger.info(
                    "URL item -> '%s' (%s) does already exist.",
                    item_name,
                    site["webUrl"],
                )
        # end for group in self._groups:

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._groups,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_admin_settings")
    def process_admin_settings(
        self,
        admin_settings: list,
        section_name: str = "adminSettings",
    ) -> bool:
        """Process admin settings in payload and import them to Content Server.

            The payload section is a list of dicts with these items:
            {
                enabled: True or False to enable or disable the payload item
                filename: The filename of the XML file with admin settings.
                          It needs to be the plain filename like "admin.xml".
                          The files reside inside the container in /settings root
                          directory. They are placed there by the Terraform automation
                          and are taken from the ./settings/payload directory.
                description: Some description about the purpose of the settings.
                             Just for information and optional.
            }

        Args:
            admin_settings (list):
                List of admin settings. We need this parameter
                as we process two different lists.
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if a restart of the OTCS pods is required. False otherwise.

        """

        if not admin_settings:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return False  # important to return False here as otherwise we are triggering a restart of services!!

        # If this payload section has been processed successfully before we
        # can return False and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return False  # important to return False here as otherwise we are triggering a restart of services!!

        restart_required: bool = False
        success: bool = True

        for admin_setting in admin_settings:
            # Sanity checks:
            if "filename" not in admin_setting:
                self.logger.error(
                    "Filename is missing. Skipping to next admin setting...",
                )
                continue
            filename = admin_setting["filename"]

            # Check if admin setting has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not admin_setting.get("enabled", True):
                self.logger.info(
                    "Payload for setting file -> '%s' is disabled. Skipping...",
                    filename,
                )
                continue

            settings_file = self._custom_settings_dir + filename
            if os.path.exists(settings_file):
                description = admin_setting.get("description")
                self.logger.info(
                    "Processing admin settings from file -> '%s'%s...",
                    filename,
                    " ({})".format(description) if description else "",
                )

                # Read the config file:
                with open(settings_file, encoding="utf-8") as file:
                    file_content = file.read()

                self.logger.debug(
                    "Replace Placeholder -> '%s' in file -> %s",
                    self._placeholder_values,
                    file_content,
                )

                file_content = self.replace_placeholders(content=file_content)

                # Write the updated config file:
                tmpfile = os.path.join(tempfile.gettempdir(), os.path.basename(settings_file))
                with open(tmpfile, "w", encoding="utf-8") as file:
                    file.write(file_content)

                response = self._otcs.apply_config(xml_file_path=tmpfile)
                if response and response["results"]["data"]["restart"]:
                    self.logger.info(
                        "A restart of the Content Server service is required.",
                    )
                    restart_required = True

                    if admin_setting.get("restart", False):
                        self.logger.info(
                            "Immediate restart requested - restart of OTCS services...",
                        )
                        # Restart OTCS frontend and backend pods:
                        self._otcs_restart_callback(
                            backend=self._otcs_backend,
                            frontend=self._otcs_frontend,
                        )

                        restart_required = False

            else:
                self.logger.error(
                    "Admin settings file -> '%s' not found!",
                    settings_file,
                )
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=admin_settings,
        )

        return restart_required

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="check_external_system")
    def check_external_system(self, external_system: dict) -> bool:
        """Check if external system is reachable.

        Args:
            external_system (dict):
                The payload data structure of external system.
                We assume here that sanity check for valid data is already done before.

        Returns:
            bool:
                True = system is reachable, False otherwise.

        """

        as_url = external_system["as_url"]

        # Extract the hostname:
        external_system_hostname = urlparse(as_url).hostname
        # Write this information back into the data structure:
        external_system["external_system_hostname"] = external_system_hostname
        # Extract the port:
        external_system_port = urlparse(as_url).port if urlparse(as_url).port else 80
        # Write this information back into the data structure:
        external_system["external_system_port"] = external_system_port

        if self._http_object.check_host_reachable(
            hostname=external_system_hostname,
            port=external_system_port,
        ):
            self.logger.info(
                "Mark external system -> '%s' as reachable for later workspace creation and user synchronization...",
                external_system["external_system_name"],
            )
            external_system["reachable"] = True
            return True
        else:
            external_system["reachable"] = False
            return False

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_external_systems")
    def process_external_systems(self, section_name: str = "externalSystems") -> bool:
        """Process external systems in payload and create them in Content Server.

            The payload section is a list of dicts (each representing one external
            system) with these items:
            {
                enabled: True or False to enable or disable the payload item
                external_system_name: Name of the external systen.
                external_system_type: Type of the external system.
                                      Possible values are
                                      * SAP
                                      * SuccessFactors
                                      * Salesforce
                                      * Guidewire
                                      * AppWorks Platform
                                      * Business Scenario Sample
                base_url: Base URL of the external system
                as_url: Application Server URL of the external system
                username: (Technical) User Name for the connection
                password: Passord of the (technical) user
                oauth_client_id: OAuth client ID
                oauth_client_secret: OAuth client secret
                archive_logical_name: Logical name of Archive for SAP
                archive_certificate_file: Path and filename to certificate file.
                                          This file is inside the customizer
                                          pof file system.
                skip_connection_test: Should we skip the connection test for this
                                      external system?
            }
            If OAuth Client ID and Client Secret are provided then username
            and password are no longer used.

            In the payload for SAP external systems there are additional
            items "client", "destination" that are processed by init_sap()

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        Side Effects:
            - based on system_type different other settings in the dict are set
            - reachability is tested and a flag is set in the payload dict

        """

        if not self._external_systems:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # Even if this payload section has been processed successfully before we
        # cannot skip processing it once more to re-initialize self._sap and self._salesforce!

        success: bool = True

        for external_system in self._external_systems:
            #
            # 1: Do sanity checks for the payload:
            #
            if "external_system_name" not in external_system:
                self.logger.error(
                    "External System connection needs a logical system name! Skipping to next external system...",
                )
                success = False
                continue
            system_name = external_system["external_system_name"]

            if "external_system_type" not in external_system:
                self.logger.error(
                    "External System connection -> '%s' needs a type (SAP, Salesfoce, SuccessFactors, AppWorks Platform)! Skipping...",
                    system_name,
                )
                success = False
                continue
            system_type = external_system["external_system_type"]

            self._log_header_callback(
                text="Process External System -> '{}' ({})".format(system_name, system_type),
                char="-",
            )

            # Check if external system has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not external_system.get("enabled", True):
                self.logger.info(
                    "Payload for External System -> '%s' (%s) is disabled. Skipping...",
                    system_name,
                    system_type,
                )
                continue

            """
            Possible Connection Types for external systems:
            "Business Scenario Sample" (Business Scenarios Sample Adapter)
            "ot.sap.c4c.SpiAdapter" (SAP C4C SPI Adapter)
            "ot.sap.c4c.SpiAdapterV2" (C4C SPI Adapter V2)
            "HTTP" (Default WebService Adapter)
            "ot.sap.S4HANAAdapter" (S/4HANA SPI Adapter)
            "SF" (SalesForce Adapter)
            "SFInstance" (SFWebService)
            "GWInstance" (Guidewire Adapter)
            """

            # Set the default settings for the different system types:
            match system_type:
                # Check if we have a SuccessFactors system:
                case "SuccessFactors":
                    connection_type = "SFInstance"
                    auth_method = "OAUTH"
                    username = None
                    password = None
                case "SAP":
                    connection_type = "HTTP"
                    auth_method = "BASIC"
                    oauth_client_id = None
                    oauth_client_secret = None
                case "Salesforce":
                    connection_type = "SF"
                    auth_method = "OAUTH"
                    username = None
                    password = None
                case "Guidewire":
                    connection_type = "GWInstance"
                    auth_method = "BASIC"
                    oauth_client_id = None
                    oauth_client_secret = None
                case "AppWorks Platform":
                    connection_type = "HTTP"
                    auth_method = "BASIC"
                    oauth_client_id = None
                    oauth_client_secret = None
                case "Business Scenario Sample":
                    connection_type = "Business Scenario Sample"
                    auth_method = "BASIC"
                    oauth_client_id = None
                    oauth_client_secret = None
                case _:
                    self.logger.error(
                        "Unsupported system_type defined -> '%s'",
                        system_type,
                    )
                    return False

            base_url = external_system.get("base_url", "")

            if "as_url" not in external_system:
                self.logger.warning(
                    "External system connection -> '%s' needs an application server URL! Skipping to next external system...",
                    system_name,
                )
                success = False
                continue
            as_url = external_system["as_url"]

            self.logger.info(
                "Processing external system -> '%s' (type -> '%s', connection type -> '%s', endpoint -> '%s')...",
                system_name,
                system_type,
                connection_type,
                as_url,
            )

            skip_connection_test = external_system.get("skip_connection_test", False)
            # If skip_connection_test is not enabled in payload and the external system is
            # not of type 'Business Scenario Sample', we run the external system check:
            if not skip_connection_test and system_type != "Business Scenario Sample":
                # Check if external system is reachable and
                # update the payload dict with a "reachable" key/value pair:
                if not self.check_external_system(external_system=external_system):
                    self.logger.warning(
                        "External System connection -> '%s' (%s) is not reachable! Skipping to next external system...",
                        system_name,
                        system_type,
                    )
                    success = False
                    continue
            else:
                self.logger.info(
                    "Skipped reachability check for external system -> '%s' (%s)...",
                    system_name,
                    system_type,
                )

            # Read either username/password (BASIC) or client ID / secret (OAuth)
            match auth_method:
                case "BASIC":
                    if "username" not in external_system:
                        self.logger.warning(
                            "External System connection -> '%s' needs a user name for BASIC authentication! Skipping to next external system...",
                            system_name,
                        )
                        continue
                    if "password" not in external_system:
                        self.logger.warning(
                            "External System connection -> '%s' needs a password for BASIC authentication! Skipping to next external system...",
                            system_name,
                        )
                        continue
                    username = external_system["username"]
                    password = external_system["password"]
                    oauth_client_id = ""
                    oauth_client_secret = ""

                case "OAUTH":
                    if "oauth_client_id" not in external_system:
                        self.logger.error(
                            "External System connection -> '%s' is missing OAuth client ID! Skipping to next external system...",
                            system_name,
                        )
                        success = False
                        continue
                    if "oauth_client_secret" not in external_system:
                        self.logger.error(
                            "External System connection -> '%s' is missing OAuth client secret! Skipping to next external system...",
                            system_name,
                        )
                        success = False
                        continue
                    oauth_client_id = external_system["oauth_client_id"]
                    oauth_client_secret = external_system["oauth_client_secret"]
                    # For backward compatibility we also read username/password
                    # with OAuth settings:
                    username = external_system["username"] if external_system.get("username") else None
                    password = external_system["password"] if external_system.get("password") else None
                case _:
                    self.logger.error(
                        "Unsupported authorization method specified (%s) , Skipping ... ",
                        auth_method,
                    )
                    return False
            # end match

            # We do this existance test late in this function to make sure the payload
            # datastructure is properly updated for debugging purposes.
            self.logger.info(
                "Test if external system -> '%s' does already exist...",
                system_name,
            )
            if self._otcs.get_external_system_connection(connection_name=system_name):
                self.logger.info(
                    "External system connection -> '%s' already exists.",
                    system_name,
                )
                # This is for handling re-runs of customizer pod where the transports
                # are skipped and thus self._sap or self._salesforce may not be initialized:
                if system_type == "SAP" and not self._sap:
                    self.logger.info(
                        "Re-Initialize SAP connection for external system -> '%s'...",
                        system_name,
                    )
                    # Initialize SAP object responsible for communication to SAP:
                    self._sap = self.init_sap(external_system)
                if system_type == "Salesforce" and not self._salesforce:
                    self.logger.info(
                        "Re-Initialize Salesforce connection for external system -> '%s'...",
                        system_name,
                    )
                    # Initialize Salesforce object responsible for communication to Salesforce:
                    self._salesforce = self.init_salesforce(
                        salesforce_external_system=external_system,
                    )
                if system_type == "SuccessFactors" and not self._successfactors:
                    self.logger.info(
                        "Re-Initialize SuccessFactors connection for external system -> '%s'...",
                        system_name,
                    )
                    # Initialize SuccessFactors object responsible for communication to SuccessFactors:
                    self._successfactors = self.init_successfactors(
                        sucessfactors_external_system=external_system,
                    )
                if system_type == "Guidewire" and "policy" in system_name.lower() and not self._guidewire_policy_center:
                    self.logger.info(
                        "Re-Initialize Guidewire connection for external system -> '%s'...",
                        system_name,
                    )
                    # Initialize Guidewire object responsible for communication to Guidewire Policy Center:
                    self._guidewire_policy_center = self.init_guidewire(
                        guidewire_external_system=external_system,
                    )
                if system_type == "Guidewire" and "claim" in system_name.lower() and not self._guidewire_claims_center:
                    self.logger.info(
                        "Re-Initialize Guidewire connection for external system -> '%s'...",
                        system_name,
                    )
                    # Initialize Guidewire object responsible for communication to Guidewire Claims Center:
                    self._guidewire_claims_center = self.init_guidewire(
                        guidewire_external_system=external_system,
                    )

                continue

            #
            # Create External System:
            #

            response = self._otcs.add_external_system_connection(
                connection_name=system_name,
                connection_type=connection_type,
                as_url=as_url,
                base_url=base_url,
                username=str(username),
                password=str(password),
                authentication_method=auth_method,
                client_id=oauth_client_id,
                client_secret=oauth_client_secret,
            )
            if response is None:
                self.logger.error(
                    "Failed to create external system -> '%s'; type -> '%s'!",
                    system_name,
                    connection_type,
                )
                success = False
            else:
                self.logger.info(
                    "Successfully created external system -> '%s'.",
                    system_name,
                )

            #
            # In case of an SAP external system we also initialize the SAP object
            #
            if system_type == "SAP":
                # Initialize SAP object responsible for communication to SAP:
                self._sap = self.init_sap(sap_external_system=external_system)

            #
            # In case of an SuccessFactors external system we also initialize the SuccessFactors object
            #
            if system_type == "SuccessFactors":
                # Initialize SuccessFactors object responsible for communication to SuccessFactors:
                self._successfactors = self.init_successfactors(
                    sucessfactors_external_system=external_system,
                )

            #
            # In case of an Salesforce external system we also initialize the Salesforce object
            #
            if system_type == "Salesforce":
                # Initialize Salesforce object responsible for communication to Salesforce:
                self._salesforce = self.init_salesforce(
                    salesforce_external_system=external_system,
                )

            #
            # In case of an Guidewire external system we also initialize the Guidewire objects:
            #
            if system_type == "Guidewire":
                if "claim" in system_name.lower():
                    # Initialize Guidewire Claim Center object:
                    self._guidewire_claims_center = self.init_guidewire(
                        guidewire_external_system=external_system,
                    )
                elif "policy" in system_name.lower():
                    # Initialize Guidewire Policy Center object:
                    self._guidewire_policy_center = self.init_guidewire(
                        guidewire_external_system=external_system,
                    )

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._external_systems,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="lookup_external_system")
    def lookup_external_system(self, ext_system_id: str, prefix: str = "success_payload_") -> dict | None:
        """Lookup an external system.

        Considers the current payload but also payload processed before (which is stored
        as JSON file in the Admin Personal Workspace).

        TODO: This is a workaround as the REST API for external systems is too limited
        and does not return the actual configuration settings of external systems.
        Check for newer OTCS versions.

        Args:
            ext_system_id (str):
                The external system name to lookup.
            prefix (str):
                The prefix of the success file in the Admin personal workspace.

        Returns:
            dict | None:
                The configuration data of the external system.

        """

        # First check if external system is in current payload:
        external_system = next(
            (item for item in self._external_systems if item["external_system_name"] == ext_system_id), None
        )
        if external_system:
            self.logger.info("Found external system -> '%s' declared in current payload.", ext_system_id)
            return external_system

        # Now we try to load it from previous processed payloads:

        if not self.check_status_file(payload_section_name="externalSystems", payload_specific=False, prefix=prefix):
            self.logger.error("Cannot find external system -> '%s'", ext_system_id)
            return None

        additional_systems = self.get_status_file(
            payload_section_name="externalSystems", payload_specific=False, prefix=prefix
        )

        # Merge avoiding duplicates and only enabled entries. existing_names is a set:
        existing_names = {item["external_system_name"] for item in self._external_systems}
        for sys in additional_systems:
            if sys["enabled"] and sys["external_system_name"] not in existing_names:
                self._external_systems.append(sys)
                existing_names.add(sys["external_system_name"])

        # Try finding the external system payload again after merging:
        external_system = next(
            (item for item in self._external_systems if item["external_system_name"] == ext_system_id), None
        )

        if external_system:
            self.logger.info("Found external system -> '%s' in previously processed payload file.", ext_system_id)
        else:
            self.logger.error(
                "Cannot find external system -> '%s' in list of external systems -> %s!",
                ext_system_id,
                str(existing_names),
            )

        return external_system

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="download_transport_package")
    def download_transport_package(
        self,
        package_url: str,
        download_dir: str | None = None,
    ) -> str | None:
        """Download the transort package from the given URL.

        Args:
            package_url (str):
                The URL to the transport package.
            package_name (str):
                The name of the transport package.
            download_dir (str, optional):
                The file system directory to download to. If None,
                a temporary directory is automatically determined.

        Returns:
            str | None:
                Path of the transport package in local file system or None in case of an error.

        """

        if not download_dir:
            download_dir = os.path.join(tempfile.gettempdir(), "transports/")

        if not self._http_object:
            self._http_object = HTTP(logger=self.logger)

        # Parse the URL
        parsed_url = urlparse(package_url)

        # Extract the path from the URL
        path = parsed_url.path

        # Get the file name from the path
        file_name = os.path.basename(path)

        download_name = os.path.join(download_dir, file_name)

        os.makedirs(download_dir, exist_ok=True)

        if not self._http_object.download_file(
            url=package_url,
            filename=download_name,
            show_error=True,
        ):
            return None

        self.logger.info(
            "Successfully downloaded transport package from -> '%s' to local file -> '%s'.",
            package_url,
            download_name,
        )

        return download_name

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_transport_packages")
    def process_transport_packages(
        self,
        transport_packages: list,
        section_name: str = "transportPackages",
    ) -> bool:
        """Process transport packages in payload and import them to Content Server.

        Args:
            transport_packages (list):
                A list of transport packages. As we have three different lists (transport,
                content_transport, transport_post) we need a parameter to pass it.
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not transport_packages:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for transport_package in transport_packages:
            name = transport_package.get("name")
            if not name:
                self.logger.error(
                    "Transport package needs a name! Skipping to next transport...",
                )
                success = False
                continue

            # Check if transport package has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not transport_package.get("enabled", True):
                self.logger.info(
                    "Payload for transport package -> '%s' is disabled. Skipping...",
                    name,
                )
                continue

            url = transport_package.get("url")
            if not url:
                self.logger.error(
                    "Transport package -> '%s' needs a URL! Skipping to next transport...",
                    name,
                )
                success = False
                continue

            description = transport_package.get("description", "")
            if not description:
                self.logger.warning(
                    "Transport package -> '%s' is missing a description.",
                    name,
                )

            # if the transport is not in the local file system
            # but given by an URL we download it first:
            if url.startswith("http"):
                package_path = self.download_transport_package(package_url=url)
                if not package_path:
                    success = False
                    continue
                url = package_path

            # For some transports there can be string replacements
            # configured:
            if "replacements" in transport_package:
                replacements = transport_package["replacements"]
                self.logger.info(
                    "Transport -> '%s' has replacements -> %s.",
                    name,
                    str(replacements),
                )
            else:
                replacements = None

            # For some transports there can be data extractions
            # configured:
            if "extractions" in transport_package:
                extractions = transport_package["extractions"]
                self.logger.info(
                    "Transport -> '%s' has extractions -> %s.",
                    name,
                    str(extractions),
                )
            else:
                extractions = None

            if description:
                self.logger.info("Deploy transport -> '%s' ('%s')...", name, description)
            else:
                self.logger.info("Deploy transport -> '%s'...", name)

            response = self._otcs.deploy_transport(
                package_url=url,
                package_name=name,
                package_description=description,
                replacements=replacements,
                extractions=extractions,
            )
            if response is None:
                self.logger.error("Failed to deploy transport -> '%s'!", name)
                success = False
                if self._stop_on_error:
                    break  # stop the for loop
            else:
                self.logger.info("Successfully deployed transport -> '%s'.", name)
                # Save the extractions for later processing, e.g. in process_business_object_types()
                if extractions:
                    self.add_transport_extractions(extractions=extractions)
        # end for transports

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=transport_packages,
        )
        self.write_status_file(
            success=success,
            payload_section_name=section_name + "Extractions",
            payload_section=self._transport_extractions,
        )

        # If stop_on_error is enabled we want to completely
        # stop the execution of the customizer and bail out:
        if not success and self._stop_on_error:
            raise StopOnError(message="STOP_ON_ERROR enabled -> Stopping execution")

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_photos")
    def process_user_photos(self, section_name: str = "userPhotos") -> bool:
        """Process user photos in payload and assign them to Content Server users.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._users:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # we assume the nickname of the photo item equals the login name of the user
        # we also assume that the photos have been uploaded / transported into the target system
        for user in self._users:
            user_name = user.get("name")
            if not user_name:
                self.logger.error("User is missing a login. Skipping to next user...")
                success = False
                continue

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_name,
                )
                continue

            # We skip also user of type "ServiceUser":
            if user.get("type", "User") == "ServiceUser":
                self.logger.info("Skipping service user -> '%s'...", user_name)
                continue

            user_id = user.get("id")
            if not user_id:
                self.logger.error(
                    "User -> '%s' does not have an ID. The user creation may have failed before. Skipping...",
                    user_name,
                )
                success = False
                continue

            response = self._otcs.get_node_from_nickname(nickname=user_name)
            if response is None:
                self.logger.warning(
                    "Missing photo for user -> '%s' - nickname not found. Skipping...",
                    user_name,
                )
                continue
            photo_id = self._otcs.get_result_value(response=response, key="id")
            response = self._otcs.update_user_photo(user_id=user_id, photo_id=photo_id)
            if not response:
                self.logger.error("Failed to add photo for user -> '%s'!", user_name)
                success = False
            else:
                self.logger.info("Successfully added photo for user -> '%s'.", user_name)

        # Check if Admin has a photo as well (nickname needs to be "admin"):
        response = self._otcs.get_node_from_nickname(nickname="admin")
        if response is None:
            self.logger.warning(
                "Missing photo for admin - nickname not found. Skipping...",
            )
        else:
            photo_id = self._otcs.get_result_value(response=response, key="id")
            response = self._otcs.update_user_photo(user_id=1000, photo_id=photo_id)
            if response is None:
                self.logger.warning("Failed to add photo for user -> 'admin'!")
            else:
                self.logger.info("Successfully added photo for user -> 'admin'.")

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._users,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_photos_m365")
    def process_user_photos_m365(self, section_name: str = "userPhotosM365") -> bool:
        """Process user photos in payload and assign them to Microsoft 365 users.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not isinstance(self._m365, M365):
            self.logger.error(
                "Microsoft 365 connection not setup properly. Skipping payload section -> '%s'...",
                section_name,
            )
            return False

        if not self._users:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # we assume the nickname of the photo item equals the login name of the user
        # we also assume that the photos have been uploaded / transported into the target system
        for user in self._users:
            user_name = user.get("name")
            if not user_name:
                self.logger.error("User is missing a login. Skipping to next user...")
                success = False
                continue

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_name,
                )
                continue

            if "id" not in user:
                self.logger.error(
                    "User -> '%s' does not have an ID. The user creation may have failed before. Skipping...",
                    user_name,
                )
                success = False
                continue

            if "enable_o365" not in user or not user["enable_o365"]:
                self.logger.info(
                    "Microsoft 365 is not enabled in payload for user -> '%s'. Skipping...",
                    user_name,
                )
                continue

            # If the customizer pod is restarted it may be that
            # the M365 user exists even if the M365 user ID is
            # not yet written back into the payload. So we use the
            # determine_user_id_m365() method that handles both cases
            # and updates the payload if the user exists in M365 already.
            user_m365_id = self.determine_user_id_m365(user=user)
            if not user_m365_id:
                self.logger.error(
                    "M365 user -> '%s' does not exist. Skipping...",
                    user_name,
                )
                success = False
                continue

            if self._m365.get_user_photo(user_id=user_m365_id, show_error=False):
                self.logger.info(
                    "User -> '%s' (%s) has already a photo in Microsoft 365. Skipping...",
                    user_name,
                    user_m365_id,
                )
                continue
            self.logger.info(
                "User -> '%s' (%s) has not yet a photo in Microsoft 365. Uploading...",
                user_name,
                user_m365_id,
            )

            response = self._otcs.get_node_from_nickname(nickname=user_name)
            if response is None:
                self.logger.warning(
                    "Missing photo for user -> '%s' - nickname not found. Skipping...",
                    user_name,
                )
                continue
            photo_id = self._otcs.get_result_value(response=response, key="id")
            photo_name = self._otcs.get_result_value(response=response, key="name")

            photo_path = os.path.join(tempfile.gettempdir(), photo_name)
            result = self._otcs.download_document(
                node_id=photo_id,
                file_path=photo_path,
            )
            # download_document() delivers a boolean result:
            if result is False:
                self.logger.warning(
                    "Failed to download photo for user -> '%s' from Content Server!",
                    user_name,
                )
                success = False
                continue
            self.logger.info(
                "Successfully downloaded photo for user -> '%s' from Content Server to file -> '%s'.",
                user_name,
                photo_path,
            )

            # Upload photo to M365:
            response = self._m365.update_user_photo(user_m365_id, photo_path)
            if response is None:
                self.logger.error(
                    "Failed to upload photo for user -> '%s' to Microsoft 365!",
                    user_name,
                )
                success = False
            else:
                self.logger.info(
                    "Successfully uploaded photo for user -> '%s' to Microsoft 365.",
                    user_name,
                )
        # end for loop

        # Check if Admin has a photo as well (nickname needs to be "admin")
        # Then we want this to be applied in M365 as well:
        response = self._otcs.get_node_from_nickname(nickname="admin")
        if response is None:
            self.logger.warning(
                "Missing photo for admin - nickname not found. Skipping...",
            )
        else:
            photo_id = self._otcs.get_result_value(response=response, key="id")
            photo_name = self._otcs.get_result_value(response=response, key="name")
            photo_path = os.path.join(tempfile.gettempdir(), photo_name)
            result = self._otcs.download_document(
                node_id=photo_id,
                file_path=photo_path,
            )
            # download_document() delivers a boolean result:
            if result is False:
                self.logger.warning(
                    "Failed to download photo for user -> 'admin' from Content Server!",
                )
                success = False
            else:
                self.logger.info(
                    "Successfully downloaded photo for user -> 'admin' from Content Server to file -> '%s'.",
                    photo_path,
                )
                m365_admin_email = "admin@" + self._m365.config()["domain"]
                response = self._m365.update_user_photo(
                    user_id=m365_admin_email,
                    photo_path=photo_path,
                )
                if response is None:
                    self.logger.warning("Failed to add photo for Microsoft 365 user -> '%s'!", m365_admin_email)
                else:
                    self.logger.info(
                        "Successfully added photo for Microsoft 365 user -> '%s'.",
                        m365_admin_email,
                    )

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._users,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_photos_salesforce")
    def process_user_photos_salesforce(
        self,
        section_name: str = "userPhotosSalesforce",
    ) -> bool:
        """Process user photos in payload and assign them to Salesforce users.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not isinstance(self._salesforce, Salesforce):
            self.logger.error(
                "Salesforce connection not setup properly. Skipping payload section -> '%s'...",
                section_name,
            )
            return False

        if not self._users:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # we assume the nickname of the photo item equals the login name of the user
        # we also assume that the photos have been uploaded / transported into the target system
        for user in self._users:
            user_name = user.get("name")
            if not user_name:
                self.logger.error("User is missing a login. Skipping to next user...")
                success = False
                continue

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_name,
                )
                continue

            # Check if the user is explicitly enabled for Salesforce:
            if not user.get("enable_salesforce", False):
                self.logger.info(
                    "User -> '%s' is not enabled for Salesforce. Skipping...",
                    user_name,
                )
                continue

            extra_attributes = user.get("extra_attributes", None)
            if not extra_attributes or len(extra_attributes) == 0:
                self.logger.info(
                    "User -> '%s' does not have the extra attributes for Salesforce. Skipping...",
                    user_name,
                )
                continue
            user_login = extra_attributes[0].get("value", "")
            if not user_login:
                self.logger.info(
                    "User -> '%s' does not have the extra attributes value for Salesforce. Skipping...",
                    user_name,
                )
                continue

            user_id = self._salesforce.get_user_id(username=user_login)
            if user_id is None:
                self.logger.error(
                    "Failed to get Salesforce user ID of user -> '%s'!",
                    user_login,
                )
                success = False
                continue

            response = self._otcs.get_node_from_nickname(nickname=user_name)
            if response is None:
                self.logger.warning(
                    "Missing photo for user -> '%s' - nickname not found. Skipping...",
                    user_name,
                )
                continue
            photo_id = self._otcs.get_result_value(response=response, key="id")
            photo_name = self._otcs.get_result_value(response=response, key="name")
            photo_path = os.path.join(tempfile.gettempdir(), photo_name)

            # Check if it is not yet downloaded:
            if not os.path.isfile(photo_path):
                # download the profile picture into the tmp directory:
                result = self._otcs.download_document(
                    node_id=photo_id,
                    file_path=photo_path,
                )
                # download_document() delivers a boolean result:
                if result is False:
                    self.logger.warning(
                        "Failed to download photo for user -> '%s' from Content Server to file -> '%s'!",
                        user_name,
                        photo_path,
                    )
                    success = False
                    continue
                self.logger.info(
                    "Successfully downloaded photo for user -> '%s' from Content Server to file -> '%s'.",
                    user_name,
                    photo_path,
                )
            else:
                self.logger.info(
                    "Reusing downloaded photo -> '%s' for Salesforce user -> '%s' (%s).",
                    photo_path,
                    user_name,
                    user_id,
                )

            response = self._salesforce.update_user_photo(
                user_id=user_id,
                photo_path=photo_path,
            )
            if response:
                self.logger.info(
                    "Successfully updated profile photo of Salesforce user -> '%s' (%s).",
                    user_login,
                    user_id,
                )
            else:
                self.logger.error(
                    "Failed to update profile photo of Salesforce user -> '%s' (%s)! Skipping...",
                    user_login,
                    user_id,
                )
                success = False
                continue

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._users,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_photos_core_share")
    def process_user_photos_core_share(
        self,
        section_name: str = "userPhotosCoreShare",
    ) -> bool:
        """Process user photos in payload and assign them to Core Share users.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not isinstance(self._core_share, CoreShare):
            self.logger.error(
                "Core Share connection not setup properly. Skipping payload section -> '%s'...",
                section_name,
            )
            return False

        if not self._users:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # we assume the nickname of the photo item equals the login name of the user
        # we also assume that the photos have been uploaded / transported into the target system
        for user in self._users:
            if "name" not in user:
                self.logger.error(
                    "User is missing login name. Skipping to next user...",
                )
                success = False
                continue
            user_login = user["name"]
            user_last_name = user.get("lastname", "")
            user_first_name = user.get("firstname", "")
            user_name = "{} {}".format(user_first_name, user_last_name).strip()

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_login,
                )
                continue

            # Check if the user is enabled for Core Share.
            # Core Share is disabled per default for the users.
            # There needs to be "enable_core_share" in user payload
            # and it needs to be True:
            if not user.get("enable_core_share", False):
                self.logger.info(
                    "User -> '%s' is not enabled for Core Share. Skipping...",
                    user_login,
                )
                continue

            core_share_user_id = self.determine_user_id_core_share(user=user)

            if core_share_user_id is None:
                self.logger.error(
                    "Failed to get ID of Core Share user -> '%s'!",
                    user_name,
                )
                success = False
                continue

            response = self._otcs.get_node_from_nickname(nickname=user_login)
            if response is None:
                self.logger.warning(
                    "Missing photo for user -> '%s' - nickname not found. Skipping...",
                    user_login,
                )
                continue
            photo_id = self._otcs.get_result_value(response=response, key="id")
            photo_name = self._otcs.get_result_value(response=response, key="name")
            photo_path = os.path.join(tempfile.gettempdir(), photo_name)

            # Check if it is not yet downloaded:
            if not os.path.isfile(photo_path):
                # download the profile picture into the tmp directory:
                result = self._otcs.download_document(
                    node_id=photo_id,
                    file_path=photo_path,
                )
                # download_document() delivers a boolean result:
                if result is False:
                    self.logger.warning(
                        "Failed to download photo for user -> '%s' from Content Server to file -> '%s'!",
                        user_name,
                        photo_path,
                    )
                    success = False
                    continue
                self.logger.info(
                    "Successfully downloaded photo for user -> '%s' from Content Server to file -> '%s'.",
                    user_name,
                    photo_path,
                )
            else:
                self.logger.info(
                    "Reusing downloaded photo -> '%s' for Core Share user -> '%s' (%s).",
                    photo_path,
                    user_name,
                    core_share_user_id,
                )

            response = self._core_share.update_user_photo(
                user_id=core_share_user_id,
                photo_path=photo_path,
            )
            if response:
                self.logger.info(
                    "Successfully updated profile photo of Core Share user -> '%s' (%s).",
                    user_name,
                    core_share_user_id,
                )
            else:
                self.logger.error(
                    "Failed to update profile photo of Core Share user -> '%s' (%s)! Skipping...",
                    user_name,
                    core_share_user_id,
                )
                success = False
                continue

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._users,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="extract_properties_from_transport_packages")
    def extract_properties_from_transport_packages(
        self, business_object_type: dict, bo_type_name: str, bo_type_id: int
    ) -> bool:
        """Extract properties from transport packages for a given Business Object Type.

        Args:
            business_object_type (dict):
                The Business Object Type dictionary to be enriched with properties.
            bo_type_name (str):
                The name of the Business Object Type.
            bo_type_id (int):
                The ID of the Business Object Type.

        Returns:
            bool:
                True if properties were successfully extracted and added, False otherwise.

        """

        business_object_type["business_properties"] = []
        business_object_type["business_property_groups"] = []

        # Now we complete the data with what we have extracted from the transport packages
        # for Business Object Types. This is a workaround for the insufficient REST API
        # implementation (see otcs.get_business_object_type)
        if self._transport_extractions:
            self.logger.info(
                "Enrich business object type -> '%s' (%d) with extractions from transport packages (found '%s' extractions)...",
                bo_type_name,
                bo_type_id,
                str(len(self._transport_extractions)),
            )
        else:
            self.logger.info(
                "No transport extractions are recorded. This may be because of customizer restart.",
            )
            extraction_status_file = "transportPackagesExtractions"
            if self.check_status_file(payload_section_name=extraction_status_file):
                self.logger.info(
                    "Try to load extractions from success file -> '%s'...",
                    extraction_status_file,
                )
                self._transport_extractions = self.get_status_file(
                    payload_section_name=extraction_status_file,
                )

        for extraction in self._transport_extractions:
            xpath = extraction.get("data")
            data_list = extraction.get("data")
            if not data_list:
                self.logger.error(
                    "Extraction -> '%s' is missing the data element. Skipping...",
                    xpath,
                )
                return False
            if not isinstance(data_list, list):
                self.logger.warning(
                    "Extracted data for -> '%s' is not a list. Cannot process it. Skipping...",
                    xpath,
                )
                return False

            """
            The following loop processes a dictionary of this structure:

            llnode: {
                '@created': '2017-11-23T16:43:35',
                '@createdby': '1000',
                '@createdbyname': 'Terrarium Admin',
                '@description': '',
                '@id': '16013',
                '@modified': '2023-12-09T12:08:21',
                '@name': 'SFDC Order',
                '@objname': 'Business Object Type',
                '@objtype': '889',
                '@ownedby': '1000',
                '@ownedbyname': 'Terrarium Admin',
                '@parentguid': '95F96645-057D-4EAF-9083-BE9F24C0CB6C',
                '@parentid': '2898',
                '@parentname': 'Business Object Types',
                ...
                'Nickname': {'@domain': ''},
                'name': {'@xml:lang': 'en', '#text': 'SFDC Order'},
                'description': {'@xml:lang': 'en'},
                'businessObjectTypeInfo': {
                    'basicInfo': {
                        '@businessObjectId': '9',
                        '@businessobjectType': 'Order',
                        '@deleted': 'false',
                        '@name': 'SFDC Order',
                        '@subtype': '889',
                        '@useBusWorkspace': 'true',
                        'displayUrl': {...}
                    },
                    'businessApplication': {
                        'businessObjectTypeReference': {...}},
                        'businessAttachmentInfo': {
                            '@automaticAddingOfBusinessObject': 'false',
                            '@canbeAddedAsBusinessObject': 'false',
                            '@enableBADIBeforeAddingBO': 'false',
                            '@enableBADIBeforeRemovingBO': 'false',
                            '@enableMetadataMapping': 'false'
                        },
                        'managedObjectTypes': {
                            'managedObjectType': [...]
                        },
                        'multilingualNames': {'language': [...]},
                        'callbacks': {'callback': [...]},
                        'workspaceTypeReference': {'@isDefaultDisplay': 'false', '@isDefaultSearch': 'false', 'businessObjectTypeReference': {...}},
                        'businessPropertyMappings': {
                            'propertyMapping': [...]
                        },
                        'businessPropertyGroupMappings': {
                            'propertyGroupMapping': [...]
                        },
                        'documentTypes': {
                            'documentType': [...]
                        },
                        'CustomBOTypeInfo': None
                    }
            }
            """

            for data in data_list:
                #
                # Level 1: llnode
                #
                llnode = data.get("llnode")
                if not llnode:
                    self.logger.error(
                        "Missing llnode structure in data. Skipping...",
                    )
                    return False

                #
                # Level 2: businessobjectTypeInfo
                #
                business_object_type_info = llnode.get(
                    "businessobjectTypeInfo",
                    None,
                )
                if not business_object_type_info:
                    self.logger.error(
                        "Information is missing for business object type -> '%s'. Skipping...",
                        bo_type_name,
                    )
                    return False

                # Check if this extraction is for the current business object type:
                basic_info = business_object_type_info.get("basicInfo", None)
                if not basic_info:
                    self.logger.error(
                        "Cannot find basic info of business object type -> '%s'. Skipping...",
                        bo_type_name,
                    )
                    return False
                name = basic_info.get("@businessobjectType", "")
                if not name:
                    self.logger.error(
                        "Cannot find name of business object type -> '%s'. Skipping...",
                        bo_type_name,
                    )
                    return False
                obj_type = llnode.get("@objtype", None)
                # we need to compare bo_type and NOT bo_type_name here!
                # Otherwise we don't find the SAP and SuccessFactors data:
                if name != bo_type_name or obj_type != "889":
                    continue

                #
                # Level 3: businessPropertyMappings - plain, non-grouped properties
                #
                business_property_mappings = business_object_type_info.get(
                    "businessPropertyMappings",
                    None,
                )
                if not business_property_mappings:
                    self.logger.info(
                        "No property mapping for business object type -> '%s'. Skipping...",
                        bo_type_name,
                    )
                    continue
                property_mappings = business_property_mappings.get(
                    "propertyMapping",
                    [],
                )
                # This can happen if there's only 1 propertyMapping;
                if not isinstance(property_mappings, list):
                    self.logger.debug(
                        "Found a single property mapping in a dictionary (not in a list). Package it into a list...",
                    )
                    property_mappings = [property_mappings]

                for property_mapping in property_mappings:
                    property_name = property_mapping.get("@propertyName")
                    attribute_name = property_mapping.get("@attributeName")
                    category_id = property_mapping.get("@categoryId")
                    mapping_type = property_mapping.get("@type")
                    self.logger.debug(
                        "%s Property Mapping for Business Object -> '%s' property -> '%s' is mapped to attribute -> '%s' (category -> %s)",
                        mapping_type,
                        bo_type_name,
                        property_name,
                        attribute_name,
                        category_id,
                    )
                    business_object_type["business_properties"].append(
                        property_mapping,
                    )

                #
                # Level 3: businessPropertyGroupMappings - grouped properties
                #
                business_property_group_mappings = business_object_type_info.get(
                    "businessPropertyGroupMappings",
                    None,
                )
                if not business_property_group_mappings:
                    self.logger.info(
                        "No property group mapping for business object type -> '%s'. Skipping...",
                        bo_type_name,
                    )
                    continue

                property_group_mappings = business_property_group_mappings.get(
                    "propertyGroupMapping",
                    [],
                )
                # This can happen if there's only 1 propertyMapping;
                if isinstance(property_group_mappings, dict):
                    self.logger.debug(
                        "Found a single property group mapping in a dictionary (not in a list). Pack it into a list...",
                    )
                    property_group_mappings = [property_group_mappings]

                for property_group_mapping in property_group_mappings:
                    group_name = property_group_mapping.get("@groupName")
                    set_name = property_group_mapping.get("@setName")
                    category_id = property_group_mapping.get("@categoryId")
                    mapping_type = property_group_mapping.get("@type")
                    self.logger.debug(
                        "%s property group mapping for business object -> %s: group -> '%s' is mapped to set -> '%s' (category -> %s)",
                        mapping_type,
                        bo_type_name,
                        group_name,
                        set_name,
                        category_id,
                    )

                    property_mappings = property_group_mapping.get(
                        "propertyMapping",
                        [],
                    )
                    # This can happen if there's only 1 propertyMapping;
                    if not isinstance(property_mappings, list):
                        self.logger.debug(
                            "Found a single property mapping in a dictionary (not in a list). Package it into a list...",
                        )
                        property_mappings = [property_mappings]

                    for property_mapping in property_mappings:
                        # for nested mappings we only have 2 fields - the rest is on the group level - see above
                        property_name = property_mapping.get("@propertyName")
                        attribute_name = property_mapping.get("@attributeName")
                        self.logger.debug(
                            "%s Property Mapping inside group for Business Object -> '%s', group -> '%s', property -> '%s' is mapped to set -> %s, attribute -> '%s' (category -> %s)",
                            mapping_type,
                            bo_type_name,
                            group_name,
                            property_name,
                            set_name,
                            attribute_name,
                            category_id,
                        )
                        # we write the group / set information also in the property mapping
                        # tp have a plain list with all information:
                        property_mapping["@groupName"] = group_name
                        property_mapping["@setName"] = set_name
                        property_mapping["@type"] = mapping_type
                        business_object_type["business_property_groups"].append(
                            property_mapping,
                        )
                # end for property_group_mapping in property_group_mappings
            # end for data in data_list
        # end for extraction in self._transport_extractions

        return True

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_business_object_types")
    def process_business_object_types(
        self,
        section_name: str = "businessObjectTypes",
    ) -> list:
        """Create a data structure for all business object types in the Extended ECM system.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            list:
                A list of business object types. Each list element is a dict with these values:
                - id (str)
                - name (str)
                - type (str)
                - ext_system_id (str)
                - business_properties (list)
                - business_property_groups (list)

        """

        # If this payload section has been processed successfully before we
        # still need to read the data structure from the status file and
        # initialize self._business_object_types:
        if self.check_status_file(payload_section_name=section_name):
            # read the list from the json file in admin Home
            # this is important for restart of customizer pod
            # as this data structure is used later on for workspace processing
            self.logger.info(
                "Re-Initialize business object types list from status file -> '%s' for later use...",
                self.get_status_file_name(payload_section_name=section_name),
            )
            self._business_object_types = self.get_status_file(
                payload_section_name=section_name,
            )
            if self._business_object_types:
                self.logger.info(
                    "Found -> %s business object types.",
                    str(len(self._business_object_types)),
                )
                self.logger.debug(
                    "Business object types -> %s",
                    str(self._business_object_types),
                )
                return self._business_object_types
            else:
                self.logger.warning(
                    "Couldn't read business object types from status file -> '%s'. Regenerate list...",
                    self.get_status_file_name(payload_section_name=section_name),
                )

        success: bool = True

        # Read payload_section "businessObjectTypes" if available.
        # For some (external) payload we may want to skip this time-
        # consuming processing as there are not BO specific payload
        # elements:
        for section in self._payload_sections:
            # Check if the section does exist and it is explicitly disabled
            # (enabled = False). Then we can skip further processing:
            if section.get("name", "") == "businessObjectTypes" and not section.get(
                "enabled",
                True,
            ):
                self.logger.info(
                    "Skip business object type processing as its section is disabled in payload.",
                )
                return self._business_object_types

        # get all business object types (these have been created by the transports and are not in the payload!)
        # we need to do this each time as it needs to work across multiple payload files...
        response = self._otcs.get_business_object_types()
        if response is None:
            self.logger.info("No business object types found!")
            self._business_object_types = []
        else:
            # Store result list in self._business_object_types for later use:
            self._business_object_types = response["results"]
            self.logger.info(
                "Found -> %s business object types.",
                str(len(self._business_object_types)),
            )
            self.logger.debug(
                "Business object types -> %s",
                str(self._business_object_types),
            )

        otcs_version = float(self._otcs.get_server_version())
        if otcs_version >= 25.3:
            self.logger.info(
                "Using new REST API of OTCS %s to retrieve business object type information...",
                str(otcs_version),
            )
        else:
            self.logger.info(
                "Have to use transport extraction with OTCS %s to retrieve business object type information...",
                str(otcs_version),
            )

        # now we enrich the business_object_type list elments (which are dicts)
        # with additional dict elements for further processing:
        for business_object_type in self._business_object_types:
            # Flatten the response structure for more easy retrieval:
            # Get BO Type (e.g. KNA1):
            bo_type = business_object_type["data"]["properties"]["bo_type"]
            self.logger.debug("Business object type -> %s", bo_type)
            business_object_type["type"] = bo_type
            # Get BO Type ID:
            bo_type_id = business_object_type["data"]["properties"]["bo_type_id"]
            self.logger.debug("Business object type ID -> %s", bo_type_id)
            business_object_type["id"] = bo_type_id
            # Get BO Type Name:
            bo_type_name = business_object_type["data"]["properties"]["bo_type_name"]
            self.logger.debug("Business object type name -> %s", bo_type_name)
            business_object_type["name"] = bo_type_name
            # Get External System ID:
            ext_system_id = business_object_type["data"]["properties"]["ext_system_id"]
            self.logger.debug("External system ID -> %s", ext_system_id)
            business_object_type["ext_system_id"] = ext_system_id
            # Get External System ID:
            workspace_type_id = business_object_type["data"]["properties"]["workspace_type_id"]
            self.logger.debug("Workspace type ID -> %s", workspace_type_id)
            business_object_type["workspace_type_id"] = workspace_type_id

            if not bo_type_name or not bo_type_id:
                self.logger.info(
                    "Business object type %sis void (dummy data for workspace types without a business object connection)! Skipping...",
                    "for workspace type ID -> {} ".format(workspace_type_id) if workspace_type_id else "",
                )
                continue
            self.logger.info(
                "Processing business object type -> '%s' (%s) with ID -> %s for workspace type with ID -> %d...",
                bo_type_name,
                bo_type,
                bo_type_id,
                workspace_type_id,
            )

            # Get additional information per BO Type (before 25.3 REST API is severly
            # limited) - it does not return Property names from External System
            # and is also missing Business Property Groups. Thus, for older versions
            # we extract that information from the Transport XML files (see else case):
            if otcs_version >= 25.3:
                response = self._otcs.get_business_object_type(type_id=bo_type_id)
                if response is None or not response["results"]:
                    self.logger.warning(
                        "Cannot retrieve additional information for business object type -> %s (%s). Skipping...",
                        bo_type,
                        bo_type_id,
                    )
                    success = False
                    continue

                business_properties = response["results"]["GeneralTab"]["data"]["properties"]["fWkspCreationConfig"][
                    "propertyMappings"
                ]
                business_object_type["business_properties"] = business_properties

                business_property_groups = response["results"]["GeneralTab"]["data"]["properties"][
                    "fWkspCreationConfig"
                ]["propertyGroups"]
                business_object_type["business_properties_groups"] = business_property_groups
            elif not self.extract_properties_from_transport_packages(business_object_type, bo_type_id, bo_type_name):
                success = False
        # end for business_object_type in self._business_object_types

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._business_object_types,
        )

        return self._business_object_types

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_business_object_properties")
    def get_business_object_properties(self, bo_type_name: str) -> dict | None:
        """Get a dictionary with all property mapping of a business object type.

        We contruct this dictionary from the two lists for the given
        business object types (property mapping and property group mappings)
        These two lists have been created before by process_business_object_types()

        This method is used for creation of business objects in Salesforce.

        Args:
            bo_type_name (str):
                The name of the business object type

        Returns:
            dict | None:
                A dictionary with keys that are either the attribute name or
                a key that is contructed like this: set name + "-" + attribute name.
                This allows for an easy lookup in methods that have access to
                the category data of business workspaces.

        """

        if not self._business_object_types:
            self.logger.warning(
                "List of business object types is empty / not initialized! Cannot lookup type -> '%s'!",
                bo_type_name,
            )
            return None

        # Find the matching business object type:
        business_object_type = next(
            (item for item in self._business_object_types if item["name"] == bo_type_name),
            None,
        )
        if not business_object_type:
            self.logger.warning(
                "Cannot find business object type -> '%s'!",
                bo_type_name,
            )
            return None

        business_properties = business_object_type.get("business_properties")
        business_property_groups = business_object_type.get("business_property_groups")

        lookup_dict = {}

        # 25.3 uses a new REST API to retreive the business object type details.
        # Pre 25.3 we use extractions from the transport package to get this information
        # See process_business_object_type() method.
        otcs_version = float(self._otcs.get_server_version())
        if otcs_version < 25.3:
            att_key = "@attributeName"
            set_key = "@setName"
        else:
            att_key = "att_name"
            set_key = "set_name"

        for mapping in business_properties:
            attribute_name = mapping.get(att_key)
            lookup_dict[attribute_name] = mapping

        for mapping in business_property_groups:
            set_name = mapping.get(set_key)
            attribute_name = mapping.get(att_key)
            lookup_dict[set_name + "-" + attribute_name] = mapping

        return lookup_dict

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_types")
    def process_workspace_types(self, section_name: str = "workspaceTypes") -> list:
        """Create a data structure for all workspace types in the OTCS.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            list:
                A list of workspace types. Each list element is a dict with these values:
                - id (str)
                - name (str)
                - templates (list)
                    + name (str)
                    + id

        """

        # If this payload section has been processed successfully before we
        # still need to read the data structure from the status file and
        # initialize self._workspace_types:
        if self.check_status_file(payload_section_name=section_name):
            # read the list from the json file in admin Home
            # this is important for restart of customizer pod
            # as this data structure is used later on for workspace processing
            self.logger.info(
                "Re-Initialize workspace types list from status file -> '%s' for later use...",
                self.get_status_file_name(payload_section_name=section_name),
            )
            self._workspace_types = self.get_status_file(
                payload_section_name=section_name,
            )
            if self._workspace_types:
                self.logger.info(
                    "Found -> %s workspace types.",
                    str(len(self._workspace_types)),
                )
                self.logger.debug("Workspace types -> %s", str(self._workspace_types))
                return self._workspace_types
            else:
                self.logger.error(
                    "Couldn't read workspace types from status file -> '%s'. Regenerate list...",
                    self.get_status_file_name(payload_section_name=section_name),
                )

        # Read payload_section "workspaceTypes" if available
        payload_section = {}
        for section in self._payload_sections:
            if section["name"] == "workspaceTypes":
                payload_section = section

        # get all workspace types (these have been created by the transports and are not in the payload!)
        # we need to do this each time as it needs to work across multiple payload files...
        response = self._otcs.get_workspace_types()
        if response is None:
            self.logger.error("No workspace types found!")
            self._workspace_types = []
        else:
            self._workspace_types = response["results"]
            self.logger.info(
                "Found -> %s workspace types.",
                str(len(self._workspace_types)),
            )
            self.logger.debug("Workspace types -> %s", str(self._workspace_types))

        # now we enrich the workspace_type list elments (which are dicts)
        # with additional dict elements for further processing:
        for workspace_type in self._workspace_types:
            workspace_type_id = workspace_type["data"]["properties"]["wksp_type_id"]
            self.logger.debug("Workspace Type ID -> %s", workspace_type_id)
            workspace_type["id"] = workspace_type_id
            workspace_type_name = workspace_type["data"]["properties"]["wksp_type_name"]
            workspace_type["name"] = workspace_type_name
            workspace_templates = workspace_type["data"]["properties"]["templates"]
            if not workspace_templates:
                self.logger.warning(
                    "Workspace type -> '%s' has no workspace templates to process! Skipping...",
                    workspace_type_name,
                )
                continue
            self.logger.info(
                "Workspace type -> '%s' has %d template%s to process...",
                workspace_type_name,
                len(workspace_templates),
                "s" if len(workspace_templates) > 1 else "",
            )

            # Create empty lists of dicts with template names and node IDs:
            workspace_type["templates"] = []
            # Determine available templates per workspace type (there can be multiple!)
            # and record them in a simplified data structure:
            for workspace_template in workspace_templates:
                workspace_template_id = workspace_template["id"]
                workspace_template_name = workspace_template["name"]
                self.logger.info(
                    "Found workspace template -> '%s' (%s).",
                    workspace_template_name,
                    workspace_template_id,
                )
                template = {
                    "name": workspace_template_name,
                    "id": workspace_template_id,
                }
                workspace_type["templates"].append(template)

                if payload_section.get("inherit_permissions", False):
                    # TODO: Workaround for problem with workspace role inheritance
                    # which may be related to Transport or REST API: to work-around this we
                    # push down the workspace roles to the workspace folders explicitly:
                    response = self._otcs.get_workspace_roles(
                        workspace_id=workspace_template_id,
                    )
                    for roles in response["results"]:
                        role_name = roles["data"]["properties"]["name"]
                        role_id = roles["data"]["properties"]["id"]
                        permissions = roles["data"]["properties"]["perms"]
                        # as get_workspace_roles() delivers permissions as a value (bit encoded)
                        # we need to convert it to a permissions string list:
                        permission_string_list = self._otcs.convert_permission_value_to_permission_string(
                            permission_value=permissions,
                        )

                        self.logger.info(
                            "Inherit permissions of workspace template -> '%s' (%s) and role -> '%s' (%s) to workspace folders...",
                            workspace_template_name,
                            workspace_template_id,
                            role_name,
                            role_id,
                        )

                        # Inherit permissions to folders of workspace template:
                        response = self._otcs.assign_workspace_permissions(
                            workspace_id=workspace_template_id,
                            role_id=role_id,
                            permissions=permission_string_list,
                            apply_to=1,  # 1 = only sub items - workspace node itself is OK
                        )
                    # end for roles in response["results"]:
                # end if payload_section.get("inherit_permissions", False):
            # end for workspace_template in workspace_templates:
        # end for workspace_type in self._workspace_types:

        self.write_status_file(success=True, payload_section_name=section_name, payload_section=self._workspace_types)

        return self._workspace_types

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_templates")
    def process_workspace_templates(
        self,
        section_name: str = "workspaceTemplates",
    ) -> bool:
        """Process workspace template playload.

        This allows to define role members on template basis.
        This avoids having to "pollute" workspace templates
        with user or group information and instead controls this via payload.

        This method also allows to assign (additional) categories to
        a workspace template controlled by payload.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for workspace_template in self._workspace_templates:
            # Read Workspace Type Name from payload:
            type_name = workspace_template.get("type_name")
            if not type_name:
                self.logger.error(
                    "Workspace template needs a type name! Skipping to next workspace template...",
                )
                success = False
                continue

            # Check if workspace template has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not workspace_template.get("enabled", True):
                self.logger.info(
                    "Payload for workspace template -> '%s' is disabled. Skipping to next workspace template...",
                    type_name,
                )
                continue

            # Read Workspace Template Name from payload:
            template_name = workspace_template.get("template_name")
            if not template_name:
                self.logger.error(
                    "Workspace template for workspace type -> '%s' needs a template name! Skipping...",
                    type_name,
                )
                success = False
                continue

            # Read members from payload:
            members = workspace_template.get("members", [])

            # Read categories from payload:
            categories = workspace_template.get("categories", [])

            # Find the workspace type with the name given in the _workspace_types
            # datastructure that has been generated by process_workspace_types() method before:
            workspace_type = next(
                (item for item in self._workspace_types if item["name"] == type_name),
                None,
            )
            if workspace_type is None:
                self.logger.error(
                    "Workspace type -> '%s' not found. Skipping...",
                    type_name,
                )
                success = False
                continue
            if workspace_type["templates"] == []:
                self.logger.error(
                    "Workspace type -> '%s' does not have templates. Skipping...",
                    type_name,
                )
                success = False
                continue

            workspace_template = next(
                (item for item in workspace_type["templates"] if item["name"] == template_name),
                None,
            )
            if workspace_template:  # does this template exist?
                self.logger.info(
                    "Workspace template -> '%s' has been specified in payload and it does exist.",
                    template_name,
                )
            else:
                self.logger.error(
                    "Workspace template -> '%s' has been specified in payload but it doesn't exist!",
                    template_name,
                )
                self.logger.error(
                    "Workspace type -> '%s' has only these templates -> %s",
                    type_name,
                    workspace_type["templates"],
                )
                success = False
                continue

            template_id = workspace_template["id"]

            workspace_roles = self._otcs.get_workspace_roles(workspace_id=template_id)
            if workspace_roles is None:
                self.logger.info(
                    "Workspace template -> '%s' with node Id -> %s has no roles. Skipping to next workspace...",
                    template_name,
                    template_id,
                )
                continue

            for member in members:
                # read user list, group list, and role name from payload:
                member_users = member.get("users", [])
                member_groups = member.get("groups", [])
                member_role_name = member.get("role", "")

                if not member_role_name:  # role name is required
                    self.logger.error(
                        "Members of workspace template -> '%s' is missing the role name.",
                        template_name,
                    )
                    success = False
                    continue
                if (
                    member_users == [] and member_groups == []
                ):  # we either need users or groups (or both) in the payload
                    self.logger.debug(
                        "Payload for workspace template -> '%s' and role -> '%s' does not have any members (no users nor groups).",
                        template_name,
                        member_role_name,
                    )
                    continue

                role_id = self._otcs.lookup_result_value(
                    response=workspace_roles,
                    key="name",
                    value=member_role_name,
                    return_key="id",
                )
                if role_id is None:
                    #    if member_role is None:
                    self.logger.error(
                        "Workspace template -> '%s' does not have a role -> '%s'",
                        template_name,
                        member_role_name,
                    )
                    success = False
                    continue

                # Process users as workspace template members:
                for member_user in member_users:
                    # find member user in current payload:
                    member_user_id = next(
                        (item for item in self._users if item["name"] == member_user),
                        {"name": member_user},  # we construct a payload on the fly to make determine_user_id() work
                    )
                    user_id = self.determine_user_id(user=member_user_id) if member_user_id else None
                    if not user_id:
                        self.logger.error(
                            "Cannot find member user with login -> '%s'. Skipping...",
                            member_user,
                        )
                        success = False
                        continue

                    # Add member if it does not yet exists - suppress warning
                    # message if user is already in role:
                    response = self._otcs.add_workspace_member(
                        workspace_id=template_id,
                        role_id=int(role_id),
                        member_id=user_id,
                        show_warning=False,
                    )
                    if response is None:
                        self.logger.error(
                            "Failed to add user -> '%s' (%s) as member to role -> '%s' (%s) of workspace template -> '%s' (%s)!",
                            member_user,
                            user_id,
                            member_role_name,
                            role_id,
                            template_name,
                            template_id,
                        )
                        success = False
                    else:
                        self.logger.info(
                            "Successfully added user -> '%s' (%s) as member to role -> '%s' (%s) of workspace template -> '%s' (%s).",
                            member_user,
                            user_id,
                            member_role_name,
                            role_id,
                            template_name,
                            template_id,
                        )
                # end for member_user in member_users:

                # Process groups as workspace template members:
                for member_group in member_groups:
                    member_group_id = next(
                        (item for item in self._groups if item["name"] == member_group),
                        None,
                    )

                    group_id = self.determine_group_id(group=member_group_id) if member_group_id else None
                    if not group_id:
                        self.logger.error(
                            "Cannot find member group -> '%s'. Skipping...",
                            member_group,
                        )
                        success = False
                        continue

                    response = self._otcs.add_workspace_member(
                        workspace_id=template_id,
                        role_id=int(role_id),
                        member_id=group_id,
                        show_warning=False,
                    )
                    if response is None:
                        self.logger.error(
                            "Failed to add group -> '%s' (%s) as member to role -> '%s' (%s) of workspace template -> '%s' (%s)!",
                            member_group_id["name"],
                            group_id,
                            member_role_name,
                            role_id,
                            template_name,
                            template_id,
                        )
                        success = False
                    else:
                        self.logger.info(
                            "Successfully added group -> '%s' (%s) as member to role -> '%s' (%s) of workspace template -> '%s' (%s).",
                            member_group_id["name"],
                            group_id,
                            member_role_name,
                            role_id,
                            template_name,
                            template_id,
                        )
                # end for member_group in member_groups:
            # end for member in members:

            existing_template_categories = None
            for category in categories:
                category_path = category.get("path")
                category_nickname = category.get("nickname")
                inheritance = category.get("inheritance")
                apply_to_sub_items = category.get("apply_to_sub_items")
                category_id = None

                if not category_nickname and not category_path:
                    self.logger.error(
                        "Category assignment to workspace template needs eith the category nickname or the category path in the category volume!",
                    )
                    success = False
                    continue
                if category_path:
                    response = self._otcs.get_node_by_volume_and_path(
                        volume_type=self._otcs.VOLUME_TYPE_CATEGORIES_VOLUME,
                        path=category_path,
                    )
                    category_id = self._otcs.get_result_value(response=response, key="id")
                elif category_nickname:
                    response = self._otcs.get_node_from_nickname(nickname=category_nickname, show_error=False)
                    category_id = self._otcs.get_result_value(response=response, key="id")
                if not category_id:
                    self.logger.error(
                        "Cannot find category for workspace template -> '%s' (%s). Tried category %s.",
                        template_name,
                        template_id,
                        "path {}".format(category_path) if category_path else "nickname {}".format(category_nickname),
                    )
                    success = False
                    continue
                if existing_template_categories is None:
                    existing_template_categories = self._otcs.get_node_category_ids(node_id=template_id)
                if category_id not in existing_template_categories:
                    result = self._otcs.assign_category(
                        node_id=template_id,
                        category_id=category_id,
                        inheritance=inheritance,
                        apply_to_sub_items=apply_to_sub_items,
                    )
                    if not result:
                        self.logger.error(
                            "Cannot assign category with ID -> %s to workspace template -> '%s' (%s)!",
                            category_id,
                            template_name,
                            template_id,
                        )
                        success = False
                        continue
                    self.logger.info(
                        "Successfully assigned category with ID -> %s to workspace template -> '%s' (%s).",
                        category_id,
                        template_name,
                        template_id,
                    )
                    # Write ID back into the payload dictionary:
                    category["id"] = category_id
                else:
                    self.logger.info(
                        "Category with ID -> %s is already assigned to workspace template -> '%s' (%s).",
                        category_id,
                        template_name,
                        template_id,
                    )

            # end for category in categories:
        # end for workspace_template in self._workspace_templates:

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._workspace_templates,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="resolve_attribute_values")
    def resolve_attribute_values(
        self,
        attribute_name: str,
        attribute_type: str,
        attribute_values: str | list[str],
    ) -> str | list[str]:
        """Resolve user logins or group names to user / group IDs.

        This method handles the "special" cases where the actual
        attribute values need to be IDs but the payload has names.

        This method is called for all attributes. If no special handling
        is required it just return an unmodified attribute value.

        Args:
            attribute_name (str):
                The name of the attribute.
            attribute_type (str):
                The type of the attribute.
            attribute_values (str | list[str]):
                The value of the attribute. A single value or a list of values.

        Returns:
            str | list[str]:
                A resolved attribute value or a list of resolved attribute values.

        """

        # Special Case 1: handle special case where attribute type is a user picker.
        # we expect that the payload includes the login name for this
        # (as user IDs are not stable across systems) but then we need
        # to lookup the real user ID here:
        if attribute_type in ("otcs_user_picker", "otcs_member_picker"):
            self.logger.debug(
                "Attribute -> '%s' is is of type -> 'User Picker' (%s). Looking up user ID for user login name -> %s",
                attribute_name,
                attribute_type,
                attribute_values,
            )

            # Do we have a single value (not a list)?
            if not isinstance(attribute_values, list):
                user = self._otcs_frontend.get_user(name=attribute_values)
                user_id = self._otcs_frontend.lookup_result_value(
                    response=user,
                    key="name",
                    value=attribute_values,
                    return_key="id",
                )
                if user_id:
                    # User has been found - determine ID:
                    self.logger.debug(
                        "User -> '%s' has ID -> %s",
                        attribute_values,
                        user_id,
                    )
                    # Put the user ID into data structure
                    return str(user_id)
                else:
                    self.logger.error(
                        "User with login name -> '%s' does not exist!",
                        attribute_values,
                    )
                    # Clear the value to avoid workspace create failure
                    return ""
            # Multi-value user attribute:
            else:
                user_ids = []
                for value in attribute_values:
                    user = self._otcs_frontend.get_user(name=value)
                    user_id = self._otcs_frontend.lookup_result_value(
                        response=user,
                        key="name",
                        value=value,
                        return_key="id",
                    )
                    if user_id:
                        # User has been found - determine ID:
                        self.logger.debug(
                            "User -> '%s' has ID -> %s",
                            value,
                            user_id,
                        )
                        # Put the user ID into the result list:
                        user_ids.append(str(user_id))
                    else:
                        self.logger.error(
                            "User with login name -> '%s' does not exist!",
                            value,
                        )
                return user_ids

        # Special Case 2: handle Extended ECM for Government attribute type "Organizational Unit" (OU).
        # This is referring to a group ID which is not stable across deployments. So we need to lookup
        # the Group ID and add it to the data structure. This expects that the payload has the
        # group name and not the group ID:
        elif attribute_type == str(11480):
            self.logger.debug(
                "Attribute -> '%s' is is of type -> 'Organizational Unit' (%s). Looking up group ID for group name -> %s",
                attribute_name,
                attribute_type,
                attribute_values,
            )
            # Do we have a single value (not a list)?
            if not isinstance(attribute_values, list):
                group = self._otcs_frontend.get_group(
                    name=attribute_values,
                )
                group_id = self._otcs_frontend.lookup_result_value(
                    response=group,
                    key="name",
                    value=attribute_values,
                    return_key="id",
                )

                if group_id:
                    self.logger.debug(
                        "Group for Organizational Unit -> '%s' has ID -> %s",
                        attribute_values,
                        group_id,
                    )
                    # Put the group ID as a string:
                    return str(group_id)
                else:
                    self.logger.error(
                        "Group for Organizational Unit -> '%s' does not exist!",
                        attribute_values,
                    )
                    # Return an empty value string:
                    return ""
            # Multi-value org group attribute:
            else:
                group_ids = []
                for value in attribute_values:
                    group = self._otcs_frontend.get_group(
                        name=value,
                    )
                    group_id = self._otcs_frontend.lookup_result_value(
                        response=group,
                        key="name",
                        value=value,
                        return_key="id",
                    )

                    if group_id:
                        self.logger.debug(
                            "Group for Organizational Unit -> '%s' has ID -> %s",
                            value,
                            group_id,
                        )
                        # Put the group ID into the result list:
                        group_ids.append(str(group_id))
                    else:
                        self.logger.error(
                            "Group for Organizational Unit -> '%s' does not exist!",
                            value,
                        )
                return group_ids

        # This is the default case - we return the unchanged attribute values:
        return attribute_values

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="prepare_workspace_create_form")
    def prepare_workspace_create_form(
        self,
        categories: list,
        template_id: int,
        ext_system_id: str | None = None,
        bo_type: str | None = None,
        bo_id: str | None = None,
        parent_workspace_node_id: int | None = None,
    ) -> dict | None:
        """Prepare the category structure for the workspace creation.

        Args:
            categories (list):
                The categories list from the workspace payload.
            template_id (int):
                The workspace template ID.
            ext_system_id (str, optional):
                The optional external system ID.
            bo_type (str, optional):
                The optional business object type ID.
            bo_id (str, optional):
                the business object ID.
            parent_workspace_node_id (int, optional):
                The optional parent Workspace ID.

        Returns:
            dict | None:
                Category structure for workspace creation or None
                in case of an error.

        """

        category_create_data = {}

        response = self._otcs_frontend.get_workspace_create_form(
            template_id=template_id,
            external_system_id=ext_system_id,
            bo_type=bo_type,
            bo_id=bo_id,
            parent_id=parent_workspace_node_id,
        )
        if response is None or "forms" not in response:
            self.logger.error(
                "Failed to retrieve create information for template ID -> %s!",
                template_id,
            )
            return None

        self.logger.debug(
            "Successfully retrieved create information for template ID -> %d!",
            template_id,
        )

        # Process category information
        forms = response["forms"]

        categories_form = {}

        # Typically the the create workspace form delivers 4-5 forms:
        # 1. Form for System Attributes (has no role name)
        # 2. Form for Category Data (role name = "categories")
        # 3. Form for Classifications (role name = "classifications")
        # 4. Form for importing team members from another existing team
        # 5. Form for Microsoft Teams integration (role name = "microsoftteams")
        for form in forms:
            if "role_name" not in form:
                self.logger.debug("Found 'system attributes' form -> %s", str(form))
                continue
            match form["role_name"]:
                case "categories":
                    categories_form = form
                    self.logger.debug("Found 'categories' form -> %s", str(form))
                    continue
                case "classifications":
                    self.logger.debug("Found 'classifications' form -> %s", str(form))
                    continue
                case "rmclassifications":
                    self.logger.debug("Found 'RM classifications' form -> %s", str(form))
                    continue
                case "importteam":
                    self.logger.debug("Found 'import team participants' form -> %s", str(form))
                    continue
                case "microsoftteams":
                    self.logger.debug("Found 'microsoft teams' form -> %s", str(form))
                    continue
                case _:
                    # the remaining option is that this form is the system attributes form:
                    self.logger.warning("Unknown form -> %s", str(form))
            # end match form["role_name"]:
        # end for form in forms:

        # We are just interested in the single category data set (role_name = "categories"):
        data = categories_form.get("data", None)

        if not data:
            self.logger.debug("No categories data found.")
            return category_create_data

        self.logger.debug("Categories data found -> %s", data)
        schema = categories_form["schema"]["properties"]
        self.logger.debug("Categories schema found -> %s", schema)
        # Loop over categories:
        for cat_id in data:  # these are the category IDs (dict keys):
            self.logger.debug("Category ID -> %s", cat_id)
            data_attributes = data[cat_id]
            self.logger.debug("Data attributes -> %s", data_attributes)
            schema_attributes = schema[cat_id]["properties"]
            self.logger.debug("Schema attributes -> %s", schema_attributes)
            cat_name = schema[cat_id]["title"]
            self.logger.debug("Category name -> %s", cat_name)
            # Loop over attributes:
            # * Sets with one (fixed) row have type = object
            # * Multi-value Sets with (multiple) rows have type = array and "properties" in "items" schema
            # * Multi-value attributes have also type = array but NO "properties" in "items" schema
            for attr_id in data_attributes:  # these a attribute IDs (dict keys)
                self.logger.debug("Attribute ID -> %s", attr_id)
                self.logger.debug("Attribute data -> %s", data_attributes[attr_id])
                self.logger.debug(
                    "Attribute schema -> %s",
                    str(schema_attributes[attr_id]),
                )
                attr_type = schema_attributes[attr_id]["type"]
                self.logger.debug("Attribute type -> %s", attr_type)
                if "title" not in schema_attributes[attr_id]:
                    self.logger.debug("Attribute has no title in schema. Skipping...")
                    continue

                #
                # Handle multi-line set:
                #
                if attr_type == "array" and ("properties" in schema_attributes[attr_id]["items"]):
                    set_name = schema_attributes[attr_id]["title"]
                    self.logger.debug("Multi-line set -> '%s'", set_name)
                    set_data_attributes = data_attributes[attr_id]  # this is a list []
                    if set_data_attributes is None:
                        self.logger.debug("Attribute has no value in data. Skipping...")
                        continue
                    self.logger.debug("Set data attributes -> %s", set_data_attributes)
                    set_schema_attributes = schema_attributes[attr_id]["items"]["properties"]
                    self.logger.debug(
                        "Set schema attributes -> %s",
                        set_schema_attributes,
                    )
                    set_schema_max_rows = schema_attributes[attr_id]["items"]["maxItems"]
                    self.logger.debug("Set schema max rows -> %s", set_schema_max_rows)
                    set_data_max_rows = len(set_data_attributes)
                    self.logger.debug("Set data max rows -> %s", set_data_max_rows)
                    row = 1
                    # it can happen that the payload contains more rows than the
                    # initial rows in the set data structure. In this case we use
                    # a copy of the data structure from row 0 as template...
                    first_row = dict(set_data_attributes[0])
                    # We don't know upfront how many rows of data we will find in payload
                    # but we at max process the maxItems specified in the schema:
                    while row <= set_schema_max_rows:
                        # Test if we have any payload for this row:
                        attribute = next(
                            (
                                item
                                for item in categories
                                if (
                                    item["name"] == cat_name
                                    and "set" in item  # not all items may have a "set" key
                                    and item["set"] == set_name
                                    and "row" in item  # not all items may have a "row" key
                                    and item["row"] == row
                                )
                            ),
                            None,
                        )
                        # stop if there's no payload for the row:
                        if attribute is None:
                            self.logger.debug(
                                "No payload found for set -> %s, row -> %s",
                                set_name,
                                row,
                            )
                            # we assume that if there's no payload for row n there will be no payload for rows > n
                            # and break the while loop:
                            break
                        # do we need to create a new row in the data frame?
                        if row > set_data_max_rows:
                            # use the row we stored above to create a new empty row:
                            self.logger.debug(
                                "Found payload for row -> %s, we need a new data row for it",
                                row,
                            )
                            self.logger.debug(
                                "Adding an additional row -> %s to set data -> '%s'",
                                row,
                                set_name,
                            )
                            # add the empty dict to the list:
                            set_data_attributes.append(dict(first_row))
                            set_data_max_rows += 1
                        else:
                            self.logger.debug(
                                "Found payload for row -> %s %s we can store in existing data row",
                                row,
                                set_name,
                            )
                        # traverse all attributes in a single set row:
                        for set_attr_id in set_schema_attributes:
                            self.logger.debug(
                                "Set attribute ID -> %s (row -> %s)",
                                set_attr_id,
                                row,
                            )
                            self.logger.debug(
                                "Set attribute schema -> %s (row -> %s)",
                                set_schema_attributes[set_attr_id],
                                row,
                            )
                            set_attr_type = set_schema_attributes[set_attr_id]["type"]
                            self.logger.debug(
                                "Set attribute type -> %s (row -> %s)",
                                set_attr_type,
                                row,
                            )
                            set_attr_name = set_schema_attributes[set_attr_id]["title"]
                            self.logger.debug(
                                "Set attribute name -> %s (row -> %s)",
                                set_attr_name,
                                row,
                            )
                            # Lookup the attribute with the right category, set, attribute name, and row number in payload:
                            attribute = next(
                                (
                                    item
                                    for item in categories
                                    if (
                                        item["name"] == cat_name
                                        and "set" in item  # not all items may have a "set" key
                                        and item["set"] == set_name
                                        and item["attribute"] == set_attr_name
                                        and "row" in item  # not all items may have a "row" key
                                        and item["row"] == row
                                    )
                                ),
                                None,
                            )
                            if attribute is None:
                                self.logger.debug(
                                    "Set -> '%s', attribute -> '%s', row -> %s not found in payload.",
                                    set_name,
                                    set_attr_name,
                                    row,
                                )

                                # need to use row - 1 as index starts with 0 but payload rows start with 1
                                set_data_attributes[row - 1][set_attr_id] = ""
                            else:
                                if set_attr_type == "array" and "items" in set_schema_attributes[set_attr_id]:
                                    set_attr_type = set_schema_attributes[set_attr_id]["items"].get(
                                        "type",
                                        set_attr_type,
                                    )

                                self.logger.debug(
                                    "Set -> '%s', attribute -> '%s', row -> %s found in payload, value -> '%s'",
                                    set_name,
                                    set_attr_name,
                                    row,
                                    attribute["value"],
                                )
                                set_data_attributes[row - 1][set_attr_id] = self.resolve_attribute_values(
                                    attribute_name=set_attr_name,
                                    attribute_type=set_attr_type,
                                    attribute_values=attribute["value"],
                                )
                        row += 1  # continue the while loop with the next row
                    # end while row <= set_schema_max_rows:
                # end if attr_type == "array" and ("properties" in schema_attributes[attr_id]["items"]):

                #
                # Handle single-line set:
                #
                elif attr_type == "object":
                    set_name = schema_attributes[attr_id]["title"]
                    self.logger.debug("Single-line set -> '%s'", set_name)
                    set_data_attributes = data_attributes[attr_id]
                    self.logger.debug("Set data attributes -> %s", set_data_attributes)

                    set_schema_attributes = schema_attributes[attr_id]["properties"]
                    self.logger.debug(
                        "Set schema attributes -> %s",
                        str(set_schema_attributes),
                    )
                    # Loop over set attributes:
                    for set_attr_id in set_data_attributes:
                        self.logger.debug("Set attribute ID -> %s", set_attr_id)
                        self.logger.debug(
                            "Set attribute data -> %s",
                            str(set_data_attributes[set_attr_id]),
                        )
                        self.logger.debug(
                            "Set attribute schema -> %s",
                            str(set_schema_attributes[set_attr_id]),
                        )
                        set_attr_type = set_schema_attributes[set_attr_id]["type"]
                        self.logger.debug("Set attribute type -> %s", set_attr_type)
                        set_attr_name = set_schema_attributes[set_attr_id]["title"]
                        self.logger.debug("Set attribute name -> '%s'", set_attr_name)
                        # Lookup the attribute with the right category, set and attribute name in payload:
                        attribute = next(
                            (
                                item
                                for item in categories
                                if (
                                    item["name"] == cat_name
                                    and "set" in item  # not all items may have a "set" key
                                    and item["set"] == set_name
                                    and item["attribute"] == set_attr_name
                                )
                            ),
                            None,
                        )
                        if attribute is None:
                            self.logger.debug(
                                "Category -> '%s', set -> %s, attribute -> '%s' not found in payload.",
                                cat_name,
                                set_name,
                                set_attr_name,
                            )
                            set_data_attributes[set_attr_id] = ""
                        else:
                            self.logger.debug(
                                "Category -> '%s', set -> %s, attribute -> '%s' found in payload, value -> %s",
                                cat_name,
                                set_name,
                                set_attr_name,
                                attribute["value"],
                            )
                            # Resolve any special cases (e.g. user picker, group picker):
                            set_data_attributes[set_attr_id] = self.resolve_attribute_values(
                                attribute_name=set_attr_name,
                                attribute_type=set_attr_type,
                                attribute_values=attribute["value"],
                            )
                    # end for set_attr_id in set_data_attributes:
                # end elif attr_type == "object":

                #
                # Handle plain attribute (not inside a set):
                #
                else:
                    attr_name = schema_attributes[attr_id]["title"]
                    self.logger.debug("Attribute name -> '%s'", attr_name)
                    # Lookup the attribute with the right category and attribute name in payload:
                    attribute = next(
                        (item for item in categories if (item["name"] == cat_name and item["attribute"] == attr_name)),
                        None,
                    )

                    if attr_type == "array" and "items" in schema_attributes[attr_id]:
                        attr_type = schema_attributes[attr_id]["items"].get("type", attr_type)

                    if attribute is None:
                        self.logger.debug(
                            "Category -> '%s', attribute -> '%s' not found in payload.",
                            cat_name,
                            attr_name,
                        )
                        data_attributes[attr_id] = ""
                    else:
                        self.logger.debug(
                            "Category -> '%s', attribute -> '%s' found in payload, value -> %s",
                            cat_name,
                            attr_name,
                            attribute["value"],
                        )
                        # Resolve any special cases (e.g. user picker, group picker):
                        data_attributes[attr_id] = self.resolve_attribute_values(
                            attribute_name=attr_name,
                            attribute_type=attr_type,
                            attribute_values=attribute["value"],
                        )
                # end else (plain attribute)
            # end for attr_data, attr_schema
            category_create_data[cat_id] = data_attributes
        # end for cat_data, cat_schema

        self.logger.debug("Category create data -> %s", category_create_data)

        return category_create_data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_salesforce_business_object")
    def get_salesforce_business_object(
        self,
        workspace: dict,
        object_type: str,
        search_field: str,
        search_value: str,
    ) -> str | None:
        """Get the Salesforce ID (str) of an Salesforce object by querying the Salesforce API.

        Args:
            workspace (dict):
                The workspace payload element.
            object_type (str):
                The business object type.
            search_field (str):
                Search field to find business object in external system.
            search_value (str):
                Search value to find business object in external system.

        Returns:
            str | None:
                The technical ID of the business object in Salesforce.

        """

        if not self._salesforce:
            self.logger.error(
                "Salesforce connection not initialized! Cannot connect to Salesforce API!",
            )
            return None

        self.logger.debug(
            "Workspaces is connected to Salesforce and we need to lookup the business object ID...",
        )
        salesforce_token = self._salesforce.authenticate()
        if not salesforce_token:
            self.logger.error("Failed to authenticate with Salesforce!")
            return None

        response = self._salesforce.get_object(
            object_type=object_type,
            search_field=search_field,
            search_value=search_value,
            result_fields=["Id"],
        )
        bo_id = self._salesforce.get_result_value(response=response, key="Id")
        num_of_bos = int(response.get("totalSize", 0)) if (response is not None and "totalSize" in response) else 0
        if num_of_bos > 1:
            self.logger.warning(
                "Salesforce lookup delivered %s business objects of type -> '%s'! Picking the first one with ID -> %s...",
                str(num_of_bos),
                str(object_type),
                bo_id,
            )
        if not bo_id:
            self.logger.warning(
                "Business object of type -> '%s' with '%s' = '%s' does not exist in Salesforce!",
                object_type,
                search_field,
                search_value,
            )
            self.logger.info(
                "We try to create the Salesforce object of type -> '%s'...",
                object_type,
            )

            # Get a helper dict to quickly lookup Salesforce properties
            # for given set + attribute name:
            property_lookup = self.get_business_object_properties(
                bo_type_name=object_type,
            )
            # In case we couldn't find properties for the given Business Object Type
            # we bail out...
            if not property_lookup:
                self.logger.warning(
                    "Cannot create Salesforce object - no business object properties found!",
                )
                return None

            categories = workspace.get("categories", [])
            parameter_dict = {}
            # We process all category entries in workspace payload
            # and see if we have a matching mapping to a business property
            # in the BO Type definition:
            for category in categories:
                # generate the lookup key:
                key = ""
                if "set" in category:
                    key += category["set"] + "-"
                key += category.get("attribute")
                # get the attribute value:
                value = category.get("value")
                # lookup the mapping
                mapping = property_lookup.get(key, None)
                # Check if we have a mapping:
                if mapping:
                    property_name = mapping.get("@propertyName", None)
                    self.logger.debug(
                        "Found business property -> '%s' for attribute -> '%s'",
                        property_name,
                        category.get("attribute"),
                    )
                    parameter_dict[property_name] = value
                else:
                    self.logger.debug(
                        "Attribute -> '%s' (key -> %s) does not have a mapped business property.",
                        category.get("attribute"),
                        key,
                    )

            if not parameter_dict:
                self.logger.warning(
                    "Cannot create Salesforce object of type -> '%s' - no parameters found!",
                    object_type,
                )
                return None

            self.logger.info(
                "Create Salesforce object of type -> '%s' with parameters -> %s...",
                object_type,
                str(parameter_dict),
            )
            #
            # Now we try to create the Salesforce object
            #
            response = self._salesforce.add_object(
                object_type=object_type,
                **parameter_dict,
            )
            bo_id = self._salesforce.get_result_value(response=response, key="id")
            if bo_id:
                self.logger.info(
                    "Successfully created Salesforce business object with ID -> %s of type -> '%s'.",
                    bo_id,
                    object_type,
                )
            else:
                self.logger.error(
                    "Failed to create Salesforce business object of type -> '%s'!",
                    object_type,
                )
        else:  # BO found
            self.logger.debug(
                "Retrieved ID -> %s for Salesforce object type -> '%s' (looking up -> '%s' in field -> '%s')",
                bo_id,
                object_type,
                search_value,
                search_field,
            )

        return bo_id

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_guidewire_business_object")
    def get_guidewire_business_object(
        self,
        external_system: dict,
        object_type: str,
        search_field: str,
        search_value: str,
    ) -> str | None:
        """Get the Guidewire ID (str) of an Guidewire object by querying the Guidewire API.

        Args:
            external_system (dict):
                Payload of the external System. Required if Guidewire python object
                needs to be re-initialized.
            object_type (str):
                The business object type (like "Claim", "Policy", "Account").
            search_field (str):
                Search field to find business object in external system.
            search_value (str):
                Search value to find business object in external system.

        Returns:
            str | None:
                The technical ID of the business object in Guidewire. None in case of an error.

        """

        if object_type in ["Account", "Policy", "account", "policy", "gw.account", "gw.policy"]:
            if not self._guidewire_policy_center:
                self._guidewire_policy_center = self.init_guidewire(guidewire_external_system=external_system)
            guidewire_object = self._guidewire_policy_center
        elif object_type in ["Claim", "claim", "gw.claim"]:
            if not self._guidewire_claims_center:
                self._guidewire_claims_center = self.init_guidewire(guidewire_external_system=external_system)
            guidewire_object = self._guidewire_claims_center
        else:
            self.logger.error("Not supported Guidewire object type -> '%s'", object_type)
            return None

        if not guidewire_object:
            self.logger.error(
                "Guidewire connection not initialized! Cannot connect to Guidewire API!",
            )
            return None

        self.logger.debug(
            "Workspaces is connected to Guidewire and we need to lookup the business object ID...",
        )
        guidewire_token = guidewire_object.authenticate()
        if not guidewire_token:
            self.logger.error("Failed to authenticate with Guidewire!")
            return None

        match object_type:
            case "Account" | "account" | "gw.account":
                response = guidewire_object.search_account(
                    attributes={search_field: search_value},
                )
                bo_id = guidewire_object.get_result_value(response=response, key="id")
            case "Policy" | "policy" | "gw.policy":
                response = guidewire_object.search_policy(
                    attributes={search_field: search_value},
                )
                bo_id = guidewire_object.get_result_value(response=response, key="policyId")
            case _:
                self.logger.warning("Currently we only support lookup of 'Account' and 'Policy' objects in Guidewire!")
                return None

        num_of_bos = int(response.get("count", 0)) if (response is not None and "count" in response) else 0
        if num_of_bos > 1:
            self.logger.warning(
                "Guidewire lookup delivered %s business objects of type -> '%s'. Picking the first one with ID -> %s.",
                str(num_of_bos),
                str(object_type),
                bo_id,
            )
        if not bo_id:
            self.logger.warning(
                "Business object of type -> '%s' with '%s' = '%s' does not exist in Guidewire!",
                object_type,
                search_field,
                search_value,
            )
        else:  # BO found
            self.logger.info(
                "Retrieved ID -> %s for Guidewire object type -> '%s' (looking up -> '%s' in field -> '%s').",
                bo_id,
                object_type,
                search_value,
                search_field,
            )

        return bo_id

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="prepare_item_create_form")
    def prepare_item_create_form(
        self,
        parent_id: int,
        categories: list,
        subtype: int = OTCS.ITEM_TYPE_DOCUMENT,
    ) -> dict | None:
        """Prepare the category structure for the item creation.

        Args:
            parent_id (int):
                The node the category should be applied to.
            categories (list):
                The categories list from the document payload.
            subtype (int):
                The subtype of the new node. Default is document.

        Returns:
            dict | None:
                Category structure for workspace creation or None
                in case of an error.

        """

        category_create_data = {}

        category_ids = []

        # Get unique combination of volume, path, name to get a list of all categories
        unique_categories = {
            (entry.get("volume", OTCS.VOLUME_TYPE_CATEGORIES_VOLUME), entry.get("path", None), entry["name"])
            for entry in categories
        }

        # Determine the names of all categories inherited from the parent_id:
        response = self._otcs_frontend.get_node_categories(node_id=parent_id)
        if response and response["results"]:
            category_ids = [
                next(iter(item["metadata_order"]["categories"])) for item in response["results"] if item.get("metadata")
            ]

        for volume, path, name in unique_categories:
            self.logger.debug("%s : %s", name, path)

            if isinstance(path, str):
                path = [path, name]
            elif isinstance(path, list):
                path.append(name)
            else:
                continue

            response = self._otcs_frontend.get_node_by_volume_and_path(
                volume_type=volume,
                path=path,
            )
            cat_id = self._otcs_frontend.get_result_value(response=response, key="id")
            if cat_id and cat_id not in category_ids:
                category_ids.append(cat_id)
        # end for volume, path, name in unique_categories:

        response = self._otcs_frontend.get_node_create_form(
            parent_id=parent_id,
            subtype=subtype,
            category_ids=category_ids,
        )
        if response is None or "forms" not in response:
            self.logger.error(
                "Failed to retrieve create information for subtype -> %s in parent with ID -> %d!",
                subtype,
                parent_id,
            )
            return None

        self.logger.debug(
            "Successfully retrieved create information for subtype -> %s in parent with ID -> %d.",
            subtype,
            parent_id,
        )

        # Process category information
        forms = response["forms"]

        categories_form = {}

        # Typically the the create item form delivers 4-5 forms:
        # 1. Form for System Attributes (has no role name)
        # 2. Form for Category Data (role name = "categories")
        # 3. Form for Classifications (role name = "classifications")
        # 4. Form for importing team members from another existing team
        # 5. Form for Microsoft Teams integration (role name = "microsoftteams")
        for form in forms:
            if "role_name" not in form:
                self.logger.debug("Found 'system attributes' form -> %s", str(form))
                continue
            match form["role_name"]:
                case "categories":
                    categories_form = form
                    self.logger.debug("Found 'categories' form -> %s", form)
                    continue
                case "classifications":
                    self.logger.debug("Found 'classifications' form -> %s", form)
                    continue
                case "rmclassifications":
                    self.logger.debug("Found 'RM classifications' form -> %s", form)
                    continue
                case "importteam":
                    self.logger.debug("Found 'import team participants' form -> %s", form)
                    continue
                case "microsoftteams":
                    self.logger.debug("Found Import Team participants form -> %s", form)
                    continue
                case _:
                    # the remaining option is that this form is the system attributes form:
                    self.logger.warning("Unknown form -> %s", str(form))
            # end match form["role_name"]:
        # end for form in forms:

        # We are just interested in the single category data set (role_name = "categories"):
        data = categories_form.get("data", None)

        if not data:
            self.logger.debug("No categories data found.")
            return category_create_data

        self.logger.debug("Categories data found -> %s", data)
        schema = categories_form["schema"]["properties"]
        self.logger.debug("Categories schema found -> %s", schema)
        # Loop over categories:
        for cat_id in data:  # these are the category IDs (dict keys):
            self.logger.debug("Category ID -> %s", cat_id)
            data_attributes = data[cat_id]
            self.logger.debug("Data attributes -> %s", data_attributes)
            schema_attributes = schema[cat_id]["properties"]
            self.logger.debug("Schema attributes -> %s", schema_attributes)
            cat_name = schema[cat_id]["title"]
            self.logger.debug("Category name -> %s", cat_name)
            # Loop over attributes:
            # * Sets with one (fixed) row have type = object
            # * Multi-value Sets with (multiple) rows have type = array and "properties" in "items" schema
            # * Multi-value attributes have also type = array but NO "properties" in "items" schema
            for attr_id in data_attributes:  # these a attribute IDs (dict keys)
                self.logger.debug("Attribute ID -> %s", attr_id)
                self.logger.debug("Attribute data -> %s", data_attributes[attr_id])
                self.logger.debug(
                    "Attribute schema -> %s",
                    schema_attributes[attr_id],
                )
                attr_type = schema_attributes[attr_id]["type"]
                self.logger.debug("Attribute type -> %s", attr_type)
                if "title" not in schema_attributes[attr_id]:
                    self.logger.debug("Attribute has no title. Skipping...")
                    continue

                #
                # Handle multi-line set:
                #
                if attr_type == "array" and ("properties" in schema_attributes[attr_id]["items"]):
                    set_name = schema_attributes[attr_id]["title"]
                    self.logger.debug("Multi-line set -> %s", set_name)
                    set_data_attributes = data_attributes[attr_id]  # this is a list []
                    if set_data_attributes is None:
                        self.logger.debug("Attribute has no value in data. Skipping...")
                        continue
                    self.logger.debug("Set data attributes -> %s", set_data_attributes)
                    set_schema_attributes = schema_attributes[attr_id]["items"]["properties"]
                    self.logger.debug(
                        "Set schema attributes -> %s",
                        set_schema_attributes,
                    )
                    set_schema_max_rows = schema_attributes[attr_id]["items"]["maxItems"]
                    self.logger.debug("Set schema max rows -> %s", set_schema_max_rows)
                    set_data_max_rows = len(set_data_attributes)
                    self.logger.debug("Set data xax rows -> %s", set_data_max_rows)
                    row = 1
                    # it can happen that the payload contains more rows than the
                    # initial rows in the set data structure. In this case we use
                    # a copy of the data structure from row 0 as template...
                    first_row = dict(set_data_attributes[0])
                    # We don't know upfront how many rows of data we will find in payload
                    # but we at max process the maxItems specified in the schema:
                    while row <= set_schema_max_rows:
                        # Test if we have any payload for this row:
                        attribute = next(
                            (
                                item
                                for item in categories
                                if (
                                    item["name"] == cat_name
                                    and "set" in item  # not all items may have a "set" key
                                    and item["set"] == set_name
                                    and "row" in item  # not all items may have a "row" key
                                    and item["row"] == row
                                )
                            ),
                            None,
                        )
                        # stop if there's no payload for the row:
                        if attribute is None:
                            self.logger.debug(
                                "No payload found for set -> %s, row -> %s",
                                set_name,
                                row,
                            )
                            # we assume that if there's no payload for row n there will be no payload for rows > n
                            # and break the while loop:
                            break
                        # do we need to create a new row in the data frame?
                        if row > set_data_max_rows:
                            # use the row we stored above to create a new empty row:
                            self.logger.debug(
                                "Found payload for row -> %s, we need a new data row for it",
                                row,
                            )
                            self.logger.debug(
                                "Adding an additional row -> %s to set data -> '%s'",
                                row,
                                set_name,
                            )
                            # add the empty dict to the list:
                            set_data_attributes.append(dict(first_row))
                            set_data_max_rows += 1
                        else:
                            self.logger.debug(
                                "Found payload for row -> %s %s we can store in existing data row",
                                row,
                                set_name,
                            )
                        # traverse all attributes in a single row:
                        for set_attr_id in set_schema_attributes:
                            self.logger.debug(
                                "Set attribute ID -> %s (row -> %s)",
                                set_attr_id,
                                row,
                            )
                            self.logger.debug(
                                "Set attribute schema -> %s (row -> %s)",
                                set_schema_attributes[set_attr_id],
                                row,
                            )
                            set_attr_type = set_schema_attributes[set_attr_id]["type"]
                            self.logger.debug(
                                "Set attribute type -> %s (row -> %s)",
                                set_attr_type,
                                row,
                            )
                            set_attr_name = set_schema_attributes[set_attr_id]["title"]
                            self.logger.debug(
                                "Set attribute name -> %s (row -> %s)",
                                set_attr_name,
                                row,
                            )
                            # Lookup the attribute with the right category, set, attribute name, and row number in payload:
                            attribute = next(
                                (
                                    item
                                    for item in categories
                                    if (
                                        item["name"] == cat_name
                                        and "set" in item  # not all items may have a "set" key
                                        and item["set"] == set_name
                                        and item["attribute"] == set_attr_name
                                        and "row" in item  # not all items may have a "row" key
                                        and item["row"] == row
                                    )
                                ),
                                None,
                            )
                            if attribute is None:
                                self.logger.debug(
                                    "Set -> '%s', attribute -> '%s', row -> %s not found in payload.",
                                    set_name,
                                    set_attr_name,
                                    row,
                                )

                                # need to use row - 1 as index starts with 0 but payload rows start with 1
                                set_data_attributes[row - 1][set_attr_id] = ""
                            else:
                                if set_attr_type == "array" and "items" in set_schema_attributes[set_attr_id]:
                                    set_attr_type = set_schema_attributes[set_attr_id]["items"].get(
                                        "type",
                                        set_attr_type,
                                    )

                                self.logger.debug(
                                    "Set -> '%s', attribute -> '%s', row -> %s found in payload, value -> '%s'",
                                    set_name,
                                    set_attr_name,
                                    row,
                                    attribute["value"],
                                )
                                set_data_attributes[row - 1][set_attr_id] = self.resolve_attribute_values(
                                    attribute_name=set_attr_name,
                                    attribute_type=set_attr_type,
                                    attribute_values=attribute["value"],
                                )
                        row += 1  # continue the while loop with the next row
                    # end while row <= set_schema_max_rows:
                # end if attr_type == "array" and ("properties" in schema_attributes[attr_id]["items"]):

                #
                # Handle single-line set:
                #
                elif attr_type == "object":
                    set_name = schema_attributes[attr_id]["title"]
                    self.logger.debug("Single-line set -> '%s'", set_name)
                    set_data_attributes = data_attributes[attr_id]
                    self.logger.debug("Set data attributes -> %s", set_data_attributes)

                    set_schema_attributes = schema_attributes[attr_id]["properties"]
                    self.logger.debug(
                        "Set schema attributes -> %s",
                        str(set_schema_attributes),
                    )
                    # Loop over set attributes:
                    for set_attr_id in set_data_attributes:
                        self.logger.debug("Set attribute ID -> %s", set_attr_id)
                        self.logger.debug(
                            "Set attribute data -> %s",
                            str(set_data_attributes[set_attr_id]),
                        )
                        self.logger.debug(
                            "Set attribute schema -> %s",
                            str(set_schema_attributes[set_attr_id]),
                        )
                        set_attr_type = set_schema_attributes[set_attr_id]["type"]
                        self.logger.debug("Set attribute type -> %s", set_attr_type)
                        set_attr_name = set_schema_attributes[set_attr_id]["title"]
                        self.logger.debug("Set attribute name -> '%s'", set_attr_name)
                        # Lookup the attribute with the right category, set and attribute name in payload:
                        attribute = next(
                            (
                                item
                                for item in categories
                                if (
                                    item["name"] == cat_name
                                    and "set" in item  # not all items may have a "set" key
                                    and item["set"] == set_name
                                    and item["attribute"] == set_attr_name
                                )
                            ),
                            None,
                        )
                        if attribute is None:
                            self.logger.debug(
                                "Category -> '%s', set -> %s, attribute -> '%s' not found in payload.",
                                cat_name,
                                set_name,
                                set_attr_name,
                            )
                            set_data_attributes[set_attr_id] = ""
                        else:
                            self.logger.debug(
                                "Category -> '%s', set -> %s, attribute -> '%s' found in payload, value -> %s",
                                cat_name,
                                set_name,
                                set_attr_name,
                                attribute["value"],
                            )
                            # Resolve any special cases (e.g. user picker, group picker):
                            set_data_attributes[set_attr_id] = self.resolve_attribute_values(
                                attribute_name=set_attr_name,
                                attribute_type=set_attr_type,
                                attribute_values=attribute["value"],
                            )
                    # end for set_attr_id in set_data_attributes:
                # end elif attr_type == "object":

                #
                # Handle plain attribute (not inside a set):
                #
                else:
                    attr_name = schema_attributes[attr_id]["title"]
                    self.logger.debug("Attribute name -> '%s'", attr_name)
                    # Lookup the attribute with the right category and attribute name in payload:
                    attribute = next(
                        (item for item in categories if (item["name"] == cat_name and item["attribute"] == attr_name)),
                        None,
                    )

                    if attr_type == "array" and "items" in schema_attributes[attr_id]:
                        attr_type = schema_attributes[attr_id]["items"].get("type", attr_type)

                    if attribute is None:
                        self.logger.debug(
                            "Category -> '%s', attribute -> '%s' not found in payload.",
                            cat_name,
                            attr_name,
                        )
                        data_attributes[attr_id] = ""
                    else:
                        self.logger.debug(
                            "Category -> '%s', attribute -> '%s' found in payload, value -> %s",
                            cat_name,
                            attr_name,
                            attribute["value"],
                        )
                        # Resolve any special cases (e.g. user picker, group picker):
                        data_attributes[attr_id] = self.resolve_attribute_values(
                            attribute_name=attr_name,
                            attribute_type=attr_type,
                            attribute_values=attribute["value"],
                        )
                # end else (plain attribute)
            # end for attr_data, attr_schema
            category_create_data[cat_id] = data_attributes
        # end for cat_data, cat_schema

        self.logger.debug("Category Create Data -> %s", category_create_data)

        return category_create_data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="prepare_workspace_business_objects")
    def prepare_workspace_business_objects(
        self,
        workspace: dict,
        business_objects: list,
    ) -> list | None:
        """Prepare the business object data for the workspace creation.

        This supports multiple external system connections. This methods
        also checks if the external system is reachable and tries to create
        missing business objects in the leading system if they are missing.

        Args:
            workspace (dict):
                The payload data for the workspace.
            business_objects (list):
                The payload data for the business object connections.

        Returns:
            list | None:
                A list of business object data connections
                (list elements are dictionaries).

        """

        business_object_list = []

        for business_object_data in business_objects:
            business_object = {}

            name = workspace.get("name")

            # Read business object data from workspace payload.
            # business_object_data is a dict with 3-5 elements:
            if "external_system" in business_object_data:
                ext_system_id = business_object_data["external_system"]
            else:
                self.logger.error(
                    "Missing External System in Business Object payload for workspace -> '%s'.",
                    name,
                )
                continue
            if "bo_type" in business_object_data:
                bo_type = business_object_data["bo_type"]
            else:
                self.logger.error(
                    "Missing type in Business Object payload for workspace -> '%s'.",
                    name,
                )
                continue

            if "bo_id" in business_object_data:
                bo_id = business_object_data["bo_id"]
                bo_search_field = None
                bo_search_value = None
            elif "bo_search_field" not in business_object_data or "bo_search_value" not in business_object_data:
                self.logger.error(
                    "Missing BO search fields (bo_search_field, bo_search_value) in Business Object payload without BO ID for workspace -> '%s'.",
                    name,
                )
                continue
            else:
                bo_search_field = business_object_data["bo_search_field"]
                bo_search_value = business_object_data["bo_search_value"]
                bo_id = None

            # Lookup external system in this payload or a former processed payload:
            external_system = self.lookup_external_system(ext_system_id=ext_system_id)

            # If the external system is not in the current payload but in the system
            # we try to avoid errors in the following code.
            # TODO: review REST APIs in OTCS 26.1 version to see if we can improve this.
            if not external_system and self._otcs.get_external_system_connection(connection_name=ext_system_id):
                # As the REST API for reading external system data is pretty much limited
                # we try to do the bare minimum here:
                if "Guidewire" in business_object_data["external_system"]:
                    external_system_type = "Guidewire"
                elif "Salesforce" in business_object_data["external_system"]:
                    external_system_type = "Salesforce"
                external_system = {"enabled": True, "reachable": True, "external_system_type": external_system_type}
                self.logger.info(
                    "External system -> '%s' is not found in current payload but it exists in the system. Construct minimum external system information -> %s.",
                    ext_system_id,
                    str(external_system),
                )

            if not external_system:
                self.logger.warning(
                    "External System -> '%s' does not exist. Cannot connect workspace -> '%s' to -> %s. Create workspace without connection.",
                    ext_system_id,
                    name,
                    ext_system_id,
                )
                continue
            if not external_system.get("enabled", True):
                self.logger.info(
                    "External System -> '%s' is disabled in payload. Cannot connect workspace -> '%s' to -> (%s, %s, %s, %s, %s). Create workspace without connection...",
                    ext_system_id,
                    name,
                    ext_system_id,
                    bo_type,
                    bo_id,
                    bo_search_field,
                    bo_search_value,
                )
                continue
            if not external_system.get("reachable"):
                self.logger.warning(
                    "External System -> '%s' is not reachable. Cannot connect workspace -> '%s' to -> (%s, %s, %s, %s, %s). Create workspace without connection...",
                    ext_system_id,
                    name,
                    ext_system_id,
                    bo_type,
                    bo_id,
                    bo_search_field,
                    bo_search_value,
                )
                continue
            external_system_type = external_system.get("external_system_type", "")

            # For Salesforce we try to determine the actual business object ID (technical ID) if it is
            # not specified directly (but instead a search field and search value):
            if external_system_type == "Salesforce" and not bo_id:
                # If Salesforce external system is used across payloads it may not be initialized here:
                if not self._salesforce:
                    self._salesforce = self.init_salesforce(external_system)
                bo_id = self.get_salesforce_business_object(
                    workspace,
                    object_type=bo_type,
                    search_field=bo_search_field,
                    search_value=bo_search_value,
                )
                if not bo_id:
                    self.logger.warning(
                        "Workspace -> '%s' will not be connected to Salesforce as the Business Object ID couldn't be determined (type -> '%s', search_field -> '%s', search_value -> '%s')!",
                        name,
                        bo_type,
                        bo_search_field,
                        bo_search_value,
                    )
                    continue
            # end if external_system_type == "Salesforce" and not bo_id

            # For Guidewire we try to determine the actual business object ID (technical ID) if it is
            # not specified in the payload (but instead a search field and search value):
            if external_system_type == "Guidewire" and not bo_id:
                bo_id = self.get_guidewire_business_object(
                    external_system=external_system,
                    object_type=bo_type,
                    search_field=bo_search_field,
                    search_value=bo_search_value,
                )
                if not bo_id:
                    self.logger.warning(
                        "Workspace -> '%s' will not be connected to Guidewire as the business object ID couldn't be determined (type -> '%s', search_field -> '%s', search_value -> '%s')!",
                        name,
                        bo_type,
                        bo_search_field,
                        bo_search_value,
                    )
                    continue
            # end if external_system_type == "Guidewire" and not bo_id

            self.logger.info(
                "Workspace -> '%s' will be connected with external system -> '%s' (%s) with (type -> '%s', ID -> '%s').",
                name,
                external_system_type,
                ext_system_id,
                bo_type,
                bo_id,
            )

            business_object["ext_system_id"] = ext_system_id
            business_object["bo_type"] = bo_type
            business_object["bo_id"] = bo_id

            business_object_list.append(business_object)

        return business_object_list

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace")
    def process_workspace(
        self,
        workspace: dict,
    ) -> bool:
        """Worker thread for workspace creation.

        Args:
            workspace (dict):
                Dictionary with payload of a single workspace.

        Returns:
            bool:
                True = Success, False = Failure

        """

        # Read name from payload:
        if "name" not in workspace:
            self.logger.error("Workspace needs a name! Skipping to next workspace...")
            return False
        name = workspace["name"]

        # Check if workspace has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not workspace.get("enabled", True):
            self.logger.info(
                "Payload for workspace -> '%s' is disabled. Skipping...",
                name,
            )
            return True

        # Read optional description from payload:
        description = workspace.get("description", "")

        # Read Type Name from payload:
        if "type_name" not in workspace:
            self.logger.error(
                "Workspace -> '%s' needs a type name! Skipping to next workspace...",
                name,
            )
            return False
        type_name = workspace["type_name"]

        # We need to do this early to find out if we have a cross-application workspace
        # and need to continue even if the workspace does exist...
        if workspace.get("business_objects"):
            business_objects = workspace["business_objects"]

            business_object_list = self.prepare_workspace_business_objects(
                workspace=workspace,
                business_objects=business_objects,
            )
            # Check if any of the external systems are avaiable:
            if business_object_list:
                self.logger.info(
                    "Workspace -> '%s' will be connected to -> %s business object%s.",
                    name,
                    str(len(business_object_list)),
                    "s" if len(business_object_list) > 1 else "",
                )
        else:
            self.logger.debug(
                "Workspace -> '%s' is not connected to any business object.",
                name,
            )
            business_object_list = []

        # Intialize cross-application workspace to "off":
        ibo_workspace_id = None

        # check if the workspace has been created before (effort to make the customizing code idem-potent)
        self.logger.debug(
            "Check if workspace -> '%s' of type -> '%s' does already exist...",
            name,
            type_name,
        )
        # Check if workspace does already exist
        # In case the workspace exists, determine_workspace_id()
        # also stores the node ID into workspace["nodeId"]
        workspace_id = self.determine_workspace_id(workspace=workspace)
        if workspace_id:
            self.logger.info(
                "Workspace -> '%s' of type -> '%s' does already exist and has ID -> %d.",
                name,
                type_name,
                workspace_id,
            )
            # Check if we have an existing workspace that is cross-application.
            # In this case we cannot just skip (return).
            if len(business_object_list) > 1:
                ibo_workspace_id = workspace_id
                self.logger.info(
                    "Workspace -> '%s' is a cross-application workspace so we cannot skip the creation...",
                    name,
                )
            elif not business_object_list:
                self.logger.info(
                    "Workspace -> '%s' does already exist and has no business object references to update - skipping the creation...",
                    name,
                )
                return True

        # Parent ID is optional and only required if workspace type does not specify a create location.
        # This is typically the case if it is a nested workspace or workspaces of the same type can be created
        # in different locations in the Enterprise Workspace:
        parent_id = workspace.get("parent_id")

        if parent_id is not None:
            parent_workspace = next(
                (item for item in self._workspaces if item["id"] == parent_id),
                None,
            )
            if parent_workspace is None:
                self.logger.error(
                    "Parent workspace with logical ID -> %s not found.",
                    parent_id,
                )
                return False

            parent_workspace_node_id = self.determine_workspace_id(
                workspace=parent_workspace,
            )
            if not parent_workspace_node_id:
                self.logger.error(
                    "Parent workspace without node ID (parent workspace creation may have failed). Skipping to next workspace...",
                )
                return False

            self.logger.debug(
                "Parent workspace with logical ID -> %s has node ID -> %s",
                parent_id,
                parent_workspace_node_id,
            )
        else:
            # Alternatively a path could be specified in the payload:
            parent_path = workspace.get("parent_path")
            if parent_path:
                self.logger.info(
                    "Workspace -> '%s' has parent path -> %s specified in payload.",
                    name,
                    parent_path,
                )
                response = self._otcs.get_node_by_volume_and_path(
                    volume_type=self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE,
                    path=parent_path,
                    create_path=True,
                )
                parent_workspace_node_id = self._otcs.get_result_value(
                    response=response,
                    key="id",
                )
            else:
                # if no parent_id is specified the workspace location is determined by the workspace type definition
                # and we pass None as parent ID to the get_workspace_create_form and create_workspace methods below:
                parent_workspace_node_id = None
                self.logger.info(
                    "Workspace -> '%s' has no parent path specified in payload. Using the default defined in the workspace type...",
                    name,
                )

        # Find the workspace type with the name given in the payload:
        workspace_type = next(
            (item for item in self._workspace_types if item["name"] == type_name),
            None,
        )
        if workspace_type is None:
            self.logger.error(
                "Workspace type -> '%s' not found. Skipping to next workspace...",
                type_name,
            )
            return False
        if workspace_type["templates"] == []:
            self.logger.error(
                "Workspace type -> '%s' does not have templates. Skipping to next workspace...",
                type_name,
            )
            return False

        # check if the template to be used is specified in the payload:
        if "template_name" in workspace:
            template_name = workspace["template_name"]
            workspace_template = next(
                (item for item in workspace_type["templates"] if item["name"] == template_name),
                None,
            )
            if workspace_template:  # does this template exist?
                self.logger.debug(
                    "Workspace template -> '%s' has been specified in payload and it does exist.",
                    template_name,
                )
            else:
                self.logger.error(
                    "Workspace template -> '%s' has been specified in payload but it doesn't exist!",
                    template_name,
                )
                self.logger.error(
                    "Workspace type -> '%s' has only these templates -> %s",
                    type_name,
                    workspace_type["templates"],
                )
                return False
        # template to be used is NOT specified in the payload - then we just take the first one:
        else:
            workspace_template = workspace_type["templates"][0]
            self.logger.info(
                "Workspace template has not been specified in payload - taking the first one (%s)...",
                workspace_template,
            )

        template_id = workspace_template["id"]
        template_name = workspace_template["name"]
        workspace_type_id = workspace_type["id"]

        if not workspace_id:
            self.logger.info(
                "Create workspace -> '%s' (type -> '%s') from workspace template -> '%s' (%s)%s...",
                name,
                type_name,
                template_name,
                template_id,
                " with business object references -> {}".format(business_object_list) if business_object_list else "",
            )
        elif business_object_list:
            self.logger.info(
                "Update workspace -> '%s' (type -> '%s') business object references with -> %s...",
                name,
                type_name,
                str(business_object_list),
            )

        # Handle the case where the workspace is not connected
        # to any external system / business object:
        if not business_object_list:
            business_object_list.append(
                {
                    "ext_system_id": None,
                    "bo_type": None,
                    "bo_id": None,
                },
            )

        for business_object in business_object_list:
            external_system_id = business_object["ext_system_id"]
            bo_type = business_object["bo_type"]
            bo_id = business_object["bo_id"]

            if workspace_id and not ibo_workspace_id and bo_id:
                # See if the existing workspace does not yet have a business object references
                # that is given in the payload:
                self.logger.info("Get existing workspace references for workspace -> '%s' (%d)...", name, workspace_id)
                workspace_references = self._otcs.get_workspace_references(node_id=workspace_id)
                workspace_reference = next(
                    (
                        item
                        for item in workspace_references or []
                        if item["external_system_id"] == external_system_id
                        and item["business_object_id"] == bo_id
                        and item["business_object_type"].lower() == bo_type.lower()
                    ),
                    None,
                )
                if not workspace_reference:
                    self.logger.info(
                        "Set workspace reference for workspace -> '%s' (%d) to business object -> ('%s', '%s', %s)...",
                        name,
                        workspace_id,
                        external_system_id,
                        bo_type,
                        bo_id,
                    )
                    response = self._otcs.set_workspace_reference(
                        workspace_id=workspace_id, external_system_id=external_system_id, bo_type=bo_type, bo_id=bo_id
                    )
                    if not response:
                        self.logger.error(
                            "Failed to set a reference for workspace -> '%s' (%d) to business object -> ('%s', '%s', %s)!",
                            name,
                            workspace_id,
                            external_system_id,
                            bo_type,
                            bo_id,
                        )
                        return False
                else:
                    self.logger.info(
                        "Workspace -> '%s' (%d) has already a reference to business object -> ('%s', '%s', %s).",
                        name,
                        workspace_id,
                        external_system_id,
                        bo_type,
                        bo_id,
                    )
                continue

            # Read categories from payload:
            if "categories" in workspace:
                categories = workspace["categories"]
                workspace_category_data = self.prepare_workspace_create_form(
                    categories=categories,
                    template_id=template_id,
                    ext_system_id=business_object["ext_system_id"],
                    bo_type=business_object["bo_type"],
                    bo_id=business_object["bo_id"],
                    parent_workspace_node_id=parent_workspace_node_id,
                )
                if not workspace_category_data:
                    self.logger.error(
                        "Failed to prepare the category data for workspace -> '%s'!",
                        name,
                    )
                    return False
            else:
                self.logger.debug(
                    "Workspace payload has no category data! Will leave category attributes empty...",
                )
                workspace_category_data = {}

            if ibo_workspace_id:
                self.logger.info(
                    "Connect existing workspace -> '%s' to an additional business object of type -> '%s' and ID -> '%s' (IBO)...",
                    name,
                    business_object["bo_type"],
                    business_object["bo_id"],
                )
            # Create the workspace with all provided information:
            response = self._otcs.create_workspace(
                workspace_template_id=template_id,
                workspace_name=name,
                workspace_description=description,
                workspace_type=workspace_type_id,
                category_data=workspace_category_data,
                external_system_id=business_object["ext_system_id"],
                bo_type=business_object["bo_type"],
                bo_id=business_object["bo_id"],
                parent_id=parent_workspace_node_id,
                ibo_workspace_id=ibo_workspace_id,
                show_error=(
                    not self._sap
                ),  # if SAP is active it may produce workspaces concurrently (race condition). Then we don't want to issue errors.
            )
            if response is None:
                # Check if workspace has been concurrently created by some other
                # process (e.g. via SAP or Salesforce). This would be a race condition
                # that seems to really occur.

                # we wait a bit to give the concurrent thread the chance to fully finish the creation:
                time.sleep(5)

                workspace_id = self.determine_workspace_id(workspace=workspace)
                if workspace_id:
                    self.logger.info(
                        "Workspace -> '%s' of type -> '%s' has been created by an external process and has ID -> %s.",
                        name,
                        type_name,
                        workspace_id,
                    )
                    workspace["nodeId"] = workspace_id
                else:
                    self.logger.error(
                        "Failed to create workspace -> '%s' of type -> '%s'!",
                        name,
                        type_name,
                    )
                    return False
            # Now we add the node ID of the new workspace to the payload data structure
            # This will be reused when creating the workspace relationships!
            elif not ibo_workspace_id:
                workspace["nodeId"] = self._otcs.get_result_value(
                    response=response,
                    key="id",
                )
                workspace_id = workspace["nodeId"]
                ibo_workspace_id = workspace["nodeId"]

                # We also get the name the workspace was finally created with.
                # This can be different form the name in the payload as additional
                # naming conventions from the Workspace Type definitions may apply.
                # This is important to make the python container idem-potent.
                response = self._otcs.get_workspace(node_id=workspace["nodeId"])
                workspace["name"] = self._otcs.get_result_value(
                    response=response,
                    key="name",
                )
                # Also update the 'name' variable accordingly, as it is used below.
                name = workspace["name"]

                self.logger.info(
                    "Successfully created workspace with final name -> '%s', type -> '%s', and node ID -> %s.",
                    workspace["name"],
                    type_name,
                    workspace["nodeId"],
                )
        # end for business_object in business_object_list

        # if the workspace creation has failed - e.g. error in lookup of business
        # object in external system then we continue with the next workspace:
        if "nodeId" not in workspace:
            self.logger.error(
                "Couldn't create the workspace -> '%s'. Skipping to next workspace...",
                workspace["name"],
            )
            return False

        # Check if there's an workspace nickname configured:
        if "nickname" in workspace:
            nickname = workspace["nickname"]
            self.logger.info(
                "Assign nickname -> '%s' to workspace -> '%s' (%s)...",
                nickname,
                name,
                workspace["nodeId"],
            )
            response = self._otcs.set_node_nickname(
                node_id=workspace["nodeId"],
                nickname=nickname,
                show_error=True,
            )
            if not response:
                self.logger.error(
                    "Failed to assign nickname -> '%s' to workspace -> '%s' (%s)!",
                    nickname,
                    name,
                    workspace["nodeId"],
                )

        # Check if there's an workspace icon/image configured:
        if "image_nickname" in workspace:
            image_nickname = workspace["image_nickname"]

            response = self._otcs.get_node_from_nickname(nickname=image_nickname)
            node_id = self._otcs.get_result_value(response=response, key="id")
            if node_id:
                mime_type = self._otcs.get_result_value(
                    response=response,
                    key="mime_type",
                )
                if not mime_type:
                    self.logger.warning(
                        "Missing mime type information - assuming 'image/png'...",
                    )
                    mime_type = "image/png"
                file_path = os.path.join(tempfile.gettempdir(), image_nickname)
                result = self._otcs.download_document(node_id=node_id, file_path=file_path)
                if not result:
                    self.logger.error(
                        "Failed to download workspace image with nickname -> '%s' to file -> '%s'!",
                        image_nickname,
                        file_path,
                    )
                else:
                    response = self._otcs.update_workspace_icon(
                        workspace_id=workspace["nodeId"],
                        file_path=file_path,
                        file_mimetype=mime_type,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to assign icon -> '%s' to workspace -> '%s' from file -> '%s'!",
                            image_nickname,
                            name,
                            file_path,
                        )
            else:
                self.logger.error(
                    "Cannot find workspace image with nickname -> '%s' for workspace -> '%s'!",
                    image_nickname,
                    name,
                )

        # Check if an RM classification is specified for the workspace:
        # RM Classification is specified as list of path elements (top-down)
        if "rm_classification_path" in workspace and workspace["rm_classification_path"] != []:
            rm_class_node = self._otcs.get_node_by_volume_and_path(
                volume_type=self._otcs.VOLUME_TYPE_CLASSIFICATION_VOLUME,
                path=workspace["rm_classification_path"],
            )
            rm_class_node_id = self._otcs.get_result_value(
                response=rm_class_node,
                key="id",
            )
            if rm_class_node_id:
                response = self._otcs.assign_rm_classification(
                    node_id=workspace["nodeId"],
                    rm_classification=rm_class_node_id,
                    apply_to_sub_items=False,
                )
                if response is None:
                    self.logger.error(
                        "Failed to assign RM classification -> '%s' (%s) to workspace -> '%s'!",
                        workspace["rm_classification_path"][-1],
                        rm_class_node_id,
                        name,
                    )
                else:
                    self.logger.info(
                        "Assigned RM Classification -> '%s' to workspace -> '%s'.",
                        workspace["rm_classification_path"][-1],
                        name,
                    )
        # Check if one or multiple classifications are specified for the workspace
        # Classifications are specified as list of path elements (top-down)
        if "classification_pathes" in workspace and workspace["classification_pathes"] != []:
            for classification_path in workspace["classification_pathes"]:
                class_node = self._otcs.get_node_by_volume_and_path(
                    volume_type=self._otcs.VOLUME_TYPE_CLASSIFICATION_VOLUME,
                    path=classification_path,
                )
                class_node_id = self._otcs.get_result_value(
                    response=class_node,
                    key="id",
                )
                if class_node_id:
                    response = self._otcs.assign_classifications(
                        node_id=workspace["nodeId"],
                        classifications=[class_node_id],
                        apply_to_sub_items=False,
                    )
                    if response is None:
                        self.logger.error(
                            "Failed to assign classification -> '%s' to workspace -> '%s'!",
                            class_node_id,
                            name,
                        )
                    else:
                        self.logger.info(
                            "Successfully assigned Classification -> '%s' to workspace -> '%s'.",
                            classification_path[-1],
                            name,
                        )

        return True

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspaces_worker")
    def process_workspaces_worker(
        self,
        partition: pd.DataFrame,
        results: list,
    ) -> None:
        """Multi-threading worker method to process a partition of the workspaces.

        Args:
            partition (pd.DataFrame):
                The partition of the workspaces to process by this thread.
            results (list):
                A mutable (shared) list of all workers to collect the results.

        """

        thread_id = threading.get_ident()
        thread_name = threading.current_thread().name

        t = trace.get_current_span()
        t.set_attributes(
            {
                "thread.id": thread_id,
                "thread.name": thread_name,
            }
        )

        result = {}
        result["thread_id"] = thread_id
        result["thread_name"] = thread_name
        result["success"] = True
        result["failure_counter"] = 0
        result["success_counter"] = 0

        total_rows = len(partition)

        # Process all datasets in the partion that was given to the thread:
        for index, row in partition.iterrows():
            # Calculate percentage of completion
            percent_complete = ((partition.index.get_loc(index) + 1) / total_rows) * 100

            self.logger.info(
                "Processing data row -> %s to create workspace -> '%s' (%.2f %% complete)...",
                str(index),
                row["name"],
                percent_complete,
            )
            # Convert the row to a dictionary - omitting any empty column:
            workspace = row.dropna().to_dict()
            # workspace is a mutable dictionary that may be updated
            # by process_workspace():
            try:
                success = self.process_workspace(workspace=workspace)
            except Exception:
                self.logger.exception("Failed process workspace -> %s", workspace)
                success = False
            # We need to make sure the row (and the whole data frame)
            # gets these updates back (and adds new columns such as "nodeId"):
            for key, value in workspace.items():
                row[key] = value  # This will update existing keys and add new ones
            self.logger.debug("Final values of row %s -> %s", str(index), str(row))

            # As iterrows() creates a copy of the data we need to
            # write the changes back into the partition
            partition.loc[index] = row

            if success:
                result["success_counter"] += 1
            else:
                self.logger.error(
                    "Failed to process row -> %s for workspace -> '%s'!",
                    str(index),
                    row["name"],
                )
                result["failure_counter"] += 1
                result["success"] = False

        results.append(result)

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspaces")
    def process_workspaces(self, section_name: str = "workspaces") -> bool:
        """Process workspaces in payload and create them in Content Server.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        Side Effects:
            Set workspace["nodeId"] to the node ID of the created workspace and update
            the workspace["name"] to the final name of the workspaces (which may be different
            from the ones in the payload depending on workspace type configutrations)

        """

        if not self._workspaces:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            # Read the list of created workspaces from the json file in admin Home
            # This is important in case of restart / rerun of customizer pod
            # as this data structure is used later on for workspace relationship
            # processing (and other) and the workspaces dictionaries have been
            # updated with "nodeId" and "name" (the final names of the workspaces
            # that can be different from the names in the payload)
            self.logger.info(
                "Re-Initialize workspace list from status file -> '%s' to have final names and node IDs...",
                self.get_status_file_name(payload_section_name=section_name),
            )
            self._workspaces = self.get_status_file(payload_section_name=section_name)

            return True

        success: bool = True

        if ENABLE_MULTI_THREADING:
            # Create a list to hold the threads
            threads = []
            # And another list to collect the results
            results = []

            df = Data(self._workspaces, logger=self.logger)

            # Add empty column for "nodeId" so that the worker threads can properly fill it:
            df.get_data_frame()["nodeId"] = None

            self.logger.info(
                "Created a data frame with -> %s rows from the workspaces list with -> %s elements.",
                str(len(df)),
                str(len(self._workspaces)),
            )
            df.print_info()

            partitions = df.partitionate(THREAD_NUMBER)

            # Create and start a thread for each partition
            for index, partition in enumerate(partitions, start=1):
                # start a thread executing the process_workspaces_worker() method below:
                thread = threading.Thread(
                    name=f"{section_name}_{index:02}",
                    target=self.thread_wrapper,
                    args=(self.process_workspaces_worker, partition, results),
                )
                self.logger.info("Starting thread -> '%s'...", str(thread.name))
                thread.start()
                threads.append(thread)
                # Avoid that all threads start at the exact same time with
                # potentially expired cookies that cases race conditions:
                time.sleep(1)
            # end for index, partition in enumerate(partitions, start=1)

            # Wait for all threads to complete
            for thread in threads:
                self.logger.info(
                    "Waiting for thread -> '%s' to complete...",
                    str(thread.name),
                )
                thread.join()
                self.logger.info("Thread -> '%s' has completed.", str(thread.name))

            # As we have basically created a copy of self._workspaces into the Pandas
            # data frame (see df = Data(...) above) and the workspace processing
            # updates the workspaces data with "nodeID" and the final
            # workspace names, we need to write the Pandas Data frame
            # back into the self._workspaces data structure for further processing
            # e.g. in the process_workspace_relationships. Otherwise the
            # changes to "nodeId" or "name" would be lost. We need to do it
            # in 2 steps as we want to avoid to have NaN values in the resulting dicts:
            # 1. Convert the data frame back to a list of dictionaries:
            updated_workspaces = df.get_data_frame().to_dict(orient="records")
            # 2. Remove any dictionary item that has a "NaN" scalar value
            # (pd.notna() only works on scalar values, not on lists!):
            self._workspaces = [
                #                {k: v for k, v in w.items() if pd.notna(v)} for w in updated_workspaces
                {
                    key: value
                    for key, value in updated_workspace.items()
                    if not pd.api.types.is_scalar(value) or pd.notna(value)
                }
                for updated_workspace in updated_workspaces
            ]

            # Print statistics for each thread. In addition,
            # check if all threads have completed without error / failure.
            # If there's a single failure in on of the thread results we
            # set 'success' variable to False.
            results.sort(key=lambda x: x["thread_name"])
            for result in results:
                if not result["success"]:
                    self.logger.info(
                        "Thread -> '%s' had %s failures and completed %s workspaces successfully.",
                        result["thread_name"],
                        result["failure_counter"],
                        result["success_counter"],
                    )
                    success = False  # mark the complete processing as "failure" for the status file.
                else:
                    self.logger.info(
                        "Thread -> '%s' completed %s workspaces successfully.",
                        result["thread_name"],
                        result["success_counter"],
                    )
        else:  # no multi-threading
            for workspace in self._workspaces:
                try:
                    result = self.process_workspace(workspace=workspace)
                except Exception:
                    self.logger.exception("Failed process workspace -> %s", workspace)
                    result = False
                success = success and result  # if a single result is False then mark this in 'success' variable.

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._workspaces,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_relationship")
    def process_workspace_relationship(self, workspace: dict) -> bool:
        """Worker thread for workspace relationship creation.

        Args:
            workspace (dict):
                Dictionary with payload of a single workspace.

        Returns:
            bool:
                True = Success, False = Failure

        """

        # Read name from payload. If it does not exist then bail out:
        name = workspace.get("name")
        if not name:
            return False

        # Check if element has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not workspace.get("enabled", True):
            self.logger.info(
                "Payload for workspace -> '%s' is disabled. Skipping...",
                name,
            )
            return True

        # Read relationships from payload:
        if "relationships" not in workspace:
            self.logger.debug(
                "Workspace -> '%s' has no relationships. Skipping to next workspace...",
                name,
            )
            return True

        # Check that workspaces actually have a logical ID -
        # otherwise we cannot establish the relationship:
        workspace_id = workspace.get("id")
        if not workspace_id:
            self.logger.warning(
                "Workspace without logical ID in payload cannot have a relationship. Skipping to next workspace...",
            )
            return False

        self.logger.info(
            "Workspace -> '%s' (type -> '%s') has relationships - creating...",
            name,
            workspace["type_name"],
        )

        # now determine the actual node ID of the workspace (which should have been created before):
        workspace_node_id = int(self.determine_workspace_id(workspace=workspace))
        if not workspace_node_id:
            self.logger.warning(
                "Workspace -> '%s' (type -> '%s') has no node ID and cannot have a relationship (workspace creation may have failed or final name is different from payload). Skipping to next workspace...",
                name,
                workspace["type_name"],
            )
            return False

        self.logger.debug(
            "Workspace with logical ID -> %s has node ID -> %s",
            str(workspace_id),
            str(workspace_node_id),
        )

        success: bool = True

        for related_workspace in workspace["relationships"]:
            # Initialize variable to determine if we found a related workspace:
            related_workspace_node_id = None
            found_by = ""

            if isinstance(related_workspace, (str, int)):
                #
                # 1. Option: Find the related workspace with the logical ID given in the payload:
                #
                related_workspace_payload = next(
                    (item for item in self._workspaces if str(item["id"]) == str(related_workspace)),
                    None,
                )
                if related_workspace_payload:
                    if not related_workspace_payload.get("enabled", True):
                        self.logger.info(
                            "Payload for related workspace -> '%s' is disabled. Skipping...",
                            related_workspace_payload["name"],
                        )
                        continue

                    related_workspace_node_id = self.determine_workspace_id(
                        workspace=related_workspace_payload,
                    )
                    if not related_workspace_node_id:
                        self.logger.warning(
                            "Related workspace -> '%s' (type -> '%s') has no node ID (workspaces creation may have failed or name is different from payload). Skipping to next workspace...",
                            related_workspace_payload["name"],
                            related_workspace_payload["type_name"],
                        )
                        continue
                    found_by = "logical ID -> '{}' in payload".format(related_workspace)
                # end if related_workspace_payload:

                #
                # 2. Option: Find the related workspace with nickname:
                #
                else:
                    # See if a nickname exists the the provided related_workspace:
                    response = self._otcs.get_node_from_nickname(nickname=related_workspace)
                    related_workspace_node_id = self._otcs.get_result_value(
                        response=response,
                        key="id",
                    )
                    if related_workspace_node_id:
                        found_by = "nickname -> '{}'".format(related_workspace)
            # end if isinstance(related_workspace_id, (str, int)):

            #
            # 3. Option: Find the related workspace type and name:
            #
            elif isinstance(related_workspace, dict):
                related_workspace_type = related_workspace.get("type", None)
                related_workspace_name = related_workspace.get("name", None)
                if related_workspace_type and related_workspace_name:
                    response = self._otcs.get_workspace_by_type_and_name(
                        type_name=related_workspace_type, name=related_workspace_name
                    )
                    related_workspace_node_id = self._otcs.get_result_value(
                        response=response,
                        key="id",
                    )
                    if related_workspace_node_id:
                        found_by = "type -> '{}' and name -> '{}'".format(
                            related_workspace_type, related_workspace_name
                        )
            #
            # 4. Option: Find the related workspace volume and path:
            #
            elif isinstance(related_workspace, list):
                response = self._otcs.get_node_by_volume_and_path(
                    volume_type=self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE, path=related_workspace
                )
                related_workspace_node_id = self._otcs.get_result_value(
                    response=response,
                    key="id",
                )
                if related_workspace_node_id:
                    found_by = "path -> {}".format(related_workspace)

            if related_workspace_node_id is None:
                self.logger.error(
                    "Related workspace -> %s not found.",
                    related_workspace,
                )
                success = False
                continue

            self.logger.debug(
                "Related workspace with %s has node ID -> %s",
                found_by,
                related_workspace_node_id,
            )

            # Check if relationship does already exists:
            response = self._otcs.get_workspace_relationships(
                workspace_id=workspace_node_id,
            )

            existing_workspace_relationship = self._otcs.exist_result_item(
                response=response,
                key="id",
                value=related_workspace_node_id,
            )
            if existing_workspace_relationship:
                self.logger.info(
                    "Workspace relationship between workspace ID -> %s and related workspace ID -> %s does already exist. Skipping...",
                    str(workspace_node_id),
                    related_workspace_node_id,
                )
                continue

            self.logger.info(
                "Create workspace relationship between workspace node ID -> %s and workspace node ID -> %s...",
                str(workspace_node_id),
                related_workspace_node_id,
            )

            response = self._otcs.create_workspace_relationship(
                workspace_id=workspace_node_id,
                related_workspace_id=related_workspace_node_id,
            )
            if not response:
                self.logger.error("Failed to create workspace relationship!")
                success = False
            else:
                self.logger.info("Successfully created workspace relationship.")

        # end for relationships

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_relationships_worker")
    def process_workspace_relationships_worker(
        self,
        partition: pd.DataFrame,
        results: list,
    ) -> None:
        """Multi-threading worker method to process a partition of the workspaces to create workspace relationships.

        Args:
            partition (pd.DataFrame):
                The partition of the workspace relationships to process by this thread.
            results (list):
                A mutable (shared) list of all workers to collect the results.

        """

        thread_id = threading.get_ident()
        thread_name = threading.current_thread().name

        result = {}
        result["thread_id"] = thread_id
        result["thread_name"] = thread_name
        result["success"] = True
        result["failure_counter"] = 0
        result["success_counter"] = 0

        total_rows = len(partition)

        # Process all datasets in the partion that was given to the thread:
        for index, row in partition.iterrows():
            # Calculate percentage of completion
            percent_complete = ((partition.index.get_loc(index) + 1) / total_rows) * 100

            self.logger.info(
                "Processing data row -> %s to create relationships for workspace -> '%s' (%.2f %% complete)...",
                str(index),
                row["name"],
                percent_complete,
            )
            success = self.process_workspace_relationship(
                workspace=row.dropna().to_dict(),
            )
            if success:
                result["success_counter"] += 1
            else:
                self.logger.error(
                    "Failed to process row -> %s for relationships of workspace -> '%s'!",
                    str(index),
                    row["name"],
                )
                result["failure_counter"] += 1
                result["success"] = False

        results.append(result)

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_relationships")
    def process_workspace_relationships(
        self,
        section_name: str = "workspaceRelationships",
    ) -> bool:
        """Process workspaces relationships in payload and create them in Content Server.

        Relationships can only be created if all workspaces have been created before.
        Once a workspace got created, the node ID of that workspaces has been added
        to the payload["workspaces"] data structure (see process_workspaces())
        Relationships are created between the node IDs of two business workspaces
        (and not the logical IDs in the inital payload specification)

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._workspaces:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        if ENABLE_MULTI_THREADING:
            # Create a list to hold the threads
            threads = []
            # And another list to collect the results
            results = []

            df = Data(self._workspaces, logger=self.logger)

            partitions: list = df.partitionate(number=THREAD_NUMBER)

            # Create and start a thread for each partition
            for index, partition in enumerate(partitions, start=1):
                # start a thread executing the process_workspace_relationships_worker() method below:
                thread = threading.Thread(
                    name=f"{section_name}_{index:02}",
                    target=self.thread_wrapper,
                    args=(
                        self.process_workspace_relationships_worker,
                        partition,
                        results,
                    ),
                )
                self.logger.info("Starting thread -> '%s'...", str(thread.name))
                thread.start()
                threads.append(thread)
                # Avoid that all threads start at the exact same time with
                # potentially expired cookies that cases race conditions:
                time.sleep(1)
            # end for index, partition in enumerate(partitions, start=1)

            # Wait for all threads to complete
            for thread in threads:
                self.logger.info(
                    "Waiting for thread -> '%s' to complete...",
                    str(thread.name),
                )
                thread.join()
                self.logger.info("Thread -> '%s' has completed.", str(thread.name))

            # Print statistics for each thread. In addition,
            # check if all threads have completed without error / failure.
            # If there's a single failure in on of the thread results we
            # set 'success' variable to False.
            results.sort(key=lambda x: x["thread_name"])
            for result in results:
                if not result["success"]:
                    self.logger.info(
                        "Thread -> '%s' had %s failures and completed relationships for %s workspaces successfully.",
                        result["thread_name"],
                        result["failure_counter"],
                        result["success_counter"],
                    )
                    success = False  # mark the complete processing as "failure" for the status file.
                else:
                    self.logger.info(
                        "Thread -> '%s' completed relationships for %s workspace successfully.",
                        result["thread_name"],
                        result["success_counter"],
                    )
        else:  # no multi-threading
            for workspace in self._workspaces:
                result = self.process_workspace_relationship(workspace=workspace)
                success = (
                    success and result
                )  # if a single result is False then the 'success' variable becomes 'False' as well.

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._workspaces,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_members")
    def process_workspace_members(self, section_name: str = "workspaceMembers") -> bool:
        """Process workspaces members in payload and create them in Content Server.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._workspaces:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for workspace in self._workspaces:
            # Read name from payload (just for logging):
            if "name" not in workspace:
                continue
            workspace_name = workspace["name"]

            # Check if workspace has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not workspace.get("enabled", True):
                self.logger.info(
                    "Payload for workspace -> '%s' is disabled. Skipping...",
                    workspace_name,
                )
                continue

            # Read members from payload:
            members = workspace.get("members")
            if not members:
                self.logger.info(
                    "Workspace -> '%s' has no members in payload. Skipping to next workspace...",
                    workspace_name,
                )
                continue

            workspace_id = workspace["id"]

            workspace_node_id = int(self.determine_workspace_id(workspace=workspace))
            if not workspace_node_id:
                self.logger.warning(
                    "Workspace without node ID cannot have members (workspaces creation may have failed). Skipping to next workspace...",
                )
                continue

            self.logger.info(
                "Workspace -> '%s' (%s) has memberships in payload - establishing...", workspace_name, workspace_node_id
            )

            # now determine the actual node IDs of the workspaces (have been created by process_workspaces()):
            workspace_node = self._otcs.get_node(node_id=workspace_node_id)
            workspace_owner_id = self._otcs.get_result_value(
                response=workspace_node,
                key="owner_user_id",
            )
            workspace_owner_name = self._otcs.get_result_value(
                response=workspace_node,
                key="owner",
            )

            workspace_roles = self._otcs.get_workspace_roles(
                workspace_id=workspace_node_id,
            )
            if workspace_roles is None:
                self.logger.debug(
                    "Workspace -> '%s' (%s) has no roles. Skipping to next workspace...",
                    workspace_name,
                    workspace_node_id,
                )
                continue

            # We don't want the workspace creator to be in the leader role
            # of automatically created workspaces - this can happen because the
            # creator gets added to the leader role automatically if
            # the workspace type advanved configuration setting
            # "Add the creator of a business workspace to the Lead role" is
            # enabled:
            roles_iterator = self._otcs.get_result_values_iterator(response=workspace_roles)
            for role in roles_iterator:
                # We can have two leader roles if in a sub-workspaces a leader
                # roles is inherited from the parent workspace. As we want
                # don't want to consider leader role of the parent workspace
                # we check that 'inherited_from_id' is not set:
                if role["leader"] and role["inherited_from_id"] is None:
                    leader_role_id = role["id"]
                    break
            else:
                leader_role_id = None

            if leader_role_id:
                leader_role_name = self._otcs.lookup_result_value(
                    response=workspace_roles,
                    key="leader",
                    value=True,
                    return_key="name",
                )
                response = self._otcs.remove_workspace_member(
                    workspace_id=workspace_node_id,
                    role_id=leader_role_id,
                    member_id=workspace_owner_id,
                    show_warning=False,
                )
                if response:
                    self.logger.info(
                        "Removed creator user -> '%s' (%s) from leader role -> '%s' (%s) of workspace -> '%s' (%s).",
                        workspace_owner_name,
                        workspace_owner_id,
                        leader_role_name,
                        leader_role_id,
                        workspace_name,
                        workspace_node_id,
                    )
                else:
                    self.logger.info(
                        "Creator user -> '%s' (%s) is not in leader role -> '%s' (%s) of workspace -> '%s' (%s). No need to remove it.",
                        workspace_owner_name,
                        workspace_owner_id,
                        leader_role_name,
                        leader_role_id,
                        workspace_name,
                        workspace_node_id,
                    )

            self.logger.info(
                "Adding members to workspace -> '%s' (%s) defined in payload...",
                workspace_name,
                workspace_node_id,
            )

            for member in members:
                # read user list and role name from payload:
                member_users = member.get("users", [])
                member_groups = member.get("groups", [])
                member_role_name = member.get("role", "")
                member_clear = member.get("clear", False)

                if member_role_name == "":  # role name is required
                    self.logger.error(
                        "Members of workspace -> '%s' is missing the role name in the payload.",
                        workspace_name,
                    )
                    success = False
                    continue

                role_id = self._otcs.lookup_result_value(
                    response=workspace_roles,
                    key="name",
                    value=member_role_name,
                    return_key="id",
                )
                if role_id is None:
                    #    if member_role is None:
                    self.logger.error(
                        "Workspace -> '%s' does not have a role -> '%s'",
                        workspace_name,
                        member_role_name,
                    )
                    success = False
                    continue
                inherited_role_id = self._otcs.lookup_result_value(
                    response=workspace_roles,
                    key="name",
                    value=member_role_name,
                    return_key="inherited_from_id",
                )
                if inherited_role_id is not None:
                    self.logger.error(
                        "The role -> '%s' (%s) of workspace -> '%s' (%s) is inherited from role with ID -> %d and members cannot be set in this sub-workspace.",
                        member_role_name,
                        role_id,
                        workspace_name,
                        workspace_node_id,
                        inherited_role_id,
                    )
                    success = False
                    continue
                self.logger.debug(
                    "Role -> '%s' has ID -> %s",
                    member_role_name,
                    role_id,
                )

                # check if we want to clear (remove) existing members of this role:
                if member_clear:
                    self.logger.info(
                        "Clear existing members of role -> '%s' (%s) for workspace -> '%s' (%s)...",
                        member_role_name,
                        role_id,
                        workspace_name,
                        workspace_id,
                    )
                    self._otcs.remove_workspace_members(
                        workspace_id=workspace_node_id,
                        role_id=role_id,
                    )

                if member_users == [] and member_groups == []:  # we either need users or groups (or both)
                    self.logger.debug(
                        "Role -> '%s' of workspace -> '%s' does not have any members (no users nor groups).",
                        member_role_name,
                        workspace_name,
                    )
                    continue

                # Process users as workspaces members:
                for member_user in member_users:
                    # find member user in current payload:
                    member_user_id = next(
                        (item for item in self._users if item["name"] == member_user),
                        {"name": member_user},  # we construct a payload on the fly to make determine_user_id() work
                    )
                    user_id = self.determine_user_id(user=member_user_id)
                    if not user_id:
                        self.logger.error(
                            "Cannot find member user with login -> '%s'. Skipping...",
                            member_user,
                        )
                        continue

                    # Add member if it does not yet exists - suppress warning
                    # message if user is already in role:
                    response = self._otcs.add_workspace_member(
                        workspace_id=workspace_node_id,
                        role_id=int(role_id),
                        member_id=user_id,
                        show_warning=False,
                    )
                    if response is None:
                        self.logger.error(
                            "Failed to add user -> '%s' (%s) as member to role -> '%s' of workspace -> '%s' (%s)!",
                            member_user,
                            user_id,
                            member_role_name,
                            workspace_name,
                            workspace_node_id,
                        )
                        success = False
                    else:
                        self.logger.info(
                            "Successfully added user -> '%s' (%s) as member to role -> '%s' of workspace -> '%s' (%s).",
                            member_user,
                            user_id,
                            member_role_name,
                            workspace_name,
                            workspace_node_id,
                        )

                # Process groups as workspaces members:
                for member_group in member_groups:
                    member_group_id = next(
                        (item for item in self._groups if item["name"] == member_group),
                        None,
                    )

                    group_id = self.determine_group_id(group=member_group_id) if member_group_id else None
                    if not group_id:
                        self.logger.error(
                            "Cannot find member group -> '%s'. Skipping...",
                            member_group,
                        )
                        success = False
                        continue

                    response = self._otcs.add_workspace_member(
                        workspace_id=workspace_node_id,
                        role_id=int(role_id),
                        member_id=group_id,
                        show_warning=False,
                    )
                    if response is None:
                        self.logger.error(
                            "Failed to add group -> '%s' (%s) to role -> '%s' of workspace -> '%s'!",
                            member_group_id["name"],
                            group_id,
                            member_role_name,
                            workspace_name,
                        )
                        success = False
                    else:
                        self.logger.info(
                            "Successfully added group -> '%s' (%s) to role -> '%s' of workspace -> '%s'.",
                            member_group_id["name"],
                            group_id,
                            member_role_name,
                            workspace_name,
                        )

                # Optionally the payload may have a permission list for the role
                # to change the default permission from the workspace template
                # to something more specific:
                member_permissions = member.get("permissions", [])
                if member_permissions == []:
                    self.logger.debug(
                        "No permission change for workspace -> '%s' and role -> '%s'",
                        workspace_name,
                        member_role_name,
                    )
                    continue

                self.logger.info(
                    "Update permissions of workspace -> '%s' (%s) and role -> '%s' to -> %s...",
                    workspace_name,
                    str(workspace_node_id),
                    member_role_name,
                    str(member_permissions),
                )
                response = self._otcs.assign_permission(
                    node_id=workspace_node_id,
                    assignee_type="custom",
                    assignee=role_id,
                    permissions=member_permissions,
                    apply_to=2,
                )
                if not response:
                    self.logger.error(
                        "Failed to update permissions of workspace -> '%s' (%s) and role -> '%s' to -> %s!",
                        workspace_name,
                        str(workspace_node_id),
                        member_role_name,
                        str(member_permissions),
                    )
                    success = False
            # end for member in members:
        # end for workspace in self._workspaces:

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._workspaces,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_member_permissions")
    def process_workspace_member_permissions(
        self,
        section_name: str = "workspaceMemberPermissions",
    ) -> bool:
        """Process workspaces members in payload and set their permissions.

        We need this separate from process_workspace_members() which also
        sets permissions (if in payload) as we add documents to workspaces with
        content transports and these documents don't inherit role permissions
        (this is a transport limitation)

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._workspaces:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for workspace in self._workspaces:
            # Read name from payload (just for logging):
            workspace_name = workspace.get("name")
            if not workspace_name:
                continue

            # Check if workspace has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not workspace.get("enabled", True):
                self.logger.info(
                    "Payload for workspace -> '%s' is disabled. Skipping...",
                    workspace_name,
                )
                continue

            # Read members from payload:
            members = workspace.get("members")
            if not members:
                self.logger.info(
                    "Workspace -> '%s' has no members in payload. No need to update permissions. Skipping to next workspace...",
                    workspace_name,
                )
                continue

            workspace_id = workspace["id"]
            workspace_node_id = int(self.determine_workspace_id(workspace=workspace))
            if not workspace_node_id:
                self.logger.warning(
                    "Workspace without node ID cannot cannot get permission changes (workspaces creation may have failed). Skipping to next workspace...",
                )
                continue

            workspace_roles = self._otcs.get_workspace_roles(
                workspace_id=workspace_node_id,
            )
            if workspace_roles is None:
                self.logger.info(
                    "Workspace with payload ID -> %s and node Id -> %s has no roles to update permissions. Skipping to next workspace...",
                    workspace_id,
                    workspace_node_id,
                )
                continue

            for member in members:
                # read user list and role name from payload:
                member_users = (
                    member["users"] if member.get("users") else []
                )  # be careful to avoid key errors as users are optional
                member_groups = (
                    member["groups"] if member.get("groups") else []
                )  # be careful to avoid key errors as groups are optional
                member_role_name = member["role"]

                if member_role_name == "":  # role name is required
                    self.logger.error(
                        "Members of workspace -> '%s' is missing the role name.",
                        workspace_name,
                    )
                    success = False
                    continue
                if member_users == [] and member_groups == []:  # we either need users or groups (or both)
                    self.logger.debug(
                        "Role -> '%s' of workspace -> '%s' does not have any members (no users nor groups).",
                        member_role_name,
                        workspace_name,
                    )
                    continue

                role_id = self._otcs.lookup_result_value(
                    response=workspace_roles,
                    key="name",
                    value=member_role_name,
                    return_key="id",
                )
                if role_id is None:
                    self.logger.error(
                        "Workspace -> '%s' does not have a role -> '%s'",
                        workspace_name,
                        member_role_name,
                    )
                    success = False
                    continue
                self.logger.debug(
                    "Role -> '%s' has ID -> %s",
                    member_role_name,
                    role_id,
                )

                member_permissions = member.get("permissions", [])
                if member_permissions == []:
                    self.logger.debug(
                        "No permission change for workspace -> '%s' and role -> '%s'.",
                        workspace_name,
                        member_role_name,
                    )
                    continue

                self.logger.info(
                    "Update permissions of workspace -> '%s' (%s) and role -> '%s' to -> %s...",
                    workspace_name,
                    str(workspace_node_id),
                    member_role_name,
                    str(member_permissions),
                )
                response = self._otcs.assign_permission(
                    node_id=workspace_node_id,
                    assignee_type="custom",
                    assignee=role_id,
                    permissions=member_permissions,
                    apply_to=2,  # inherit to all sub-folders
                )
                if not response:
                    self.logger.error(
                        "Failed to update permissions of workspace -> '%s' (%s) and role -> '%s' to -> %s!",
                        workspace_name,
                        str(workspace_node_id),
                        member_role_name,
                        str(member_permissions),
                    )
                    success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._workspaces,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_aviators")
    def process_workspace_aviators(
        self,
        section_name: str = "workspaceAviators",
    ) -> bool:
        """Process workspaces Content Aviator settings in payload and enable Aviator for selected workspaces.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._workspaces:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for workspace in self._workspaces:
            # Read name from payload (just for logging):
            workspace_name = workspace.get("name")
            if not workspace_name:
                continue

            # Check if workspace has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not workspace.get("enabled", True):
                self.logger.info(
                    "Payload for workspace -> '%s' is disabled. Skipping...",
                    workspace_name,
                )
                continue

            # Read Aviator setting from payload:
            if not workspace.get("enable_aviator", False):
                self.logger.info(
                    "Aviator is not enabled for workspace -> '%s'. Skipping to next workspace...",
                    workspace_name,
                )
                continue

            # We cannot just lookup with workspace.get("nodeId") as the customizer
            # may have been restarted inbetween - so we use our proper determine_workspace_id
            # here...
            workspace_id = self.determine_workspace_id(workspace=workspace)
            if not workspace_id:
                self.logger.error(
                    "Cannot find node ID for workspace -> '%s'. Workspace creation may have failed. Skipping to next workspace...",
                    workspace_name,
                )
                success = False
                continue

            # Make code idem-potent and check if Aviator is already enabled
            # for this workspace:
            if self._otcs.check_workspace_aviator(workspace_id=workspace_id):
                self.logger.info(
                    "Skip workspace -> '%s' (%s) as Aviator is already enabled...",
                    workspace_name,
                    workspace_id,
                )
                continue

            # Now enable the Content Aviator for the workspace:
            response = self._otcs.update_workspace_aviator(
                workspace_id=workspace_id,
                status=True,
            )
            if not response:
                self.logger.error(
                    "Failed to enable Content Aviator for workspace -> '%s' (%s)!",
                    workspace_name,
                    workspace_id,
                )
                success = False
                continue
            # Also embed the workspace metadata:
            if not self._otcs.aviator_embed_metadata(
                node_id=workspace_id,
                workspace_metadata=True,
                wait_for_completion=False,
            ):
                success = False
                continue
        # end for workspace in self._workspaces

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._workspaces,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_web_reports")
    def process_web_reports(
        self,
        web_reports: list,
        section_name: str = "webReports",
    ) -> bool:
        """Process web reports in payload and run them in Content Server.

        Args:
            web_reports (list):
                The payload list of web reports. As we have two different list (pre and post)
                we need to pass the actual list as parameter.
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if a restart of the OTCS pods is required. False otherwise.

        """

        if not web_reports:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return False  # important to return False here as otherwise we are triggering a restart of services!!

        # If this payload section has been processed successfully before we
        # can return False and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return False  # important to return False here as otherwise we are triggering a restart of services!!

        restart_required: bool = False
        success: bool = True

        for web_report in web_reports:
            nickname = web_report.get("nickname")
            if not nickname:
                self.logger.error("Web Report payload needs a nickname! Skipping...")
                continue

            # Check if web report has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not web_report.get("enabled", True):
                self.logger.info(
                    "Payload for Web Report -> '%s' is disabled. Skipping...",
                    nickname,
                )
                continue

            description = web_report.get("description", "")
            restart = web_report.get("restart", False)

            if not self._otcs.get_node_from_nickname(nickname=nickname):
                self.logger.error(
                    "Web Report with nickname -> '%s' does not exist! Skipping...",
                    nickname,
                )
                success = False
                continue

            # be careful to avoid key errors as Web Report parameters are optional:
            actual_params = web_report["parameters"] if web_report.get("parameters") else {}
            formal_params = self._otcs.get_web_report_parameters(nickname=nickname)
            if actual_params:
                self.logger.info(
                    "Running Web Report -> '%s' (%s) with parameters -> %s ...",
                    nickname,
                    description,
                    actual_params,
                )
                # Do some sanity checks to see if the formal and actual parameters are matching...
                # Check 1: are there formal parameters at all?
                if not formal_params:
                    self.logger.error(
                        "Web Report -> '%s' is called with actual parameters but it does not expect parameters! Skipping...",
                        nickname,
                    )
                    success = False
                    continue
                lets_continue = False
                # Check 2: Iterate through the actual parameters given in the payload
                # and see if there's a matching formal parameter expected by the Web Report:
                for key, value in actual_params.items():
                    # Check if there's a matching formal parameter defined on the Web Report node:
                    formal_param = next(
                        (item for item in formal_params if item["parm_name"] == key),
                        None,
                    )
                    if formal_param is None:
                        self.logger.error(
                            "Web Report -> '%s' is called with parameter -> '%s' that is not expected! Value: %s) Skipping...",
                            nickname,
                            key,
                            value,
                        )
                        success = False
                        lets_continue = True  # we cannot do a "continue" here directly as we are in an inner loop
                # Check 3: Iterate through the formal parameters and validate there's a matching
                # actual parameter defined in the payload for each mandatory formal parameter
                # that does not have a default value:
                for formal_param in formal_params:
                    if (
                        (formal_param["mandatory"] is True)
                        and (formal_param["default_value"] is None)
                        and not actual_params.get(formal_param["parm_name"])
                    ):
                        self.logger.error(
                            "Web Report -> '%s' is called without mandatory parameter -> %s! Skipping...",
                            nickname,
                            formal_param["parm_name"],
                        )
                        success = False
                        lets_continue = True  # we cannot do a "continue" here directly as we are in an inner loop
                # Did any of the checks fail?
                if lets_continue:
                    continue
                # Actual parameters are validated, we can run the Web Report:
                response = self._otcs.run_web_report(
                    nickname=nickname,
                    web_report_parameters=actual_params,
                )
            # end if actual_params
            else:
                self.logger.info(
                    "Running Web Report -> '%s' (%s) without parameters...",
                    nickname,
                    description,
                )
                # Check if there's a formal parameter that is mandatory but
                # does not have a default value:
                if formal_params:
                    required_param = next(
                        (item for item in formal_params if (item["mandatory"] is True) and (not item["default_value"])),
                        None,
                    )
                    if required_param:
                        self.logger.error(
                            "Web Report -> '%s' is called without parameters but has a mandatory parameter -> '%s' without a default value! Skipping...",
                            nickname,
                            required_param["parm_name"],
                        )
                        success = False
                        continue
                    # we are good to proceed!
                    self.logger.debug(
                        "Web Report -> '%s' does not have a mandatory parameter without a default value!",
                        nickname,
                    )
                response = self._otcs.run_web_report(nickname=nickname)
            # end else
            if response is None:
                self.logger.error(
                    "Failed to run web report with nickname -> '%s'!",
                    nickname,
                )
                success = False

            if restart:
                restart_required = True

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=web_reports,
        )

        return restart_required

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_cs_applications")
    def process_cs_applications(
        self,
        otcs_object: OTCS,
        section_name: str = "csApplications",
    ) -> bool:
        """Process CS applications in payload and install them in Content Server.

        The CS Applications need to be installed in all frontend and backends.

        Args:
            otcs_object (object):
                This can either be the OTCS frontend or OTCS backend. If None
                then the otcs_backend is used.
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._cs_applications:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # OTCS backend is the default:
        if not otcs_object:
            otcs_object = self._otcs_backend

        for cs_application in self._cs_applications:
            application_name = cs_application.get("name")
            if not application_name:
                self.logger.error("Missing CS application name in payload! Skipping...")
                continue

            # Check if CS Application has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not cs_application.get("enabled", True):
                self.logger.info(
                    "Payload for CS Application -> '%s' is disabled. Skipping...",
                    application_name,
                )
                continue

            application_description = cs_application.get("description", "")

            self.logger.info(
                "Install CS Application -> '%s'%s...",
                application_name,
                " ({})".format(application_description) if application_description else "",
            )
            response = otcs_object.install_cs_application(
                application_name=application_name,
            )
            if response is None:
                self.logger.error(
                    "Failed to install CS Application -> '%s'!",
                    application_name,
                )
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._cs_applications,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_settings")
    def process_user_settings(self, section_name: str = "userSettings") -> bool:
        """Process user settings in payload and apply them in OTDS.

        This includes password settings and user display settings.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._users:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for user in self._users:
            user_name = user.get("name")
            if not user_name:
                self.logger.error(
                    "Missing user name - cannot apply user settings. Skipping user...",
                )
                continue

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_name,
                )
                continue

            self._log_header_callback(
                text="Process settings for user -> '{}'".format(user_name),
                char="-",
            )

            user_partition = self._otcs.config().get("partition", None)
            if not user_partition:
                self.logger.error("User partition not found!")
                success = False
                continue

            # Set the OTDS display name. Content Server does not use this but
            # it makes AppWorks display users correctly (and it doesn't hurt)
            # We only set this if firstname _and_ last name are in the payload:
            if "firstname" in user and "lastname" in user:
                user_display_name = user["firstname"] + " " + user["lastname"]
                response = self._otds.update_user(
                    partition=user_partition,
                    user_id=user_name,
                    attribute_name="displayName",
                    attribute_value=user_display_name,
                )
                if response:
                    self.logger.info(
                        "Successfully updated display name of user -> '%s' to -> '%s'.",
                        user_name,
                        user_display_name,
                    )
                else:
                    self.logger.error(
                        "Failed to update display name of user -> '%s' to -> '%s'!",
                        user_name,
                        user_display_name,
                    )
                    success = False

            # Don't enforce the user to reset password at first login (settings in OTDS):
            self.logger.info(
                "Don't enforce password change for user -> '%s'...",
                user_name,
            )
            response = self._otds.update_user(
                partition=user_partition,
                user_id=user_name,
                attribute_name="UserMustChangePasswordAtNextSignIn",
                attribute_value="False",
            )
            if not response:
                success = False

            response = self._otds.update_user(
                partition=user_partition,
                user_id=user_name,
                attribute_name="UserCannotChangePassword",
                attribute_value="True",
            )
            if not response:
                success = False

            # Set user password to never expire
            response = self._otds.update_user(
                partition=user_partition,
                user_id=user_name,
                attribute_name="PasswordNeverExpires",
                attribute_value="True",
            )
            if not response:
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._users,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_favorites_and_profiles")
    def process_user_favorites_and_profiles(
        self,
        section_name: str = "userFavoritesAndProfiles",
    ) -> bool:
        """Process user favorites in payload and create them in Content Server.

        This method also simulates browsing the favorites to populate the
        widgets on the landing pages and sets personal preferences.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._users:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        for dic in self._payload_sections:
            if dic.get("name") == "users":
                users_payload = dic
                break
        smartui_theme = "jato" if users_payload.get("jato_enabled", True) else "cf"

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # We can only set favorites if we impersonate / authenticate as the user.
        # The following code (for loop) will change the authenticated user - we need to
        # switch it back to admin user later so we safe the admin credentials for this:

        for user in self._users:
            with tracer.start_as_current_span("process_user_favorites_and_profiles-user") as t:
                t.set_attribute("class", "otcs")
                user_name = user.get("name")
                if not user_name:
                    self.logger.error(
                        "Missing user name - cannot configure user favorites and profile. Skipping user...",
                    )
                    continue

                self._log_header_callback(
                    text="Process Favorites and Profile for user -> '{}'".format(user_name),
                    char="-",
                )

                # Check if user has been explicitly disabled in payload
                # (enabled = false). In this case we skip the element:
                if not user.get("enabled", True):
                    self.logger.info(
                        "Payload for user -> '%s' is disabled. Skipping...",
                        user_name,
                    )
                    continue

                # We skip also user of type "ServiceUser":
                if user.get("type", "User") == "ServiceUser":
                    self.logger.info("Skipping service user -> '%s'...", user_name)
                    continue

                # Impersonate as the user:
                self.logger.info("Impersonate user -> '%s'...", user_name)
                result = self.start_impersonation(username=user_name)
                if not result:
                    self.logger.error("Couldn't impersonate user -> '%s'!", user_name)
                    success = False
                    continue

                # Configure default Theme to be Jato if configured
                user_smartui_theme = user.get("smartui_theme", smartui_theme)
                if user_smartui_theme != "cf":
                    response = self._otcs.update_user_profile(
                        config_section="SmartUI",
                        field="theme",
                        value=user_smartui_theme,
                    )
                    if response is None:
                        self.logger.warning(
                            "Profile for user -> '%s' couldn't be updated to Smart View theme -> '%s'!",
                            user_name,
                            user_smartui_theme,
                        )
                    else:
                        self.logger.info(
                            "Profile for user -> '%s' has been updated to Smart View theme -> '%s'.",
                            user_name,
                            user_smartui_theme,
                        )

                # Update the user profile to activate responsive (dynamic) containers:
                response = self._otcs.update_user_profile(
                    field="responsiveContainerMode",
                    value=True,
                    config_section="SmartUI",
                )
                if response is None:
                    self.logger.warning(
                        "Profile for user -> '%s' couldn't be updated with responsive container mode!",
                        user_name,
                    )
                else:
                    self.logger.info(
                        "Profile for user -> '%s' has been updated to enable responsive container mode.",
                        user_name,
                    )
                response = self._otcs.update_user_profile(
                    field="responsiveContainerMessageMode",
                    value=True,
                    config_section="SmartUI",
                )
                if response is None:
                    self.logger.warning(
                        "Profile for user -> '%s' couldn't be updated with responsive container message mode!",
                        user_name,
                    )
                else:
                    self.logger.info(
                        "Profile for user -> '%s' has been updated to enable messages for responsive container mode.",
                        user_name,
                    )

                restrict_personal_workspace = user.get("restrict_personal_workspace", False)
                if restrict_personal_workspace:
                    # Let the user restrict itself to have read-only access to its
                    # personal workspace:
                    node = self._otcs.get_node_by_volume_and_path(
                        volume_type=self._otcs.VOLUME_TYPE_PERSONAL_WORKSPACE,
                        path=[],
                    )
                    node_id = self._otcs.get_result_value(response=node, key="id")
                    if node_id:
                        self.logger.info(
                            "Restricting Personal Workspace of user -> '%s' to read-only.",
                            user_name,
                        )
                        response = self._otcs.assign_permission(
                            node_id=int(node_id),
                            assignee_type="owner",
                            assignee=0,
                            permissions=["see", "see_contents"],
                            apply_to=2,
                        )

                # We work through the list of favorites defined for the user:
                favorites = user.get("favorites", [])
                for favorite in favorites:
                    # We try three things to determine the favorite node:
                    # 1. If the favorite is a path (a python list) then we try to resolve
                    #    this path in the Enterprise Workspace.
                    # 2. We try to find the item (string) as a logical
                    #    workspace ID inside the payload.
                    # 3. We try to find the item (string) as a nickname in OTCS.
                    favorite_id = None
                    is_workspace = False
                    # 1. Check if the favorite item is itself a list,
                    #    then we try to interpret it as a path in
                    #    the enterprise workspace:
                    if isinstance(favorite, list):
                        favorite_item = self._otcs.get_node_by_volume_and_path(
                            volume_type=self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE,
                            path=favorite,
                        )
                        if not favorite_item:
                            self.logger.error("Cannot find path -> %s for favorite item!", str(favorite))
                            success = False
                            continue
                        favorite_id = self._otcs.get_result_value(
                            response=favorite_item,
                            key="id",
                        )
                        favorite_name = self._otcs.get_result_value(
                            response=favorite_item,
                            key="name",
                        )
                        favorite_type = self._otcs.get_result_value(
                            response=favorite_item,
                            key="type",
                        )
                        if favorite_type == self._otcs.ITEM_TYPE_BUSINESS_WORKSPACE:
                            is_workspace = True
                    # 2. Check if it a logical workspace identifier in the payload:
                    if not favorite_id:
                        # check if favorite is a logical workspace name
                        favorite_item = next(
                            (item for item in self._workspaces if item["id"] == favorite),
                            None,
                        )
                        if favorite_item:
                            if favorite_item.get("enabled", True):
                                self.logger.debug(
                                    "Found favorite item (workspace) -> '%s' in payload and it is enabled.",
                                    favorite_item["name"],
                                )
                            else:
                                self.logger.info(
                                    "Found favorite item (workspace) -> '%s' in payload but it is not enabled. Skipping...",
                                    favorite_item["name"],
                                )
                                continue
                            favorite_id = self.determine_workspace_id(workspace=favorite_item)
                            if not favorite_id:
                                self.logger.warning(
                                    "Workspace of type -> '%s' and name -> '%s' does not exist. Cannot create favorite. Skipping...",
                                    favorite_item["type_name"],
                                    favorite_item["name"],
                                )
                                continue
                            is_workspace = True
                            favorite_name = favorite_item["name"]
                    # 3. Check if it is a nickname:
                    if not favorite_id:
                        favorite_item = self._otcs.get_node_from_nickname(nickname=favorite)
                        favorite_id = self._otcs.get_result_value(
                            response=favorite_item,
                            key="id",
                        )
                        favorite_name = self._otcs.get_result_value(
                            response=favorite_item,
                            key="name",
                        )
                        favorite_type = self._otcs.get_result_value(
                            response=favorite_item,
                            key="type",
                        )
                        if favorite_type == self._otcs.ITEM_TYPE_BUSINESS_WORKSPACE:
                            is_workspace = True

                    # If nothing has worked then skip this payload favorite.
                    if not favorite_id:
                        self.logger.warning(
                            "Favorite -> '%s' neither found as workspace payload ID, not as a path, nor as nickname. Skipping to next favorite...",
                            favorite,
                        )
                        continue

                    response = self._otcs.add_favorite(node_id=favorite_id)
                    if response is None:
                        self.logger.warning(
                            "Favorite -> '%s' (%s) couldn't be added for user -> '%s'!",
                            favorite_name,
                            favorite_id,
                            user_name,
                        )
                    else:
                        self.logger.info(
                            "Added favorite -> '%s' (%s) for user -> '%s'.",
                            favorite_name,
                            favorite_id,
                            user_name,
                        )
                        self.logger.info(
                            "Simulate user -> '%s' browsing node -> '%s' (%s)...",
                            user_name,
                            favorite_name,
                            favorite_id,
                        )
                        # simulate a browse by the user to populate recently accessed items
                        response = (
                            self._otcs.get_workspace(node_id=favorite_id, fields="properties{id}")
                            if is_workspace
                            else self._otcs.get_node(node_id=favorite_id, fields="properties{id}")
                        )
                # end for favorite in favorites

                # we work through the list of proxies defined for the user
                # (we need to consider that not all users have the proxies element):
                proxies = user.get("proxies", [])

                for proxy in proxies:
                    proxy_user = next(
                        (item for item in self._users if item["name"] == proxy),
                        None,
                    )
                    if not proxy_user or "id" not in proxy_user:
                        self.logger.error(
                            "The proxy -> '%s' for user -> '%s' does not exist! Skipping proxy...",
                            proxy,
                            user_name,
                        )
                        success = False
                        continue
                    proxy_user_id = proxy_user["id"]

                    # Check if the proxy is already set:
                    if not self._otcs.is_proxy(user_name=proxy):
                        self.logger.info(
                            "Set user -> '%s' (%s) as proxy for user -> '%s'.",
                            proxy,
                            proxy_user_id,
                            user_name,
                        )
                        # set the user proxy - currently we don't support time based proxies in payload.
                        # The called method is ready to support this.
                        response = self._otcs.add_user_proxy(proxy_user_id=proxy_user_id)
                    else:
                        self.logger.info(
                            "User -> '%s' (%s) is already proxy for user -> '%s'. Skipping...",
                            proxy,
                            proxy_user_id,
                            user_name,
                        )
            # end for user in self._users

        if self._users:
            # Impersonate as the admin user:
            self.logger.info(
                "Impersonate as admin user -> '%s'...",
                self._otcs.credentials()["username"],
            )
            # Stop the impersonation as a user:
            result = self.stop_impersonation()
            if not result:
                self.logger.error(
                    "Couldn't impersonate as admin user -> '%s'!",
                    self._otcs.credentials()["username"],
                )
                success = False

        # Also for the admin user we want to update the user profile to activate responsive (dynamic) containers:
        response = self._otcs.update_user_profile(
            field="responsiveContainerMode",
            value=True,
            config_section="SmartUI",
        )
        if response is None:
            self.logger.warning(
                "Profile for 'admin' user couldn't be updated with responsive container mode!",
            )
        else:
            self.logger.info(
                "Profile for 'admin' user has been updated to enable responsive container mode.",
            )
        response = self._otcs.update_user_profile(
            field="responsiveContainerMessageMode",
            value=True,
            config_section="SmartUI",
        )
        if response is None:
            self.logger.warning(
                "Profile for 'admin' user couldn't be updated with responsive container message mode!",
            )
        else:
            self.logger.info(
                "Profile for 'admin' user has been updated to enable messages for responsive container mode.",
            )

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._users,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_security_clearances")
    def process_security_clearances(
        self,
        section_name: str = "securityClearances",
    ) -> bool:
        """Process Security Clearances for Content Server.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._security_clearances:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for security_clearance in self._security_clearances:
            clearance_level = security_clearance.get("level")
            if not clearance_level:
                self.logger.error(
                    "Security clearance requires a level in the payload. Skipping.",
                )
                continue
            clearance_name = security_clearance.get("name")
            if not clearance_name:
                self.logger.error(
                    "Security clearance requires a name in the payload. Skipping.",
                )
                continue

            # Check if security clearance has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not security_clearance.get("enabled", True):
                self.logger.info(
                    "Payload for security clearance -> '%s' is disabled. Skipping...",
                    clearance_name,
                )
                continue

            clearance_description = security_clearance.get("description", "")
            if clearance_level and clearance_name:
                self.logger.info(
                    "Creating security clearance -> '%s' : %s%s...",
                    clearance_level,
                    clearance_name,
                    " ({})".format(clearance_description) if clearance_description else "",
                )
                self._otcs.run_web_report(
                    nickname="web_report_security_clearance",
                    web_report_parameters=security_clearance,
                )
            else:
                self.logger.error(
                    "Cannot create security clearance - either level or name is missing!",
                )
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._security_clearances,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_supplemental_markings")
    def process_supplemental_markings(
        self,
        section_name: str = "supplementalMarkings",
    ) -> bool:
        """Process Supplemental Markings for Content Server.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._supplemental_markings:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for supplemental_marking in self._supplemental_markings:
            code = supplemental_marking.get("code")

            # Check if supplemental marking has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not supplemental_marking.get("enabled", True):
                self.logger.info(
                    "Payload for supplemental marking -> '%s' is disabled. Skipping...",
                    code,
                )
                continue

            description = supplemental_marking.get("description", "")

            if code:
                self.logger.info(
                    "Creating supplemental marking -> '%s'%s...",
                    code,
                    " ({})".format(description) if description else "",
                )
                self._otcs.run_web_report(
                    nickname="web_report_supplemental_marking",
                    web_report_parameters=supplemental_marking,
                )
            else:
                self.logger.error(
                    "Cannot create supplemental marking - either code or description is missing!",
                )
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._supplemental_markings,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_security")
    def process_user_security(self, section_name: str = "userSecurity") -> bool:
        """Process Security Clearance and Supplemental Markings for Content Server users.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._users:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for user in self._users:
            user_name = user.get("name")

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_name,
                )
                continue

            user_id = user.get("id")
            if not user_id:
                self.logger.error(
                    "User is missing an ID. It was likely not created. Skipping to next user...",
                )
                success = False
                continue

            # Read security clearance from user payload (it is optional!)
            user_security_clearance = user.get("security_clearance")
            if user_id and user_security_clearance:
                self._otcs.assign_user_security_clearance(
                    user_id=user_id,
                    security_clearance=user_security_clearance,
                )

            # Read supplemental markings from user payload (it is optional!)
            user_supplemental_markings = user.get("supplemental_markings")
            if user_id and user_supplemental_markings:
                self._otcs.assign_user_supplemental_markings(
                    user_id=user_id,
                    supplemental_markings=user_supplemental_markings,
                )

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._users,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_records_management_settings")
    def process_records_management_settings(
        self,
        section_name: str = "recordsManagementSettings",
    ) -> bool:
        """Process Records Management Settings for Content Server.

        The setting files need to be placed in the OTCS file system file via
        a transport into the Support Asset Volume.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._records_management_settings:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        if (
            "records_management_system_settings" in self._records_management_settings
            and self._records_management_settings["records_management_system_settings"] != ""
        ):
            filename = (
                self._custom_settings_dir + self._records_management_settings["records_management_system_settings"]
            )
            response = self._otcs.import_records_management_settings(file_path=filename)
            if not response:
                success = False

        if (
            "records_management_codes" in self._records_management_settings
            and self._records_management_settings["records_management_codes"] != ""
        ):
            filename = self._custom_settings_dir + self._records_management_settings["records_management_codes"]
            response = self._otcs.import_records_management_codes(file_path=filename)
            if not response:
                success = False

        if (
            "records_management_rsis" in self._records_management_settings
            and self._records_management_settings["records_management_rsis"] != ""
        ):
            filename = self._custom_settings_dir + self._records_management_settings["records_management_rsis"]
            response = self._otcs.import_records_management_rsis(file_path=filename)
            if not response:
                success = False

        if (
            "physical_objects_system_settings" in self._records_management_settings
            and self._records_management_settings["physical_objects_system_settings"] != ""
        ):
            filename = self._custom_settings_dir + self._records_management_settings["physical_objects_system_settings"]
            response = self._otcs.import_physical_objects_settings(file_path=filename)
            if not response:
                success = False

        if (
            "physical_objects_codes" in self._records_management_settings
            and self._records_management_settings["physical_objects_codes"] != ""
        ):
            filename = self._custom_settings_dir + self._records_management_settings["physical_objects_codes"]
            response = self._otcs.import_physical_objects_codes(file_path=filename)
            if not response:
                success = False

        if (
            "physical_objects_locators" in self._records_management_settings
            and self._records_management_settings["physical_objects_locators"] != ""
        ):
            filename = self._custom_settings_dir + self._records_management_settings["physical_objects_locators"]
            response = self._otcs.import_physical_objects_locators(file_path=filename)
            if not response:
                success = False

        if (
            "security_clearance_codes" in self._records_management_settings
            and self._records_management_settings["security_clearance_codes"] != ""
        ):
            filename = self._custom_settings_dir + self._records_management_settings["security_clearance_codes"]
            response = self._otcs.import_security_clearance_codes(file_path=filename)
            if not response:
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._records_management_settings,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_holds")
    def process_holds(self, section_name: str = "holds") -> bool:
        """Process Records Management Holds.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._holds:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for hold in self._holds:
            hold_name = hold.get("name")
            if not hold_name:
                self.logger.error("Cannot create a hold without a name! Skipping...")
                continue

            hold_type = hold.get("type")
            if not hold_type:
                self.logger.error(
                    "Cannot create hold -> '%s' without a type! Skipping...",
                    hold_name,
                )
                success = False
                continue

            # Check if hold has been explicitly disabled in payload
            # (enabled = false). In this case we skip this payload element:
            if not hold.get("enabled", True):
                self.logger.info(
                    "Payload for hold -> '%s' (%s) is disabled. Skipping...",
                    hold_name,
                    hold_type,
                )
                continue

            hold_group = hold.get("group")
            hold_comment = hold.get("comment", "")
            hold_alternate_id = hold.get("alternate_id")
            hold_date_applied = hold.get("date_applied")
            hold_date_to_remove = hold.get("date_to_remove")

            response = self._otcs.get_node_by_volume_and_path(
                volume_type=self._otcs.VOLUME_TYPE_RECORDS_MANAGEMENT,
                path=["Hold Maintenance"],
            )
            if not response:
                self.logger.error("Cannot find 'Records Management' volume!")
                continue
            holds_maintenance_id = self._otcs.get_result_value(
                response=response,
                key="id",
            )
            if not holds_maintenance_id:
                self.logger.error(
                    "Cannot find 'Holds Maintenance' folder in 'Records Management' volume!",
                )
                continue

            if hold_group:
                # Check if the Hold Group (folder) does already exist.
                response = self._otcs.get_node_by_parent_and_name(
                    parent_id=holds_maintenance_id,
                    name=hold_group,
                )
                parent_id = self._otcs.get_result_value(response=response, key="id")
                if not parent_id:
                    response = self._otcs.create_item(
                        parent_id=holds_maintenance_id,
                        item_type=self._otcs.ITEM_TYPE_HOLD,
                        item_name=hold_group,
                    )
                    parent_id = self._otcs.get_result_value(response=response, key="id")
                    if not parent_id:
                        self.logger.error(
                            "Failed to create hold group -> '%s'!",
                            hold_group,
                        )
                        continue
            else:
                parent_id = holds_maintenance_id

            # Holds are special - they ahve folders that cannot be traversed
            # in the normal way - we need to get the whole list of holds and use
            # specialparameters for the exist_result_items() method as the REST
            # API calls delivers a results->data->holds structure (not properties)
            response = self._otcs.get_records_management_holds()
            if self._otcs.exist_result_item(
                response=response,
                key="HoldName",
                value=hold_name,
                property_name="holds",
            ):
                self.logger.info(
                    "Hold -> '%s' does already exist. Skipping...",
                    hold_name,
                )
                continue

            hold = self._otcs.create_records_management_hold(
                hold_type=hold_type,
                name=hold_name,
                comment=hold_comment,
                alternate_id=hold_alternate_id,
                parent_id=int(parent_id),
                date_applied=hold_date_applied,
                date_to_remove=hold_date_to_remove,
            )

            if hold and hold["holdID"]:
                self.logger.info(
                    "Successfully created hold -> '%s' with ID -> %s.",
                    hold_name,
                    hold["holdID"],
                )
            else:
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._holds,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_additional_group_members")
    def process_additional_group_members(
        self,
        section_name: str = "additionalGroupMemberships",
    ) -> bool:
        """Process additional groups memberships we want to have in OTDS.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._additional_group_members:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for additional_group_member in self._additional_group_members:
            parent_group = additional_group_member.get("parent_group")
            if not parent_group:
                self.logger.error("Missing parent group! Skipping...")
                continue

            # Check if additional group member has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not additional_group_member.get("enabled", True):
                self.logger.info(
                    "Payload for additional group member with parent group -> '%s' is disabled. Skipping...",
                    parent_group,
                )
                continue

            if ("user_name" not in additional_group_member) and ("group_name" not in additional_group_member):
                self.logger.error(
                    "Either group name or user name need to be specified! Skipping additional group members...",
                )
                success = False
                continue
            if "group_name" in additional_group_member:
                group_name = additional_group_member["group_name"]
                self.logger.info(
                    "Adding group -> '%s' to parent group -> '%s' in OTDS.",
                    group_name,
                    parent_group,
                )
                response = self._otds.add_group_to_parent_group(
                    group=group_name,
                    parent_group=parent_group,
                )
                if not response:
                    self.logger.error(
                        "Failed to add group -> '%s' to parent group -> '%s' in OTDS!",
                        group_name,
                        parent_group,
                    )
                    success = False
            elif "user_name" in additional_group_member:
                user_name = additional_group_member["user_name"]
                self.logger.info(
                    "Adding user -> '%s' to group -> '%s' in OTDS.",
                    user_name,
                    parent_group,
                )
                response = self._otds.add_user_to_group(
                    user=user_name,
                    group=parent_group,
                )
                if not response:
                    self.logger.error(
                        "Failed to add user -> '%s' to group -> '%s' in OTDS!",
                        user_name,
                        parent_group,
                    )
                    success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._additional_group_members,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(
        attributes=OTEL_TRACING_ATTRIBUTES, name="process_additional_application_role_assignments"
    )
    def process_additional_application_role_assignments(
        self,
        section_name: str = "additionalApplicationRoleAssignments",
    ) -> bool:
        """Process additional application role assignments we want to have in OTDS.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._additional_application_role_assignments:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for additional_role_assignment in self._additional_application_role_assignments:
            role = additional_role_assignment.get("role_name")
            role_parts = role.split("@", 1)
            role_name = role_parts[0]
            role_partition = role_parts[1] if len(role_parts) > 1 else "OAuthClients"

            if not role_name:
                self.logger.error(
                    "Missing application role! Skipping additional role members...",
                )
                continue

            # Check if additional access role member has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not additional_role_assignment.get("enabled", True):
                self.logger.info(
                    "Payload for additional assignment for application role -> '%s' is disabled. Skipping...",
                    role_name,
                )
                continue

            if ("user_name" not in additional_role_assignment) and ("group_name" not in additional_role_assignment):
                self.logger.error(
                    "Either group_name or user_name need to be specified for application role assignment! Skipping...",
                )
                success = False
                continue

            if "group_name" in additional_role_assignment:
                group = additional_role_assignment["group_name"]
                group_parts = group.split("@", 1)
                group_name = group_parts[0]
                group_partition = group_parts[1] if len(group_parts) > 1 else self._otcs.config()["partition"]

                self.logger.info(
                    "Adding group -> '%s' in partition -> '%s' to application role -> '%s' in partition -> '%s'.",
                    group_name,
                    group_partition,
                    role_name,
                    role_partition,
                )

                response = self._otds.assign_group_to_application_role(
                    group_id=group_name,
                    group_partition=group_partition,
                    role_name=role_name,
                    role_partition=role_partition,
                )

                if not response:
                    self.logger.error(
                        "Failed to assign group -> '%s' in partition -> '%s' to application role -> '%s' in partition -> '%s'!",
                        group_name,
                        group_partition,
                        role_name,
                        role_partition,
                    )
                    success = False

            elif "user_name" in additional_role_assignment:
                user = additional_role_assignment["user_name"]
                user_parts = user.split("@", 1)
                user_name = user_parts[0]
                user_partition = user_parts[1] if len(user_parts) > 1 else self._otcs.config()["partition"]

                self.logger.info(
                    "Adding user -> '%s' in partition -> '%s' to application role -> '%s' in partition -> '%s'!",
                    user_name,
                    user_partition,
                    role_name,
                    role_partition,
                )

                response = self._otds.assign_user_to_application_role(
                    user_id=user_name,
                    user_partition=user_partition,
                    role_name=role_name,
                    role_partition=role_partition,
                )
                if not response:
                    self.logger.error(
                        "Failed to assign user -> '%s' in partition -> '%s' to application role -> '%s' in partition -> '%s'!",
                        user_name,
                        user_partition,
                        role_name,
                        role_partition,
                    )
                    success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._additional_application_role_assignments,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_additional_access_role_members")
    def process_additional_access_role_members(
        self,
        section_name: str = "additionalAccessRoleMemberships",
    ) -> bool:
        """Process additional access role memberships we want to have in OTDS.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._additional_access_role_members:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for additional_access_role_member in self._additional_access_role_members:
            access_role = additional_access_role_member.get("access_role")
            if not access_role:
                self.logger.error(
                    "Missing access role! Skipping additional role members...",
                )
                continue

            # Check if additional access role member has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not additional_access_role_member.get("enabled", True):
                self.logger.info(
                    "Payload for additional member for access role -> '%s' is disabled. Skipping...",
                    access_role,
                )
                continue

            if (
                ("user_name" not in additional_access_role_member)
                and ("group_name" not in additional_access_role_member)
                and ("partition_name" not in additional_access_role_member)
            ):
                self.logger.error(
                    "Either group_name or user_name need to be specified! Skipping...",
                )
                success = False
                continue
            if "group_name" in additional_access_role_member:
                group_name = additional_access_role_member["group_name"]
                self.logger.info(
                    "Adding group -> '%s' to access role -> '%s' in OTDS.",
                    group_name,
                    access_role,
                )
                response = self._otds.add_group_to_access_role(
                    access_role=access_role,
                    group=group_name,
                )
                if not response:
                    self.logger.error(
                        "Failed to add group -> '%s' to access role -> '%s' in OTDS!",
                        group_name,
                        access_role,
                    )
                    success = False
            elif "user_name" in additional_access_role_member:
                user_name = additional_access_role_member["user_name"]
                self.logger.info(
                    "Adding user -> '%s' to access role -> '%s' in OTDS.",
                    user_name,
                    access_role,
                )
                response = self._otds.add_user_to_access_role(
                    access_role=access_role,
                    user_id=user_name,
                )
                if not response:
                    self.logger.error(
                        "Failed to add user -> '%s' to access role -> '%s' in OTDS!",
                        user_name,
                        access_role,
                    )
                    success = False
            elif "partition_name" in additional_access_role_member:
                partition_name = additional_access_role_member["partition_name"]
                self.logger.info(
                    "Adding partition -> '%s' to access role -> '%s' in OTDS.",
                    partition_name,
                    access_role,
                )
                response = self._otds.add_partition_to_access_role(
                    access_role=access_role,
                    partition=partition_name,
                )
                if not response:
                    self.logger.error(
                        "Failed to add partition -> '%s' to access role -> '%s' in OTDS!",
                        partition_name,
                        access_role,
                    )
                    success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._additional_access_role_members,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_renamings")
    def process_renamings(self, section_name: str = "renamings") -> bool:
        """Process renamings specified in payload and rename existing Content Server items.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._renamings:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for renaming in self._renamings:
            if "name" not in renaming:
                self.logger.error("Renamings require the new name!")
                continue
            if "nodeid" in renaming:
                node_id = renaming["nodeid"]
            elif "volume" in renaming:
                path = renaming.get("path")
                volume = renaming.get("volume")
                if path:
                    self.logger.info(
                        "Found path -> '%s' in renaming payload. Determine node ID by volume and path...",
                        path,
                    )
                    node = self._otcs.get_node_by_volume_and_path(
                        volume_type=volume,
                        path=path,
                    )
                else:
                    # Determine object ID of volume:
                    node = self._otcs.get_volume(volume_type=volume)
                node_id = self._otcs.get_result_value(response=node, key="id")
            elif "nickname" in renaming:
                nickname = renaming["nickname"]
                self.logger.info(
                    "Found nickname -> '%s' in renaming payload. Determine node ID by nickname...",
                    nickname,
                )
                node = self._otcs.get_node_from_nickname(nickname=nickname)
                node_id = self._otcs.get_result_value(response=node, key="id")
            else:
                self.logger.error(
                    "Renamings require either a node ID or a volume (with an optional path) or a nickname! Skipping to next renaming...",
                )
                continue

            # Check if renaming has been explicitly disabled in payload
            # (enabled = false). In this case we skip this payload element:
            if not renaming.get("enabled", True):
                self.logger.info("Payload for renaming is disabled. Skipping...")
                continue

            response = self._otcs.rename_node(
                node_id=int(node_id),
                name=renaming["name"],
                description=renaming.get("description", ""),
            )
            if not response:
                self.logger.error(
                    "Failed to rename node ID -> '%s' to new name -> '%s'!",
                    node_id,
                    renaming["name"],
                )
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._renamings,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_items")
    def process_items(self, items: list, section_name: str = "items") -> bool:
        """Process items specified in payload and create them in Content Server.

        Args:
            items (list):
                List of items to create (need this as parameter as we
                have multiple lists).
                Each list item in the payload is a dict with this structure:
                {
                    enabled = "..."
                    name = "..."
                    description = "..."
                    nickname = "..."
                    parent_nickname = "..."
                    parent_path = [...]
                    parent_volume = ...
                    original_nickname = "..."
                    original_path = []
                    type = ...
                    url = "..."
                    details = {
                        "scheduledbotdetails" : ...
                    } # additional parameters
                }
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not items:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )

            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for item in items:
            item_name = item.get("name")
            if not item_name:
                self.logger.error("Item needs a name. Skipping...")
                continue

            # Check if element has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not item.get("enabled", True):
                self.logger.info(
                    "Payload for item -> '%s' is disabled. Skipping...",
                    item_name,
                )
                continue

            item_description = item.get("description", "")
            item_nickname = item.get("nickname", None)
            parent_nickname = item.get("parent_nickname", None)
            parent_path = item.get("parent_path", None)

            if parent_nickname:  # parent nickname has preference over parent path
                parent_node = self._otcs.get_node_from_nickname(
                    nickname=parent_nickname,
                )
                parent_id = self._otcs.get_result_value(response=parent_node, key="id")
                if not parent_id:
                    self.logger.error(
                        "Item -> '%s' has a parent nickname -> '%s' that does not exist. Skipping...",
                        item_name,
                        parent_nickname,
                    )
                    success = False
                    continue
            elif parent_path is not None:  # parent_path can be [] which is valid for top-level items!
                parent_volume = item.get("parent_volume", self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE)
                parent_node = self._otcs.get_node_by_volume_and_path(
                    volume_type=parent_volume,
                    path=parent_path,
                    create_path=True,
                )
                parent_id = self._otcs.get_result_value(response=parent_node, key="id")
                if not parent_id:
                    # if not parent_node:
                    self.logger.error(
                        "Item -> '%s' has a parent path -> %s that does not exist and couldn't be created in volume -> %d. Skipping...",
                        item_name,
                        parent_path,
                        self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE,
                    )
                    success = False
                    continue
            else:
                self.logger.error("The parent for the item -> '%s' is not specified by nickname nor path!", item_name)
                success = False
                continue

            # Handling for shortcut items that have an orginal node:
            original_nickname = item.get("original_nickname")
            original_path = item.get("original_path")

            if original_nickname:
                original_node = self._otcs.get_node_from_nickname(
                    nickname=original_nickname,
                )
                original_id = self._otcs.get_result_value(
                    response=original_node,
                    key="id",
                )
                if not original_id:
                    # if not original_node:
                    self.logger.error(
                        "Item -> '%s' has a original nickname -> '%s' that does not exist. Skipping...",
                        item_name,
                        original_nickname,
                    )
                    success = False
                    continue
            elif original_path is not None:  # original_path can be [] which is valid for top-level items!
                original_volume = item.get("original_volume", self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE)
                original_node = self._otcs.get_node_by_volume_and_path(
                    volume_type=original_volume,
                    path=original_path,
                )
                original_id = self._otcs.get_result_value(
                    response=original_node,
                    key="id",
                )
                if not original_id:
                    # if not original_node:
                    self.logger.error(
                        "Item -> '%s' has a original path that does not exist. Skipping...",
                        item_name,
                    )
                    success = False
                    continue
            else:
                original_id = 0

            if "type" not in item:
                self.logger.error("Item -> '%s' needs a type. Skipping...", item_name)
                success = False
                continue

            item_type = int(item.get("type", self._otcs.ITEM_TYPE_FOLDER))
            item_url = item.get("url", "")
            item_details = item.get("details", {})
            create_item_details = item.get("create_details", {})
            # check that we have the required information
            # for the given item type:
            match item_type:
                case self._otcs.ITEM_TYPE_URL:  # URL
                    if item_url == "":
                        self.logger.error(
                            "Item -> '%s' has type URL but the URL is not in the payload. Skipping...",
                            item_name,
                        )
                        success = False
                        continue
                case self._otcs.ITEM_TYPE_SHORTCUT:  # Shortcut
                    if not original_id:
                        self.logger.error(
                            "Item -> '%s' has type Shortcut but the original item is not in the payload. Skipping...",
                            item_name,
                        )
                        success = False
                        continue
                case self._otcs.ITEM_TYPE_COLLECTION:  # Collection
                    item_ids = item.get("ids", None)
                    if item_ids is None:
                        self.logger.error(
                            "Item -> '%s' has type Collection but the list of collected items is not provided in payload. Skipping...",
                            item_name,
                        )
                        success = False
                        continue
                case self._otcs.ITEM_TYPE_SCHEDULED_BOT:  # Scheduled Bots
                    if any(
                        key not in item_details for key in ["xecmpfJobType", "scheduledbotdetails"]
                    ):  #  not in item_details:
                        self.logger.error(
                            "Item -> '%s' has type Scheduled Bot but the mandatory details are not provided in payload (xecmpfJobType, scheduledbotdetails). Skipping...",
                            item_name,
                        )
                        success = False
                        continue

                    create_item_details["xecmpfJobType"] = item_details["xecmpfJobType"]

            # Check if an item with the same name does already exist.
            # This can also be the case if the python container runs a 2nd time.
            # For this reason we are also not issuing an error but just an info (False):
            response = self._otcs.get_node_by_parent_and_name(
                parent_id=int(parent_id),
                name=item_name,
                show_error=False,
            )
            if self._otcs.get_result_value(response=response, key="name") == item_name:
                self.logger.info(
                    "Item with name -> '%s' does already exist in parent folder with ID -> %s.",
                    item_name,
                    parent_id,
                )
                continue
            response = self._otcs.create_item(
                parent_id=int(parent_id),
                item_type=item_type,
                item_name=item_name,
                item_description=item_description,
                url=item_url,
                original_id=int(original_id),
                **create_item_details,
            )
            node_id = self._otcs.get_result_value(response=response, key="id")
            if not node_id:
                self.logger.error(
                    "Failed to create item -> '%s' under parent%s!",
                    item_name,
                    " with nickname -> '{}'".format(parent_nickname)
                    if parent_nickname
                    else " path -> {} in volume -> {}".format(parent_path, parent_volume),
                )
                success = False
                continue

            self.logger.info(
                "Successfully created item -> '%s' with ID -> %s under parent%s.",
                item_name,
                node_id,
                " with nickname -> '{}'".format(parent_nickname)
                if parent_nickname
                else " path -> {} in volume -> {}".format(parent_path, parent_volume),
            )

            # Special handling for scheduled bot items:
            if item_type == self._otcs.ITEM_TYPE_SCHEDULED_BOT:
                scheduled_bot_details = item_details.get("scheduledbotdetails", {})
                if not scheduled_bot_details:
                    self.logger.error("Failed to get details of scheduled bot item -> '%s'!", item_name)
                    success = False
                    continue
                start_mode = scheduled_bot_details.get("startmodus")
                if not start_mode:
                    self.logger.error("Failed to get start mode of scheduled bot item -> '%s'!", item_name)
                    success = False
                    continue
                start_mode = start_mode.get("startMode")
                if not start_mode:
                    self.logger.error("Failed to get start mode of scheduled bot item -> '%s'!", item_name)
                    success = False
                    continue
                old_schedule_data = scheduled_bot_details.get("oldscheduleData")
                # Check if this bot should start after another bot:
                if start_mode == "AfterJob":
                    after_job_nickname = scheduled_bot_details["startmodus"].get("afterJob")
                    if not after_job_nickname:
                        after_job_nickname = item_details.get("xecmpfAfterJobDataId")
                    self.logger.info(
                        "Scheduled bot item -> '%s' starts after another scheduled bot with nickname -> '%s'. Resolving nickname...",
                        item_name,
                        after_job_nickname,
                    )
                    # Get the Scheduled Bot node that this bot depends on:
                    response = self._otcs.get_node_from_nickname(nickname=after_job_nickname)
                    after_job_id = self._otcs.get_result_value(
                        response=response,
                        key="id",
                    )
                    # Update the bot details - changing the name coming from the
                    # payload to the actual node ID (both in details and in 'old' details):
                    scheduled_bot_details["startmodus"]["afterJob"] = after_job_id
                    old_schedule_data["afterJob"] = after_job_id

                    item_details["xecmpfAfterJobDataId"] = after_job_id
                # Check if this bot should start based on a schedule.
                # Then we need to configure the agent to run it:
                elif str(start_mode) == "7558":  # 7558 is NOT a object ID but an internal agent type ID!
                    self.logger.info(
                        "Scheduled bot item -> '%s' starts based on a schedule. Setting the agent ID to '7558'...",
                        item_name,
                    )
                    # Make sure the agent ID is configured:
                    item_details["xecmpfAgentId"] = 7558
                self.logger.debug("Scheduled bot details -> %s", str(scheduled_bot_details))

                # The REST API requires to have the scheduled bot details wrapped into a JSON structure:
                item_details["scheduledbotdetails"] = json.dumps(item_details.get("scheduledbotdetails", {}))

                response = self._otcs.update_item(node_id=node_id, body=False, **item_details)
                if not response:
                    self.logger.error("Failed to update scheduled bot item -> '%s'!", item_name)
                    success = False
                    continue

                # Check if we want to execute an action immediately after creation, like "Runnow":
                actions = item_details.get("actions", [])
                for action in actions:
                    self.logger.info("Execute action -> '%s' for scheduled bot -> '%s'...", action, item_name)
                    response = self._otcs.update_item(node_id=node_id, body=False, actionName=action)
                    if not response:
                        self.logger.error(
                            "Failed to execute action -> '%s' for scheduled bot item -> '%s'!", action, item_name
                        )
                        success = False
                        continue
                if not actions:
                    self.logger.info("No immediate actions specified for scheduled bot -> '%s'.", item_name)

            # end if item_type == self._otcs.ITEM_TYPE_SCHEDULED_BOT:

            # Special handling for collection items:
            elif item_type == self._otcs.ITEM_TYPE_COLLECTION:
                item_node_ids = []
                for collection_item in item_ids or []:
                    # First see if the collection item is a workspace we can
                    # lookup in the payload by its logical payload ID
                    member_item = next(
                        (item for item in self._workspaces if item["id"] == collection_item),
                        None,
                    )
                    if member_item:
                        if member_item.get("enabled", True):
                            self.logger.info(
                                "Found collection item (workspace) -> '%s' in payload and it is enabled.",
                                member_item["name"],
                            )
                        else:
                            self.logger.info(
                                "Found collection item (workspace) -> '%s' in payload but it is not enabled. Skipping...",
                                member_item["name"],
                            )
                            continue
                        member_id = self.determine_workspace_id(workspace=member_item)
                        if not member_id:
                            self.logger.warning(
                                "Workspace of type -> '%s' and name -> '%s' does not exist. Cannot create collection item. Skipping...",
                                member_item["type_name"],
                                member_item["name"],
                            )
                            continue
                    else:
                        # alternatively try to find the item as a nickname:
                        member_item = self._otcs.get_node_from_nickname(
                            nickname=collection_item,
                        )
                        member_id = self._otcs.get_result_value(
                            response=member_item,
                            key="id",
                        )
                        if not member_id:
                            self.logger.warning(
                                "Item -> '%s' does not exist. Cannot create collection item. Skipping...",
                                member_item["name"],
                            )
                            continue
                    item_node_ids.append(member_id)
                # end for collection_item in item_ids
                if item_node_ids:
                    response = self._otcs.add_node_to_collection(
                        collection_id=node_id,
                        node_ids=item_node_ids,
                    )
            # end if item_type == self._otcs.ITEM_TYPE_COLLECTION

            # Do we have a nickname for the item in the payload? Then assign it:
            if item_nickname:
                self.logger.info("Assign nickname -> '%s' to item -> '%s' (%s)...", item_nickname, item_name, node_id)
                self._otcs.set_node_nickname(node_id=node_id, nickname=item_nickname)
        # end for item in items:

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=items,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_permission")
    def process_permission(
        self,
        node_id: int,
        node_name: str,
        permission: dict,
        apply_to: int,
        workspace_id: int | None = None,
    ) -> bool:
        """Process a single permission payload item for a given node.

        Args:
            node_id (int):
                The ID of the node.
            node_name (str):
                The name of the node.
            permission (dict):
                The permission payload.
            apply_to (int):
                - 0 = Apply to this item only
                - 1 = Apply to sub-items only
                - 2 = Apply to this item and its sub-items (default)
                - 3 = Apply to this item and its immediate sub-items
            workspace_id (int | None):
                If role permissions should be set we also need the workspace_id.
                Use None if node is not part of a workspace or no role permissions
                should be set.

        Returns:
            bool:
                True = success, False = Error

        """

        # 1. Process Owner Permissions
        if "owner_permissions" in permission:
            owner_permissions = permission["owner_permissions"]
            self.logger.info(
                "Update owner permissions for item -> '%s' (%d) to -> %s.",
                node_name,
                node_id,
                str(owner_permissions),
            )
            response = self._otcs.assign_permission(
                node_id=int(node_id),
                assignee_type="owner",
                assignee=0,
                permissions=owner_permissions,
                apply_to=apply_to,
            )
            if not response:
                self.logger.error(
                    "Failed to update owner permissions for item -> '%s' (%d)!",
                    node_name,
                    node_id,
                )
                return False

        # 2. Process Owner Group Permissions
        if "owner_group_permissions" in permission:
            owner_group_permissions = permission["owner_group_permissions"]
            self.logger.info(
                "Update owner group permissions for item -> '%s' (%d) to -> %s.",
                node_name,
                node_id,
                str(owner_group_permissions),
            )
            response = self._otcs.assign_permission(
                node_id=int(node_id),
                assignee_type="group",
                assignee=0,
                permissions=owner_group_permissions,
                apply_to=apply_to,
            )
            if not response:
                self.logger.error(
                    "Failed to update group permissions for item -> '%s' (%d)!",
                    node_name,
                    node_id,
                )
                return False

        # 3. Process Public Permissions
        if "public_permissions" in permission:
            public_permissions = permission["public_permissions"]
            self.logger.info(
                "Update public permissions for item -> '%s' (%d) to -> %s.",
                node_name,
                node_id,
                str(public_permissions),
            )
            response = self._otcs.assign_permission(
                node_id=int(node_id),
                assignee_type="public",
                assignee=0,
                permissions=public_permissions,
                apply_to=apply_to,
            )
            if not response:
                self.logger.error(
                    "Failed to update public permissions for item -> '%s' (%d)!",
                    node_name,
                    node_id,
                )
                return False
        # end if "public_permissions" in permission

        # 4. Process Assigned User Permissions (if specified and not empty)
        users = permission.get("users", [])
        for user in users:
            if "name" not in user or "permissions" not in user:
                self.logger.error(
                    "Missing user name in user permission specificiation. Cannot set user permissions. Skipping...",
                )
                return False
            user_name = user["name"]
            if "permissions" not in user:
                self.logger.error(
                    "Missing permissions in user -> '%s' permission specificiation. Cannot set user permissions. Skipping...",
                    user_name,
                )
                return False
            user_permissions = user["permissions"]
            response = self._otcs.get_user(name=user_name, show_error=True)
            user_id = self._otcs.get_result_value(response=response, key="id")
            if not user_id:
                self.logger.error(
                    "Cannot find user -> '%s'; cannot set user permissions. Skipping user...",
                    user_name,
                )
                return False
            user["id"] = user_id  # write ID back into payload

            self.logger.info(
                "Update permission of user -> '%s' for item -> '%s' (%d) to -> %s.",
                user_name,
                node_name,
                node_id,
                str(user_permissions),
            )
            response = self._otcs.assign_permission(
                node_id=int(node_id),
                assignee_type="custom",
                assignee=user_id,
                permissions=user_permissions,
                apply_to=apply_to,
            )
            if not response:
                self.logger.error(
                    "Failed to update assigned user permissions for item -> %s!",
                    node_id,
                )
                return False
        # end for user in users

        # 5. Process Assigned Group Permissions (if specified and not empty)
        groups = permission.get("groups", [])
        for group in groups:
            if "name" not in group:
                self.logger.error(
                    "Missing group name in group permission specificiation. Cannot set group permissions. Skipping...",
                )
                return False
            group_name = group["name"]
            if "permissions" not in group:
                self.logger.error(
                    "Missing permissions in group -> '%s' permission specificiation. Cannot set group permissions. Skipping...",
                    group_name,
                )
                return False
            group_permissions = group["permissions"]
            self.logger.info(
                "Update permissions of group -> '%s' for item -> '%s' (%d) to -> %s.",
                group_name,
                node_name,
                node_id,
                str(group_permissions),
            )
            otcs_group = self._otcs.get_group(name=group_name, show_error=True)
            group_id = self._otcs.get_result_value(response=otcs_group, key="id")
            if not group_id:
                self.logger.error(
                    "Cannot find group -> '%s'; cannot set group permissions. Skipping group...",
                    group_name,
                )
                return False
            group["id"] = group_id  # write ID back into payload
            response = self._otcs.assign_permission(
                node_id=int(node_id),
                assignee_type="custom",
                assignee=group_id,
                permissions=group_permissions,
                apply_to=apply_to,
            )
            if not response:
                self.logger.error(
                    "Failed to update assigned group permissions for item -> '%s' (%d)!",
                    node_name,
                    node_id,
                )
                return False
        # end for group in groups

        # 6. Process Workspace Role Permissions (if specified and not empty)
        roles = permission.get("roles", [])
        if roles and not workspace_id:
            self.logger.error(
                "Cannot set workspace roles if no workspace ID is provided!",
            )
            return False
        for role in roles:
            if "name" not in role:
                self.logger.error(
                    "Missing role name in role permission specificiation. Cannot set role permissions. Skipping...",
                )
                return False
            role_name = role["name"]
            if "permissions" not in role:
                self.logger.error(
                    "Missing permissions in role -> '%s' permission specificiation. Cannot set role permissions. Skipping...",
                    group_name,
                )
                return False
            role_permissions = role["permissions"]
            self.logger.info(
                "Update permissions of role -> '%s' for workspace item -> '%s' (%d) in workspace with ID -> %d to -> %s.",
                role_name,
                node_name,
                node_id,
                workspace_id,
                str(role_permissions),
            )
            response = self._otcs.get_workspace_roles(workspace_id=workspace_id)
            role_id = self._otcs.lookup_result_value(
                response=response,
                key="name",
                value=role_name,
                return_key="id",
            )
            if not role_id:
                self.logger.error(
                    "Cannot find role -> '%s'; cannot set role permissions.",
                    role_name,
                )
                return False
            response = self._otcs.assign_permission(
                node_id=int(node_id),
                assignee_type="custom",
                assignee=role_id,
                permissions=role_permissions,
                apply_to=apply_to,
            )
            if not response:
                self.logger.error(
                    "Failed to update role permissions for item -> '%s' (%d)!",
                    node_name,
                    node_id,
                )
                return False
        # end for role in roles

        return True

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_permissions")
    def process_permissions(
        self,
        permissions: list,
        section_name: str = "permissions",
    ) -> bool:
        """Process items specified in payload and upadate permissions.

        Args:
            permissions (list):
                A list of items to apply permissions to.
                Each list item in the payload is a dict with this structure:
                {
                    nodeid = "..."
                    volume = "..."
                    nickname = "..."
                    public_permissions = ["see", "see_content", ...]
                    owner_permissions = []
                    owner_group_permissions = []
                    groups = [
                        {
                            name = "..."
                            permissions = []
                        }
                    ]
                    users = [
                        {
                            name = "..."
                            permissions = []
                        }
                    ]
                    apply_to = 2
                }
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not permissions:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for permission in permissions:
            if "path" not in permission and "volume" not in permission and "nickname" not in permission:
                self.logger.error(
                    "Item to change permission is not specified (needs path, volume, or nickname). Skipping...",
                )
                success = False
                continue

            # Check if permission has been explicitly disabled in payload
            # (enabled = false). In this case we skip this payload element:
            if not permission.get("enabled", True):
                self.logger.info("Payload for Permission is disabled. Skipping...")
                continue

            node_id = 0

            # Check if "volume" is in payload and not empty string
            # we try to get the node ID from the volume type:
            if permission.get("volume"):
                volume_type = permission["volume"]
                self.logger.info(
                    "Found volume type -> '%s' in permission payload. Determine volume ID...",
                    volume_type,
                )
                node = self._otcs.get_volume(volume_type=volume_type)
                node_id = self._otcs.get_result_value(response=node, key="id")
                if not node_id:
                    self.logger.error(
                        "Illegal volume -> '%s' in permission payload. Skipping...",
                        volume_type,
                    )
                    success = False
                    continue
            else:
                # the following path block requires a value for the volume - if it is
                # not specified we take the Enterprise Workspace:
                volume_type = self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE

            # Check if "path" is in payload and not empty list
            # (path can be combined with volume so we need to take volume into account):
            if permission.get("path"):
                path = permission["path"]
                self.logger.info(
                    "Found path -> '%s' in permission payload. Determine node ID...",
                    path,
                )
                node = self._otcs.get_node_by_volume_and_path(
                    volume_type=volume_type,
                    path=path,
                )
                node_id = self._otcs.get_result_value(response=node, key="id")
                if not node_id:
                    self.logger.error("Path -> '%s' does not exist. Skipping...", path)
                    success = False
                    continue

            # Check if "nickname" is in payload and not empty string:
            if permission.get("nickname"):
                nickname = permission["nickname"]
                self.logger.info(
                    "Found nickname -> '%s' in permission payload. Determine node ID...",
                    nickname,
                )
                node = self._otcs.get_node_from_nickname(nickname=nickname)
                node_id = self._otcs.get_result_value(response=node, key="id")
                if not node_id:
                    self.logger.error(
                        "Nickname -> '%s' does not exist. Skipping...",
                        nickname,
                    )
                    success = False
                    continue

            # Now we should have a value for node_id:
            if not node_id:
                self.logger.error("No node ID found! Skipping permission...")
                success = False
                continue

            node_name = self._otcs.get_result_value(response=node, key="name")
            self.logger.info(
                "Found node -> '%s' with ID -> %s to apply permission to.",
                node_name,
                node_id,
            )
            # write node information back into payload
            # for better debugging
            permission["node_name"] = node_name
            permission["node_id"] = node_id

            # Make item + sub-items (2) the default:
            apply_to = permission.get("apply_to", 2)

            # Prcess a single permission payload item:
            if not self.process_permission(
                node_id=node_id,
                node_name=node_name,
                permission=permission,
                apply_to=apply_to,
            ):
                success = False
                continue
        # end for permission in permissions

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=permissions,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_permissions")
    def process_workspace_permissions(
        self,
        section_name: str = "workspacePermissions",
    ) -> bool:
        """Process items specified in payload and upadate workspace permissions.

        Args:
            workspace_permissions (list):
                List of items to apply permissions to.
                Each list item in the payload is a dict with this structure:
                {
                    workspace_type = "..."
                    workspace_folder = "..."
                    regex = True
                    public_permissions = ["see", "see_content", ...]
                    owner_permissions = []
                    owner_group_permissions = []
                    groups = [
                        {
                            name = "..."
                            permissions = []
                        }
                    ]
                    users = [
                        {
                            name = "..."
                            permissions = []
                        }
                    ]
                    roles = [
                        {
                            name = "..."
                            permissions = []
                        }
                    ]
                    apply_to = 2
                }
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections like "permissionsPost").
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise

        """

        if not self._workspace_permissions:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for workspace_permission in self._workspace_permissions:
            workspace_type_name = workspace_permission.get("workspace_type")
            if not workspace_type_name:
                self.logger.error(
                    "Workspaces type to change permissions is not specified. Skipping...",
                )
                success = False
                continue

            # Check if workspace permission has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not workspace_permission.get("enabled", True):
                self.logger.info(
                    "Payload for workspace permission for workspace type -> '%s' is disabled. Skipping...",
                    workspace_type_name,
                )
                continue

            workspace_folder_name = workspace_permission.get("workspace_folder", None)
            regex = workspace_permission.get("regex", False)

            # Make item + sub-items (2) the default:
            apply_to = workspace_permission.get("apply_to", 2)

            # Find the workspace type with the name given in the payload:
            workspace_type_id = next(
                (item["id"] for item in self._workspace_types if item["name"] == workspace_type_name),
                None,
            )
            if workspace_type_id is None:
                self.logger.error(
                    "Workspace type -> '%s' not found. Skipping...",
                    workspace_type_name,
                )
                success = False
                continue

            # The workspace type name is used as a "starts with" search on the
            # workspace type name. This can cause issues, so we prefer to use the type ID
            # if we have it. Otherwise the REST API prefers the name over the type:
            workspace_instances = self._otcs.get_workspace_instances_iterator(
                type_name=workspace_type_name if not workspace_type_id else None,
                type_id=workspace_type_id,
            )
            for workspace_instance in workspace_instances:
                workspace = workspace_instance.get("data").get("properties")
                workspace_id = workspace["id"]
                workspace_name = workspace["name"]
                if workspace_folder_name:
                    if not regex:
                        workspace_folder = self._otcs.get_node_by_parent_and_name(
                            parent_id=workspace_id,
                            name=workspace_folder_name,
                        )
                    else:
                        workspace_folder = self._otcs.lookup_node_by_regex(
                            parent_node_id=workspace_id,
                            regex_list=[workspace_folder_name],
                        )
                    workspace_folder_id = self._otcs.get_result_value(
                        response=workspace_folder,
                        key="id",
                    )
                    if not workspace_folder_id:
                        self.logger.info(
                            "Workspace folder name -> '%s' was not found in workspace -> '%s' (%s). Skipping this workspace...",
                            workspace_folder_name,
                            workspace_name,
                            workspace_id,
                        )
                        continue
                else:
                    # if no folder is specified we apply the permission on the workspace
                    workspace_folder_id = workspace_id
                    workspace_folder_name = workspace_name

                # Process a single workspace permission payload item:
                if not self.process_permission(
                    node_id=workspace_folder_id,
                    node_name=workspace_folder_name,
                    permission=workspace_permission,
                    apply_to=apply_to,
                ):
                    success = False
                    continue
            # end for workspace_instance in workspace_instances:

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._workspace_permissions,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_assignments")
    def process_assignments(self, section_name: str = "assignments") -> bool:
        """Process assignments and assign items (such as workspaces and items with nicknames) to users or groups.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise

        """

        if not self._assignments:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for assignment in self._assignments:
            # Sanity check: we need a subject - it's mandatory:
            subject = assignment.get("subject")
            if not subject:
                self.logger.error("Assignment needs a subject! Skipping assignment...")
                success = False
                continue

            # Check if assignment has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not assignment.get("enabled", True):
                self.logger.info(
                    "Payload for assignment -> '%s' is disabled. Skipping...",
                    subject,
                )
                continue

            # instruction is optional but we give a warning if they are missing:
            instruction = assignment.get("instruction", "")
            if not instruction:
                self.logger.warning(
                    "Assignment -> '%s' should have an instruction!",
                    subject,
                )

            # Sanity check: we either need users or groups (or both):
            if "groups" not in assignment and "users" not in assignment:
                self.logger.error(
                    "Assignment -> '%s' needs groups or users! Skipping assignment...",
                    subject,
                )
                success = False
                continue

            # Check if a workspace is specified for the assignment and check it does exist:
            if assignment.get("workspace"):
                workspace = next(
                    (item for item in self._workspaces if item["id"] == assignment["workspace"]),
                    None,
                )
                if not workspace:
                    self.logger.error(
                        "Assignment -> '%s' has specified a not existing workspace -> %s! Skipping assignment...",
                        subject,
                        assignment["workspace"],
                    )
                    success = False
                    continue
                node_id = self.determine_workspace_id(workspace=workspace)
                if not node_id:
                    self.logger.error(
                        "Assignment -> '%s' has specified a not existing workspace -> %s! Skipping assignment...",
                        subject,
                        assignment["workspace"],
                    )
                    success = False
                    continue
            # If we don't have a workspace then check if a nickname is specified for the assignment:
            elif "nickname" in assignment:
                response = self._otcs.get_node_from_nickname(
                    nickname=assignment["nickname"],
                )
                node_id = self._otcs.get_result_value(response=response, key="id")
                if not node_id:
                    # if response == None:
                    self.logger.error(
                        "Assignment item with nickname -> '%s' not found",
                        assignment["nickname"],
                    )
                    success = False
                    continue
            else:
                self.logger.error(
                    "Assignment -> '%s' needs a workspace or nickname! Skipping assignment...",
                    subject,
                )
                success = False
                continue

            assignees = []

            if "groups" in assignment:
                group_assignees = assignment["groups"]
                for group_assignee in group_assignees:
                    # find the group in the group list
                    group = next(
                        (item for item in self._groups if item["name"] == group_assignee),
                        None,
                    )
                    if not group:
                        self.logger.error(
                            "Assignment group -> '%s' does not exist! Skipping group...",
                            group_assignee,
                        )
                        success = False
                        continue
                    if "id" not in group:
                        self.logger.error(
                            "Assignment group -> '%s' does not have an ID. Skipping group...",
                            group_assignee,
                        )
                        success = False
                        continue
                    group_id = group["id"]
                    # add the group ID to the assignee list:
                    assignees.append(group_id)

            if "users" in assignment:
                user_assignees = assignment["users"]
                for user_assignee in user_assignees:
                    # find the user in the user list
                    user = next(
                        (item for item in self._users if item["name"] == user_assignee),
                        None,
                    )
                    if not user:
                        self.logger.error(
                            "Assignment user -> '%s' does not exist! Skipping user...",
                            user_assignee,
                        )
                        success = False
                        continue
                    if "id" not in user:
                        self.logger.error(
                            "Assignment user -> '%s' does not have an ID. Skipping user...",
                            user_assignee,
                        )
                        success = False
                        continue
                    user_id = user["id"]
                    # add the group ID to the assignee list:
                    assignees.append(user_id)

            if not assignees:
                self.logger.error(
                    "Cannot add assignment -> '%s' for node ID -> %s because no assignee was found.",
                    subject,
                    node_id,
                )
                success = False
                continue

            response = self._otcs.assign_item_to_user_group(
                node_id=int(node_id),
                subject=subject,
                instruction=instruction,
                assignees=assignees,
            )
            if not response:
                self.logger.error(
                    "Failed to add assignment -> '%s' for node ID -> %s with assignees -> %s!",
                    subject,
                    node_id,
                    assignees,
                )
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._assignments,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_licenses")
    def process_user_licenses(
        self,
        resource_name: str,
        license_feature: str,
        license_name: str = "EXTENDED_ECM",
        user_specific_payload_field: str = "licenses",
        section_name: str = "userLicenses",
    ) -> bool:
        """Assign a specific OTDS license feature to all Content Server users.

        This method is used for OTIV and OTCS licenses.

        Args:
            resource_name (str):
                The name of the OTDS resource.
            license_feature (str):
                The license feature to assign to the user (product specific).
            license_name (str):
                The name of the license Key (e.g. "EXTENDED_ECM" or "INTELLIGENT_VIEWING").
            user_specific_payload_field (str, optional):
                The name of the user specific field in payload
                (if empty it will be ignored).
            section_name (str, optional):
                The name of the section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise.

        """

        if not self._users:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        otds_resource = self._otds.get_resource(resource_name)
        if not otds_resource:
            self.logger.error(
                "OTDS Resource -> '%s' not found. Cannot assign licenses to users.",
                resource_name,
            )
            return False

        user_partition = self._otcs.config()["partition"]
        if not user_partition:
            self.logger.error("OTCS user partition not found in OTDS!")
            return False

        for user in self._users:
            user_name = user.get("name")
            if not user_name:
                self.logger.error(
                    "Missing user name - cannot assign license to user. Skipping user...",
                )
                continue

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_name,
                )
                continue

            if user_specific_payload_field and user_specific_payload_field in user:
                self.logger.info(
                    "Found specific license feature -> %s for user -> '%s'. Overwriting default license feature -> '%s'.",
                    user[user_specific_payload_field],
                    user_name,
                    license_feature,
                )
                user_license_features = user[user_specific_payload_field]
            else:  # use the default feature from the actual parameter
                user_license_features = [license_feature]

            for user_license_feature in user_license_features:
                if isinstance(user_license_feature, dict):
                    user_license_feature_dict = user_license_feature
                    user_license_feature = user_license_feature_dict.get("feature", license_feature)
                    license_name = user_license_feature_dict.get("name", license_name)
                    if "enabled" in user_license_feature_dict and not user_license_feature_dict["enabled"]:
                        self.logger.info(
                            "Payload for License '%s' -> '%s' is disabled. Skipping...",
                            license_name,
                            license_feature,
                        )
                        continue

                    if "resource" in user_license_feature_dict:
                        try:
                            resource_id = self._otds.get_resource(name=user_license_feature_dict["resource"])[
                                "resourceID"
                            ]
                        except Exception:
                            self.logger.error(
                                "Error retrieving resourceID for -> %s", user_license_feature_dict["resource"]
                            )
                            continue
                            success = False
                    else:
                        resource_id = otds_resource["resourceID"]

                elif isinstance(user_license_feature, str):
                    resource_id = otds_resource["resourceID"]

                else:
                    self.logger.error("Invalid License feature specified -> %s", user_license_feature)
                    continue

                if self._otds.is_user_licensed(
                    user_name=user_name,
                    resource_id=resource_id,
                    license_feature=user_license_feature,
                    license_name=license_name,
                ):
                    self.logger.info(
                        "User -> '%s' is already licensed for -> '%s' (%s). Skipping...",
                        user_name,
                        license_name,
                        user_license_feature,
                    )
                    continue
                assigned_license = self._otds.assign_user_to_license(
                    partition=user_partition,
                    user_id=user_name,  # we want the plain login name here
                    resource_id=resource_id,
                    license_feature=user_license_feature,
                    license_name=license_name,
                )

                if not assigned_license:
                    self.logger.error(
                        "Failed to assign license feature -> '%s' to user -> %s!",
                        user_license_feature,
                        user_name,
                    )
                    success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._users,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_exec_commands")
    def process_exec_commands(self, section_name: str = "execCommands") -> bool:
        """Process Payload items to execute a command.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise

        """

        if not self._exec_commands:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for exec_command in self._exec_commands:
            if "command" not in exec_command or not exec_command.get("command"):
                self.logger.error(
                    "Command is not specified! It needs to be a non-empty list! Skipping to next command...",
                )
                success = False

                continue
            command = exec_command["command"]

            # Check if exec command has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not exec_command.get("enabled", True):
                self.logger.info(
                    "Payload for exec command is disabled. Skipping...",
                )
                continue

            description = exec_command.get("description")
            if not description:
                self.logger.info(
                    "Executing command -> %s...",
                    command,
                )
            else:
                self.logger.info(
                    "Executing command -> %s (%s)...",
                    command,
                    description,
                )

            try:
                output = subprocess.run(command, stdout=subprocess.PIPE, check=False)
                result = output.stdout.decode("utf-8")
            except Exception:
                self.logger.error(
                    "Execution of command -> %s failed!",
                    command,
                )
                success = False

            # we need to differentiate 3 cases here:
            # 1. result = None is returned - this is an error (exception)
            # 2. result is empty string - this is OK
            # 3. result is a non-empty string - this is OK - print it to log
            if result is None:
                self.logger.error(
                    "Execution of command -> %s failed",
                    command,
                )
                success = False
            elif result != "":
                self.logger.info(
                    "Execution of command -> %s returned result -> %s.",
                    command,
                    result,
                )
            else:
                # It is not an error if no result is returned. It depends on the nature of the command
                # if a result is written to stdout or stderr.
                self.logger.info(
                    "Execution of command -> %s did not return a result.",
                    command,
                )

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._exec_pod_commands,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_exec_pod_commands")
    def process_exec_pod_commands(self, section_name: str = "execPodCommands") -> bool:
        """Process commands that should be executed in the Kubernetes pods.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise

        """

        self.logger.warning("execPodCommand is deprecated - use kubernetes section with action 'execPodCommands'!")

        if not isinstance(self._k8s, K8s):
            self.logger.error(
                "K8s not setup properly -> Skipping payload section -> '%s'...",
                section_name,
            )
            return False

        if not self._exec_pod_commands:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for exec_pod_command in self._exec_pod_commands:
            pod_name = exec_pod_command.get("pod_name")
            if not pod_name:
                self.logger.error(
                    "To execute a command in a pod the pod name needs to be specified in the payload! Skipping to next pod command...",
                )
                success = False
                continue

            command = exec_pod_command.get("command", [])
            if not command:
                self.logger.error(
                    "Command is not specified for pod -> %s! It needs to be a non-empty list! Skipping to next pod command...",
                    pod_name,
                )
                success = False
                continue

            container = exec_pod_command.get("container", None)
            timeout = int(exec_pod_command.get("timeout", 60))

            # Check if exec pod command has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not exec_pod_command.get("enabled", True):
                self.logger.info(
                    "Payload for exec pod command in pod -> '%s' is disabled. Skipping...",
                    pod_name,
                )
                continue

            if "description" not in exec_pod_command:
                self.logger.info(
                    "Executing command -> %s in pod -> '%s'...",
                    command,
                    pod_name,
                )
            else:
                description = exec_pod_command["description"]
                self.logger.info(
                    "Executing command -> %s in pod -> '%s' (%s)...",
                    command,
                    pod_name,
                    description,
                )

            if "interactive" not in exec_pod_command or exec_pod_command["interactive"] is False:
                result = self._k8s.exec_pod_command(
                    pod_name=pod_name, command=command, container=container, timeout=timeout
                )
            else:
                result = self._k8s.exec_pod_command_interactive(
                    pod_name=pod_name,
                    commands=command,
                    timeout=timeout,
                )

            # we need to differentiate 3 cases here:
            # 1. result = None is returned - this is an error (exception)
            # 2. result is empty string - this is OK
            # 3. result is a non-empty string - this is OK - print it to log
            if result is None:
                self.logger.error(
                    "Execution of command -> %s in pod -> '%s' failed",
                    command,
                    pod_name,
                )
                success = False
            elif result != "":
                self.logger.info(
                    "Execution of command -> %s in pod -> '%s' returned result -> %s.",
                    command,
                    pod_name,
                    result,
                )
            else:
                # It is not an error if no result is returned. It depends on the nature of the command
                # if a result is written to stdout or stderr.
                self.logger.info(
                    "Execution of command -> %s in pod -> '%s' did not return a result.",
                    command,
                    pod_name,
                )

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._exec_pod_commands,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_kubernetes")
    def process_kubernetes(self, section_name: str = "kubernetes") -> bool:
        """Process actions that should be executed in the Kubernetess.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise.

        """

        if not isinstance(self._k8s, K8s):
            self.logger.error(
                "K8s not setup properly -> Skipping payload section -> '%s'...",
                section_name,
            )
            return False

        if not self._kubernetes:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for item in self._kubernetes:
            # Check if element has been disabled in payload (enabled = false).
            # In this case we skip the element:
            if isinstance(item, dict) and not item.get("enabled", True):
                self.logger.info("Skipping disabled item -> %s", item)
                continue

            match item.get("action"):
                case "execPodCommand":
                    pod_name = item.get("pod_name")

                    if not pod_name:
                        self.logger.error(
                            "To execute a command in a pod the pod name needs to be specified in the payload! Skipping to next kubernetes item...",
                        )
                        success = False
                        continue

                    command = item.get("command", [])
                    if not command:
                        self.logger.error(
                            "Command is not specified for pod -> %s! It needs to be a non-empty list! Skipping to next kubernetes item...",
                            pod_name,
                        )
                        success = False
                        continue

                    container = item.get("container", None)
                    timeout = int(item.get("timeout", 60))

                    # Check if exec pod command has been explicitly disabled in payload
                    # (enabled = false). In this case we skip the element:
                    if not item.get("enabled", True):
                        self.logger.info(
                            "Payload for exec pod command in pod -> '%s' is disabled. Skipping...",
                            pod_name,
                        )
                        continue

                    if "description" not in item:
                        self.logger.info(
                            "Executing command -> %s in pod -> '%s'",
                            command,
                            pod_name,
                        )

                    else:
                        description = item["description"]
                        self.logger.info(
                            "Executing command -> %s in pod -> '%s' (%s)...",
                            command,
                            pod_name,
                            description,
                        )

                    if "interactive" not in item or item["interactive"] is False:
                        result = self._k8s.exec_pod_command(
                            pod_name=pod_name, command=command, container=container, timeout=timeout
                        )
                    else:
                        result = self._k8s.exec_pod_command_interactive(
                            pod_name=pod_name,
                            commands=command,
                            timeout=timeout,
                        )

                    # we need to differentiate 3 cases here:
                    # 1. result = None is returned - this is an error (exception)
                    # 2. result is empty string - this is OK
                    # 3. result is a non-empty string - this is OK - print it to log
                    if result is None:
                        self.logger.error(
                            "Execution of command -> %s in pod -> '%s' failed!",
                            command,
                            pod_name,
                        )
                        success = False
                    elif result != "":
                        self.logger.info(
                            "Execution of command -> %s in pod -> '%s' returned result -> %s.",
                            command,
                            pod_name,
                            result,
                        )
                    else:
                        # It is not an error if no result is returned. It depends on the nature of the command
                        # if a result is written to stdout or stderr.
                        self.logger.info(
                            "Execution of command -> %s in pod -> '%s' did not return a result.",
                            command,
                            pod_name,
                        )

                case "restart":
                    k8s_type = item.get("type")
                    name = item.get("name")
                    message = item.get("message")

                    if not name:
                        self.logger.error("Name not specified for kubernetes item -> %s", item)
                    if not k8s_type:
                        self.logger.error("Type not specified for kubernetes item -> %s", item)

                    if message:
                        self.logger.info("%s", message)

                    if k8s_type.lower() == "statefulset":
                        self.logger.info("Restarting statefulset -> %s...", name)
                        restart_result = self._k8s.restart_stateful_set(sts_name=name)

                    elif k8s_type.lower() == "deployment":
                        self.logger.info("Restarting deployment -> %s...", name)
                        restart_result = self._k8s.restart_deployment(deployment_name=name)

                    elif k8s_type.lower() == "pod":
                        self.logger.info("Deleting pod -> %s...", name)
                        restart_result = self._k8s.delete_pod(pod_name=name)

                    if not restart_result:
                        success = False

                case _:
                    self.logger.error("Action not defined for work item -> %s", item)
                    continue

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._kubernetes,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_exec_database_commands")
    def process_exec_database_commands(self, section_name: str = "execDatabaseCommands") -> bool:
        """Process commands that should be executed in the PostgreSQL database.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise.

        """

        if not self._exec_database_commands:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        if not psycopg_installed:
            self.logger.warning("Python module 'psycopg' not installed. Cannot execute database commands! Skipping...")
            return False

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for database_command_set in self._exec_database_commands:
            # Check if exec pod command has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not database_command_set.get("enabled", True):
                self.logger.info(
                    "Payload for database command set is disabled. Skipping...",
                )
                continue

            db_connection = database_command_set.get("db_connection")
            if not db_connection:
                self.logger.error(
                    "To execute a command in a database the connection information needs to be specified in the payload! Skipping to next database command set...",
                )
                success = False
                continue

            db_name = db_connection.get("db_name", None)
            if not db_name:
                self.logger.error(
                    "Database connection information is missing the database name! Skipping to next database command set...",
                )
                success = False
                continue
            db_hostname = db_connection.get("db_hostname", None)
            if not db_hostname:
                self.logger.error(
                    "Database connection information is missing the database hostname! Skipping to next database command set...",
                )
                success = False
                continue
            db_port = int(db_connection.get("db_port", 5432))
            db_username = db_connection.get("db_username", None)
            if not db_username:
                self.logger.error(
                    "Database connection information is missing the database username! Skipping to next database command set...",
                )
                success = False
                continue
            db_password = db_connection.get("db_password", None)
            if not db_password:
                self.logger.error(
                    "Database connection information is missing the database password! Skipping to next database command set...",
                )
                success = False
                continue

            connect_string = "dbname={} user={} password={} host={} port={}".format(
                db_name, db_username, db_password, db_hostname, db_port
            )

            db_commands = database_command_set.get("db_commands", [])

            # TODO: Add support for sql file

            db_connection = None  # Predefine for safe access in except
            allowed_verbs = {"SELECT", "INSERT", "UPDATE", "CREATE"}

            try:
                # Using a context managers (with ...) for automatic resource management:
                with psycopg.connect(connect_string) as db_connection:
                    self.logger.info(
                        "Connected to database -> '%s' (%s) with user -> '%s'...", db_name, db_hostname, db_username
                    )
                    with db_connection.cursor() as cursor:
                        for db_command in db_commands:
                            cmd = db_command.get("command", None)
                            if not cmd:
                                self.logger.warning(
                                    "Cannot execute database command without SQL statement. Skipping..."
                                )
                                continue
                            params = db_command.get("params", None)
                            if params is not None and isinstance(params, (list, tuple)):
                                self.logger.error(
                                    "Database parameters -> %s must be given as a list or tuple!", str(params)
                                )
                                continue
                            # Get the command verb (like "SELECT", "CREATE")
                            verb = cmd.strip().split()[0].upper()
                            if verb not in allowed_verbs:
                                self.logger.error("Database command -> '%s' is not allowed!", verb)
                                continue
                            if params:
                                self.logger.info(
                                    "Execute database command -> '%s' with parameters -> %s...", cmd, str(params)
                                )
                            else:
                                self.logger.info("Execute database command -> '%s' without parameters...", cmd)
                            cursor.execute(cmd, params)
                            if verb == "SELECT":
                                response = cursor.fetchall()
                                self.logger.debug("Database response -> '%s'", response)
                    db_connection.commit()
            except psycopg.Error as e:
                success = False
                self.logger.error("Database error -> %s", e)
                if db_connection is not None:
                    db_connection.rollback()

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._exec_database_commands,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_document_generators")
    def process_document_generators(
        self,
        section_name: str = "documentGenerators",
    ) -> bool:
        """Generate documents for a defined workspace type based on template.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:,
                True if payload has been processed without errors, False otherwise.

        """

        if not self._doc_generators:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # save admin credentials for later switch back to admin user:
        #        admin_credentials = self._otcs.credentials()
        authenticated_user = "admin"

        for doc_generator in self._doc_generators:
            if "workspace_type" not in doc_generator:
                self.logger.error(
                    "To generate documents for workspaces the workspace type needs to be specified in the payload! Skipping to next document generator...",
                )
                success = False
                continue
            workspace_type = doc_generator["workspace_type"]

            self._log_header_callback(
                text="Process Document Generator for workspace type -> '{}'".format(workspace_type),
                char="-",
            )

            # Check if doc generator has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not doc_generator.get("enabled", True):
                self.logger.info(
                    "Payload for document generator of workspace type -> '%s' is disabled. Skipping...",
                    workspace_type,
                )
                continue

            template_path = doc_generator.get("template_path")
            if not template_path:
                self.logger.error(
                    "To generate documents for workspaces of type -> '%s' the path to the document template needs to be specified in the payload! Skipping to next document generator...",
                    workspace_type,
                )
                success = False
                continue

            template = self._otcs.get_node_by_volume_and_path(
                volume_type=self._otcs.VOLUME_TYPE_CONTENT_SERVER_DOCUMENT_TEMPLATES,
                path=template_path,
            )
            if not template:
                self.logger.error(
                    "Cannot find document template in path -> %s. Skipping to next document generator...",
                    template_path,
                )
                success = False
                continue
            template_id = self._otcs.get_result_value(response=template, key="id")
            template_name = self._otcs.get_result_value(response=template, key="name")

            if "classification_path" not in doc_generator:
                self.logger.error(
                    "To generate documents for workspaces of type -> '%s' the path to the document classification needs to be specified in the payload! Skipping to next document generator...",
                    workspace_type,
                )
                success = False
                continue
            classification_path = doc_generator["classification_path"]
            classification = self._otcs.get_node_by_volume_and_path(
                volume_type=self._otcs.VOLUME_TYPE_CLASSIFICATION_VOLUME,
                path=classification_path,
            )
            if not classification:
                self.logger.error(
                    "Cannot find document classification in path -> %s. Skipping to next document generator...",
                    classification_path,
                )
                success = False
                continue
            classification_id = self._otcs.get_result_value(
                response=classification,
                key="id",
            )

            # "category_name" is optional. But if it is specified
            # then also "attributes" needs to be specified:
            if "category_name" not in doc_generator:
                self.logger.info(
                    "No metadata (category name) specified in the payload for this document generator.",
                )
                category_name = ""
                attributes = {}
                category_data = {}
            else:
                category_name = doc_generator["category_name"]
                if "attributes" not in doc_generator:
                    self.logger.error(
                        "To generate documents for workspaces of type -> '%s' with metadata, the attributes needs to be specified in the payload! Skipping to next document generator...",
                        workspace_type,
                    )
                    success = False
                    continue
                attributes = doc_generator["attributes"]

                # The following method returns two values: the category ID and
                # a dict of the attributes. If the category is not found
                # on the document template it returns -1 for the category ID
                # and an empty dict for the attribute definitions:
                (
                    category_id,
                    attribute_definitions,
                ) = self._otcs.get_node_category_definition(
                    node_id=template_id,
                    category_name=category_name,
                )
                if category_id == -1:
                    self.logger.error(
                        "The document template -> '%s' does not have the specified category -> %s. Skipping to next document generator...",
                        template_name,
                        category_name,
                    )
                    success = False
                    continue

                category_data = {str(category_id): {}}

                # now we fill the prepared (but empty) category_data
                # with the actual attribute values from the payload:
                for attribute in attributes:
                    attribute_name = attribute["name"]
                    attribute_value = attribute["value"]
                    attribute_type = attribute_definitions[attribute_name]["type"]
                    attribute_id = attribute_definitions[attribute_name]["id"]

                    # Special treatment for type user: determine the ID for the login name.
                    # the ID is the actual value we have to put in the attribute:
                    if attribute_type == "user":
                        user = self._otcs.get_user(
                            name=attribute_value,
                            show_error=True,
                        )
                        user_id = self._otcs.get_result_value(response=user, key="id")
                        if not user_id:
                            self.logger.error(
                                "Cannot find user with login name -> '%s'. Skipping...",
                                attribute_value,
                            )
                            success = False
                            continue
                        attribute_value = user_id
                    category_data[str(category_id)][attribute_id] = attribute_value

            if "workspace_folder_path" not in doc_generator:
                self.logger.info(
                    "No workspace folder path defined for workspaces of type -> '%s'. Documents will be stored in workspace root.",
                    workspace_type,
                )
                workspace_folder_path = []
            else:
                workspace_folder_path = doc_generator["workspace_folder_path"]

            if "exec_as_user" in doc_generator:
                exec_as_user = doc_generator["exec_as_user"]

                # Find the user in the users payload:
                exec_user = next(
                    (item for item in self._users if item["name"] == exec_as_user),
                    None,
                )
                # Have we found the user in the payload?
                if exec_user is not None:
                    self.logger.info(
                        "Executing document generator with user -> '%s'...",
                        exec_as_user,
                    )
                    # Impersonate as the user specified in the payload:
                    self.logger.info("Impersonate user -> '%s'...", exec_as_user)
                    result = self.start_impersonation(username=exec_as_user)
                    if not result:
                        self.logger.error(
                            "Couldn't impersonate user -> '%s'!",
                            exec_as_user,
                        )
                        continue
                    admin_context = False
                    authenticated_user = exec_as_user
                else:
                    self.logger.error(
                        "Cannot find user with login name -> '%s' for executing document generator. Executing as admin...",
                        exec_as_user,
                    )
                    admin_context = True
                    success = False
            else:
                admin_context = True
                exec_as_user = "admin"

            if admin_context and authenticated_user != "admin":
                # Impersonate as the admin user:
                self.logger.info(
                    "Impersonate as admin user -> '%s'...",
                    self._otcs.credentials()["username"],
                )
                # Stop the impersonation as a user:
                result = self.stop_impersonation()
                authenticated_user = "admin"

            if category_data:
                self.logger.info(
                    "Generate documents for workspace type -> '%s' based on template -> '%s' with metadata -> %s...",
                    workspace_type,
                    template_name,
                    category_data,
                )
            else:
                self.logger.info(
                    "Generate documents for workspace type -> '%s' based on template -> '%s' without metadata...",
                    workspace_type,
                    template_name,
                )

            # Find the workspace type with the name given in the _workspace_types
            # datastructure that has been generated by process_workspace_types() method before:
            workspace_type_id = next(
                (item["id"] for item in self._workspace_types if item["name"] == workspace_type),
                None,
            )

            # The workspace type name is used as a "starts with" search on the
            # workspace type name. This can cause issues, so we prefer to use the type ID
            # if we have it. Otherwise the REST API prefers the name over the type:
            workspace_instances = self._otcs.get_workspace_instances_iterator(
                type_name=workspace_type if not workspace_type_id else None,
                type_id=workspace_type_id,
            )
            for workspace_instance in workspace_instances:
                workspace = workspace_instance.get("data").get("properties")
                workspace_id = workspace["id"]
                workspace_name = workspace["name"]
                if workspace_folder_path:
                    workspace_folder = self._otcs.get_node_by_workspace_and_path(
                        workspace_id=workspace_id,
                        path=workspace_folder_path,
                    )
                    if workspace_folder:
                        workspace_folder_id = self._otcs.get_result_value(
                            response=workspace_folder,
                            key="id",
                        )
                    else:
                        # If the workspace template is not matching
                        # the path we may have an error here. Then
                        # we fall back to workspace root level.
                        self.logger.warning(
                            "Folder path does not exist in workspace -> '%s'. Using workspace root level instead...",
                            workspace_name,
                        )
                        workspace_folder_id = workspace_id
                else:
                    workspace_folder_id = workspace_id

                document_name = workspace_name + " - " + template_name
                self.logger.info("Generate document -> '%s'...", document_name)

                response = self._otcs.check_node_name(
                    parent_id=int(workspace_folder_id),
                    node_name=document_name,
                )
                if response["results"]:
                    self.logger.warning(
                        "Node -> '%s' does already exist in workspace folder with ID -> %s",
                        document_name,
                        workspace_folder_id,
                    )
                    continue
                response = self._otcs.create_document_from_template(
                    template_id=int(template_id),
                    parent_id=int(workspace_folder_id),
                    classification_id=int(classification_id),
                    category_data=category_data,
                    doc_name=document_name,
                    doc_description="This document has been auto-generated by Terrarium",
                )
                if not response:
                    self.logger.error(
                        "Failed to generate document -> '%s' in workspace -> '%s' (%s) as user -> '%s'!",
                        document_name,
                        workspace_name,
                        workspace_id,
                        exec_as_user,
                    )
                    success = False
                else:
                    self.logger.info(
                        "Successfully generated document -> '%s' in workspace -> '%s' (%s) as user -> '%s'.",
                        document_name,
                        workspace_name,
                        workspace_id,
                        exec_as_user,
                    )

        if authenticated_user != "admin":
            # Impersonate as the admin user:
            self.logger.info(
                "Impersonate as admin user -> '%s'...",
                self._otcs.credentials()["username"],
            )
            result = self.stop_impersonation()
            if not result:
                self.logger.error(
                    "Couldn't impersonate as admin user -> '%s'!",
                    self._otcs.credentials()["username"],
                )
                success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._doc_generators,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workflow_attributes")
    def process_workflow_attributes(
        self,
        attributes: list,
        workflow_attribute_definition: dict,
    ) -> None:
        """Process the attributes in the workflow steps.

        This method adds the IDs for the attribute to the payload dicts.
        The IDs are needed for the workflow REST API calls.

        Args:
            attributes (list):
                The list of attributes (payload) processed in the workflow step.
            workflow_attribute_definition (dict):
                The workflow attribute definition.

        Returns:
            None. The mutable dictionary in the workflow_step is updated with the IDs.

        """

        # now we need to get the technical attribute IDs from
        # the workflow definition and map them
        # with the attribute names from the payload:
        for attribute in attributes:
            attribute_name = attribute["name"]
            attribute_value = attribute["value"]
            attribute_type = attribute.get("type", None)

            # Special treatment for type user: determine the ID for the login name.
            # the ID is the actual value we have to put in the attribute:
            if attribute_type and attribute_type.lower() == "user":
                user = self._otcs.get_user(name=attribute_value, show_error=True)
                user_id = self._otcs.get_result_value(response=user, key="id")
                if not user_id:
                    self.logger.error(
                        "Cannot find user with login name -> '%s'. Skipping...",
                        attribute_value,
                    )
                    continue
                attribute_value = user_id
                attribute["value"] = user_id

            attribute_definition = workflow_attribute_definition.get(attribute_name)
            if not attribute_definition:
                self.logger.error(
                    "Cannot find the attribute -> '%s' in the workflow definition. Skipping...",
                )
                continue
            # Enrich the attribute dictionary with the attribute ID from the workflow definition:
            attribute["id"] = attribute_definition["id"]
            # Enrich the attribute dictionary with the attribute form ID from the workflow definition:
            attribute["form_id"] = attribute_definition["form_id"]

        if attributes:
            self.logger.info(
                "Updated workflow step attributes with IDs -> %s.",
                str(attributes),
            )

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workflow_step")
    def process_workflow_step(
        self,
        workflow_id: int,
        workflow_step: dict,
        workflow_attribute_definition: dict,
        documents: list | None = None,
        process_id: int | None = None,
    ) -> bool:
        """Process a workflow step of a workflow.

        Args:
            workflow_id (int):
                The node ID of the workflow (the workflow map).
            workflow_step (dict):
                Payload dictionary for a single workflow step.
                Keys: "action", "exec_as_user", "attributes"
            workflow_attribute_definition (dict):
                The workflow attribute definition.
            documents (list | None, optional):
                The list of workflow documents (attachments9).
            process_id (int | None, optional):
                The process ID of the workflow.

        Returns:
            bool:
                True, if the workflow step has been processed successfully, False otherwise.

        """

        if "action" not in workflow_step:
            self.logger.error("Missing workflow action in workflow step.")
            return False
        action = workflow_step["action"]

        if "exec_as_user" not in workflow_step:
            self.logger.error("Missing workflow user in workflow step.")
            return False
        exec_as_user = workflow_step["exec_as_user"]

        # Find the user in the users payload:
        exec_user = next(
            (item for item in self._users if item["name"] == exec_as_user),
            None,
        )
        # Have we found the user in the payload?
        if exec_user is None:
            self.logger.error(
                "Cannot find user with login name -> '%s' for workflow processing.",
                exec_as_user,
            )
            return False

        self.logger.info("Executing workflow step as user -> '%s'...", exec_as_user)

        # Impersonate as the user:
        self.logger.info("Impersonate user -> '%s'...", exec_as_user)
        # True = force new login with new user
        result = self.start_impersonation(username=exec_as_user)
        if not result:
            self.logger.error("Couldn't impersonate user -> '%s'", exec_as_user)
            return False

        # "attributes" is optional:
        if "attributes" not in workflow_step:
            self.logger.warning(
                "No workflow attributes specified in the payload for this workflow step.",
            )
            attributes = []
            workflow_step_values = None
        else:
            attributes = workflow_step["attributes"]
            self.logger.info(
                "Workflow step has attributes -> %s. Adding attribute IDs to the payload names...",
                str(attributes),
            )
            # Update / enrich the attributes in the workflow step with the IDs
            # from the workflow definition (this CHANGES the attributes!)
            self.process_workflow_attributes(
                attributes=attributes,
                workflow_attribute_definition=workflow_attribute_definition,
            )
            # Prepare the data for the REST call to
            # update the process:
            workflow_step_values = {
                attr["form_id"]: attr["value"] for attr in attributes if "form_id" in attr and "value" in attr
            }

        if action == "Initiate":
            # Create a draft process in preparation for the workflow initiation:
            response = self._otcs.create_draft_process(
                workflow_id=workflow_id,
                documents=documents,
                attach_documents=True,
            )
            draftprocess_id = self._otcs.get_result_value(
                response=response,
                key="draftprocess_id",
                property_name="",
            )
            if not draftprocess_id:
                self.logger.error(
                    "Failed to create draft process for workflow ID -> %s as user -> '%s'!",
                    str(workflow_id),
                    exec_as_user,
                )
                # Stop the impersonation as a user:
                result = self.stop_impersonation()
                return False
            else:
                self.logger.info(
                    "Successfully generated draft process with ID -> %s%s.",
                    str(draftprocess_id),
                    " attching document IDs -> " + str(documents) if documents else "",
                )
            workflow_step["draftprocess_id"] = draftprocess_id

            # Check if a due date is specified. The payload has
            # a relative offset in number of days that we add to
            # the current date:
            due_in_days = workflow_step.get("due_in_days")
            if due_in_days:
                due_date = datetime.now(UTC) + timedelta(days=int(due_in_days))
                due_date = due_date.strftime("%Y-%m-%d")
            else:
                due_date = None
            # Record the due date in the workflow step dictionary
            workflow_step["due_date"] = due_date

            # Update the draft process with title, due date
            # and workflow attribute values from the payload:
            response = self._otcs.update_draft_process(
                draftprocess_id=draftprocess_id,
                title=workflow_step.get("title"),
                due_date=due_date,
                values=workflow_step_values,
            )

            # Initiate the draft process. This creates
            # the running workflow instance:
            response = self._otcs.initiate_draft_process(
                draftprocess_id=draftprocess_id,
                comment=workflow_step.get("comment"),
            )
            process_id = self._otcs.get_result_value(
                response=response,
                key="process_id",
                property_name="",
            )
            if not process_id:
                self.logger.error(
                    "Failed to initiate process for workflow with ID -> %s as user -> '%s'!",
                    str(workflow_id),
                    exec_as_user,
                )
                # Stop the impersonation as a user:
                result = self.stop_impersonation()
                return False
            self.logger.info(
                "Successfully initiated process with ID -> %s for workflow with ID -> %s as user -> '%s'.",
                str(process_id),
                str(workflow_id),
                exec_as_user,
            )
            workflow_step["process_id"] = process_id
        # end if action == "Initiate"
        else:
            if not process_id:
                self.logger.error(
                    "Workflow step cannot be executed as process is not initiated (process ID not set)",
                )
                # Stop the impersonation as a user:
                result = self.stop_impersonation()
                return False
            response = self._otcs.get_process_task(
                process_id=process_id,
            )
            # Are there any workflow attributes to update with new values?
            if attributes:
                response = self._otcs.update_process_task(
                    process_id=process_id,
                    values=workflow_step_values,
                )
            # Execute the step action defined in the payload
            response = self._otcs.update_process_task(
                process_id=process_id,
                action=action,
            )

        # Impersonate as the admin user:
        self.logger.info(
            "Impersonate as admin user -> '%s'...",
            self._otcs.credentials()["username"],
        )
        # Stop the impersonation as a user:
        result = self.stop_impersonation()

        return True

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workflows")
    def process_workflows(self, section_name: str = "workflows") -> bool:
        """Initiate and process workflows for a defined workspace type and folder path.

        Args:
            section_name (str, optional):
                The name of the section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise.

        """

        if not self._workflows:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        # save admin credentials for later switch back to admin user:
        admin_credentials = self._otcs.credentials()

        for workflow in self._workflows:
            workflow_nickname = workflow.get("workflow_nickname")
            if not workflow_nickname:
                self.logger.error(
                    "To initiate and process workflows for documents in workspaces the workflow nickname needs to be specified in the payload! Skipping to next workflow initiation...",
                )
                success = False
                continue
            workflow_node = self._otcs.get_node_from_nickname(
                nickname=workflow_nickname,
            )
            workflow_id = self._otcs.get_result_value(response=workflow_node, key="id")
            workflow_name = self._otcs.get_result_value(
                response=workflow_node,
                key="name",
            )
            if not workflow_id:
                self.logger.error(
                    "Cannot find workflow by nickname -> '%s'! Skipping to next workflow...",
                    workflow_nickname,
                )
                success = False
                continue

            workspace_type = workflow.get("workspace_type")
            if not workspace_type:
                self.logger.error(
                    "To process workflow -> '%s' for documents in workspaces the workspace type needs to be specified in the payload! Skipping to next workflow...",
                    workflow_name,
                )
                success = False
                continue

            # Check if workflow has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not workflow.get("enabled", True):
                self.logger.info(
                    "Payload for workflow -> '%s' of workspace type -> '%s' is disabled. Skipping...",
                    workflow_name,
                    workspace_type,
                )
                continue

            # Find the workspace type with the name given in the _workspace_types
            # datastructure that has been generated by process_workspace_types() method before:
            workspace_type_id = next(
                (item["id"] for item in self._workspace_types if item["name"] == workspace_type),
                None,
            )

            workspace_folder_path = workflow.get("workspace_folder_path", [])
            if not workspace_folder_path:
                self.logger.info(
                    "No workspace folder path defined for workspaces of type -> '%s'. Workflows will be started for documents in workspace root.",
                    workspace_type,
                )

            workflow_steps = workflow.get("steps")
            if not workflow_steps:
                self.logger.error(
                    "To process workflow -> '%s', workflow steps ('steps') needs to be specified in the payload! Skipping to next workflow initiation...",
                    workflow_name,
                )
                success = False
                continue

            # Get the attribute details (name, ID, type) from the workflow definition:
            workflow_attribute_definition = self._otcs.get_workflow_attributes(
                workflow_id=workflow_id,
            )

            # The workspace type name is used as a "starts with" search on the
            # workspace type name. This can cause issues, so we prefer to use the type ID
            # if we have it. Otherwise the REST API prefers the name over the type:
            workspace_instances = self._otcs.get_workspace_instances_iterator(
                type_name=workspace_type if not workspace_type_id else None,
                type_id=workspace_type_id,
            )
            for workspace_instance in workspace_instances:
                workspace = workspace_instance.get("data").get("properties")
                workspace_id = workspace["id"]
                workspace_name = workspace["name"]

                if workspace_folder_path:
                    workspace_folder = self._otcs.get_node_by_workspace_and_path(
                        workspace_id=workspace_id,
                        path=workspace_folder_path,
                    )
                    if workspace_folder:
                        workspace_folder_id = self._otcs.get_result_value(
                            response=workspace_folder,
                            key="id",
                        )
                    else:
                        # If the workspace template is not matching
                        # the path we may have an error here. Then
                        # we fall back to workspace root level.
                        self.logger.warning(
                            "Folder path does not exist in workspace -> '%s'. Using workspace root level instead...",
                            workspace_name,
                        )
                        workspace_folder_id = workspace_id
                else:
                    workspace_folder_id = workspace_id

                # Get all documents (-3 = non-container) from the defined folder:
                response = self._otcs.get_subnodes(
                    parent_node_id=workspace_folder_id,
                    filter_node_types=-3,
                )
                documents = self._otcs.get_result_values(response=response, key="id")

                process_id = None
                for workflow_step in workflow_steps:
                    result = self.process_workflow_step(
                        workflow_id=workflow_id,
                        workflow_step=workflow_step,
                        workflow_attribute_definition=workflow_attribute_definition,
                        documents=documents,
                        process_id=process_id,
                    )
                    # If the step fails we are bailing out as it doesn't make
                    # sense to continue with further steps:
                    if not result:
                        success = False
                        break
                    if "process_id" in workflow_step:
                        process_id = workflow_step["process_id"]
            # end for workspace_instance in workspace_instances:
        # end for workflow in self._workflows

        # Set back admin credentials:
        self._otcs.set_credentials(
            username=admin_credentials["username"],
            password=admin_credentials["password"],
        )
        # Authenticate back as the admin user:
        self.logger.info(
            "Authenticate as admin user -> '%s'...",
            admin_credentials["username"],
        )
        # True = force new login with new user
        self._otcs.authenticate(revalidate=True)

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._workflows,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_browser_automations")
    def process_browser_automations(
        self,
        browser_automations: list,
        section_name: str = "browserAutomations",
        check_status: bool = True,
    ) -> bool:
        """Process Playwright-based browser automations and tests.

        Args:
            browser_automations (list):
                A list of browser automations (need this as parameter as we
                have multiple lists).
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.
            check_status (bool, optional):
                Defines whether or not this needs to re-run
                for each customizer run (even if it has been successful before).
                If check_status is True (default) then it is only re-run
                if it has NOT been successfully before.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise.

        """

        if not browser_automations:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if check_status and self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        automation_type = "Browser" if "browser" in section_name else "Test"

        for browser_automation in browser_automations:
            name = browser_automation.get("name")
            if not name:
                self.logger.error("%s automation is missing a unique name. Skipping...", automation_type)
                success = False
                continue

            self._log_header_callback(
                text="Process {} Automation -> '{}'".format(automation_type, name),
                char="-",
            )

            description = browser_automation.get("description", "")
            if description:
                self.logger.info(
                    "%s automation description -> '%s'.",
                    automation_type,
                    description,
                )

            # Check if browser automation has been explicitly disabled in payload
            # (enabled = false). In this case we skip this payload element:
            if not browser_automation.get("enabled", True):
                self.logger.info(
                    "Payload for %s automation -> '%s' is disabled. Skipping...",
                    automation_type.lower(),
                    name,
                )
                continue

            base_url = browser_automation.get("base_url")
            if not base_url:
                self.logger.error(
                    "%s automation -> '%s' is missing 'base_url' parameter. Skipping...", automation_type, name
                )
                browser_automation["result"] = "failure"
                success = False
                continue

            user_name = browser_automation.get("user_name", "")
            if not user_name:
                self.logger.info("%s automation -> '%s' is not having user name.", automation_type, name)

            password = browser_automation.get("password", "")
            if not password:
                self.logger.info("%s automation -> '%s' is not having password.", automation_type, name)

            automation_steps = browser_automation.get("automations", [])
            if not automation_steps:
                self.logger.error(
                    "%s automation -> '%s' is missing list of automations. Skipping...",
                    automation_type,
                    name,
                )
                browser_automation["result"] = "failure"
                success = False
                continue

            debug_automation: bool = browser_automation.get("debug", False)

            # Create Selenium Browser Automation:
            self.logger.info("%s automation base URL -> %s.", automation_type, base_url)
            self.logger.info("%s automation user -> '%s'.", automation_type, user_name)
            wait_until = browser_automation.get("wait_until")  # it is OK to be None
            if "wait_until" in browser_automation:
                # Only log the "wait until" value if it is specified in the payload:
                self.logger.info(
                    "%s Automation page navigation strategy is to wait until -> '%s'.",
                    automation_type,
                    wait_until,
                )
            browser_automation_object = BrowserAutomation(
                base_url=base_url,
                user_name=user_name,
                user_password=password,
                automation_name=name,
                take_screenshots=debug_automation,
                headless=browser_automation.get("headless", self._browser_headless),
                logger=self.logger,
                wait_until=wait_until,
                browser=browser_automation.get("browser"),  # None is acceptable
            )
            if not browser_automation_object:
                self.logger.error(
                    "Cannot execute browser automation -> '%s'. Initialization of browser automation object failed!",
                    name,
                )
                browser_automation["result"] = "failure"
                success = False
                continue
            # Wait time is a global setting (for whole browser session)
            # This makes sure a page is fully loaded and elements are present
            # before accessing them. We set 30.0 seconds as default if not
            # otherwise specified by "wait_time" in the payload.
            wait_time = float(browser_automation.get("wait_time", 30.0))
            browser_automation_object.set_timeout(wait_time=wait_time)
            if "wait_time" in browser_automation:
                self.logger.info(
                    "%s automation wait time -> '%s' configured.",
                    automation_type,
                    str(wait_time),
                )

            # Initialize overall result status:
            result = True
            first_step = True

            for automation_step in automation_steps:
                automation_step_type = automation_step.get("type", "")
                if not automation_step_type:
                    self.logger.error(
                        "%s automation step -> %s in browser automation -> '%s' is missing 'type' parameter. Stopping automation -> '%s'.",
                        automation_type,
                        str(automation_step),
                        name,
                        name,
                    )
                    success = False
                    break
                # Check if browser automation step has been explicitly disabled in payload
                # (enabled = false). In this case we skip this automation step:
                if not automation_step.get("enabled", True):
                    self.logger.info(
                        "Automation step -> '%s' in automation -> '%s' is disabled. Skipping...",
                        automation_step_type,
                        name,
                    )
                    continue
                dependent = automation_step.get("dependent", True)
                if not dependent and not result:
                    self.logger.warning(
                        "Ignore result of previous step as current step -> '%s' is NOT dependent on it.",
                        automation_step_type,
                    )
                    result = True
                elif not result:
                    # In this case a proceeding automation step has failed
                    # and this step is marked as dependent. Then it does not make sense
                    # to continue with this automation step after the proceeding step failed.
                    self.logger.warning(
                        "Step -> '%s' is dependent on a proceeding step that failed. Skipping this step...",
                        automation_step_type,
                    )
                    continue
                elif not first_step:
                    self.logger.debug(
                        "Current step -> '%s' is %s on proceeding step.",
                        automation_step_type,
                        "dependent" if dependent else "not dependent",
                    )

                match automation_step_type:
                    case "login":
                        page = automation_step.get("page", "")
                        user_field = automation_step.get("user_field", "otds_username")
                        password_field = automation_step.get(
                            "password_field",
                            "otds_password",
                        )
                        login_button = automation_step.get("login_button", "loginbutton")
                        # Do we have a step-specific wait mechanism? If not, we pass None
                        # then the browser automation will take the default configured for
                        # the whole browser automation (see BrowserAutomation() constructor above):
                        wait_until = automation_step.get("wait_until", None)
                        self.logger.info(
                            "Login to -> %s as user -> '%s' (%s page navigation strategy is to wait until -> '%s')...",
                            base_url + page,
                            user_name,
                            "specific" if wait_until is not None else "default",
                            wait_until if wait_until is not None else browser_automation_object.wait_until,
                        )
                        result = browser_automation_object.run_login(
                            page=page,
                            user_field=user_field,
                            password_field=password_field,
                            login_button=login_button,
                            wait_until=wait_until,
                        )
                        if not result:
                            self.logger.error(
                                "Cannot log into -> %s. Skipping to next automation step...",
                                base_url + page,
                            )
                            automation_step["result"] = "failure"
                            success = False
                            continue
                        self.logger.info(
                            "Successfully logged into page -> %s. Page title -> '%s'.",
                            base_url + page,
                            browser_automation_object.get_title(),
                        )
                    case "get_page":
                        page = automation_step.get("page", "")
                        if not page:
                            self.logger.error(
                                "Automation step type -> '%s' requires 'page' parameter. Stopping automation.",
                                automation_step_type,
                            )
                            automation_step["result"] = "failure"
                            success = False
                            break
                        volume = automation_step.get("volume", OTCS.VOLUME_TYPE_ENTERPRISE_WORKSPACE)
                        path = automation_step.get("path", [])
                        if path and volume:
                            page_node = self._otcs.get_node_by_volume_and_path(
                                volume_type=volume,
                                path=path,
                                create_path=False,
                            )
                            page_id = self._otcs.get_result_value(response=page_node, key="id")
                            if not page_id:
                                # if not parent_node:
                                self.logger.error(
                                    "%s automation -> '%s' has a page path that does not exist. Skipping...",
                                    automation_type,
                                    name,
                                )
                                automation_step["result"] = "failure"
                                success = False
                                continue
                            self.logger.info(
                                "Resolved volume -> %s and page path -> %s to node ID -> %s.",
                                str(volume),
                                str(path),
                                str(page_id),
                            )
                        else:
                            page_id = None
                        if "{}" in page and page_id:
                            page = page.format(page_id)
                        # Do we have a step-specific wait mechanism? If not, we pass None
                        # then the browser automation will take the default configured for
                        # the whole browser automation (see BrowserAutomation() constructor called above):
                        wait_until = automation_step.get("wait_until", None)
                        self.logger.info(
                            "Load page -> %s (%s page navigation strategy is to wait until -> '%s')...",
                            base_url + page,
                            "specific" if wait_until is not None else "default",
                            wait_until if wait_until is not None else browser_automation_object.wait_until,
                        )
                        result = browser_automation_object.get_page(url=page, wait_until=wait_until)
                        if not result:
                            self.logger.error(
                                "Cannot load page -> %s! Skipping this step...",
                                page,
                            )
                            automation_step["result"] = "failure"
                            success = False
                            continue
                        self.logger.info(
                            "Successfully loaded page -> %s. Page title -> '%s'.",
                            base_url + page,
                            browser_automation_object.get_title(),
                        )
                    case "click_elem":
                        # We keep the deprecated "elem" syntax supported (for now)
                        selector = automation_step.get("selector", automation_step.get("elem", ""))
                        if not selector:
                            self.logger.error(
                                "Automation step type -> '%s' requires 'selector' parameter. Stopping automation.",
                                automation_step_type,
                            )
                            automation_step["result"] = "failure"
                            success = False
                            break
                        # We keep the deprecated "find" syntax supported (for now)
                        selector_type = automation_step.get("selector_type", automation_step.get("find", "id"))
                        show_error = automation_step.get("show_error", True)
                        # Do we navigate away from the current page with the click?
                        navigation = automation_step.get("navigation", False)
                        # Do we open a new browser (popup) window with the click?
                        popup_window = automation_step.get("popup_window", False)
                        # De we close the current (popup) window with the click?
                        close_window = automation_step.get("close_window", False)
                        # Do we have a 'desired' state for clicking a checkbox?
                        checkbox_state = automation_step.get("checkbox_state", None)
                        wait_until = automation_step.get("wait_until", None)
                        wait_time = automation_step.get("wait_time", 0.0)
                        role_type = automation_step.get("role_type", None)
                        occurrence = automation_step.get("occurrence", 1)
                        scroll_to_element = automation_step.get("scroll_to_element", True)
                        exact_match = automation_step.get("exact_match", None)
                        regex = automation_step.get("regex", None)
                        hover_only = automation_step.get("hover_only", False)
                        iframe = automation_step.get("iframe", None)
                        force = automation_step.get("force", None)
                        click_button = automation_step.get("click_button", None)
                        click_count = automation_step.get("click_count", None)
                        click_modifiers = automation_step.get("click_modifiers", None)
                        repeat_reload = automation_step.get("repeat_reload", None)
                        repeat_reload_delay = automation_step.get("repeat_reload_delay", None)
                        result = browser_automation_object.find_elem_and_click(
                            selector=selector,
                            selector_type=selector_type,
                            role_type=role_type,
                            occurrence=occurrence,
                            scroll_to_element=scroll_to_element,
                            desired_checkbox_state=checkbox_state,
                            is_navigation_trigger=navigation,
                            is_popup_trigger=popup_window,
                            is_page_close_trigger=close_window,
                            wait_until=wait_until,
                            wait_time=wait_time,
                            exact_match=exact_match,
                            regex=regex,
                            hover_only=hover_only,
                            iframe=iframe,
                            force=force,
                            click_button=click_button,
                            click_count=click_count,
                            click_modifiers=click_modifiers,
                            repeat_reload=repeat_reload,
                            repeat_reload_delay=repeat_reload_delay,
                            show_error=show_error,
                        )
                        if not result:
                            message = "Cannot find clickable element with selector -> '{}' ({}) on current page. Skipping this step...".format(
                                selector, selector_type
                            )
                            if show_error:
                                self.logger.error(message)
                                automation_step["result"] = "failure"
                                success = False
                            else:
                                self.logger.warning(message)
                            continue
                        self.logger.info(
                            "Successfully %s %s element selected by -> '%s' (%s%s).",
                            "clicked" if not hover_only else "hovered over",
                            "navigational" if navigation else "non-navigational",
                            selector,
                            "selector type -> '{}'".format(selector_type),
                            ", role type -> '{}'".format(role_type) if role_type else "",
                        )
                    case "set_elem":
                        # We keep the deprecated "elem" syntax supported (for now)
                        selector = automation_step.get("selector", automation_step.get("elem", ""))
                        if not selector:
                            self.logger.error(
                                "Automation step type -> '%s' requires 'selector' parameter. Stopping automation.",
                                automation_step_type,
                            )
                            automation_step["result"] = "failure"
                            success = False
                            break
                        # We keep the deprecated "find" syntax supported (for now)
                        selector_type = automation_step.get("selector_type", automation_step.get("find", "id"))
                        role_type = automation_step.get("role_type", None)
                        occurrence = automation_step.get("occurrence", 1)
                        exact_match = automation_step.get("exact_match", None)
                        regex = automation_step.get("regex", None)
                        iframe = automation_step.get("iframe", None)
                        value = automation_step.get("value", "")
                        typing = automation_step.get("typing", False)
                        press_enter = automation_step.get("press_enter", False)
                        if not value:
                            self.logger.error(
                                "Automation step type -> '%s' for element selected by -> '%s' (%s) requires 'value' parameter. Stopping automation.",
                                automation_step_type,
                                selector,
                                selector_type,
                            )
                            automation_step["result"] = "failure"
                            success = False
                            break
                        # we also support replacing placeholders that are
                        # enclosed in double % characters like %%OTCS_RESOURCE_ID%%:
                        if isinstance(value, str):
                            value = self.replace_placeholders(value)
                        show_error = automation_step.get("show_error", True)
                        result = browser_automation_object.find_elem_and_set(
                            selector=selector,
                            value=value,
                            selector_type=selector_type,
                            role_type=role_type,
                            occurrence=occurrence,
                            press_enter=press_enter,
                            exact_match=exact_match,
                            regex=regex,
                            iframe=iframe,
                            typing=typing,
                            show_error=show_error,
                        )
                        if not result:
                            message = "Cannot set element{} selected by -> '{}' ({}{}) to value -> '{}'. Skipping this step...".format(
                                " (occurrence -> {})".format(occurrence) if occurrence > 1 else "",
                                selector,
                                "selector type -> '{}'".format(selector_type),
                                ", role type -> '{}'".format(role_type) if role_type else "",
                                value,
                            )
                            if show_error:
                                self.logger.error(message)
                                automation_step["result"] = "failure"
                                success = False
                            else:
                                self.logger.warning(message)
                            continue
                        self.logger.info(
                            "Successfully set element%s selected by -> '%s' (%s%s) to value -> '%s'.",
                            " (occurrence -> {})".format(occurrence) if occurrence > 1 else "",
                            selector,
                            "selector type -> '{}'".format(selector_type),
                            ", role type -> '{}'".format(role_type) if role_type else "",
                            value,
                        )
                    case "check_elem":
                        # We keep the deprecated "elem" syntax supported (for now)
                        selector = automation_step.get("selector", automation_step.get("elem", ""))
                        if not selector:
                            self.logger.error(
                                "Automation step type -> '%s' requires 'selector' parameter. Stopping automation.",
                                automation_step_type,
                            )
                            automation_step["result"] = "failure"
                            success = False
                            break
                        # We keep the deprecated "find" syntax supported (for now)
                        selector_type = automation_step.get("selector_type", automation_step.get("find", "id"))
                        role_type = automation_step.get("role_type", None)
                        value = automation_step.get("value", None)
                        attribute = automation_step.get("attribute", "")
                        substring = automation_step.get("substring", False)
                        iframe = automation_step.get("iframe", None)
                        min_count = automation_step.get("min_count", 1)
                        want_exist = automation_step.get("want_exist", True)
                        wait_time = automation_step.get("wait_time", 0.0)
                        if value:
                            # we also support replacing placeholders that are
                            # enclosed in double % characters like %%OTCS_RESOURCE_ID%%:
                            value = self.replace_placeholders(value)
                        (result, count) = browser_automation_object.check_elems_exist(
                            selector=selector,
                            selector_type=selector_type,
                            role_type=role_type,
                            value=value,
                            attribute=attribute,
                            substring=substring,
                            iframe=iframe,
                            min_count=min_count,
                            wait_time=wait_time,  # time to wait before the check is actually done
                            show_error=not want_exist,  # if element is not found that we do not want to find it is not an error
                        )
                        # Check if we didn't get what we want:
                        if (not result and want_exist) or (result and not want_exist):
                            self.logger.error(
                                "%s %s%s%s on current page. Test failed.%s",
                                "Cannot find" if not result and want_exist else "Found",
                                "{} elements with selector -> '{}' ({}{})".format(
                                    min_count if want_exist else count,
                                    selector,
                                    "selector type -> '{}'".format(selector_type),
                                    ", role type -> '{}'".format(role_type) if role_type else "",
                                )
                                if (min_count > 1 and want_exist) or (count > 1 and not want_exist)
                                else "an element with selector -> '{}' ({}{})".format(
                                    selector,
                                    "selector type -> '{}'".format(selector_type),
                                    ", role type -> '{}'".format(role_type) if role_type else "",
                                ),
                                " with {}value -> '{}'".format("substring-" if substring else "", value)
                                if value
                                else "",
                                " in attribute -> '{}'".format(attribute) if attribute else "",
                                " Found {}{} occurences.".format(
                                    count,
                                    " undesirable" if not want_exist else " from a minimum of {}".format(min_count),
                                ),
                            )
                            success = False
                            continue
                            # Don't break here! We want to do all existance tests!
                        self.logger.info(
                            "Successfully passed %sexistence test for %s%s%s on current page.",
                            "non-" if not want_exist else "",
                            "{} elements with selector -> '{}' ({})".format(min_count, selector, selector_type)
                            if min_count > 1
                            else "an element with selector -> '{}' ({})".format(selector, selector_type),
                            " with {}value -> '{}'".format("substring-" if substring else "", value) if value else "",
                            " in attribute -> '{}'".format(attribute) if attribute else "",
                        )
                    case _:
                        self.logger.error(
                            "Illegal automation step type -> '%s' in %s automation!",
                            automation_step_type,
                            automation_type.lower(),
                        )
                        automation_step["result"] = "failure"
                        success = False
                        break
                # end match automation_step_type:
                first_step = False
            # end for automation_step in automation_steps:

            # Cleanup session and and remove reference to the object:
            browser_automation_object.end_session()
            browser_automation_object = None

            browser_automation["result"] = (
                "failure" if any(step.get("result", "success") == "failure" for step in automation_steps) else "success"
            )
        # end for browser_automation in browser_automations:

        if check_status:
            self.write_status_file(
                success=success,
                payload_section_name=section_name,
                payload_section=browser_automations,
            )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="init_sap")
    def init_sap(
        self,
        sap_external_system: dict,
        direct_application_server_login: bool = True,
    ) -> SAP | None:
        """Initialize SAP object for RFC communication with SAP S/4HANA.

        Args:
            sap_external_system (dict):
                SAP external system created before
            direct_application_server_login (bool, optional):
                Flag to control whether we comminicate directly with
                SAP application server or via a load balancer

        Returns:
            SAP | None:
                The SAP object or None in case of an error.

        """

        if not sap_external_system:
            return None

        username = sap_external_system["username"]
        password = sap_external_system["password"]
        # "external_system_hostname" is extracted from as_url in process_external_systems()
        host = sap_external_system["external_system_hostname"]
        client = sap_external_system.get("client", "100")
        system_number = sap_external_system.get("external_system_number", "00")
        system_id = sap_external_system["external_system_name"]
        group = sap_external_system.get("group", "PUBLIC")
        destination = sap_external_system.get("destination", "")

        self.logger.info("Connection parameters SAP:")
        self.logger.info("SAP Hostname             = %s", host)
        self.logger.info("SAP Client               = %s", client)
        self.logger.info("SAP System Number        = %s", system_number)
        self.logger.info("SAP System ID            = %s", system_id)
        self.logger.info("SAP User Name            = %s", username)
        if not direct_application_server_login:
            self.logger.info("SAP Group Name (for RFC) = %s", group)
        if destination:
            self.logger.info("SAP Destination          = %s", destination)

        if direct_application_server_login:
            self.logger.info(
                "SAP Login                = Direct Application Server (ashost)",
            )
            sap_object = SAP(
                username=username,
                password=password,
                ashost=host,
                client=client,
                system_number=system_number,
                system_id=system_id,
                destination=destination,
                logger=self.logger,
            )
        else:
            self.logger.info(
                "SAP Login                = Logon with load balancing (mshost)",
            )
            sap_object = SAP(
                username=username,
                password=password,
                mshost=host,
                group=group,
                client=client,
                system_number=system_number,
                system_id=system_id,
                destination=destination,
                logger=self.logger,
            )

        self._sap = sap_object

        if (
            "archive_logical_name" in sap_external_system
            and "archive_certificate_file" in sap_external_system
            and self._otac
        ):
            self.logger.info(
                "Put certificate file -> '%s' for logical archive -> '%s' into Archive Center...",
                sap_external_system["archive_certificate_file"],
                sap_external_system["archive_logical_name"],
            )
            response = self._otac.put_cert(
                sap_external_system["external_system_name"],
                sap_external_system["archive_logical_name"],
                sap_external_system["archive_certificate_file"],
            )
            if not response:
                self.logger.error("Failed to install Archive Center certificate!")
            else:
                self.logger.info(
                    "Enable certificate file -> '%s' for logical archive -> '%s'...",
                    sap_external_system["archive_certificate_file"],
                    sap_external_system["archive_logical_name"],
                )
                response = self._otac.enable_cert(
                    auth_id=sap_external_system["external_system_name"],
                    logical_archive=sap_external_system["archive_logical_name"],
                    enable=True,
                )
                if not response:
                    self.logger.debug("Failed to enable Archive Center certificate!")

        return sap_object

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_sap_rfcs")
    def process_sap_rfcs(self, sap_rfcs: list, section_name: str = "sapRFCs") -> bool:
        """Process SAP RFCs in payload and run them in SAP S/4HANA.

        Args:
            sap_rfcs (list):
                The payload list of SAP RFCs. As we have two different list (pre and post)
                we need to pass the actual list as parameter.
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise.

        """

        if not self._sap:
            self.logger.info("SAP object is undefined. Cannot call RFCs. Bailing out.")
            return False

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for sap_rfc in sap_rfcs:
            rfc_name = sap_rfc.get("name")
            if not rfc_name:
                self.logger.error("SAP RFC needs a name! Skipping...")
                success = False
                continue

            # Check if SAP RFC has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not sap_rfc.get("enabled", True):
                self.logger.info(
                    "Payload for SAP RFC -> '%s' is disabled. Skipping...",
                    rfc_name,
                )
                continue

            rfc_description = sap_rfc.get("description", "")

            # Check if we have SAP RFC call parameters in the payload. They are optional:
            rfc_params = sap_rfc.get("parameters", {})
            # Check if we have SAP RFC call options in the payload. They are optional:
            rfc_call_options = sap_rfc.get("call_options", {})

            if rfc_params:
                self.logger.info(
                    "Calling SAP RFC -> '%s' (%s) with parameters -> %s%s...",
                    rfc_name,
                    rfc_description,
                    str(rfc_params),
                    " and options -> {}".format(rfc_call_options) if rfc_call_options else "",
                )
            else:
                self.logger.info(
                    "Calling SAP RFC -> '%s' (%s)%s...",
                    rfc_name,
                    rfc_description,
                    " with options -> {}".format(rfc_call_options) if rfc_call_options else "",
                )

            if rfc_call_options:
                self.logger.debug("Using call options -> '%s' ...", rfc_call_options)

            result = self._sap.call(rfc_name, rfc_call_options, rfc_params)
            if result is None:
                self.logger.error("Failed to call SAP RFC -> '%s'!", rfc_name)
                success = False
            elif result.get("RESULT") == "WARNING":
                self.logger.warning(
                    "Result of SAP RFC -> '%s' has a warning -> '%s'.",
                    rfc_name,
                    str(result.get("COMMENT")).strip() if result.get("COMMENT") else str(result),
                )
            elif result.get("RESULT") != "OK":
                self.logger.error(
                    "Result of SAP RFC -> '%s' is not OK, %sresult -> %s",
                    rfc_name,
                    "it returned -> '{}' failed items in".format(result.get("FAILED")) if result.get("FAILED") else "",
                    str(result),
                )
                success = False
            else:
                self.logger.info(
                    "Successfully called RFC -> '%s'. Result -> %s.",
                    rfc_name,
                    str(result),
                )
                # Save result for status file content
                sap_rfc["result"] = result

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=sap_rfcs,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="init_successfactors")
    def init_successfactors(
        self,
        sucessfactors_external_system: dict,
    ) -> SuccessFactors | None:
        """Initialize SuccessFactors object for workspace creation.

        This is needed synchronize user passwords and emails with SuccessFactors.

        Args:
            sucessfactors_external_system (dict):
                The payload of the SuccessFactors external system created before.

        Returns:
            SuccessFactors: The SuccessFactors object.

        """

        def extract_company_from_url(url: str) -> str:
            parsed_url = urlparse(url)
            query_params = parse_qs(parsed_url.query)
            company_value = query_params.get("company", [""])[0]
            return company_value

        if not sucessfactors_external_system:
            return None

        username = sucessfactors_external_system["username"]
        password = sucessfactors_external_system["password"]
        base_url = sucessfactors_external_system["base_url"]
        as_url = sucessfactors_external_system["as_url"]
        saml_url = sucessfactors_external_system.get("saml_url", "")
        company_id = extract_company_from_url(saml_url)
        client_id = sucessfactors_external_system["oauth_client_id"]
        client_secret = sucessfactors_external_system["oauth_client_secret"]

        self.logger.info("Connection parameters SuccessFactors:")
        self.logger.info("SuccessFactors base URL            = %s", base_url)
        self.logger.info("SuccessFactors application URL     = %s", as_url)
        self.logger.info("SuccessFactors username            = %s", username)
        self.logger.debug("SuccessFactors password            = %s", password)
        self.logger.info("SuccessFactors client ID           = %s", client_id)
        self.logger.debug("SuccessFactors client secret       = %s", client_secret)
        self.logger.info("SuccessFactors company ID (tenant) = %s", company_id)
        successfactors_object = SuccessFactors(
            base_url=base_url,
            as_url=as_url,
            client_id=client_id,
            client_secret=client_secret,
            username=username,
            password=password,
            company_id=company_id,
            logger=self.logger,
        )

        self._successfactors = successfactors_object

        return successfactors_object

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="init_salesforce")
    def init_salesforce(self, salesforce_external_system: dict) -> Salesforce | None:
        """Initialize Salesforce object for workspace creation.

        This is needed to query Salesforce API to lookup IDs of Salesforce objects.

        Args:
            salesforce_external_system (dict):
                The payload of the Salesforce external system created before.

        Returns:
            Salesforce | None:
                Salesforce object or None in case an error occured.

        """

        if not salesforce_external_system:
            return None

        username = salesforce_external_system["username"]
        password = salesforce_external_system["password"]
        base_url = salesforce_external_system["base_url"]
        authorization_url = salesforce_external_system.get("token_endpoint", "")
        client_id = salesforce_external_system["oauth_client_id"]
        client_secret = salesforce_external_system["oauth_client_secret"]

        self.logger.info("Connection parameters Salesforce:")
        self.logger.info("Salesforce base URL          = %s", base_url)
        self.logger.info("Salesforce authorization URL = %s", base_url)
        self.logger.info("Salesforce username          = %s", username)
        self.logger.debug("Salesforce password          = %s", password)
        self.logger.info("Salesforce client ID         = %s", client_id)
        self.logger.debug("Salesforce client secret     = %s", client_secret)
        salesforce_object = Salesforce(
            base_url=base_url,
            client_id=client_id,
            client_secret=client_secret,
            username=username,
            password=password,
            authorization_url=authorization_url,
            logger=self.logger,
        )

        self._salesforce = salesforce_object

        return salesforce_object

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="init_guidewire")
    def init_guidewire(self, guidewire_external_system: dict) -> Guidewire | None:
        """Initialize Guidewire object for workspace creation.

        This is needed to query Guidewire REST API to lookup IDs of Guidewire objects.

        Args:
            guidewire_external_system (dict):
                The payload of the Guidewire external system created before.

        Returns:
            Guidewire | None:
                Guidewire object or None in case an error occured.

        """

        if not guidewire_external_system:
            return None

        system_name = guidewire_external_system["external_system_name"]
        username = guidewire_external_system["username"]
        password = guidewire_external_system["password"]
        base_url = guidewire_external_system["base_url"]
        as_url = guidewire_external_system["as_url"]
        auth_type = guidewire_external_system.get("auth_type", "basic").lower()

        self.logger.info("Connection parameters for %s:", system_name)
        self.logger.info("Guidewire base URL        = %s", base_url)
        self.logger.info("Guidewire application URL = %s", as_url)
        self.logger.info("Guidewire username        = %s", username)
        self.logger.debug("Guidewire password        = %s", password)
        guidewire_object = Guidewire(
            base_url=base_url,
            as_url=as_url,
            auth_type=auth_type,
            username=username,
            password=password,
            logger=self.logger,
        )

        return guidewire_object

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="write_data_source_file")
    def write_data_source_file(
        self,
        data_source: dict,
        compression: bool = True,
    ) -> bool:
        """Write a data source file as JSON into the Admin Personal Workspace in Content Server.

        This allows for further analysis or to reload it in succeeding customizer runs.

        Args:
            data_source (dict):
                Data source dictionary with embedded "data" Data object.
            compression (bool):
                If True, a compressed JSON file gets saved. If False then
                an uncompressed JSON is saved.

        Returns:
            bool:
                True if the data source file (JSON) as been upladed to
                Content Server successfully, False otherwise.

        """

        data_source_name = data_source.get("name", "")
        if not data_source_name:
            self.logger.error("Missing data source name!")
            return False

        data: Data = data_source.get("data")
        if not data:
            self.logger.error(
                "Data source -> '%s' does not have data! Cannot save it.",
                data_source_name,
            )
            return False

        response = self._otcs.get_node_by_volume_and_path(
            volume_type=self._otcs.VOLUME_TYPE_PERSONAL_WORKSPACE,
        )  # write to Personal Workspace of Admin (with Volume Type ID = 142)
        target_folder_id = self._otcs.get_result_value(response=response, key="id")
        if not target_folder_id:
            target_folder_id = 2004  # use Personal Workspace of Admin as fallback

        if self._payload_source:
            payload_file_name = os.path.basename(
                self._payload_source,
            )  # remove directories
            # Split once at the first occurance of a dot
            # as the _payload_source may have multiple suffixes
            # such as .yml.gz.b64:
            payload_file_name = payload_file_name.split(".", 1)[0] + "_"
        else:
            payload_file_name = ""

        file_name = "data_source_" + payload_file_name + data_source_name + ".json"
        if compression:
            file_name += ".zip"
        full_path = os.path.join(tempfile.gettempdir(), file_name)

        # We also want to keep the row numbers (index):
        if not data.save_json_data(
            json_path=full_path,
            preserve_index=True,
            index_column="row",
            compression="zip" if compression else None,
        ):
            self.logger.error(
                "Data source -> '%s' could not be saved!",
                data_source_name,
            )
            return False

        # Check if the status file has been uploaded before.
        # This can happen if we re-run the python container.
        # In this case we add a version to the existing document:
        response = self._otcs.get_node_by_parent_and_name(
            parent_id=int(target_folder_id),
            name=file_name,
            show_error=False,
        )
        data_source_node_id = self._otcs.get_result_value(response=response, key="id")
        if data_source_node_id:
            response = self._otcs.add_document_version(
                node_id=int(data_source_node_id),
                file_url=full_path,
                file_name=file_name,
                mime_type="application/zip" if compression else "application/json",
                description="Updated data source file after re-run of customization",
            )
        else:
            response = self._otcs.upload_file_to_parent(
                file_url=full_path,
                file_name=file_name,
                mime_type="application/zip" if compression else "application/json",
                parent_id=int(target_folder_id),
            )

        if response:
            self.logger.info(
                "Data source file -> '%s' has been written to Personal Workspace of admin user.",
                file_name,
            )
            # Don't leave stuff behind:
            os.remove(full_path)

            return True

        self.logger.error(
            "Failed to write data source file -> '%s' to Personal Workspace of admin user!",
            file_name,
        )

        return False

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="read_data_source_file")
    def read_data_source_file(
        self,
        data_source: dict,
        compression: bool = True,
    ) -> bool:
        """Read a data source file from the Admin Personal Workspace in Content Server.

        Args:
            data_source (dict):
                Data source dictionary with embedded "data" Data object.
            compression (bool, optional):
                Use True, if the data source file is compressed.

        Returns:
            bool:
                True if the data source file (JSON) as been loaded from
                Content Server successfully, False otherwise

        """

        data_source_name = data_source.get("name", "")
        if not data_source_name:
            self.logger.error("Missing data source name!")
            return False

        response = self._otcs.get_node_by_volume_and_path(
            volume_type=self._otcs.VOLUME_TYPE_PERSONAL_WORKSPACE,
        )  # write to Personal Workspace of Admin (with Volume Type ID = 142)
        target_folder_id = self._otcs.get_result_value(response=response, key="id")
        if not target_folder_id:
            target_folder_id = 2004  # use Personal Workspace of Admin as fallback

        if self._payload_source:
            payload_file_name = os.path.basename(
                self._payload_source,
            )  # remove directories
            # Split once at the first occurance of a dot
            # as the _payload_source may have multiple suffixes
            # such as .yml.gz.b64:
            payload_file_name = payload_file_name.split(".", 1)[0] + "_"
        else:
            payload_file_name = ""

        file_name = "data_source_" + payload_file_name + data_source_name + ".json"
        if compression:
            file_name += ".zip"
        full_path = os.path.join(tempfile.gettempdir(), file_name)

        # Check if the data source file has been uploaded before.
        # This can happen if we re-run the python container.
        # In this case we add a version to the existing document:
        response = self._otcs.get_node_by_parent_and_name(
            parent_id=int(target_folder_id),
            name=file_name,
            show_error=False,
        )
        data_source_node_id = self._otcs.get_result_value(response=response, key="id")
        if not data_source_node_id:
            self.logger.warning(
                "Data source -> '%s' file -> '%s' not found!",
                data_source_name,
                file_name,
            )
            return False

        self.logger.info(
            "Download data source to temporary file -> '%s'...",
            full_path,
        )
        if not self._otcs.download_document(
            node_id=int(data_source_node_id),
            file_path=full_path,
        ):
            self.logger.error(
                "Error downloading data source -> '%s' to file -> '%s'!",
                data_source_name,
                full_path,
            )
            return False

        self.logger.info("Load data source file -> '%s' into data frame...", full_path)
        data = Data()
        if not data.load_json_data(
            json_path=full_path,
            index_column="row",
            compression="zip" if compression else None,
        ):
            self.logger.error(
                "Data source -> '%s' could not be loaded!",
                data_source_name,
            )
            return False

        self.logger.info(
            "Data source file -> '%s' has been loaded from Personal Workspace of admin user into the data frame.",
            file_name,
        )

        data_source["data"] = data

        # Don't leave stuff behind:
        os.remove(full_path)

        return True

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_otcs")
    def process_bulk_datasource_otcs(self, data_source: dict) -> Data:
        """Load data from Content Server (OTCS) data source into the data frame.

        See helper/data.py

        Args:
            data_source (dict):
                Payload dict element for the data source.

        Returns:
            Data:
                Data class that includes a Pandas DataFrame

        Side Effects:
            self._otcs_source is set to the OTCS object created by this method.

        """

        # 1. Read and validate values from the data source payload:
        otcs_hostname = data_source.get("otcs_hostname", "")
        if not otcs_hostname:
            self.logger.error(
                "Content Server hostname (otcs_hostname) is not specified in payload of bulk data source. Cannot load data!",
            )
            return None
        otcs_protocol = data_source.get("otcs_protocol", "https")
        otcs_port = data_source.get("otcs_port", "443")
        otcs_basepath = data_source.get("otcs_basepath", "/cs/cs")
        otcs_username = data_source.get("otcs_username", "")
        otcs_password = data_source.get("otcs_password", "")
        if not otcs_username or not otcs_password:
            self.logger.error(
                "Content Server user name (otcs_username) or password (otcs_password) are missing in payload of bulk data source. Cannot load data!",
            )
            return None
        otcs_thread_number = data_source.get("otcs_thread_number", BULK_THREAD_NUMBER)
        otcs_download_dir = data_source.get("otcs_download_dir", "/data/contentserver")
        # Root nodes for the traversal of the data loader (can be a list or single ID)
        # we keep two spellings to remain backwards compatible with the payload syntax:
        otcs_root_node_ids = data_source.get(
            "otcs_root_node_ids",
            data_source.get("otcs_root_node_id"),
        )

        if not otcs_root_node_ids:
            self.logger.error(
                "Content Server root node ID(s) for traversal are missing in payload of bulk data source. Cannot load data!",
            )
            return None

        otcs_include_workspaces = data_source.get("otcs_include_workspaces", True)
        otcs_include_items = data_source.get("otcs_include_items", True)
        otcs_include_workspace_metadata = data_source.get("otcs_include_workspace_metadata", True)
        otcs_include_item_metadata = data_source.get("otcs_include_item_metadata", True)

        # Filter workspace by depth under the given root (only consider items as workspace if they have the right depth in the hierarchy):
        otcs_filter_workspace_depth = data_source.get("otcs_filter_workspace_depth", 0)
        # Filter workspace by subtype (only consider items as workspace if they have the right technical subtype):
        # This is NOT the workspace type but the technical subtype (like 848 for workspaces and 0 for folder)
        otcs_filter_workspace_subtypes = data_source.get(
            "otcs_filter_workspace_subtypes",
        )
        # Filter workspace by category name (only consider items as workspace if they have the category):
        otcs_filter_workspace_category = data_source.get(
            "otcs_filter_workspace_category",
        )
        # Filter workspace by attribute values (only consider items as workspace if they have the attributes with the defined values):
        otcs_filter_workspace_attributes = data_source.get(
            "otcs_filter_workspace_attributes",
        )

        # Filter item by depth under the given root:
        otcs_filter_item_depth = data_source.get("otcs_filter_item_depth")
        # Filter item by subtype (only consider items if they have the right technical subtype):
        # This is the technical subtype (like 0 for folder and 144 for documents)
        otcs_filter_item_subtypes = data_source.get(
            "otcs_filter_item_subtypes",
        )
        # Filter item by category name (only consider items as workspace if they have the category):
        otcs_filter_item_category = data_source.get("otcs_filter_item_category")
        # Filter item by attribute values (only consider items if they have the attributes with the defined values):
        otcs_filter_item_attributes = data_source.get("otcs_filter_item_attributes")
        # Filter item also if the are in workspaces (default is True):
        otcs_filter_item_in_workspace = data_source.get(
            "otcs_filter_item_in_workspace",
            True,
        )
        # List of node IDs to exclude in traversing the folders in the OTCS data source:
        otcs_exclude_node_ids = data_source.get("otcs_exclude_node_ids")

        # Document handling parameters:
        otcs_download_documents = data_source.get("otcs_download_documents", True)
        otcs_skip_existing_downloads = data_source.get("otcs_skip_existing_downloads", True)
        otcs_extract_zip = data_source.get("extract_zip", False)
        # The following parameter controls how column names are constructed. If it is true, then
        # attribute columns for workspaces and items will use the category ID in the column name.
        # Wokspace attributes always start with "workspace_cat_". Item attributes start with item_cat_".
        # If the value of 'otcs_use_numeric_category_identifier' is False then the category name
        # is converted to lower-case and spaces and non-alphanumeric characters are replaced with "_".
        # Example with otcs_use_numeric_category_identifier = True: workspace_cat_47110815_10
        # Example with otcs_use_numeric_category_identifier = False: workspace_cat_customer_use_case_10
        otcs_use_numeric_category_identifier = data_source.get("otcs_use_numeric_category_identifier", True)

        # Ensure Root_node_id is a list of integers
        if not isinstance(otcs_root_node_ids, list):
            otcs_root_node_ids = [otcs_root_node_ids]
        otcs_root_node_ids = [int(item) for item in otcs_root_node_ids]

        # 2. Creating the OTCS object:
        self._otcs_source = OTCS(
            protocol=otcs_protocol,
            hostname=otcs_hostname,
            port=otcs_port,
            base_path=otcs_basepath,
            username=otcs_username,
            password=otcs_password,
            thread_number=otcs_thread_number,
            download_dir=otcs_download_dir,
            use_numeric_category_identifier=otcs_use_numeric_category_identifier,
            logger=self.logger,
        )

        # 3. Authenticate at OTCS
        auth_data = self._otcs_source.authenticate()
        if not auth_data:
            self.logger.error(
                "Failed to authenticate at Content Server -> %s!",
                otcs_protocol + "://" + otcs_hostname + otcs_basepath,
            )
            return None
        else:
            self.logger.info(
                "Successfully authenticated at Content Server -> %s.",
                otcs_protocol + "://" + otcs_hostname + otcs_basepath,
            )

        # 4. Load the Content Server data into the Data object (Pandas DataFrame):

        self.logger.info(
            "Loading data (folder, workspaces, items) from Content Server -> %s using root IDs -> %s...",
            otcs_protocol + "://" + otcs_hostname + otcs_basepath,
            otcs_root_node_ids,
        )

        for otcs_root_node_id in otcs_root_node_ids:
            if not self._otcs_source.load_items(
                node_id=otcs_root_node_id,
                workspaces=otcs_include_workspaces,
                items=otcs_include_items,
                filter_workspace_depth=otcs_filter_workspace_depth,
                filter_workspace_subtypes=otcs_filter_workspace_subtypes,
                filter_workspace_category=otcs_filter_workspace_category,
                filter_workspace_attributes=otcs_filter_workspace_attributes,
                filter_item_depth=otcs_filter_item_depth,
                filter_item_subtypes=otcs_filter_item_subtypes,
                filter_item_category=otcs_filter_item_category,
                filter_item_attributes=otcs_filter_item_attributes,
                filter_item_in_workspace=otcs_filter_item_in_workspace,
                exclude_node_ids=otcs_exclude_node_ids,
                workspace_metadata=otcs_include_workspace_metadata,
                item_metadata=otcs_include_item_metadata,
                download_documents=otcs_download_documents,
                skip_existing_downloads=otcs_skip_existing_downloads,
                extract_zip=otcs_extract_zip,
                workers=otcs_thread_number,
            ):
                self.logger.error(
                    "Failure during load of Content Server items from root node ID -> %s!",
                    str(otcs_root_node_id),
                )
                return None

        data = self._otcs_source.get_data()
        if data is None:
            self.logger.error(
                "Failure during load of Content Server items! No data loaded!",
            )
            return None

        return data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_servicenow")
    def process_bulk_datasource_servicenow(self, data_source: dict) -> Data | None:
        """Load data from ServiceNow data source into the data frame of the Data class.

        See helper/data.py

        Args:
            data_source (dict):
                Payload dict element for the data source.

        Returns:
            Data | None:
                Data class that includes a Pandas data frame. None in case of an error.

        Side Effects:
            self._servicenow is set to the ServiceNow object created by this method

        """

        # 1. Read and validate values from the data source payload:
        sn_base_url = data_source.get("sn_base_url", "")
        if not sn_base_url:
            self.logger.error(
                "ServiceNow base URL (sn_base_url) is not specified in payload of bulk data source. Cannot load data!",
            )
            return None
        sn_auth_type = data_source.get("sn_auth_type", "basic")
        sn_username = data_source.get("sn_username", "")
        sn_password = data_source.get("sn_password", "")
        sn_client_id = data_source.get("sn_client_id")
        sn_client_secret = data_source.get("sn_client_secret")
        sn_table_name = data_source.get(
            "sn_table_name",
            "u_kb_template_technical_article_public",
        )
        sn_queries = data_source.get("sn_queries", [])
        sn_query = data_source.get("sn_query")
        if sn_query is not None:
            sn_queries.append({"table": sn_table_name, "query": sn_query})

        sn_thread_number = data_source.get("sn_thread_number", BULK_THREAD_NUMBER)
        sn_download_dir = data_source.get("sn_download_dir", "/data/knowledgebase")
        sn_skip_existing_downloads = data_source.get("sn_skip_existing_downloads", True)
        sn_product_exclusions = data_source.get("sn_product_exclusions", [])

        if sn_product_exclusions:
            self.logger.info(
                "Excluding products -> %s from ServiceNow data loading.",
                sn_product_exclusions,
            )

        if sn_base_url and (sn_auth_type == "basic") and (not sn_username or not sn_password):
            self.logger.error(
                "ServiceNow basic authentication needs username and password in payload!",
            )
            return None
        if sn_base_url and (sn_auth_type == "oauth") and (not sn_client_id or not sn_client_secret):
            self.logger.error(
                "ServiceNow OAuth authentication needs client ID and client secret in payload!",
            )
            return None

        # 2. Creating the ServiceNow object:
        self._servicenow = ServiceNow(
            base_url=sn_base_url,
            auth_type=sn_auth_type,
            client_id=sn_client_id,
            client_secret=sn_client_secret,
            username=sn_username,
            password=sn_password,
            thread_number=sn_thread_number,
            download_dir=sn_download_dir,
            product_exclusions=sn_product_exclusions,
            logger=self.logger,
        )

        # 3. Authenticate at ServiceNow
        auth_data = self._servicenow.authenticate(auth_type=sn_auth_type)
        if not auth_data:
            self.logger.error("Failed to authenticate at ServiceNow -> %s!", sn_base_url)
            return None
        else:
            self.logger.info(
                "Successfully authenticated at ServiceNow -> %s.",
                sn_base_url,
            )

        # 4. Load the ServiceNow data into the Data object (Pandas DataFrame):
        for item in sn_queries:
            sn_table_name = item["sn_table_name"]
            sn_query = item["sn_query"]

            self.logger.info(
                "Loading data from ServiceNow table -> '%s' with query -> '%s'...",
                sn_table_name,
                sn_query,
            )

            if not self._servicenow.load_articles(
                table_name=sn_table_name,
                query=sn_query,
                skip_existing_downloads=sn_skip_existing_downloads,
            ):
                self.logger.error(
                    "Failure during load of ServiceNow articles from table -> '%s' using query -> '%s' !",
                    sn_table_name,
                    sn_query,
                )
                continue

        data = self._servicenow.get_data()
        if data is None:
            self.logger.error(
                "Failure during load of ServiceNow articles! No articles loaded!",
            )
            return None

        return data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_otmm")
    def process_bulk_datasource_otmm(self, data_source: dict) -> Data | None:
        """Load data from OTMM data source into the data frame of the Data class.

        See helper/data.py

        Args:
            data_source (dict):
                Payload dict element for the data source.

        Returns:
            Data | None:
                Data class that includes a Pandas data frame. None in case of an error.

        Side Effects:
            self._otmm is set to the OTMM object created by this method

        """

        # 1. Read and validate values from the data source payload:
        otmm_base_url = data_source.get("otmm_base_url", "")
        if not otmm_base_url:
            self.logger.error(
                "OTMM base URL (otmm_base_url) is not specified in payload of bulk data source. Cannot load data!",
            )
            return None
        otmm_username = data_source.get("otmm_username", "")
        otmm_password = data_source.get("otmm_password", "")
        otmm_client_id = data_source.get("otmm_client_id")
        otmm_client_secret = data_source.get("otmm_client_secret")
        otmm_thread_number = data_source.get("otmm_thread_number", BULK_THREAD_NUMBER)
        otmm_download_dir = data_source.get("otmm_download_dir", "/data/mediaassets")
        otmm_business_unit_exclusions = data_source.get(
            "otmm_business_unit_exclusions",
            [],
        )
        otmm_business_unit_inclusions = data_source.get("otmm_business_unit_inclusions")
        otmm_product_exclusions = data_source.get("otmm_product_exclusions", [])
        otmm_product_inclusions = data_source.get("otmm_product_inclusions")
        otmm_asset_exclusions = data_source.get("otmm_asset_exclusions", [])
        otmm_asset_inclusions = data_source.get("otmm_asset_inclusions", [])

        self.logger.info(
            "Loading data from OpenText Media Management -> %s (Marketing Assets)...",
            otmm_base_url,
        )

        # 2. Creating the OTMM object:
        self._otmm = OTMM(
            base_url=otmm_base_url,
            client_id=otmm_client_id,
            client_secret=otmm_client_secret,
            username=otmm_username,
            password=otmm_password,
            thread_number=otmm_thread_number,
            download_dir=otmm_download_dir,
            business_unit_exclusions=otmm_business_unit_exclusions,
            business_unit_inclusions=otmm_business_unit_inclusions,
            product_exclusions=otmm_product_exclusions,
            product_inclusions=otmm_product_inclusions,
            asset_exclusions=otmm_asset_exclusions,
            asset_inclusions=otmm_asset_inclusions,
            logger=self.logger,
        )
        if not self._otmm:
            self.logger.error("Failed to initialize OTMM object!")
            return None

        # 3. Authenticate at OTMM
        token = self._otmm.authenticate()
        if not token:
            self.logger.error(
                "Failed to authenticate at OpenText Media Management -> %s!",
                otmm_base_url,
            )
            return None
        else:
            self.logger.info(
                "Successfully authenticated at OpenText Media Management -> %s.",
                otmm_base_url,
            )

        # 4. Load the OTMM assets into the Data object (Pandas DataFrame):
        if not self._otmm.load_assets():
            self.logger.error(
                "Failure during load of OpenText Media Management assets!",
            )
            return None

        data = self._otmm.get_data()
        if data is None:
            self.logger.error(
                "Failure during load of OpenText Media Management assets! No assets loaded!",
            )
            return None

        return data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_pht")
    def process_bulk_datasource_pht(self, data_source: dict) -> Data | None:
        """Load data from OpenText PHT data source into the data frame of the Data class.

        See helper/data.py

        Args:
            data_source (dict):
                Payload dict element for the data source.

        Returns:
            Data | None:
                Data class that includes a Pandas data frame. None in case of an error.

        Side Effects:
            self._pht is set to the PHT object created by this method.

        """

        try:
            from pyxecm_internal.pht import PHT
        except ModuleNotFoundError:
            self.logger.info(
                "Python module 'pyxecm_internal' not installed. Cannot process PHT data source!",
            )
            return None

        # 1. Read and validate values from the data source payload:
        pht_base_url = data_source.get("pht_base_url", "")
        if not pht_base_url:
            self.logger.error(
                "PHT base URL (pht_base_url) is not specified in payload of bulk data source. Cannot load data!",
            )
            return None
        pht_username = data_source.get("pht_username", "")
        if not pht_username:
            self.logger.error(
                "PHT username (pht_username) is not specified in payload of bulk data source. Cannot load data!",
            )
            return None
        pht_password = data_source.get("pht_password", "")
        if not pht_password:
            self.logger.error(
                "PHT password (pht_password) is not specified in payload of bulk data source. Cannot load data!",
            )
            return None

        pht_business_unit_exclusions = data_source.get(
            "pht_business_unit_exclusions",
            [],
        )
        pht_business_unit_inclusions = data_source.get(
            "pht_business_unit_inclusions",
            [],
        )
        pht_product_exclusions = data_source.get("pht_product_exclusions", [])
        pht_product_inclusions = data_source.get("pht_product_inclusions", [])
        pht_product_category_exclusions = data_source.get(
            "pht_product_category_exclusions",
            [],
        )
        pht_product_category_inclusions = data_source.get(
            "pht_product_category_inclusions",
            [],
        )
        pht_product_status_exclusions = data_source.get(
            "pht_product_status_exclusions",
            [],
        )
        pht_product_status_inclusions = data_source.get(
            "pht_product_status_inclusions",
            [],
        )
        pht_product_attributes = data_source.get(
            "pht_product_attributes",
            [],
        )

        self.logger.info(
            "Loading data from OpenText PHT -> %s (Product Hierarchy)...",
            pht_base_url,
        )

        # 2. Creating the PHT object:
        self._pht = PHT(
            base_url=pht_base_url,
            username=pht_username,
            password=pht_password,
            business_unit_exclusions=pht_business_unit_exclusions,
            business_unit_inclusions=pht_business_unit_inclusions,
            product_exclusions=pht_product_exclusions,
            product_inclusions=pht_product_inclusions,
            product_category_exclusions=pht_product_category_exclusions,
            product_category_inclusions=pht_product_category_inclusions,
            product_status_exclusions=pht_product_status_exclusions,
            product_status_inclusions=pht_product_status_inclusions,
            logger=self.logger,
        )
        if not self._pht:
            self.logger.error("Failed to initialize PHT object!")
            return None

        # 3. Authenticate at PHT
        token = self._pht.authenticate()
        if not token:
            self.logger.error(
                "Failed to authenticate at OpenText PHT -> %s!",
                pht_base_url,
            )
            return None
        else:
            self.logger.info(
                "Successfully authenticated at OpenText PHT -> %s.",
                pht_base_url,
            )

        # 4. Load the PHT product information into the Data object (Pandas DataFrame):
        if not self._pht.load_products(attributes_to_extract=pht_product_attributes):
            self.logger.error("Failure during load of OpenText PHT products!")
            return None

        data = self._pht.get_data()
        if data is None:
            self.logger.error("Failure during load of OpenText PHT product data!")
            return None

        return data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="prepare_datasource_file")
    def prepare_datasource_file(self, data_source: dict, filename: str) -> str:
        """Download data source from HTTP/HTTPS link and store it in a local temp file.

        Args:
            data_source (dict):
                The data source object from payload.
            filename (str):
                The filename to check if it is a reference to an external file.

        Returns:
            str:
                Filename or Filename of the local temp file.

        """

        if not filename.startswith(("http://", "https://")):
            return filename

        name = data_source.get("name", "")
        tmp_filename = os.path.join(tempfile.gettempdir(), "{}_{}".format(name, os.path.basename(filename)))

        if os.path.isfile(tmp_filename):
            self.logger.info("Reusing previously downloaded file -> '%s'.", tmp_filename)
            return tmp_filename

        try:
            self.logger.info(
                "Downloading data source from -> %s to -> %s...",
                filename,
                tmp_filename,
            )
            response = requests.get(filename, stream=True, timeout=10)

            with open(tmp_filename, "wb") as f:
                for chunk in response.iter_content(chunk_size=1024):
                    if chunk:
                        f.write(chunk)

        except Exception:
            self.logger.error("Error downloading JSON data source -> '%s'!", filename)

        return tmp_filename

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_excel")
    def process_bulk_datasource_excel(self, data_source: dict) -> Data | None:
        """Load data from Excel files into the data frame of the Data class.

        See helper/data.py

        Args:
            data_source (dict):
                Payload dict element for the data source.

        Returns:
            Data | None:
                Data class that includes a Pandas data frame. None in case of an error.

        """

        # 1. Read and validate values from the data source payload:
        xlsx_files = data_source.get("xlsx_files", [])
        if not xlsx_files:
            self.logger.error(
                "Excel files not specified in payload of bulk data source. Cannot load data!",
            )
            return None
        xlsx_sheets = data_source.get("xlsx_sheets", 0)  # use 0 not None!
        xlsx_columns = data_source.get("xlsx_columns")
        xlsx_skip_rows = data_source.get("xlsx_skip_rows", 0)
        xlsx_na_values = data_source.get("xlsx_na_values")

        # 2. Initialize Data object:
        data = Data(logger=self.logger)

        # 3. Iterate of Excel files and load them into the Data object:
        for xlsx_file in xlsx_files:
            if not data.load_excel_data(
                xlsx_path=xlsx_file,
                sheet_names=xlsx_sheets,
                usecols=xlsx_columns,
                skip_rows=xlsx_skip_rows,
                na_values=xlsx_na_values,
            ):
                self.logger.error("Failed to load Excel file -> '%s'!", xlsx_file)
                return None

        return data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_xml")
    def process_bulk_datasource_xml(self, data_source: dict) -> Data | None:
        """Load data from XML files or directories or zip files into the data frame of the Data class.

        See helper/data.py

        Args:
            data_source (dict):
                Payload dict element for the data source.

        Returns:
            Data | None:
                Data class that includes a Pandas data frame. None in case of an error.

        """

        # 1. Read and validate values from the data source payload:
        xml_files = data_source.get("xml_files", [])
        xml_directories = data_source.get(
            "xml_directories",
            [],
        )  # can also be zip files
        xml_xpath = data_source.get("xml_xpath")

        # 2. Initialize Data object:
        data = Data(logger=self.logger)

        # 3. If no XML directories are specified we interpret the "xml_files"
        if not xml_directories:
            for xml_file in xml_files:
                self.logger.info("Loading XML file -> '%s'...", xml_file)
                if not data.load_xml_data(xml_path=xml_file, xpath=xml_xpath):
                    self.logger.error("Failed to load XML file -> '%s'!", xml_file)
                    return None

        # 4. If XML directories or zip files of XML files are given we traverse them instead:
        else:
            # we now produce a list of dictionaries:
            xml_data = XML.load_xml_files_from_directories(
                directories=xml_directories,
                filenames=xml_files,
                xpath=xml_xpath,
                logger=self.logger.getChild("xml"),
            )
            if xml_data is None:  # test on error
                self.logger.error(
                    "Failed to load XML files from directories or ZIP files -> %s!",
                    xml_directories,
                )
                return None
            if not xml_data:  # test on empty result
                self.logger.warning(
                    "Found no XML files in directory or ZIP files -> %s!",
                    xml_directories,
                )
            else:
                data.append(add_data=xml_data)

        return data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_json")
    def process_bulk_datasource_json(self, data_source: dict) -> Data | None:
        """Load data from JSON files into the data frame of the Data class.

        See helper/data.py

        Args:
            data_source (dict):
                Payload dict element for the data source.

        Returns:
            Data | None:
                Data class that includes a Pandas data frame. None in case of an error.

        """

        # 1. Read and validate values from the data source payload:
        json_files = data_source.get("json_files", [])
        if not json_files:
            self.logger.error(
                "JSON files not specified in payload of bulk data source. Cannot load data!",
            )
            return None

        # 2. Initialize Data object:
        data = Data(logger=self.logger)

        # 3. Iterate JSON files and load data into Data object:
        for json_file in json_files:
            json_file = self.prepare_datasource_file(
                data_source=data_source,
                filename=json_file,
            )

            if not data.load_json_data(json_path=json_file):
                self.logger.error(
                    "Invalid JSON file -> '%s'. Cannot load it!",
                    json_file,
                )
                return None

        return data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_csv")
    def process_bulk_datasource_csv(self, data_source: dict) -> Data | None:
        """Load data from CSV files (comma-separated values) into the data frame of the Data class.

        See helper/data.py

        Args:
            data_source (dict):
                Payload dict element for the data source.

        Returns:
            Data | None:
                Data class that includes a Pandas data frame. None in case of an error.

        """

        # 1. Read and validate values from the data source payload:
        csv_files = data_source.get("csv_files", [])
        if not csv_files:
            self.logger.error(
                "CSV files not specified in payload of bulk data source. Cannot load data!",
            )
            return None
        csv_delimiter = data_source.get("csv_delimiter", ",")
        # The header index is an integer. The first row / line has the index 0.
        # But 0 is not the default! The default is the CSV does not have a header line
        # at all (None):
        csv_header_index = data_source.get("csv_header_index")
        csv_column_names = data_source.get("csv_column_names")
        csv_use_columns = data_source.get("csv_use_columns")

        # If no headers is specified it means the CSV does not have column
        # names in a row (typically row 0 = first row). If we also don't
        # have the names for the columns we will end with having coumn names
        # that a index values (1, 2, 3, ...). This may not be what the payload
        # author wants - so we issue a warning:
        if not csv_column_names and csv_header_index is None:  # "is None" is important here as the index can be 0
            self.logger.warning(
                "CSV loader parameters are missing both, header line index (no 'csv_header_index') and column names (no 'csv_column_names'). This could lead to numeric column names.",
            )

        # 2. Initialize Data object:
        data = Data(logger=self.logger)

        # 3. Iterate CSV files and load data into Data object:
        for csv_file in csv_files:
            if not data.load_csv_data(
                csv_path=csv_file,
                delimiter=csv_delimiter,
                names=csv_column_names,
                header=csv_header_index,
                usecols=csv_use_columns,
            ):
                self.logger.error("failed to load CSV file -> '%s'!", csv_file)
                return None

        return data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_web")
    def process_bulk_datasource_web(self, data_source: dict) -> Data | None:
        """Load data from Web page into the data frame of the Data class.

        See helper/data.py

        Args:
            data_source (dict):
                Payload dict element for the data source

        Returns:
            Data | None:
                Data class that includes a Pandas DataFrame. None in case of an error.

        """

        # 1. Read and validate values from the data source payload:
        web_url_templates = data_source.get("web_url_templates")
        if not web_url_templates:
            self.logger.error(
                "Web URLs (templates) not specified in payload of bulk data source. Cannot load data!",
            )
            return None
        web_start_value = data_source.get("web_start_value")
        web_end_value = data_source.get("web_end_value")
        web_value_name = data_source.get(
            "web_value_name",
        )  # which column name to use for the value
        web_special_url_templates = data_source.get("web_special_url_templates", [])
        # Pattern to filter file links on the page
        web_file_url_pattern = data_source.get("web_file_url_pattern")

        web_values = list(range(web_start_value, web_end_value + 1)) if web_start_value and web_end_value else None
        web_special_values = data_source.get("web_special_values", [])

        # 2. Initialize Data object:
        data = Data(logger=self.logger)

        # 3. Iterate over Web pages and load data into Data object:
        if not data.load_web(
            values=web_values,
            value_name=web_value_name,
            url_templates=web_url_templates,
            special_values=web_special_values,
            special_url_templates=web_special_url_templates,
            pattern=web_file_url_pattern,
        ):
            self.logger.error(
                "Failed to load Web data from URLs -> %s!",
                str(web_url_templates),
            )
            return None

        return data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_nhc")
    def process_bulk_datasource_nhc(self, data_source: dict) -> Data | None:
        """Load data from National Hurricane Center data source into the data frame of the Data class.

        See helper/data.py

        Args:
            data_source (dict):
                Payload dict element for the data source

        Returns:
            Data | None:
                Data class that includes a Pandas data frame. None in cause of an error.

        Side Effects:
            self._nhc is set to the NHC object created by this method.

        """
        try:
            from pyxecm_internal.nhc import (
                NHC,
            )
        except ModuleNotFoundError:
            self.logger.info(
                "Python module 'pyxecm_internal' not installed. Cannot process NHC data source!",
            )
            return None

        # 1. Read and validate values from the data source payload:
        nhc_year_start = data_source.get("nhc_year_start", "2023")
        nhc_year_end = data_source.get("nhc_year_end", "2024")
        nhc_basin = data_source.get("nhc_basin", "north_atlantic")
        nhc_track_images = data_source.get("nhc_track_images", ["png"])
        nhc_track_data = data_source.get("nhc_track_data", [])
        nhc_rain_images = data_source.get("nhc_rain_images", ["png"])
        nhc_rain_data = data_source.get("nhc_rain_data", [])
        nhc_skip_existing_files = data_source.get("nhc_skip_existing_files", True)
        nhc_async = data_source.get("nhc_async", True)
        nhc_async_processes = data_source.get("nhc_async_processes", 5)
        nhc_async_processes = data_source.get("nhc_async_processes", 5)
        nhc_storm_plot_exclusions = data_source.get("nhc_storm_plot_exclusions", [])
        nhc_download_dir = data_source.get("nhc_download_dir", "/data/nhc")
        # We either get the download dir for images from the payload
        # directly or we construct it from the general download dir:
        nhc_download_dir_images = data_source.get(
            "nhc_download_dir_images",
            os.path.join(nhc_download_dir, "images"),
        )  # don't use "/images"
        # We either get the download dir for data files (JSON) from the
        # payload directly or we construct it from the general download dir:
        nhc_download_dir_data = data_source.get(
            "nhc_download_dir_images",
            os.path.join(nhc_download_dir, "json"),
        )  # don't use "/data"

        # 2. Creating the NHC object:
        self._nhc = NHC(
            basin=nhc_basin,
            storm_plot_exclusions=nhc_storm_plot_exclusions,
            download_dir_images=nhc_download_dir_images,
            download_dir_data=nhc_download_dir_data,
            logger=self.logger,
        )
        if not self._nhc:
            self.logger.error("Failed to initialize NHC object!")
            return None

        # 3. Load the NHC storms into the Data object (Pandas DataFrame):
        if not self._nhc.load_storms(
            year_start=nhc_year_start,
            year_end=nhc_year_end,
            save_track_images=nhc_track_images,
            save_track_data=nhc_track_data,
            save_rain_images=nhc_rain_images,
            save_rain_data=nhc_rain_data,
            skip_existing_files=nhc_skip_existing_files,
            load_async=nhc_async,
            async_processes=nhc_async_processes,
        ):
            self.logger.error("Failure during load of NHC storms!")
            return None

        data = self._nhc.get_data()
        if data is None:
            self.logger.error("Failure during load of NHC storm data!")
            return None

        return data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_filesystem")
    def process_bulk_datasource_filesystem(self, data_source: dict) -> Data | None:
        """Load data from filesystem into the data frame of the Data class.

        See helper/data.py

        Args:
            data_source (dict):
                Payload dict element for the data source.

        Returns:
            Data | None:
                Data class that includes a Pandas data frame. None in case of an error.

        """

        # 1. Read and validate values from the data source payload:
        root_folders = data_source.get("root_folders", [])
        if not root_folders:
            self.logger.error(
                "Root folders not specified in payload of bulk data source. Cannot load data from filesystem!",
            )
            return None

        if not isinstance(root_folders, list):
            self.logger.warning(
                "The payload for root folders of the 'filesytem' data source should be a list! You should adjust the 'root_folders' setting.",
            )
            root_folders: list = [root_folders]

        # 2. Initialize Data object:
        data = Data(logger=self.logger)

        # 3. Iterate root folders and load data into Data object:
        for folder in root_folders:
            if not data.load_directory(path_to_root=folder):
                self.logger.error(
                    "Failed to load data from root folder -> '%s'!",
                    folder,
                )
                return None

        return data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource")
    def process_bulk_datasource(
        self,
        data_source_name: str,
        force_reload: bool = True,
    ) -> Data | None:
        """Process a data source that is given by a payload element.

        Parse its properties and deliver a 'Data' object which is a wrapper for
        a Pandas data frame.

        Args:
            data_source_name (str):
                The data source name.
            force_reload (bool):
                Force a reload of the data source if True.

        Returns:
            Data | None:
                Data source object of type Data. None in case of an error.

        """

        if not data_source_name:
            self.logger.error("Missing data source name!")
            return None

        self._log_header_callback(
            text="Process Bulk Data Source -> '{}'".format(data_source_name),
            char="-",
        )

        self.logger.info(
            "Found specified data source name -> '%s'. Lookup the data source payload...",
            data_source_name,
        )
        data_source = next(
            (item for item in self._bulk_datasources if item["name"] == data_source_name),
            None,
        )
        if not data_source:  # does this data source not exist?
            self.logger.error(
                "Cannot find specified data source -> '%s' in payload!",
                data_source_name,
            )
            return None

        # Check if data has already been loaded for the data source:
        if "data" in data_source and not force_reload:
            self.logger.info(
                "Data for data source -> '%s' is already loaded and reload is not enforced. Return existing data...",
                data_source_name,
            )
            return data_source["data"]
        elif force_reload:
            self.logger.info(
                "Reload of data from data source -> '%s' is enforced. Building data...",
                data_source_name,
            )
        else:
            self.logger.info(
                "Data for data source -> '%s' is not yet available. Building data...",
                data_source_name,
            )
            # Try to load the data source in the Admin Personal Workspace for investigations:
            if data_source.get("load_data_source", False):
                self.logger.info(
                    "Payload allows to reload existing data source. Check if the data source -> '%s' can be reloaded...",
                    data_source_name,
                )
                if self.read_data_source_file(data_source=data_source):
                    return data_source["data"]
                else:
                    self.logger.warning(
                        "Couldn't load the data source -> '%s' from file. Fall back to reloading it from the source...",
                        data_source_name,
                    )

        data_source_type = data_source.get("type", None)
        if not data_source_type:
            self.logger.error(
                "Data source needs a type parameter. This is new - you may need to add it to your bulkDataSource payload definition file! Cannot load data.",
            )
            return None

        match data_source_type:
            case "excel":
                data = self.process_bulk_datasource_excel(data_source=data_source)
                if data is None:
                    self.logger.error("Failure during load of Excel file!")
                    return None
            case "servicenow":
                data = self.process_bulk_datasource_servicenow(data_source=data_source)
                if data is None:
                    self.logger.error("Failure during load of ServiceNow articles!")
                    return None
            case "otmm":
                data = self.process_bulk_datasource_otmm(data_source=data_source)
                if data is None:
                    self.logger.error(
                        "Failure during load of OpenText Media Management assets!",
                    )
                    return None
            case "otcs":
                data = self.process_bulk_datasource_otcs(data_source=data_source)
                if data is None:
                    self.logger.error(
                        "Failure during load of OpenText Content Server items!",
                    )
                    return None
            case "pht":
                data = self.process_bulk_datasource_pht(data_source=data_source)
                if data is None:
                    self.logger.error(
                        "Failure during load of OpenText Product Hierarchy (PHT)!",
                    )
                    return None
            case "nhc":
                data = self.process_bulk_datasource_nhc(data_source=data_source)
                if data is None:
                    self.logger.error(
                        "Failure during load of National Hurricane Center data (NHC)!",
                    )
                    return None
            case "json":
                data = self.process_bulk_datasource_json(data_source=data_source)
                if data is None:
                    self.logger.error("Failure during load of JSON data source!")
                    return None
            case "xml":
                data = self.process_bulk_datasource_xml(data_source=data_source)
                if data is None:
                    self.logger.error("Failure during load of XML data source!")
                    return None
            case "csv":
                data = self.process_bulk_datasource_csv(data_source=data_source)
                if data is None:
                    self.logger.error("Failure during load of CSV data source!")
                    return None
            case "web":
                data = self.process_bulk_datasource_web(data_source=data_source)
                if data is None:
                    self.logger.error("Failure during load of Web data source!")
                    return None
            case "filesystem":
                data = self.process_bulk_datasource_filesystem(data_source=data_source)
                if data is None:
                    self.logger.error("Failure during load of Web data source!")
                    return None
            case _:
                self.logger.error(
                    "Illegal data source type. Types supported: 'excel', 'servicenow', 'otmm', 'otcs', 'pht', 'json', 'csv', 'xml', 'web', or 'filesystem'",
                )
                return None

        if data.get_data_frame() is None or data.get_data_frame().empty:
            self.logger.warning("Data source is empty - nothing loaded.")
            return None

        self.logger.info(
            "Data Frame for source -> '%s' has %s row(s) and %s column(s) after data loading.",
            data_source_name,
            data.get_data_frame().shape[0],
            data.get_data_frame().shape[1],
        )

        cleansings = data_source.get("cleansings", {})
        columns_to_drop = data_source.get("columns_to_drop", [])
        columns_to_keep = data_source.get("columns_to_keep", [])
        columns_to_add = data_source.get("columns_to_add", [])
        columns_to_add_list = data_source.get("columns_to_add_list", [])
        columns_to_add_concat = data_source.get("columns_to_add_concat", [])
        columns_to_add_table = data_source.get("columns_to_add_table", [])
        filters = data_source.get("filters", [])
        if not filters:
            # Backward compatibility. This used to be called "conditions" before
            # but we don't want to mix this with the conditions on row level
            # we have for bulkWorkspaces and bulkDocuments:
            filters = data_source.get("conditions", [])
        explosions = data_source.get("explosions", [])

        # Cleanup data if specified in data_source. We do this first!
        if cleansings:
            self.logger.info(
                "Start cleansing for data source -> '%s'...",
                data_source_name,
            )
            data.cleanse(cleansings=cleansings)
            self.logger.info(
                "Cleansing for data source -> '%s' completed...",
                data_source_name,
            )

        # Add columns if specified in data_source:
        for add_column in columns_to_add:
            if "source_column" not in add_column or "name" not in add_column:
                self.logger.error(
                    "Add columns is missing name or source column. Column will not be added.",
                )
                continue
            data.add_column(
                source_column=add_column["source_column"],
                new_column=add_column["name"],
                reg_exp=add_column.get("regex", add_column.get("reg_exp", None)),
                prefix=add_column.get("prefix", ""),
                suffix=add_column.get("suffix", ""),
                length=add_column.get("length", None),
                group_chars=add_column.get("group_chars", None),
                group_separator=add_column.get("group_separator", "."),
            )

        # Add columns with list values from a list of other columns
        # if specified in data_source:
        for add_column in columns_to_add_list:
            if "source_columns" not in add_column or "name" not in add_column:
                self.logger.error(
                    "Add list columns is missing name or source columns. Column will not be added.",
                )
                continue
            data.add_column_list(
                source_columns=add_column["source_columns"],
                new_column=add_column["name"],
            )

        # Add columns with list values from a list of other columns
        # if specified in data_source:
        for add_column in columns_to_add_concat:
            if "source_columns" not in add_column or "name" not in add_column:
                self.logger.error(
                    "Add concatenation columns is missing name or source columns. Column will not be added.",
                )
                continue
            data.add_column_concat(
                source_columns=add_column["source_columns"],
                new_column=add_column["name"],
                concat_char=add_column.get("concat_chars", ""),
                lower=add_column.get("lower", False),
                upper=add_column.get("upper", False),
                capitalize=add_column.get("capitalize", False),
                title=add_column.get("title", False),
            )

        # Add columns with list values from a list of other columns
        # if specified in data_source:
        for add_column in columns_to_add_table:
            if "source_columns" not in add_column or "name" not in add_column:
                self.logger.error(
                    "Add table columns is missing name or source columns. Column will not be added.",
                )
                continue
            data.add_column_table(
                source_columns=add_column["source_columns"],
                new_column=add_column["name"],
                delimiter=add_column.get("list_splitter", ","),
            )

        # Drop columns if specified in data_source:
        if columns_to_drop:
            data.drop_columns(columns_to_drop)

        # Keep only selected columns if specified in data_source:
        if columns_to_keep:
            data.keep_columns(columns_to_keep)

        # Check if fields with list substructures should be exploded:
        for explosion in explosions:
            # The explode field parameter can be a string or a list.
            # Exploding multiple fields at once avoids combinatorial explosions -
            # this is VERY different from exploding columns one after the other!
            if "explode_field" not in explosion and "explode_fields" not in explosion:
                self.logger.error("Missing explosion field(s)!")
                continue
            # We support both syntax (singular + plural):
            explode_fields = (
                explosion["explode_field"] if "explode_fields" not in explosion else explosion["explode_fields"]
            )

            flatten_fields = explosion.get("flatten_fields", [])
            split_string_to_list = explosion.get("split_string_to_list", False)
            list_splitter = explosion.get("list_splitter", None)
            self.logger.info(
                "Starting explosion of data source -> '%s' by field(s) -> '%s' (type -> '%s'). Size of data frame before explosion -> %s.",
                data_source_name,
                str(explode_fields),
                type(explode_fields),
                str(len(data)),
            )
            data.explode_and_flatten(
                explode_fields=explode_fields,
                flatten_fields=flatten_fields,
                make_unique=False,
                split_string_to_list=split_string_to_list,
                separator=list_splitter,
                reset_index=True,
            )
            self.logger.info("Size of data frame after explosion -> %s.", str(len(data)))

        # Keep only selected rows if filters are specified in data_source
        # We have this after "explosions" to allow access to subfields as well:
        if filters:
            data.filter(conditions=filters)

        # Keep the Data Frame for later lookups:
        data_source["data"] = data

        self._log_header_callback(
            text="Completed Bulk Data Source -> '{}'".format(data_source_name),
            char="-",
        )

        # Save the data source in the Admin Personal Workspace for investigations:
        if data_source.get("save_data_source", False):
            self.write_data_source_file(data_source=data_source)

        return data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_workspaces")
    def process_bulk_workspaces(self, section_name: str = "bulkWorkspaces") -> bool:
        """Process workspaces in payload and bulk create them in Content Server (multi-threaded).

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise.

        """

        if not self._bulk_workspaces:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        # For efficient idem-potent operation we may want to see which workspaces
        # have already been processed before:
        if self.check_status_file(
            payload_section_name=section_name,
            payload_specific=True,
            prefix="failure_",
        ):
            # Read payload from where we left it last time
            self._bulk_workspaces = self.get_status_file(
                payload_section_name=section_name,
                prefix="failure_",
            )
            if not self._bulk_workspaces:
                self.logger.error(
                    "Cannot load existing bulkWorkspaces failure file. Bailing out!",
                )
                return False

        success: bool = True

        for bulk_workspace in self._bulk_workspaces:
            # Check if element has been disabled in payload (enabled = false).
            # In this case we skip the element:
            enabled = bulk_workspace.get("enabled", True)
            if not enabled:
                self.logger.info("Payload for bulk workspace is disabled. Skipping...")
                continue

            # Read workspace type name from payload:
            type_name = bulk_workspace.get("type_name", None)
            if not type_name:
                self.logger.error(
                    "Bulk workspace needs a workspace type name! Skipping to next workspace...",
                )
                success = False
                continue

            # The payload element must have a "data_source" key:
            data_source_name = bulk_workspace.get("data_source", None)
            if not data_source_name:
                self.logger.error("Missing data source name in bulk workspace!")
                success = False
                continue

            self._log_header_callback(
                text="Process bulk workspaces for -> '{}' using data source -> '{}'".format(
                    type_name,
                    data_source_name,
                ),
                char="-",
            )

            copy_data_source = bulk_workspace.get("copy_data_source", False)
            force_reload = bulk_workspace.get("force_reload", True)

            # Load and prepare the data source for the bulk processing:
            if copy_data_source:
                self.logger.info(
                    "Take a copy of data source -> '%s' to avoid side-effects for repeative usage of the data source...",
                    data_source_name,
                )
                data = Data(
                    self.process_bulk_datasource(
                        data_source_name=data_source_name,
                        force_reload=force_reload,
                    ),
                    logger=self.logger,
                )
            else:
                self.logger.info(
                    "Use original data source -> '%s' and not do a copy.",
                    bulk_workspace["data_source"],
                )
                data = self.process_bulk_datasource(
                    data_source_name=data_source_name,
                    force_reload=force_reload,
                )
            if data is None:  # important to use "is None" here!
                self.logger.error(
                    "Failed to load data source -> '%s' for bulk workspace type -> '%s'!",
                    data_source_name,
                    type_name,
                )
                success = False
                continue
            if data.get_data_frame() is None:  # important to use "is None" here!
                self.logger.error(
                    "Data source for bulk workspace type -> '%s' is empty!",
                    type_name,
                )
                continue

            # Check if fields with list substructures should be exploded.
            # We may want to do this outside the bulkDatasource to only
            # explode for bulkDocuments and not for bulkWorkspaces or
            # bulkWorkspaceRelationships:
            explosions = bulk_workspace.get("explosions", [])
            for explosion in explosions:
                # explode field can be a string or a list
                # exploding multiple fields at once avoids
                # combinatorial explosions - this is VERY
                # different from exploding columns one after the other!
                if "explode_field" not in explosion and "explode_fields" not in explosion:
                    self.logger.error("Missing explosion field(s)!")
                    continue
                # we want to be backwards compatible...
                explode_fields = (
                    explosion["explode_field"] if "explode_field" in explosion else explosion["explode_fields"]
                )
                flatten_fields = explosion.get("flatten_fields", [])
                split_string_to_list = explosion.get("split_string_to_list", False)
                list_splitter = explosion.get(
                    "list_splitter",
                    ",",
                )  # don't have None as default!
                self.logger.info(
                    "Starting explosion of bulk workspaces by field(s) -> %s (type -> %s). Size of data frame before explosion -> %s.",
                    explode_fields,
                    type(explode_fields),
                    str(len(data)),
                )
                data.explode_and_flatten(
                    explode_fields=explode_fields,
                    flatten_fields=flatten_fields,
                    make_unique=False,
                    split_string_to_list=split_string_to_list,
                    separator=list_splitter,
                    reset_index=True,
                )
                self.logger.info(
                    "Size of data frame after explosion -> %s.",
                    str(len(data)),
                )

            # Keep only selected rows if filters are specified in bulkWorkspaces.
            # We have this _after_ "explosions" to allow access to subfields as well.
            # We have this _before_ "sorting" and "deduplication" as may keep the wrong
            # rows otherwise (unique / deduplication always keeps the first matching row).
            # We may want to do this outside the bulkDatasource to only
            # filter for bulkDocuments and not for bulkWorkspaces or
            # bulkWorkspaceRelationships:
            filters = bulk_workspace.get("filters", [])
            if filters:
                data.filter(conditions=filters)

            # Sort the data frame if "sort" specified in payload. We may want to do this to have a
            # higher chance that rows with workspace names that may collapse into
            # one name are put into the same partition. This can avoid race conditions
            # between different Python threads.
            sort_fields = bulk_workspace.get("sort", [])
            if sort_fields:
                self.logger.info(
                    "Start sorting of data frame for workspace type -> '%s' based on fields (columns) -> %s...",
                    type_name,
                    str(sort_fields),
                )
                data.sort(sort_fields=sort_fields, inplace=True)
                self.logger.info(
                    "Sorting of data frame for workspace type -> '%s' based on fields (columns) -> %s completed.",
                    type_name,
                    str(sort_fields),
                )

            # Check if duplicate rows for given fields should be removed. It is
            # important to do this after sorting as Pandas always keep the first occurance,
            # so ordering plays an important role in deduplication!
            unique_fields = bulk_workspace.get("unique", [])
            if unique_fields:
                self.logger.info(
                    "Starting deduplication of data frame for workspace type -> '%s' with unique fields -> %s. Size of data frame before deduplication -> %s.",
                    type_name,
                    str(unique_fields),
                    str(len(data)),
                )
                data.deduplicate(unique_fields=unique_fields, inplace=True)
                self.logger.info(
                    "Size of data frame after deduplication -> %s.",
                    str(len(data)),
                )

            # Read name field from payload:
            workspace_name_field = bulk_workspace.get("name", None)
            if not workspace_name_field:
                self.logger.error(
                    "Bulk workspace needs a name field! Skipping to next workspace...",
                )
                success = False
                continue

            # Read optional description field from payload:
            workspace_description_field = bulk_workspace.get("description", None)

            # Find the workspace type with the name given in the payload:
            workspace_type = next(
                (item for item in self._workspace_types if item["name"] == type_name),
                None,
            )
            if workspace_type is None:
                self.logger.error(
                    "Workspace type -> '%s' not found. Skipping to next bulk workspace...",
                    type_name,
                )
                success = False
                continue
            if workspace_type["templates"] == []:
                self.logger.error(
                    "Workspace type -> '%s' does not have templates. Skipping to next bulk workspace...",
                    type_name,
                )
                success = False
                continue

            # check if the template to be used is specified in the payload:
            if "template_name" in bulk_workspace:
                template_name_field = bulk_workspace["template_name"]
                workspace_template = next(
                    (item for item in workspace_type["templates"] if item["name"] == template_name_field),
                    None,
                )
                if workspace_template:  # does this template exist?
                    self.logger.info(
                        "Workspace template -> '%s' has been specified in payload and it does exist.",
                        template_name_field,
                    )
                elif "{" in template_name_field and "}" in template_name_field:
                    self.logger.info(
                        "Workspace template -> '%s' has been specified in payload and contains placeholders, validation cannot be performed.",
                        template_name_field,
                    )
                else:
                    self.logger.error(
                        "Workspace template -> '%s' has been specified in payload but it doesn't exist!",
                        template_name_field,
                    )
                    self.logger.error(
                        "Workspace type -> '%s' has only these templates -> %s.",
                        type_name,
                        workspace_type["templates"],
                    )
                    success = False
                    continue
            # template to be used is NOT specified in the payload - then we just take the first one:
            else:
                workspace_template = workspace_type["templates"][0]
                template_name_field = None
                self.logger.info(
                    "Workspace template has not been specified in payload - taking the first one (%s)...",
                    workspace_template,
                )

            # Read the optional categories payload:
            categories = bulk_workspace.get("categories", None)
            if not categories:
                self.logger.info(
                    "Bulk workspace payload has no category data! Will leave category attributes empty...",
                )

            # Should existing workspaces be updated? No is the default.
            operations = bulk_workspace.get("operations", ["create"])

            self.logger.info(
                "Bulk create workspaces (name field -> %s, type -> '%s') from workspace template -> '%s'. Operations -> %s.",
                workspace_name_field,
                type_name,
                template_name_field,
                str(operations),
            )

            # See if bulkWorkspace definition has a specific thread number
            # otherwise it is read from a global environment variable
            bulk_thread_number = int(
                bulk_workspace.get("thread_number", BULK_THREAD_NUMBER),
            )

            partitions = data.partitionate(bulk_thread_number)

            # Create a list to hold the threads
            threads = []
            results = []

            # Create and start a thread for each partition
            for index, partition in enumerate(partitions, start=1):
                # start a thread executing the process_bulk_workspaces_worker() method below:
                thread = threading.Thread(
                    name=f"{section_name}_{index:02}",
                    target=self.thread_wrapper,
                    args=(
                        self.process_bulk_workspaces_worker,
                        bulk_workspace,
                        partition,
                        workspace_type,
                        template_name_field,
                        workspace_name_field,
                        workspace_description_field,
                        categories,
                        operations,
                        results,
                    ),
                )
                self.logger.info("Starting thread -> '%s'...", str(thread.name))
                thread.start()
                threads.append(thread)
                # Avoid that all threads start at the exact same time with
                # potentially expired cookies that cases race conditions:
                time.sleep(1)
            # end for index, partition in enumerate(partitions, start=1)

            # Wait for all threads to complete
            for thread in threads:
                self.logger.info(
                    "Waiting for thread -> '%s' to complete...",
                    str(thread.name),
                )
                thread.join()
                self.logger.info("Thread -> '%s' has completed.", str(thread.name))

            if "workspaces" not in bulk_workspace:
                bulk_workspace["workspaces"] = {}

            # Print statistics for each thread. In addition,
            # check if all threads have completed without error / failure.
            # If there's a single failure in on of the thread results we
            # set 'success' variable to False.
            results.sort(key=lambda x: x["thread_name"])
            for result in results:
                self.logger.info(
                    "Thread -> '%s' completed with overall status '%s'.",
                    str(result["thread_name"]),
                    "successful" if result["success"] else "failed",
                )
                self.logger.info(
                    "Thread -> '%s' processed %s data rows with %s successful, %s failed, and %s skipped.",
                    str(result["thread_name"]),
                    str(
                        result["success_counter"] + result["failure_counter"] + result["skipped_counter"],
                    ),
                    str(result["success_counter"]),
                    str(result["failure_counter"]),
                    str(result["skipped_counter"]),
                )
                self.logger.info(
                    "Thread -> '%s' created %s workspaces, updated %s workspaces, and deleted %s workspaces.",
                    str(result["thread_name"]),
                    str(result["create_counter"]),
                    str(result["update_counter"]),
                    str(result["delete_counter"]),
                )

                if not result["success"]:
                    success = False
                # Record all generated workspaces. This should allow us
                # to restart in case of failures and avoid trying to
                # create workspaces that have been created before.
                bulk_workspace["workspaces"].update(result["workspaces"])
            self._log_header_callback(
                text="Completed processing of bulk workspaces for -> '{}' using data source -> '{}'".format(
                    type_name,
                    data_source_name,
                ),
                char="-",
            )

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._bulk_workspaces,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_categories")
    def process_bulk_categories(
        self,
        row: pd.Series,
        index: str,
        categories: list,
        replacements: list,
    ) -> list:
        """Replace the value placeholders of the bulk category structures.

        The payload placeholders are replaced with values from the Pandas Series (row).
        It also processes value mappings + lookups and special treatment of list and table values.

        Args:
            row (pd.Series):
                The current row in the Pandas data frame.
            index (str):
                The index of the Pandas data frame. Just used here for logging.
            categories (list):
                List of payload categories.
            replacements (list):
                List of replacements.

        Returns:
            list:
                List of categories.

        """

        # Make sure the threads are not changing data structures that
        # are shared between threads. categories is a list of dicts.
        # list and dicts are "mutable" data structures in Python!
        worker_categories = copy.deepcopy(categories)

        # In this first loop we expand table-value attributes into a new
        # list of category / attribute payload. The value of table-value attributes
        # is a list of dictionaries (in a string we evaluate into a Python
        # datastructure)
        worker_categories_expanded = []
        for category_item in worker_categories:
            if "value_type" in category_item and category_item["value_type"] == "table":
                value_field = category_item["value_field"]

                # The following method always returns a string even if the value is actually a list.
                # This is required as we want to support multiple placeholders in one string...
                value = self.replace_bulk_placeholders(
                    input_string=value_field,
                    row=row,
                    index=None,
                    replacements=replacements,
                )
                # If value is empty (probably because placeholders could not be resolved)
                # and if the payload provides an alternative value field we try this:
                if not value and "value_field_alt" in category_item:
                    value = self.replace_bulk_placeholders(
                        input_string=category_item["value_field_alt"],
                        row=row,
                        index=None,
                        replacements=replacements,
                    )
                if not value:
                    self.logger.info(
                        "Table-type attribute -> '%s' is empty (value field -> '%s'). Cannot parse table. Skipping...",
                        category_item.get("name", ""),
                        value_field,
                    )
                    continue

                try:
                    value_table = literal_eval(value)
                except (SyntaxError, ValueError):
                    self.logger.error(
                        "Cannot parse table-type attribute -> '%s'; value field -> '%s'",
                        category_item.get("name", ""),
                        value_field,
                    )
                    continue

                if not isinstance(value_table, list):
                    self.logger.error(
                        "Table-type value -> '%s' requires a list of dictionaries!",
                        value_field,
                    )
                    continue

                """
                Get the mapping of the loader generated columns in the Data Frame to the
                attribute names in the target OTCS category. If no mapping
                is in the payload, then it is assumed that the category
                attribute names are identical to the column names in the Data Frame

                Example mapping:

                attribute_mapping = {
                  "Application": "u_product_model",
                  "Version": "u_version_name"
                }
                """

                attribute_mapping = category_item.get("attribute_mapping", None)

                row_index = 1
                for value_row in value_table:
                    if not isinstance(value_row, dict):
                        self.logger.error(
                            "Illegal data type for table row -> %s. Expected 'dict', got -> '%s' with value -> %s",
                            str(row_index),
                            type(value_row),
                            value_row,
                        )
                        continue
                    for key, value in value_row.items():
                        attribute = {}
                        attribute["name"] = category_item.get("name", "")
                        attribute["set"] = category_item.get("set", "")
                        attribute["row"] = row_index
                        # check if we have a mapping for this attribute in the payload:
                        if attribute_mapping and key in attribute_mapping:
                            attribute["attribute"] = attribute_mapping[key]
                        else:
                            attribute["attribute"] = key
                        # For tables values can be None if the number of
                        # list items in the source columns are not equal.
                        # To avoid the warning below we set the value to empty string
                        # if it is None:
                        attribute["value"] = value if value is not None else ""
                        worker_categories_expanded.append(attribute)
                    row_index += 1
            # end if "value_type" in category_item and category_item["value_type"] == "table"
            else:
                worker_categories_expanded.append(category_item)
        # end for category_item in worker_categories

        # this loop generates a "value" for each
        # "value_field". "value_field" may also contain lists
        # that are either delimited by [...] or by a "value_type" with value "list"
        for category_item in worker_categories_expanded:
            if "attribute" not in category_item:
                self.logger.error(
                    "Category item -> %s is missing the attribute field!",
                    category_item,
                )
                continue

            # per default the value is in the "value" item:
            value = category_item.get("value", None)

            # is there a value replacement for the current attribute?
            if "value_field" in category_item:
                value_field = category_item["value_field"]
                # If no row is specified we set the set index to None and not 0 to handle
                # cases where we have multi-value attributes that take a list as parameter
                set_index = int(category_item["row"]) - 1 if "row" in category_item else None

                # this method always returns a string even if the value is
                # actually a list.
                value = self.replace_bulk_placeholders(
                    input_string=value_field,
                    row=row,
                    index=set_index,
                    replacements=replacements,
                )
                # If value is empty (probably because placeholders could not be resolved)
                # and if the payload provides an alternative value field we try this:
                if not value and "value_field_alt" in category_item:
                    value = self.replace_bulk_placeholders(
                        input_string=category_item["value_field_alt"],
                        row=row,
                        index=set_index,
                        replacements=replacements,
                    )
            else:
                value_field = None

            # if we don't have a value now, then there's an issue:
            if value is None:
                value = ""
                self.logger.warning(
                    "Category item needs either a value or value_field! Skipping attribute -> '%s'",
                    category_item["attribute"],
                )

            # We have an empty string value (this is different from None!)
            if value == "":
                category_item["value"] = value
                # We can continue as any further processing (below) does not make sense for an empty string value:
                continue

            # This variable should only be true if we don't have
            # a native python string but a delimiter separated
            # value list in a string, e.g. "a, b, c" or "a | b | c" or "x;y;z":
            is_list_in_string = False

            # The data source loader may have written a real python list into the value
            # In this case the string value includes square brackets [...]
            # We only do  this if the actual type of the value is string and the
            # proposed value (value_type) is not string:
            if (
                isinstance(value, str)  # it is a string
                and value.startswith("[")  # it starts with a list indicator character
                and value.endswith("]")  # it ends with a list indicator character
                and category_item.get("value_type", "")
                != "string"  # it is NOT explicitly stated it should remain a string
            ):
                # Remove the square brackets and declare it is a list!
                try:
                    value = literal_eval(value)
                except (SyntaxError, ValueError) as e:
                    self.logger.warning(
                        "Cannot directly parse list-type attribute; value field -> %s; error -> %s. Try string processing...",
                        value_field,
                        str(e),
                    )
                    self.logger.warning(
                        "Value string -> %s has [...] - remove brackets and interpret as delimiter separated values in a string...",
                        value,
                    )
                    # In this failure case we try to remove the square brackets and hope the inner part
                    # can be treated as a string of values delimited with a delimiter (e.g. comma or semicolon)
                    value = value.strip("[]")
                    is_list_in_string = True

            # Handle this special case where we get a string that actually represents a date time format and convert it.
            # We only do this if the attribute has a "value_type" of "datetime" in the payload:
            if category_item.get("value_type", "string") == "datetime":
                old_value = value
                try:
                    date_obj = parse(value)
                    value = datetime.strftime(date_obj, "%Y-%m-%dT%H:%M:%SZ")
                except (OverflowError, ValueError):
                    self.logger.error(
                        "Cannot convert value -> '%s' into a date for attribute -> '%s'. Value will be empty.",
                        old_value,
                        category_item["attribute"],
                    )
                    value = ""
                else:
                    self.logger.debug(
                        "Attribute -> %s is declared in payload to be a datetime (convert format). Converted it from -> %s to -> %s",
                        category_item.get("attribute"),
                        old_value,
                        value,
                    )

            # Handle special case where we get a string that actually represents a list but is
            # not yet a Python list but a string. This requires that value_type == "list". The list splitter
            # can be specified via "list_splitter" in the payload:
            # Now we try to convert the string to a Python list:
            if (category_item.get("value_type", "string") == "list" or is_list_in_string) and isinstance(value, str):
                # we split the string after splitter characters:
                list_splitter = category_item.get("list_splitter", ";,")
                self.logger.info(
                    "Value -> '%s' is declared in payload to be a list but it is provided as a string. Splitting it after these characters -> '%s'.",
                    value,
                    list_splitter,
                )

                # Escape the split characters to ensure they are treated literally in the regex pattern
                escaped_splitter = re.escape(list_splitter)

                # Construct the regex pattern dynamically
                pattern = rf"[{escaped_splitter}]\s*"

                # Split the value string at the defined splitter characters:
                elements = re.split(pattern, value)

                # Remove the quotes around each element
                elements = [element.strip("'") for element in elements]
                value = elements
                self.logger.info(
                    "Found list for a multi-value category attribute -> '%s' from field -> '%s' in data row -> %s. Value -> '%s'.",
                    category_item["attribute"],
                    value_field,
                    index,
                    str(value),
                )

            # now we check if there's a data lookup configured in the payload:
            lookup_data_source = category_item.get("lookup_data_source", None)
            # Do we want to drop / clear values that fail to lookup?
            drop_value = category_item.get("lookup_data_failure_drop", False)

            if lookup_data_source:
                self.logger.info(
                    "Found lookup data source -> '%s' for attribute -> '%s'. Processing...",
                    lookup_data_source,
                    category_item["attribute"],
                )
                if not isinstance(value, list):
                    # value is a single value and not a list:
                    (_, synonym) = self.process_bulk_workspaces_synonym_lookup(
                        data_source_name=lookup_data_source,
                        workspace_name_synonym=value,
                    )
                    if synonym:
                        self.logger.info(
                            "Found synonym -> '%s' for attribute -> '%s' and value -> '%s' in data source -> '%s'.",
                            synonym,
                            category_item["attribute"],
                            value,
                            lookup_data_source,
                        )
                        value = synonym
                    elif drop_value:
                        self.logger.warning(
                            "Cannot lookup the value for attribute -> '%s' and value -> '%s' in data source -> '%s'. Clear existing value.",
                            category_item["attribute"],
                            value,
                            lookup_data_source,
                        )
                        # Clear the value:
                        value = ""
                    else:
                        self.logger.warning(
                            "Cannot lookup the value for attribute -> '%s' and value -> '%s' in data source -> '%s'. Keep existing value.",
                            category_item["attribute"],
                            value,
                            lookup_data_source,
                        )
                # end if not isinstance(value, list)
                else:
                    # value is a list - so we apply the lookup to each item:
                    # Iterate backwards to avoid index issues while popping items:
                    for i in range(len(value) - 1, -1, -1):
                        # Make sure the value does not have leading or trailing spaces:
                        value[i] = value[i].strip()
                        (_, synonym) = self.process_bulk_workspaces_synonym_lookup(
                            data_source_name=lookup_data_source,
                            workspace_name_synonym=value[i],
                            workspace_type=None,  # we don't need the workspace ID, just the workspace name
                        )
                        if synonym:
                            self.logger.info(
                                "Found synonym -> '%s' for attribute -> '%s' and value -> '%s' in data source -> '%s'.",
                                synonym,
                                category_item["attribute"],
                                value[i],
                                lookup_data_source,
                            )
                            value[i] = synonym
                        elif drop_value:
                            self.logger.warning(
                                "Cannot lookup the value -> '%s' for attribute -> '%s' in data source -> '%s'! Drop existing value from list.",
                                value[i],
                                category_item["attribute"],
                                lookup_data_source,
                            )
                            # Remove the list item we couldn't lookup as drop_value is True:
                            value.pop(i)
                        else:
                            self.logger.warning(
                                "Cannot lookup the value -> '%s' for attribute -> '%s' in data source -> '%s'! Keep existing value.",
                                value[i],
                                category_item["attribute"],
                                lookup_data_source,
                            )
            # end if lookup_data_source

            # If value is a list then we have a multi-value attribute.
            # We now want to make sure that we don't have duplicates in
            # the value list:
            if isinstance(value, list) and len(value) != len(set(value)):
                self.logger.info(
                    "The multi-value attribute -> '%s' has duplicates in its values -> %s.",
                    category_item["attribute"],
                    value,
                )
                # Remove duplicates from the list while preserving order.
                # Uses `dict.fromkeys()` since dictionaries maintain insertion order (Python 3.7+).
                # This ensures that only the first occurrence of each element is kept.
                value = list(dict.fromkeys(value))
                self.logger.info("The attribute values got de-duplicated to -> %s.", value)

            # now we check if there's a values mapping configured in the payload.
            # This is a dictionary for keys being the original values and
            # values being the mapped values. This makes most sense for
            # values with a limited / fixed domain of values. Example
            # `value_mapping = { "TS": "Tropical Storm", "HU": "Hurricane"}`
            value_mapping = category_item.get("value_mapping", None)
            if value_mapping:
                if not isinstance(value, list):
                    # value is a single value and not a list:
                    if value in value_mapping:
                        self.logger.info(
                            "Found value mapping for attribute -> '%s' from value -> '%s' to value -> '%s'.",
                            category_item["attribute"],
                            value,
                            value_mapping[value],
                        )
                        value = value_mapping[value]
                else:
                    # value is a list - so we apply the lookup to each item:
                    # Iterate backwards to avoid index issues while popping items:
                    for i in range(len(value) - 1, -1, -1):
                        # Make sure the value does not have leading or trailing spaces:
                        value[i] = value[i].strip()

                        if value[i] in value_mapping:
                            self.logger.info(
                                "Found value mapping for attribute -> '%s' from value -> '%s' to value -> '%s'",
                                category_item["attribute"],
                                value[i],
                                value_mapping[value[i]],
                            )
                            value[i] = value_mapping[value[i]]

            # If value is a list then we have a multi-value attribute.
            # If "sort_multi_values" is specified in payload and evaluate to True
            # we sort the list alphabetically (note that numbers are not necessarily sorted numerically):
            if isinstance(value, list) and len(value) > 1 and category_item.get("sort_multi_values", False):
                value.sort(key=str)
                self.logger.info(
                    "Sorting the values of multi-value attribute -> '%s' to -> %s.",
                    category_item["attribute"],
                    value,
                )

            if value_field:
                self.logger.debug(
                    "Reading category -> '%s', attribute -> '%s' from field -> '%s' in data row -> %s. Value -> '%s'.",
                    category_item["name"],
                    category_item["attribute"],
                    value_field,
                    index,
                    str(value),
                )
            else:
                self.logger.debug(
                    "Setting category -> '%s', attribute -> '%s' to value -> '%s'.",
                    category_item["name"],
                    category_item["attribute"],
                    str(value),
                )
            category_item["value"] = value
        # end for category_item in worker_categories_expanded

        # Cleanup categories_payload to remove empty rows of sets:
        rows_to_remove = {}
        for attribute in worker_categories_expanded:
            if attribute.get("set") is not None and attribute.get("row") is not None:
                set_name = attribute["set"]
                row_number = attribute["row"]
                value = attribute["value"]

                # If value is empty, mark it for removal:
                if not value or value == [""]:  # Treat empty strings or None as empty
                    # The following if statement is important to not mark a set row
                    # as True = removable that has been set to False in the else case below!
                    if (set_name, row_number) not in rows_to_remove:
                        rows_to_remove[(set_name, row_number)] = True
                else:
                    # If any value in the row is not empty, mark the row as not removable.
                    # This may change the marking that have been applied in the if case
                    # above!
                    rows_to_remove[(set_name, row_number)] = False

        # Keep only the rows marked as True (empty rows to be removed)
        rows_to_remove = {k: v for k, v in rows_to_remove.items() if v is True}

        if rows_to_remove:
            self.logger.debug("Empty rows to remove from sets -> %s.", rows_to_remove)
        else:
            self.logger.debug("No empty rows to remove from sets.")

        cleaned_categories = [
            item
            for item in worker_categories_expanded
            if "set" not in item or "row" not in item or (item["set"], item["row"]) not in rows_to_remove
        ]

        return cleaned_categories

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_classification_assignments")
    def process_bulk_classification_assignments(
        self,
        row: pd.Series,
        index: str,
        classifications: list,
        replacements: list,
    ) -> list:
        """Replace the value placeholders of the bulk classification structures.

        The payload placeholders are replaced with values from the Pandas Series (row).

        Args:
            row (pd.Series):
                The current row in the Pandas data frame.
            index (str):
                The index of the Pandas data frame. Just used here for logging.
            classifications (list):
                List of payload classifications.
            replacements (list):
                List of replacements.

        Returns:
            list:
                List of classification IDs.

        """

        result_list = []

        for classification in classifications:
            # Do we have a classification path?
            if isinstance(classification, list):
                # Replace placeholders in the list. As list is a mutable data type,
                # the replacement happens in-place:
                self.replace_bulk_placeholders_list(input_list=classification, row=row, replacements=replacements)
                class_node = self._otcs_frontend.get_node_by_volume_and_path(
                    volume_type=self._otcs.VOLUME_TYPE_CLASSIFICATION_VOLUME,
                    path=classification,
                )
            elif isinstance(classification, string):
                nickname = self.replace_bulk_placeholders(
                    input_string=classification,
                    row=row,
                    replacements=replacements,
                )
                class_node = self._otcs_frontend.get_node_from_nickname(nickname=nickname)

            class_node_id = self._otcs.get_result_value(
                response=class_node,
                key="id",
            )
            if class_node_id:
                result_list.append(class_node_id)

        if result_list:
            self.logger.debug(
                "Found classifications for data row -> %s. Value -> %s",
                index,
                str(result_list),
            )

        return result_list

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_workspaces_worker")
    def process_bulk_workspaces_worker(
        self,
        bulk_workspace: dict,
        partition: pd.DataFrame,
        workspace_type: dict,
        template_name_field: str | None,
        workspace_name_field: str,
        workspace_description_field: str,
        categories: list | None = None,
        operations: list | None = None,
        results: list | None = None,
    ) -> None:
        """Thread worker to create workspaces in bulk.

        Each worker thread gets a partition of the rows that include
        the data required for the workspace creation.

        Args:
            bulk_workspace (dict):
                The payload of the bulkWorkspace.
            partition (pd.DataFrame):
                Data partition with rows to process.
            template_id (int):
                ID of the workspace template to use.
            workspace_type (dict):
                Workspace type data.
            template_name_field (str | None):
                Field where the template name is stored.
            workspace_name_field (str):
                Field where the workspace name is stored.
            workspace_description_field (str):
                Field where the workspace description is stored.
            categories (list, optional):
                List of category dictionaries.
            operations (list, optional):
                Defines which operations should be applyed on workspaces.
                Possible values are "create", "update", "delete", "recreate".
            results (list, optional):
                A mutable list of thread results.

        """

        thread_id = threading.get_ident()
        thread_name = threading.current_thread().name

        t = trace.get_current_span()
        t.set_attributes(
            {
                "thread.id": thread_id,
                "thread.name": thread_name,
            }
        )

        self.logger.info(
            "Start working on data frame partition of size -> %s to bulk create workspaces...",
            str(len(partition)),
        )

        # Avoid linter warnings - so make parameter default None while we
        # actually want ["create"] to be the default:
        if operations is None:
            operations = ["create"]

        result = {}
        result["thread_id"] = thread_id
        result["thread_name"] = thread_name
        result["success_counter"] = 0
        result["failure_counter"] = 0
        result["skipped_counter"] = 0
        result["create_counter"] = 0
        result["update_counter"] = 0
        result["delete_counter"] = 0
        result["workspaces"] = {}
        result["success"] = True

        # Check if workspaces have been processed before, e.i. testing
        # if a "workspaces" key exists and if it is pointing to a non-empty list.
        # Additionally we check that workspace updates are not enforced:
        if (
            bulk_workspace.get("workspaces")
            and "update" not in operations
            and "delete" not in operations
            and "recreate" not in operations
        ):
            existing_workspaces = bulk_workspace["workspaces"]
            self.logger.info(
                "Found %s already processed workspaces. Try to complete the job...",
                str(len(existing_workspaces)),
            )
        else:
            existing_workspaces = {}

        # See if external creation and modification dates are in the payload data:
        external_modify_date_field = bulk_workspace.get("external_modify_date")
        external_create_date_field = bulk_workspace.get("external_create_date")

        # See if we have a key field to uniquely identify an existing workspace:
        key_field = bulk_workspace.get("key")

        # Get dictionary of replacements for bulk workspace creations
        # this we will be used of all places data is read from the
        # data frame. Each dictionary item has the field name as the
        # dictionary key and a list of regular expressions as dictionary value
        replacements = bulk_workspace.get("replacements")

        # In case the name cannot be resolved we allow to
        # specify an alternative name field in the payload.
        name_field_alt = bulk_workspace.get("name_alt")

        # In case the description cannot be resolved we allow to
        # specify an alternative description field in the payload.
        description_field_alt = bulk_workspace.get("description_alt")

        # Fetch the nickname field from the payload (if it is specified):
        nickname_field = bulk_workspace.get("nickname")

        # In case the nickname cannot be resolved we allow to
        # specify an alternative nickname field in the payload.
        nickname_field_alt = bulk_workspace.get("nickname_alt")

        # Nicknames are very limited in terms of allowed characters.
        # For nicknames we need additional regexp as we need to
        # replace all non-alphanumeric, non-space characters with ""
        # We also preserve hyphens in the first step to replace
        # them below with underscores. This is important to avoid
        # that different spellings of names produce different nicknames.
        # We want spellings with spaces match spellings with hyphens.
        # For this the workspace names have a regexp "-| " in the payload.
        nickname_additional_regex_list = [r"[^\w\s-]"]

        # Classification can either be provided by classification pathes
        # or by nicknames:
        classification_pathes = bulk_workspace.get("classification_pathes", [])
        classification_nicknames = bulk_workspace.get("classification_nicknames", [])

        total_rows = len(partition)

        # Process all rows in the partition that was given to this thread:
        for index, row in partition.iterrows():
            with tracer.start_as_current_span("process_bulk_workspaces_worker-workspace") as t:
                # Calculate percentage of completion:
                percent_complete = ((partition.index.get_loc(index) + 1) / total_rows) * 100

                self.logger.info(
                    "Processing data row -> %s for bulk workspace creation (%.2f %% complete)...",
                    str(index),
                    percent_complete,
                )

                # Clear variables to esure clean state for each row:
                workspace_id = None

                workspace_template = None
                if template_name_field is None:
                    workspace_template = workspace_type["templates"][0]

                else:
                    workspace_template_name = self.replace_bulk_placeholders(
                        input_string=template_name_field,
                        row=row,
                        replacements=replacements,
                    )

                    workspace_template = next(
                        (item for item in workspace_type["templates"] if item["name"] == workspace_template_name),
                        None,
                    )

                if workspace_template is None:
                    self.logger.error(
                        "Workspace template -> '%s' has been specified in payload but it doesn't exist!",
                        workspace_template_name,
                    )
                    self.logger.error(
                        "Workspace type -> '%s' has only these templates -> %s",
                        workspace_type["name"],
                        workspace_type["templates"],
                    )
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue

                template_id = workspace_template["id"]
                template_name = workspace_template["name"]
                workspace_type_id = workspace_type["id"]

                # Determine the workspace name:
                workspace_name = self.replace_bulk_placeholders(
                    input_string=workspace_name_field,
                    row=row,
                    replacements=replacements,
                )
                # If we cannot get the workspace name from the
                # workspace_name_field we try the alternative name field
                # (if specified in payload via "name_alt"):
                if not workspace_name and name_field_alt:
                    self.logger.debug(
                        "Row -> %s does not have the data to resolve the placeholders in workspace name -> %s! Trying alternative name field -> %s...",
                        str(index),
                        workspace_name_field,
                        name_field_alt,
                    )
                    workspace_name = self.replace_bulk_placeholders(
                        input_string=name_field_alt,
                        row=row,
                        replacements=replacements,
                    )

                if not workspace_name:
                    self.logger.warning(
                        "Row -> %s does not have the required data to resolve -> %s for the workspace name!%s",
                        str(index),
                        workspace_name_field,
                        " There's no alternative field name given in the payload (via 'name_alt')."
                        if not name_field_alt
                        else " The alternative field -> '{}' could not be resolved either!".format(
                            name_field_alt,
                        ),
                    )
                    result["skipped_counter"] += 1
                    continue

                # Cleanse the workspace name (allowed characters, maximum length):
                workspace_name = OTCS.cleanse_item_name(workspace_name)

                # Check if workspace has been created before (either in this run
                # or in a former run of the customizer):
                if (
                    workspace_name in existing_workspaces  # processed in former run?
                    or workspace_name in result["workspaces"]  # processed in current run?
                ):
                    self.logger.info(
                        "Workspace -> '%s' does already exist. Skipping...",
                        workspace_name,
                    )
                    result["skipped_counter"] += 1
                    continue

                # Determine the description field:
                if workspace_description_field:
                    description = self.replace_bulk_placeholders(
                        input_string=workspace_description_field,
                        row=row,
                    )
                    # If we cannot get the description from the
                    # workspace_description_field we try the alternative name field
                    # (if specified in payload via "description_alt"):
                    if not description and description_field_alt:
                        self.logger.debug(
                            "Row -> %s does not have the data to resolve the placeholders in workspace description -> %s! Trying alternative description field -> %s...",
                            str(index),
                            workspace_name_field,
                            description_field_alt,
                        )
                        description = self.replace_bulk_placeholders(
                            input_string=description_field_alt,
                            row=row,
                        )
                else:
                    description = ""

                # Create a copy of the mutable operations list as we may
                # want to modify it per row:
                row_operations = list(operations)

                # Check if all data conditions to create the workspace are met
                conditions = bulk_workspace.get("conditions")
                if conditions:
                    evaluated_condition = self.evaluate_conditions(
                        conditions=conditions,
                        row=row,
                        replacements=replacements,
                    )
                    if not evaluated_condition:
                        self.logger.info(
                            "Condition for bulk workspace row -> %s not met. Skipping row for workspace creation...",
                            str(index),
                        )
                        result["skipped_counter"] += 1
                        continue

                # Check if all data conditions to create or recreate the workspace are met:
                if "create" in row_operations or "recreate" in row_operations:
                    conditions_create = bulk_workspace.get("conditions_create")
                    if conditions_create:
                        evaluated_conditions_create = self.evaluate_conditions(
                            conditions=conditions_create,
                            row=row,
                            replacements=replacements,
                        )
                        if not evaluated_conditions_create:
                            self.logger.info(
                                "Create condition for bulk workspace row -> %s not met. Excluding create operation for current row...",
                                str(index),
                            )
                            if "create" in row_operations:
                                row_operations.remove("create")
                            if "recreate" in row_operations:
                                row_operations.remove("recreate")
                    elif (
                        "recreate" in row_operations
                    ):  # we still create and recreate without conditions_create. But give a warning for 'recreate' without condition.
                        self.logger.warning(
                            "No create condition provided but 'recreate' operation requested. This will recreate all existing workspaces!",
                        )

                # Check if all data conditions to delete the workspace are met:
                if "delete" in row_operations:
                    conditions_delete = bulk_workspace.get("conditions_delete")
                    if conditions_delete:
                        evaluated_conditions_delete = self.evaluate_conditions(
                            conditions=conditions_delete,
                            row=row,
                            replacements=replacements,
                        )
                        if not evaluated_conditions_delete:
                            self.logger.info(
                                "Delete condition for bulk workspace row -> %s not met. Excluding delete operation for current row...",
                                str(index),
                            )
                            row_operations.remove("delete")
                    else:  # without delete_conditions we don't delete!!
                        self.logger.warning(
                            "Delete operation requested for bulk workspaces but conditions for deletion are missing! (specify 'conditions_delete')!",
                        )
                        row_operations.remove("delete")

                # Check if all data conditions to delete the workspace are met:
                if "update" in row_operations:
                    conditions_update = bulk_workspace.get("conditions_update")
                    if conditions_update:
                        evaluated_conditions_update = self.evaluate_conditions(
                            conditions=conditions_update,
                            row=row,
                            replacements=replacements,
                        )
                        if not evaluated_conditions_update:
                            self.logger.info(
                                "Update condition for bulk workspace row -> %s not met. Excluding update operation for current row...",
                                str(index),
                            )
                            row_operations.remove("update")

                # Determine the external modification date (if any):
                if external_modify_date_field:
                    external_modify_date = self.replace_bulk_placeholders(
                        input_string=external_modify_date_field,
                        row=row,
                    )
                else:
                    external_modify_date = None

                # Determine the external creation date (if any):
                if external_create_date_field:
                    external_create_date = self.replace_bulk_placeholders(
                        input_string=external_create_date_field,
                        row=row,
                    )
                else:
                    external_create_date = None

                # Determine the key field (if any):
                key = self.replace_bulk_placeholders(input_string=key_field, row=row) if key_field else None

                # check if workspace with this nickname does already exist.
                # we also store the nickname to assign it to the new workspace:
                found_workspace_name = None
                found_workspace_id = None
                if nickname_field:
                    nickname = self.replace_bulk_placeholders(
                        input_string=nickname_field,
                        row=row,
                        replacements=replacements,
                        additional_regex_list=nickname_additional_regex_list,
                    )
                    # If we cannot get the nickname from the
                    # workspace_nickname_field we try the alternative nickname field
                    # (if specified in payload via "nickname_alt"):
                    if not nickname and nickname_field_alt:
                        nickname = self.replace_bulk_placeholders(
                            input_string=nickname_field_alt,
                            row=row,
                            replacements=replacements,
                            additional_regex_list=nickname_additional_regex_list,
                        )

                    # Nicknames for sure should not have leading or trailing spaces:
                    nickname = nickname.strip()
                    # Nicknames for sure are not allowed to include spaces:
                    nickname = nickname.replace(" ", "_")
                    # We also want to replace hyphens with underscores
                    # to make sure that workspace name spellings with
                    # spaces and with hyphens are mapped to the same
                    # workspace nicknames (aligned with the workspace names
                    # that have a regexp rule for this in the payload):
                    nickname = nickname.replace("-", "_")
                    nickname = nickname.replace("___", "_")
                    nickname = nickname.lower()
                    response = self._otcs_frontend.get_node_from_nickname(nickname=nickname)
                    if response:
                        found_workspace_name = self._otcs_frontend.get_result_value(
                            response=response,
                            key="name",
                        )
                        found_workspace_id = self._otcs_frontend.get_result_value(
                            response=response,
                            key="id",
                        )
                        if found_workspace_name != workspace_name:
                            self.logger.warning(
                                "Clash of nicknames for -> '%s' and -> '%s' (%s)! We will not create the workspace but allow for updates and deletes",
                                workspace_name,
                                found_workspace_name,
                                found_workspace_id,
                            )
                            # Remove 'create' from row operations as it would produce errors about uniqueness of nicknames.
                            if "create" in row_operations:
                                row_operations.remove("create")
                        # Only skip, if workspace 'update' or 'delete' is NOT requested:
                        elif "update" not in row_operations and "delete" not in row_operations:
                            self.logger.info(
                                "Workspace -> '%s' with nickname -> '%s' does already exist (found -> '%s'). No update or delete operations requested or allowed. Skipping...",
                                workspace_name,
                                nickname,
                                found_workspace_name,
                            )
                            result["skipped_counter"] += 1
                            continue
                # end if nickname_field
                else:
                    nickname = None

                t.add_event(name="Preparations done.", timestamp=time.time_ns())

                # Based on the evaluation of conditions_create, conditions_delete,
                # and conditions_update we could end up with an empty operations list.
                # In such cases we skip the further processing of this row:
                if not row_operations:
                    self.logger.info(
                        "Skip workspace -> '%s' as row operations is empty (no create, update, delete or recreate). This may be because conditions_create, conditions_delete, and conditions_update have been evaluated to false for this row.",
                        workspace_name,
                    )
                    result["skipped_counter"] += 1
                    continue

                self.logger.info(
                    "Bulk process workspace -> '%s' using effective operations -> %s...",
                    workspace_name,
                    str(row_operations),
                )

                # Check if the workspace does already exist.
                # First we try to look up the workspace by a unique
                # key that does not change over time.
                # For this we expect a "key" value to be defined for the
                # bulkWorkspace and one of the category / attribute item
                # to be marked with "is_key" = True. If we don't find the
                # workspace via key we try via name and type.
                self.logger.info(
                    "Check if workspace -> '%s' does already exist...",
                    workspace_name,
                )
                # Initialize variables - this is important!
                workspace_old_name = None
                workspace_id = None
                handle_name_clash = False
                workspace_modify_date = None
                parent_id = None

                # We have 4 cases to differentiate:

                # 1. key given + key found = name irrelevant, item exist
                # 2. key given + key not found = if name exist it is a name clash
                # 3. no key given + name found = item exist
                # 4. no key given + name not found = item does not exist

                # We are NOT trying to compensate for a failed key lookup with a name lookup!
                # Names are only relevant if no key is in payload!

                if key:
                    # if we have a key we need to also know which category attribute is
                    # storing the value for the key:
                    key_attribute = next(
                        (cat_attr for cat_attr in categories if cat_attr.get("is_key", False) is True),
                        None,
                    )
                    if not key_attribute:
                        self.logger.error(
                            "Bulk Workspace has key -> '%s' defined but none of the category attributes is marked as key ('is_key' is missing)!",
                            key,
                        )
                        result["success"] = False
                        result["failure_counter"] += 1
                        continue
                    cat_name = key_attribute.get("name", None)
                    att_name = key_attribute.get("attribute", None)
                    set_name = key_attribute.get("set", None)

                    # determine where workspaces of this type typically reside (IMPORTANT: this
                    # may return None if no instances of this workspace type exist!):
                    parent_id = self._otcs_frontend.get_workspace_type_location(
                        type_id=workspace_type_id,
                    )
                    if parent_id:
                        # Try to find the node that has the given key attribute value:
                        response = self._otcs_frontend.lookup_nodes(
                            parent_node_id=parent_id,
                            category=cat_name,
                            attribute=att_name,
                            attribute_set=set_name,
                            value=key,
                        )
                        workspace_id = self._otcs_frontend.get_result_value(
                            response=response,
                            key="id",
                        )
                    else:
                        # if we couldn't determine the parent ID this means there are
                        # now workspace instances for this workspace type. Then we set
                        # workspace_id = None and let the code go into the else case below:
                        workspace_id = None

                    if workspace_id:
                        # Case 1: key given + key found = name irrelevant, item exist
                        workspace_old_name = self._otcs_frontend.get_result_value(
                            response=response,
                            key="name",
                        )
                        self.logger.info(
                            "Found existing workspace -> %s (%s) in folder with ID -> %s using key value -> '%s', category -> '%s', and attribute -> '%s'.",
                            workspace_old_name,
                            workspace_id,
                            parent_id,
                            key,
                            cat_name,
                            att_name,
                        )
                        if workspace_old_name != workspace_name:
                            self.logger.info(
                                "Existing workspace has the old name -> '%s' which is different from the new name -> '%s'.",
                                workspace_old_name,
                                workspace_name,
                            )
                        else:
                            workspace_old_name = None
                        # We get the modify date of the existing workspace.
                        workspace_modify_date = self._otcs_frontend.get_result_value(
                            response=response,
                            key="modify_date",
                        )
                    else:
                        # Case 2: key given + key not found = if name exist it is a name clash
                        self.logger.info(
                            "Couldn't find existing workspace with the key value -> '%s' in category -> '%s' and attribute -> '%s' in folder with ID -> %s.",
                            key,
                            cat_name,
                            att_name,
                            parent_id,
                        )
                        handle_name_clash = True
                    # end if key_attribute
                # end if key
                else:
                    # If we haven't a key we try by type + name
                    response = self._otcs_frontend.get_workspace_by_type_and_name(
                        type_id=workspace_type_id,
                        name=workspace_name,
                    )
                    workspace_id = self._otcs_frontend.get_result_value(
                        response=response,
                        key="id",
                    )
                    if workspace_id:
                        # Case 3: no key given + name found = item exist
                        self.logger.info(
                            "Found existing workspace -> '%s' (%s) with type ID -> %s.",
                            workspace_name,
                            workspace_id,
                            workspace_type_id,
                        )
                        # We get the modify date of the existing workspace.
                        workspace_modify_date = self._otcs_frontend.get_result_value(
                            response=response,
                            key="modify_date",
                        )
                    else:
                        # Case 4: no key given + name not found = item does not exist
                        self.logger.info(
                            "No existing workspace with name -> '%s' and type ID -> %s.",
                            workspace_name,
                            workspace_type_id,
                        )
                        # Check if we found an existing workspace by the same nickname:
                        if found_workspace_id and found_workspace_name:
                            self.logger.info(
                                "But there's a workspace -> '%s' (%s) that has a matching nickname -> '%s'. Using this workspace instead...",
                                found_workspace_name,
                                found_workspace_id,
                                nickname,
                            )
                            workspace_id = found_workspace_id

                # Check if we want to recreate an existing workspace
                # then we handle the "delete" part of "recreate" first:
                if workspace_id and "recreate" in row_operations:
                    response = self._otcs_frontend.delete_node(
                        node_id=workspace_id,
                        purge=True,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to recreate existing workspace -> '%s' (%s) with type ID -> %s! Delete failed.",
                            (workspace_name if workspace_old_name is None else workspace_old_name),
                            workspace_id,
                            workspace_type_id,
                        )
                        result["success"] = False
                        result["failure_counter"] += 1
                        continue
                    result["delete_counter"] += 1
                    self.logger.info(
                        "Deleted existing workspace -> '%s' (%s) as part of the recreate operation.",
                        (workspace_name if workspace_old_name is None else workspace_old_name),
                        workspace_id,
                    )
                    workspace_id = None

                # Check if workspace does not exists - then we create a new workspace
                # if this is requested ("create" or "recreate" value in operations list in payload)
                if not workspace_id and ("create" in row_operations or "recreate" in row_operations):
                    # for Case 2 (we looked up the workspace by key but could have name collisions)
                    # we need to see if we have a name collision:
                    if handle_name_clash and parent_id:
                        response = self._otcs_frontend.check_node_name(
                            parent_id=parent_id,
                            node_name=workspace_name,
                        )
                        # result is a list of names that clash - if it is empty we have no clash
                        if response["results"]:
                            # We add the suffix with the key which should be unique:
                            self.logger.warning(
                                "Workspace -> '%s' does already exist in folder with ID -> %s and we need to handle the name clash by using name -> '%s'",
                                workspace_name,
                                parent_id,
                                workspace_name + " (" + key + ")",
                            )
                            workspace_name = workspace_name + " (" + key + ")"

                    # If category data is in payload we substitute
                    # the values with data from the current data row:
                    if categories:
                        # Make sure the threads are not changing data structures that
                        # are shared between threads. categories is a list of dicts.
                        # list and dicts are "mutable" data structures in Python!
                        worker_categories = self.process_bulk_categories(
                            row=row,
                            index=index,
                            categories=categories,
                            replacements=replacements,
                        )
                        workspace_category_data = self.prepare_workspace_create_form(
                            categories=worker_categories,
                            template_id=template_id,
                        )
                        if not workspace_category_data:
                            self.logger.error(
                                "Failed to prepare the category data for new workspace -> '%s'!",
                                workspace_name,
                            )
                            result["success"] = False
                            result["failure_counter"] += 1
                            continue  # for index, row in partition.iterrows()
                    else:
                        workspace_category_data = {}

                    self.logger.info(
                        "Bulk create workspace -> '%s'...",
                        workspace_name,
                    )

                    if classification_nicknames or classification_pathes:
                        classification_ids = self.process_bulk_classification_assignments(
                            row=row,
                            index=index,
                            classifications=classification_pathes + classification_nicknames,
                            replacements=replacements,
                        )
                    else:
                        classification_ids = None

                    # Create the workspace with all provided information:
                    response = self._otcs_frontend.create_workspace(
                        workspace_template_id=template_id,
                        workspace_name=workspace_name,
                        workspace_description=description,
                        workspace_type=workspace_type_id,
                        category_data=workspace_category_data,
                        classifications=classification_ids,
                        external_create_date=external_create_date,
                        external_modify_date=external_modify_date,
                        show_error=False,
                    )
                    if response is None:
                        # Potential race condition: see if the workspace has been created by a concurrent thread.
                        # So we better check if the workspace is there even if the create_workspace() call delivered
                        # a 'None' response:
                        response = self._otcs_frontend.get_workspace_by_type_and_name(
                            type_id=workspace_type_id,
                            name=workspace_name,
                        )
                    workspace_id = self._otcs_frontend.get_result_value(
                        response=response,
                        key="id",
                    )
                    if not workspace_id:
                        self.logger.error(
                            "Failed to bulk create workspace -> '%s' with type ID -> %s from template -> '%s' (%s)!",
                            workspace_name,
                            workspace_type_id,
                            template_name,
                            template_id,
                        )
                        result["success"] = False
                        result["failure_counter"] += 1
                        continue
                    self.logger.info(
                        "Successfully created workspace -> '%s' with ID -> %s from template -> '%s' (%s).",
                        workspace_name,
                        workspace_id,
                        template_name,
                        template_id,
                    )
                    result["create_counter"] += 1

                    # Is Content Aviator enabled system-wide and is it enabled for this
                    # bulkWorkspaces?
                    if self._aviator_enabled and bulk_workspace.get("enable_aviator", False):
                        response = self._otcs_frontend.update_workspace_aviator(
                            workspace_id=workspace_id,
                            status=True,
                        )
                        if not response:
                            self.logger.error(
                                "Failed to enable Content Aviator for workspace -> '%s' (%s)!",
                                workspace_name,
                                workspace_id,
                            )

                    # Check if metadata embeddings need to be updated
                    aviator_metadata = bulk_workspace.get("aviator_metadata", False)
                    if aviator_metadata:
                        self.logger.info(
                            "Trigger Content Aviator metadata embedding for workspace -> '%s' (%s)...",
                            workspace_name,
                            workspace_id,
                        )

                        self._otcs_frontend.aviator_embed_metadata(
                            node_id=workspace_id,
                            wait_for_completion=False,
                        )

                # end if not workspace_id and "create" or "recreate" in row_operations

                # If "updates" are an requested row operation we update the existing workspace with
                # fresh metadata from the payload. Additionally we check the external
                # modify date to support incremental load for content that has really
                # changed.
                # In addition we check that "delete" is not requested as otherwise it will
                # never go in elif "delete" ... below (and it does not make sense to update a workspace
                # that is deleted in the next step...)
                elif (
                    workspace_id
                    and "update" in row_operations
                    and "delete" not in row_operations  # note the NOT !
                    and OTCS.date_is_newer(
                        date_old=workspace_modify_date,
                        date_new=external_modify_date,
                    )
                ):
                    # get the specific update operations given in the payload
                    # if not specified we do all 4 update operations (name, description, categories and nickname)
                    update_operations = bulk_workspace.get(
                        "update_operations",
                        ["name", "description", "categories", "nickname", "classifications", "members"],
                    )

                    # If category data is in payload we substitute
                    # the values with data from the current data row.
                    # This is only done if "categories" update is not
                    # excluded from the update_operations:
                    if categories and "categories" in update_operations:
                        # Make sure the threads are not changing data structures that
                        # are shared between threads. categories is a list of dicts.
                        # list and dicts are "mutable" data structures in Python!
                        worker_categories = self.process_bulk_categories(
                            row=row,
                            index=index,
                            categories=categories,
                            replacements=replacements,
                        )
                        # response = self._otcs_frontend.get_node(node_id=workspace_id)
                        # parent_id = self._otcs_frontend.get_result_value(response=response, key="parent_id")
                        # workspace_category_data = self.prepare_item_create_form(
                        #     parent_id=parent_id,
                        #     categories=worker_categories,
                        #     subtype=self._otcs_frontend.ITEM_TYPE_BUSINESS_WORKSPACE,
                        # )
                        # Transform the payload structure into the format
                        # the OTCS REST API requires:
                        workspace_category_data = self.prepare_category_data(
                            categories_payload=worker_categories,
                            source_node_id=workspace_id,
                        )
                        if not workspace_category_data:
                            self.logger.error(
                                "Failed to prepare the updated category data for workspace -> '%s'!",
                                workspace_name,
                            )
                            result["success"] = False
                            result["failure_counter"] += 1
                            continue  # for index, row in partition.iterrows()
                    # end if categories
                    else:
                        workspace_category_data = {}

                    self.logger.info(
                        "Update existing workspace -> '%s' (%s) with operations -> %s...",
                        workspace_old_name if workspace_old_name else workspace_name,
                        str(workspace_id),
                        str(update_operations),
                    )
                    # Update the workspace with all provided information:
                    response = self._otcs_frontend.update_workspace(
                        workspace_id=workspace_id,
                        workspace_name=(workspace_name if "name" in update_operations else None),
                        workspace_description=(description if "description" in update_operations else None),
                        category_data=(workspace_category_data if "categories" in update_operations else None),
                        external_create_date=external_create_date,
                        external_modify_date=external_modify_date,
                        show_error=True,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to update existing workspace -> '%s' (%s) with type ID -> %s!",
                            (workspace_old_name if workspace_old_name else workspace_name),
                            workspace_id,
                            workspace_type_id,
                        )
                        result["success"] = False
                        result["failure_counter"] += 1
                        continue
                    self.logger.info(
                        "Updated existing workspace -> '%s' (%s) with type ID -> %s.",
                        workspace_name if "name" in update_operations or not workspace_old_name else workspace_old_name,
                        workspace_id,
                        workspace_type_id,
                    )
                    result["update_counter"] += 1

                    if "classifications" in update_operations and (classification_nicknames or classification_pathes):
                        classification_ids = self.process_bulk_classification_assignments(
                            row=row,
                            index=index,
                            classifications=classification_pathes + classification_nicknames,
                        )
                        response = self._otcs.assign_classifications(
                            node_id=workspace_id,
                            classifications=classification_ids,
                            apply_to_sub_items=False,
                        )
                        if response is None:
                            self.logger.error(
                                "Failed to assign classifications -> %s to workspace -> '%s' (%s)!",
                                classification_ids,
                                workspace_name
                                if "name" in update_operations or not workspace_old_name
                                else workspace_old_name,
                                workspace_id,
                            )
                        else:
                            self.logger.info(
                                "Successfully assigned classifications -> %s to workspace -> '%s' (%s).",
                                classification_ids,
                                workspace_name
                                if "name" in update_operations or not workspace_old_name
                                else workspace_old_name,
                                workspace_id,
                            )

                    # Check if metadata embeddings need to be updated
                    aviator_metadata = bulk_workspace.get("aviator_metadata", False)
                    if aviator_metadata:
                        self.logger.info(
                            "Update Content Aviator metadata embedding for workspace -> %s (%s)...",
                            workspace_name,
                            workspace_id,
                        )

                        self._otcs_frontend.aviator_embed_metadata(
                            node_id=workspace_id,
                            wait_for_completion=False,
                        )
                # end elif "update" in row_operations...
                elif workspace_id and "delete" in row_operations:
                    # We delete with immediate purging to keep recycle bin clean
                    # and to not run into issues with nicknames used in deleted items:
                    response = self._otcs_frontend.delete_node(
                        node_id=workspace_id,
                        purge=True,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to delete existing workspace -> '%s' (%s) with type ID -> %s!",
                            (workspace_old_name if workspace_old_name else workspace_name),
                            workspace_id,
                            workspace_type_id,
                        )
                        result["success"] = False
                        result["failure_counter"] += 1
                        continue
                    self.logger.info(
                        "Successfully deleted existing workspace -> '%s' (%s).",
                        workspace_old_name if workspace_old_name else workspace_name,
                        workspace_id,
                    )
                    result["delete_counter"] += 1
                    workspace_id = None
                # end elif workspace_id and "delete" in row_operations

                # this is the plain old "it does exist and we just skip it" case:
                elif workspace_id:
                    result["skipped_counter"] += 1
                    self.logger.info(
                        "Skipped existing workspace -> '%s' (%s)",
                        workspace_old_name if workspace_old_name else workspace_name,
                        workspace_id,
                    )
                # this is the case where we just want to operate on existing workspaces (update or delete)
                # but they do not exist:
                elif not workspace_id and ("update" in row_operations or "delete" in row_operations):
                    result["skipped_counter"] += 1
                    self.logger.info(
                        "Skipped update/delete of non-existing workspace -> '%s'.",
                        workspace_old_name if workspace_old_name else workspace_name,
                    )

                # The following code is executed for all operations
                # (create, recreate, update, delete):

                # Check if we want to set / update the nickname of the workspace.
                # Precondition is we have determined a nickname, we have the workspace ID
                # and the update of the nickname is either required (create, recreate)
                # or requested (update_operations include "nickname"):
                if (
                    nickname
                    and workspace_id
                    and (
                        "create" in row_operations
                        or "recreate" in row_operations
                        or ("update" in row_operations and "nickname" in update_operations)
                    )
                ):
                    response = self._otcs_frontend.set_node_nickname(
                        node_id=workspace_id,
                        nickname=nickname,
                        show_error=True,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to assign nickname -> '%s' to workspace -> '%s'!",
                            nickname,
                            workspace_name,
                        )

                # Check if we want to set / update the members for the workspace roles.
                # Precondition is we have members configured in the payload and setting
                # of roles is requested (workspaces is created or update_operations
                # include "members" operation):
                members = bulk_workspace.get("members")
                if (
                    members
                    and workspace_id
                    and (
                        "create" in row_operations
                        or "recreate" in row_operations
                        or ("update" in row_operations and "members" in update_operations)
                    )
                ):
                    self.looger.info(
                        "Update workspace role members for workspace -> '%s' (%s)...",
                        workspace_name,
                        workspace_id,
                    )
                    workspace_roles = self._otcs_frontend.get_workspace_roles(
                        workspace_id=workspace_id,
                    )

                    # Traverse the members payload:
                    for member in members:
                        # Get the member role from payload first. It is mandatory:
                        member_role = member.get("role", None)
                        if not member_role:
                            self.logger.warning(
                                "Members of workspace -> '%s' is missing the role name.",
                                workspace_name,
                            )
                            continue
                        member_role = self.replace_bulk_placeholders(input_string=member_role, row=row)
                        if not member_role:
                            continue
                        role_id = self._otcs.lookup_result_value(
                            response=workspace_roles,
                            key="name",
                            value=member_role,
                            return_key="id",
                        )
                        if not role_id:
                            self.logger.warning(
                                "Cannot find workspace role -> '%s' for workspace -> '%s' (%s). Skipping...",
                                member_role,
                                workspace_name,
                                workspace_id,
                            )
                            continue

                        # read user list and role name from payload:
                        member_users = member.get("users", [])
                        member_groups = member.get("groups", [])

                        if member_users:
                            self.replace_bulk_placeholders_list(input_list=member_users, row=row)
                        if member_groups:
                            self.replace_bulk_placeholders_list(input_list=member_groups, row=row)

                        if not member_users and not member_groups:
                            self.logger.debug(
                                "Role -> '%s' of workspace -> '%s' does not have any members (no users nor groups).",
                                member_role,
                                workspace_name,
                            )
                            continue

                        # check if we want to clear (remove) existing members of this role:
                        member_clear = member.get("clear", False)
                        if member_clear:
                            self.logger.info(
                                "Clear existing members of role -> '%s' (%s) for workspace -> '%s' (%s)...",
                                member_role,
                                role_id,
                                workspace_name,
                                workspace_id,
                            )
                            self._otcs.remove_workspace_members(
                                workspace_id=workspace_id,
                                role_id=role_id,
                            )
                    # TODO: complete the implementation...
                # end if members...

                # Depending on the bulk operations (create, update, delete)
                # and the related conditions it may well be that workspace_id is None.
                # Only if workspace ID is not none we want to count this row as success:
                if workspace_id is not None:
                    result["success_counter"] += 1
                    # Record the workspace name and ID to allow to read it from failure file
                    # and speedup the process.
                    result["workspaces"][workspace_name] = workspace_id
        # end for index...

        self.logger.info("End working...")

        results.append(result)

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="lookup_data_source_value")
    def lookup_data_source_value(
        self,
        data_source: dict,
        lookup_column: str,
        lookup_value: str,
        separator: str = "|",
    ) -> pd.Series | None:
        """Lookup a value in a given data source (specified by payload dict).

        If the data source has not been loaded before then load the data source.
        As this runs in a multi-threading environment we need to protect
        the data source update from multiple threads doing it at the same time.
        A global data_load_lock variable acts as a mutex.

        Args:
            data_source (dict):
                The payload dictionary of the data source definition.
            lookup_column (str):
                The name of the column in the data frame (see Data class).
            lookup_value (str):
                The value to lookup - selection criteria for the result row.
            separator (str, optional):
                The string list delimiter / separator. The pipe symbol | is the default
                as it is unlikely to appear in a normal string (other than a plain comma).
                The separator is NOT looked for in the lookup_value but in the column that
                is given by lookup_column!

        Returns:
            pd.Series | None:
                Row that matches the lookup value in the lookup column.

        """

        data_source_name = data_source.get("name")
        if not data_source_name:
            self.logger.error("Data source has no name!")
            return None

        # We don't want multiple threads to trigger a data source load at the same time,
        # so we use a lock (mutex) to avoid this:
        data_load_lock.acquire()
        try:
            # First we check if the data source has been loaded already.
            # If not, we load the data source on the fly:
            data_source_data: Data = data_source.get("data")
            if not data_source_data:
                self.logger.info(
                    "Lookup data source -> '%s' has no data yet. Reloading...",
                    data_source_name,
                )
                data_source_data = self.process_bulk_datasource(
                    data_source_name=data_source_name,
                    force_reload=True,
                )
        finally:
            # Ensure the lock is released even if an error occurs
            data_load_lock.release()

        # If we still don't have data from this data source we bail out:
        if data_source_data is None:  # important to use "is None" here!
            self.logger.error(
                "Lookup data source -> '%s' has no data and reload did not work. Cannot lookup value -> '%s' in column -> '%s'!",
                data_source_name,
                lookup_value,
                lookup_column,
            )
            return None

        # Lookup the data frame row (pd.Series) in the given
        # column with the given lookup value:
        lookup_row: pd.Series = data_source_data.lookup_value(
            lookup_column=lookup_column,
            lookup_value=lookup_value,
            separator=separator,
        )

        return lookup_row

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_workspaces_synonym_lookup")
    def process_bulk_workspaces_synonym_lookup(
        self,
        data_source_name: str,
        workspace_name_synonym: str = "",
        workspace_type: str = "",
    ) -> tuple[int | None, str | None] | None:
        """Use a data source to lookup the workspace name (or all fields) and ID using a given synonym.

        Args:
            data_source_name (str):
                The data source name.
            workspace_name_synonym (str, optional):
                The synonym of the workspace name as input for lookup.
            workspace_type (str, optional):
                The name of the workspace type. Default is "".

        Returns:
            tuple[int | None, int | None]:
                The workspace ID and the looked up workspace name or None in case the loomkup has failed.

        """

        # Determine the data source to use for synonym lookup:
        if not data_source_name:
            self.logger.error(
                "No workspace data source name specified. Cannot lookup the workspace by synonym -> '%s'!",
                workspace_name_synonym,
            )
            return (None, None)
        workspace_data_source = next(
            (item for item in self._bulk_datasources if item["name"] == data_source_name),
            None,
        )
        if not workspace_data_source:
            self.logger.error(
                "Workspace data source -> '%s' not found in payload. Cannot lookup the workspace by synonym -> '%s'!",
                data_source_name,
                workspace_name_synonym,
            )
            return (None, None)

        # Read the synonym column and the name column from the data source payload item:
        workspace_data_source_name_column = workspace_data_source.get(
            "name_column",
            None,  # e.g. "Name"
        )
        workspace_data_source_synonyms_column = workspace_data_source.get(
            "synonyms_column",
            None,  # e.g. "Synonyms"
        )

        if not workspace_data_source_name_column or not workspace_data_source_synonyms_column:
            self.logger.warning(
                "Workspace data source -> '%s' has no synonym lookup columns. Cannot find the workspace by synonym -> '%s'!",
                data_source_name,
                workspace_name_synonym,
            )
            return (None, None)

        # Get the row that has the synonym in the synonym column:
        lookup_row = self.lookup_data_source_value(
            data_source=workspace_data_source,
            lookup_column=workspace_data_source_synonyms_column,
            lookup_value=workspace_name_synonym,
        )

        if lookup_row is None:
            # Handle an edge case where the actual workspace name
            # is already correct (using the name column):
            lookup_row = self.lookup_data_source_value(
                data_source=workspace_data_source,
                lookup_column=workspace_data_source_name_column,
                lookup_value=workspace_name_synonym,
            )

        if lookup_row is not None:
            # Now we determine the real workspace name be taking it from
            # the name column in the result row:
            workspace_name = lookup_row[workspace_data_source_name_column]
            self.logger.info(
                "Found workspace name -> '%s' using synonym -> '%s'.",
                workspace_name,
                workspace_name_synonym,
            )

            # We now have the real name. If the workspace type name is
            # provided as well we should be able to lookup the workspace ID now:
            if workspace_type:
                response = self._otcs_frontend.get_workspace_by_type_and_name(
                    type_name=workspace_type,
                    name=workspace_name,
                )
                workspace_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
            else:
                # This method may be called with workspace_type=None.
                # In this case we can return the synonym but cannot
                # lookup the workspace ID:
                workspace_id = None

            # Return the tuple with workspace_id and the real workspace name
            return (workspace_id, workspace_name)

        return (None, None)

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_workspaces_lookup")
    def process_bulk_workspaces_lookup(
        self,
        workspace_nickname: str | None = None,
        workspace_name: str | None = None,
        workspace_type: str | None = None,
        parent_id: int | None = None,
        data_source_name: str | None = None,
        show_error: bool = True,
    ) -> tuple[int | None, str | None]:
        """Look the workspace name and ID.

        Use a combination of workspace name, workspace type, and workspace
        data source (using synonyms) to do so.

        Args:
            workspace_nickname (str, optional):
                The nickname of the workspace.
            workspace_name (str, optional):
                The name as input for lookup. This must be either the name of
                an existing workspace instance or one of the synonyms of the workspace name.
            workspace_type (str, optional):
                The Name of the workspace type.
            parent_id (int, optional):
                ID of parent workspace (if it is a sub-workspace) or parent folder.
            data_source_name (str, optional):
                The workspace data source name.
            show_error (bool, optional):
                Whether or not to log an error if it occurs. If False, just log a warning.

        Returns:
            tuple[int | None, str | None]:
                The workspace ID and the looked up workspace name.

        """

        # First we try to find the workspace by a nickname (if provided)
        if workspace_nickname:
            # Nicknames for sure are not allowed to include spaces and dashes:
            workspace_nickname = workspace_nickname.replace(" ", "_")
            workspace_nickname = workspace_nickname.replace("-", "_")
            workspace_nickname = workspace_nickname.replace("___", "_")
            workspace_nickname = workspace_nickname.lower()

            response = self._otcs_frontend.get_node_from_nickname(
                nickname=workspace_nickname,
                show_error=False,
            )
            workspace_id = self._otcs_frontend.get_result_value(
                response=response,
                key="id",
            )
            if workspace_id:
                # If we don't have the name yet, we can get it from the response above:
                if not workspace_name:
                    workspace_name = self._otcs_frontend.get_result_value(
                        response=response,
                        key="name",
                    )
                return (workspace_id, workspace_name)
            # DON'T RETURN FAILURE AT THIS POINT!

        # Our 2nd try is to find the workspace by a workspace name and workspace type combination:
        if workspace_name:
            workspace_name = workspace_name.strip()
        else:
            self.logger.error(
                "No workspace name specified. Cannot find the workspace by nickname%s, nor by type and name, nor by parent ID and name, nor by synonym.",
                " -> {}".format(workspace_nickname) if workspace_nickname else "",
            )
            return (None, None)

        # If we have workspace name and workspace parent ID then we try this:
        if workspace_name and parent_id is not None:
            response = self._otcs_frontend.get_node_by_parent_and_name(
                parent_id=parent_id,
                name=workspace_name,
            )
            workspace_id = self._otcs_frontend.get_result_value(
                response=response,
                key="id",
            )
            if workspace_id:
                return (workspace_id, workspace_name)

        # If we have workspace name and workspace type then we try to get
        # the workspace by type + name:
        if workspace_name and workspace_type:
            response = self._otcs_frontend.get_workspace_by_type_and_name(
                type_name=workspace_type,
                name=workspace_name,
            )
            workspace_id = self._otcs_frontend.get_result_value(
                response=response,
                key="id",
            )
            if workspace_id:
                return (workspace_id, workspace_name)

        # if the code gets to here we dont have a nickname and the workspace with given name
        # type, or parent ID was not found either. Now we see if we can find the workspace name
        # as a synonym in the workspace data source to find the real/correct workspace name:
        if data_source_name:
            self.logger.info(
                "Try to find the workspace with the synonym -> '%s' using data source -> '%s'...",
                workspace_name,
                data_source_name,
            )

            (workspace_id, workspace_synonym_name) = self.process_bulk_workspaces_synonym_lookup(
                data_source_name=data_source_name,
                workspace_name_synonym=workspace_name,  # see if workspace_name is a synonym
                workspace_type=workspace_type,
            )
            if workspace_id is not None:
                return (workspace_id, workspace_synonym_name)

        # As this message may be hunderds of times in the log
        # we invest some effort to make it look nice:
        message = "Couldn't find a workspace "
        concat_string = ""
        if workspace_nickname:
            message += "by nickname -> '{}'".format(workspace_nickname)
            concat_string = ", nor "
        if workspace_name:
            message += "{}by name -> '{}'".format(concat_string, workspace_name)
            concat_string = ", nor "
        if parent_id:
            message += "{}by parent ID -> {}".format(concat_string, parent_id)
            concat_string = ", nor "
        if data_source_name:
            message += "{}as synonym in data source -> '{}'".format(
                concat_string,
                data_source_name,
            )
        if show_error:
            self.logger.error(message)
        else:
            self.logger.warning(message)

        return (
            None,
            workspace_name,
        )  # it is important to return the name - used by process_bulk_categories()

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_workspace_relationships")
    def process_bulk_workspace_relationships(
        self,
        section_name: str = "bulkWorkspaceRelationships",
    ) -> bool:
        """Process workspaces in payload and bulk create them in Content Server (multi-threaded).

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise.

        """

        if not self._bulk_workspace_relationships:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        # For efficient idem-potent operation we may want to see which workspaces
        # have already been processed before:
        if self.check_status_file(
            payload_section_name=section_name,
            payload_specific=True,
            prefix="failure_",
        ):
            # Read payload from where we left it last time
            self._bulk_workspace_relationships = self.get_status_file(
                payload_section_name=section_name,
                prefix="failure_",
            )
            if not self._bulk_workspace_relationships:
                self.logger.error(
                    "Cannot load existing bulkWorkspaceRelationships failure file. Bailing out!",
                )
                return False

        success: bool = True

        for bulk_workspace_relationship in self._bulk_workspace_relationships:
            # Check if element has been disabled in payload (enabled = false).
            # In this case we skip the element:
            enabled = bulk_workspace_relationship.get("enabled", True)
            if not enabled:
                self.logger.info(
                    "Payload for bulk workspace relationship is disabled. Skipping...",
                )
                continue

            # Read Pattern for "From" Workspace from payload:
            from_workspace = bulk_workspace_relationship.get("from_workspace", None)
            if not from_workspace:
                self.logger.error(
                    "Bulk workspace relationship creation needs a workspace nickname in from_workspace! Skipping to next bulk workspace relationship...",
                )
                success = False
                continue
            from_sub_workspace = bulk_workspace_relationship.get(
                "from_sub_workspace_name",
                None,
            )

            # Read Pattern for "To" Workspace from payload:
            to_workspace = bulk_workspace_relationship.get("to_workspace", None)
            if not to_workspace:
                self.logger.error(
                    "Bulk workspace relationship creation needs a workspace nickname in to_workspace! Skipping to next bulk workspace relationship...",
                )
                success = False
                continue
            to_sub_workspace = bulk_workspace_relationship.get(
                "to_sub_workspace_name",
                None,
            )

            # The payload element must have a "data_source" key:
            data_source_name = bulk_workspace_relationship.get("data_source", None)
            if not data_source_name:
                self.logger.error(
                    "Missing data source name in bulk workspace relationship!",
                )
                success = False
                continue

            self._log_header_callback(
                text="Process bulk workspace relationships from -> '{}' to -> '{}'".format(
                    from_sub_workspace if from_sub_workspace else from_workspace,
                    to_sub_workspace if to_sub_workspace else to_workspace,
                ),
                char="-",
            )

            # Determine which operations should be done for this bulk workspace relationship:
            operations = bulk_workspace_relationship.get("operations", ["create"])
            if "delete" in operations and "create" in operations:
                self.logger.error(
                    "Bulk workspace relationships can either have 'create' or 'delete' operation - not both. Use separate payloads for this purpose!",
                )
                success = False
                continue

            copy_data_source = bulk_workspace_relationship.get(
                "copy_data_source",
                False,
            )
            force_reload = bulk_workspace_relationship.get("force_reload", True)

            # Load and prepare the data source for the bulk processing:
            if copy_data_source:
                self.logger.info(
                    "Take a copy of data source -> '%s' to avoid side-effects for repeative usage of the data source...",
                    bulk_workspace_relationship["data_source"],
                )
                data = Data(
                    self.process_bulk_datasource(
                        data_source_name=data_source_name,
                        force_reload=force_reload,
                    ),
                    logger=self.logger,
                )
            else:
                self.logger.info(
                    "Use original data source -> '%s' and not do a copy.",
                    bulk_workspace_relationship["data_source"],
                )
                # Load and prepare the data source for the bulk processing:
                data = self.process_bulk_datasource(
                    data_source_name=data_source_name,
                    force_reload=force_reload,
                )
            if data is None:  # important to use "is None" here!
                self.logger.error(
                    "Failed to load data source -> '%s' for bulk workspace relationships from -> '%s' to -> '%s'",
                    data_source_name,
                    from_sub_workspace if from_sub_workspace else from_workspace,
                    to_sub_workspace if to_sub_workspace else to_workspace,
                )
                success = False
                continue
            if data.get_data_frame() is None:  # important to use "is None" here!
                self.logger.warning(
                    "Data source for bulk workspace relationships from -> '%s' to -> '%s' is empty!",
                    from_sub_workspace if from_sub_workspace else from_workspace,
                    to_sub_workspace if to_sub_workspace else to_workspace,
                )
                continue

            # Check if fields with list substructures should be exploded.
            # We may want to do this outside the bulkDatasource to only
            # explode for bulkDocuments and not for bulkWorkspaces or
            # bulkWorkspaceRelationships:
            explosions = bulk_workspace_relationship.get("explosions", [])
            for explosion in explosions:
                # The explode field parameter can be a string or a list.
                # Exploding multiple fields at once avoids combinatorial explosions -
                # this is VERY different from exploding columns one after the other!
                if "explode_field" not in explosion and "explode_fields" not in explosion:
                    self.logger.error("Missing explosion field(s)!")
                    continue
                # we want to be backwards compatible...
                explode_fields = (
                    explosion["explode_field"] if "explode_field" in explosion else explosion["explode_fields"]
                )
                flatten_fields = explosion.get("flatten_fields", [])
                split_string_to_list = explosion.get("split_string_to_list", False)
                list_splitter = explosion.get(
                    "list_splitter",
                    ",",
                )  # don't have None as default!
                self.logger.info(
                    "Starting explosion of bulk relationships by field(s) -> %s (type -> %s). Size of data frame before explosion -> %s.",
                    str(explode_fields),
                    type(explode_fields),
                    str(len(data)),
                )
                data.explode_and_flatten(
                    explode_fields=explode_fields,
                    flatten_fields=flatten_fields,
                    make_unique=False,
                    split_string_to_list=split_string_to_list,
                    separator=list_splitter,
                    reset_index=True,
                )
                self.logger.info(
                    "Size of data frame after explosion -> %s.",
                    str(len(data)),
                )
            # end for explosion in explosions

            # Keep only selected rows if filters are specified in bulkWorkspaceRelationship.
            # We have this _after_ "explosions" to allow access to subfields as well.
            # We have this _before_ "sorting" and "deduplication" as we may keep the wrong
            # rows otherwise (unique / deduplication always keeps the first matching row).
            # We may want to do this filtering outside the bulkDatasource to only
            # filter the data frame for bulkDocuments and not for bulkWorkspaces or
            # bulkWorkspaceRelationships:
            filters = bulk_workspace_relationship.get("filters", [])
            if filters:
                data.filter(conditions=filters)

            # Sort the data frame if "sort" specified in payload. We may want to do this to have a
            # higher chance that rows with common values that may collapse into
            # one name are put into the same partition. This can avoid race conditions
            # between different Python threads.
            sort_fields = bulk_workspace_relationship.get("sort", [])
            if sort_fields:
                self.logger.info(
                    "Start sorting of bulk workspace relationships data frame based on fields (columns) -> %s...",
                    str(sort_fields),
                )
                data.sort(sort_fields=sort_fields, inplace=True)
                self.logger.info(
                    "Sorting of bulk workspace relationships data frame based on fields (columns) -> %s completed.",
                    str(sort_fields),
                )

            # Check if duplicate rows for given fields should be removed. It is
            # important to do this after sorting as Pandas always keep the first occurance,
            # so ordering plays an important role in deduplication!
            unique_fields = bulk_workspace_relationship.get("unique", [])
            if unique_fields:
                self.logger.info(
                    "Starting deduplication of data frame for bulk workspace relationships with unique fields -> %s. Size of data frame before deduplication -> %s",
                    str(unique_fields),
                    str(len(data)),
                )
                data.deduplicate(unique_fields=unique_fields, inplace=True)
                self.logger.info(
                    "Size of data frame after deduplication -> %s.",
                    str(len(data)),
                )

            self.logger.info(
                "Bulk create workspace relationships (from workspace -> '%s' to workspace -> '%s'). Operations -> %s.",
                from_sub_workspace if from_sub_workspace else from_workspace,
                to_sub_workspace if to_sub_workspace else to_workspace,
                str(operations),
            )

            # See if bulkWorkspace definition has a specific thread number
            # otherwise it is read from a global environment variable
            bulk_thread_number = int(
                bulk_workspace_relationship.get("thread_number", BULK_THREAD_NUMBER),
            )

            partitions = data.partitionate(bulk_thread_number)

            # Create a list to hold the threads
            threads = []
            results = []

            # Create and start a thread for each partition
            for index, partition in enumerate(partitions, start=1):
                # start a thread executing the process_bulk_workspace_relationships_worker() method below:
                thread = threading.Thread(
                    name=f"{section_name}_{index:02}",
                    target=self.thread_wrapper,
                    args=(
                        self.process_bulk_workspace_relationships_worker,
                        bulk_workspace_relationship,
                        partition,
                        operations,
                        results,
                    ),
                )
                self.logger.info("Starting thread -> '%s'...", str(thread.name))
                thread.start()
                threads.append(thread)
                # Avoid that all threads start at the exact same time with
                # potentially expired cookies that cases race conditions:
                time.sleep(1)
            # end for index, partition in enumerate(partitions, start=1)

            # Wait for all threads to complete
            for thread in threads:
                self.logger.info(
                    "Waiting for thread -> '%s' to complete...",
                    str(thread.name),
                )
                thread.join()
                self.logger.info("Thread -> '%s' has completed.", str(thread.name))

            if "relationships" not in bulk_workspace_relationship:
                bulk_workspace_relationship["relationships"] = {}

            # Print statistics for each thread. In addition,
            # check if all threads have completed without error / failure.
            # If there's a single failure in on of the thread results we
            # set 'success' variable to False.
            results.sort(key=lambda x: x["thread_name"])
            for result in results:
                if not result["success"]:
                    self.logger.info(
                        "Thread -> '%s' completed with %s failed, %s skipped, and %s created workspace relationships.",
                        str(result["thread_name"]),
                        str(result["failure_counter"]),
                        str(result["skipped_counter"]),
                        str(result["success_counter"]),
                    )
                    success = False
                else:
                    self.logger.info(
                        "Thread -> '%s' completed successful with %s skipped, and %s created workspace relationships.",
                        str(result["thread_name"]),
                        str(result["skipped_counter"]),
                        str(result["success_counter"]),
                    )
                # Record all generated workspaces relationships. This should
                # allow us to restart in case of failures and avoid trying to
                # create workspaces that have been created before.
                bulk_workspace_relationship["relationships"].update(
                    result["relationships"],
                )
            self._log_header_callback(
                text="Completed processing of bulk workspace relationships from -> '{}' to -> '{}'".format(
                    from_sub_workspace if from_sub_workspace else from_workspace,
                    to_sub_workspace if to_sub_workspace else to_workspace,
                ),
                char="-",
            )

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._bulk_workspace_relationships,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_bulk_workspace_relationship_endpoint")
    def get_bulk_workspace_relationship_endpoint(
        self,
        bulk_workspace_relationship: dict,
        row: pd.Series,
        index: int,
        endpoint: str,
        replacements: dict | None = None,
        nickname_additional_regex_list: list | None = None,
        show_error: bool = True,
    ) -> tuple[int | None, str | None]:
        """Determine the node ID of the workspace that is one of the endpoints of the workspace relationship (either 'from' or 'to').

        Args:
            bulk_workspace_relationship (dict):
                The payload element of the bulk workspace relationship
            row (pd.Series):
                The data frame row.
            index (int):
                The index of the data frame row.
            endpoint (str):
                The name of the endpoint - either "from" or "to".
            replacements (dict | None, optional):
                Replacements for placeholders. Defaults to None.
            nickname_additional_regex_list (list | None, optional):
                Additional regex replacements for nicknames. Defaults to None.
            show_error (bool, optional):
                Whether or not to log an error. If False just a warning is logged.

        Returns:
            tuple[int | None, str | None]:
                The workspace ID and the looked up workspace name.

        """

        if endpoint not in ["from", "to"]:
            self.logger.error("The endpoint must be either 'from' or 'to'!")
            return (None, None)

        # Determine the workspace nickname field:
        workspace_nickname_field = bulk_workspace_relationship.get(
            "{}_workspace".format(endpoint),
            None,
        )
        workspace_nickname = self.replace_bulk_placeholders(
            input_string=workspace_nickname_field,
            row=row,
            replacements=replacements,
            additional_regex_list=nickname_additional_regex_list,
        )
        if not workspace_nickname:
            self.logger.warning(
                "Row -> %s does not have the required data to resolve -> '%s' for the workspace nickname (endpoint -> '%s')!",
                str(index),
                workspace_nickname_field,
                endpoint,
            )
            return (None, None)

        # Get the workspace type if specified:
        workspace_type = bulk_workspace_relationship.get(
            "{}_workspace_type".format(endpoint),
            None,
        )

        # Get the workspace name if specified:
        workspace_name_field = bulk_workspace_relationship.get(
            "{}_workspace_name".format(endpoint),
            None,
        )
        if workspace_name_field:
            workspace_name = self.replace_bulk_placeholders(
                input_string=workspace_name_field,
                row=row,
                replacements=replacements,
            )
            if not workspace_name:
                self.logger.warning(
                    "Row -> %s does not have the required data to resolve -> '%s' for the workspace name (endpoint -> '%s')!",
                    str(index),
                    workspace_name_field,
                    endpoint,
                )
                return (None, None)
        else:
            workspace_name = None

        # Get the workspace data source if specified:
        workspace_data_source = bulk_workspace_relationship.get(
            "{}_workspace_data_source".format(endpoint),
            None,
        )

        # Based on the given information, we now try to determine
        # the name and the ID of the workspace that is the endpoint
        # for the workspace relationship:
        (workspace_id, workspace_name) = self.process_bulk_workspaces_lookup(
            workspace_nickname=workspace_nickname,
            workspace_name=workspace_name,
            workspace_type=workspace_type,
            data_source_name=workspace_data_source,
            show_error=show_error,
        )

        if not workspace_id:
            self.logger.warning(
                "Cannot find workspace to establish relationship (endpoint -> '%s')%s%s%s%s",
                endpoint,
                (", nickname -> '{}'".format(workspace_nickname) if workspace_nickname else ""),
                (", workspace name -> '{}'".format(workspace_name) if workspace_name else ""),
                (", workspace type -> '{}'".format(workspace_type) if workspace_type else ""),
                (", data source -> '{}'".format(workspace_data_source) if workspace_data_source else ""),
            )
            return (None, None)

        # See if a sub-workspace is configured:
        sub_workspace_name_field = bulk_workspace_relationship.get(
            "{}_sub_workspace_name".format(endpoint),
            None,
        )
        # If no sub-workspace is configured we can already
        # return the resulting workspace ID and name here:
        if not sub_workspace_name_field:
            return (workspace_id, workspace_name)

        # Otherwise we are no processing the sub-workspaces to return
        # its ID instead:
        sub_workspace_name = self.replace_bulk_placeholders(
            input_string=sub_workspace_name_field,
            row=row,
            replacements=replacements,
        )
        if not sub_workspace_name:
            self.logger.warning(
                "Row -> %s does not have the required data to resolve -> %s for the sub-workspace name (endpoint -> '%s')!",
                str(index),
                sub_workspace_name_field,
                endpoint,
            )
            return (None, None)

        # See if a sub-workspace is in a sub-path of the main workspace:
        sub_workspace_path = bulk_workspace_relationship.get(
            "{}_sub_workspace_path".format(endpoint),
            None,
        )
        if sub_workspace_path:
            # sub_workspace_path is a mutable that is changed in place!
            result = self.replace_bulk_placeholders_list(
                input_list=sub_workspace_path,
                row=row,
                replacements=replacements,
            )
            if not result:
                self.logger.warning(
                    "Row -> %s does not have the required data to resolve -> %s for the sub-workspace path (endpoint -> '%s')!",
                    str(index),
                    sub_workspace_path,
                    endpoint,
                )
                return None

            self.logger.info(
                "Endpoint has a sub-workspace -> '%s' configured. Try to find the sub-workspace in workspace path -> %s...",
                sub_workspace_name,
                sub_workspace_path,
            )

            # We now want to retrieve the folder in the main workspace that
            # includes the sub-workspace:
            response = self._otcs_frontend.get_node_by_workspace_and_path(
                workspace_id=workspace_id,
                path=sub_workspace_path,
                create_path=False,  # we don't want the path to be created if it doesn't exist
                show_error=True,
            )
            parent_id = self._otcs_frontend.get_result_value(
                response=response,
                key="id",
            )
            if not parent_id:
                self.logger.error(
                    "Failed to find path -> %s in workspace -> '%s' (%s)!",
                    str(sub_workspace_path),
                    workspace_name,
                    workspace_id,
                )
                return (None, None)
        # end if sub_workspace_path_field
        else:
            # the sub-workspace is immediately under the main workspace:
            parent_id = workspace_id

        response = self._otcs_frontend.get_node_by_parent_and_name(
            parent_id=parent_id,
            name=sub_workspace_name,
            show_error=True,
        )
        sub_workspace_id = self._otcs_frontend.get_result_value(
            response=response,
            key="id",
        )

        return (sub_workspace_id, sub_workspace_name)

    # end method definition

    @tracer.start_as_current_span(
        attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_workspace_relationships_worker"
    )
    def process_bulk_workspace_relationships_worker(
        self,
        bulk_workspace_relationship: dict,
        partition: pd.DataFrame,
        operations: list | None = None,
        results: list | None = None,
    ) -> None:
        """Thread worker to create workspaces relationships in bulk.

        Each worker thread gets a partition of the rows that include
        the data required for the workspace relationship creation.

        Args:
            bulk_workspace_relationship (dict):
                The payload of the bulkWorkspaceRelationship.
            partition (pd.DataFrame):
                The data partition with rows to process.
            from_workspace (str):
                The string pattern for nickname of workspace (from).
            to_workspace (str):
                The string pattern for nickname of workspace (to).
            operations (list, optional):
                Defines which operations should be applyed on workspace relationships.
                Possible values are "create", "delete", "recreate".
            results (list, optional):
                A mutable list of thread results.

        """

        thread_id = threading.get_ident()
        thread_name = threading.current_thread().name

        self.logger.info(
            "Start working on data frame partition of size -> %s...",
            str(len(partition)),
        )

        # Avoid linter warnings - so make parameter default None while we
        # actually want ["create"] to be the default:
        if operations is None:
            operations = ["create"]

        result = {}
        result["thread_id"] = thread_id
        result["thread_name"] = thread_name
        result["success_counter"] = 0
        result["failure_counter"] = 0
        result["skipped_counter"] = 0
        result["delete_counter"] = 0
        result["relationships"] = {}
        result["success"] = True

        # Check if workspace relationships have been processed before, e.i. testing
        # if a "relationships" key exists and if it is pointing to a non-empty list:
        if bulk_workspace_relationship.get("relationships") and "delete" not in operations:
            existing_workspace_relationships = bulk_workspace_relationship["relationships"]
            self.logger.info(
                "Found %s already processed workspace relationships. Try to complete the job...",
                str(len(existing_workspace_relationships)),
            )
        else:
            existing_workspace_relationships = {}

        # Get dictionary of replacements for bulk workspace relationships
        # this we will be used of all places data is read from the
        # current data frame row. Each dictionary item has the field name as the
        # dictionary key and a list of regular expressions as dictionary value
        replacements = bulk_workspace_relationship.get("replacements")

        # Check if all data conditions to create the workspace relationship are met
        conditions = bulk_workspace_relationship.get("conditions")

        # Type of the relationship - can either be child or parent.
        relationship_type = bulk_workspace_relationship.get("type", "child")

        # Whether or not we want to show an error if the lookup of related
        # workspaces fails.
        show_lookup_error_from = bulk_workspace_relationship.get(
            "from_workspace_lookup_error",
            True,
        )
        show_lookup_error_to = bulk_workspace_relationship.get(
            "to_workspace_lookup_error",
            True,
        )

        # Nicknames are very limited in terms of allowed characters.
        # For nicknames we need additional regexp as we need to
        # replace all non-alphanumeric, non-space characters with ""
        # We also preserve hyphens in the first step to replace
        # them below with underscores. This is important to avoid
        # that different spellings of names produce different nicknames.
        # We want spellings with spaces match spellings with hyphens.
        # For this the workspace names have a regexp "-| " in the payload.
        nickname_additional_regex_list = [r"[^\w\s-]"]

        total_rows = len(partition)

        # Process all datasets in the partion that was given to the thread:
        for index, row in partition.iterrows():
            # Calculate percentage of completion
            percent_complete = ((partition.index.get_loc(index) + 1) / total_rows) * 100

            self.logger.info(
                "Processing data row -> %s for bulk workspace relationship creation (%.2f %% complete)...",
                str(index),
                percent_complete,
            )

            # Create a copy of the mutable operations list as we may
            # want to modify it per row:
            row_operations = list(operations)

            # check if we have any exlusions that apply here:
            if conditions:
                evaluated_condition = self.evaluate_conditions(
                    conditions=conditions,
                    row=row,
                    replacements=replacements,
                )
                if not evaluated_condition:
                    self.logger.info(
                        "Condition for row -> %s not met. Skipping row for workspace relationship...",
                        str(index),
                    )
                    result["skipped_counter"] += 1
                    continue

            (from_workspace_id, from_workspace_name) = self.get_bulk_workspace_relationship_endpoint(
                bulk_workspace_relationship=bulk_workspace_relationship,
                row=row,
                index=index,
                endpoint="from",
                replacements=replacements,
                nickname_additional_regex_list=nickname_additional_regex_list,
                show_error=show_lookup_error_from,
            )

            # Check we have "from" endpoint:
            if not from_workspace_id:
                self.logger.warning(
                    "%s%s",
                    (
                        "Failed to retrieve 'from' endpoint for bulk workspace relationship! "
                        if not from_workspace_id and not from_workspace_name
                        else ""
                    ),
                    (
                        "Failed to retrieve 'from' endpoint (workspace name -> {}) for bulk workspace relationship! ".format(
                            from_workspace_name,
                        )
                        if not from_workspace_id and from_workspace_name
                        else ""
                    ),
                )
                result["skipped_counter"] += 1
                continue

            # If relationships should be deleted we do it only if the
            # relationships for this workspace ID have not been deleted before.
            # This is checked by a dictionary result["relationships"].
            # If a "delete" operation is requested there cannot be a "create"
            # operation in the same bulk workspace relationships payload!
            # (this would create too many interferences between threads)
            # Because of this we can continue with the next row then:
            if "delete" in row_operations and from_workspace_id not in result["relationships"]:
                # Get the target (to) workspace type if specified:
                to_workspace_type = bulk_workspace_relationship.get("to_workspace_type")
                if not to_workspace_type:
                    self.logger.error(
                        "Cannot delete workspace relationships! Need the target (to) workspace type!",
                    )
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue
                result = self._otcs_frontend.delete_workspace_relationships(
                    workspace_id=from_workspace_id,
                    related_workspace_type_id=to_workspace_type,
                )
                if not result:
                    result["success"] = False
                    result["failure_counter"] += 1
                else:
                    result["delete_counter"] += 1
                    # Mark that we have deleted the relationships originating from
                    # the current workspace ID (from) pointing to workspace of
                    # the given type.
                    result["relationships"][from_workspace_id] = [to_workspace_type]
                # We are continuing here as we cannot do delete and create
                # operations in one thread. This needs separate bulkRelationships
                # payloads!
                continue

            (to_workspace_id, to_workspace_name) = self.get_bulk_workspace_relationship_endpoint(
                bulk_workspace_relationship=bulk_workspace_relationship,
                row=row,
                index=index,
                endpoint="to",
                replacements=replacements,
                nickname_additional_regex_list=nickname_additional_regex_list,
                show_error=show_lookup_error_to,
            )

            # Check we have "to" endpoint:
            if not from_workspace_id or not to_workspace_id:
                self.logger.warning(
                    "%s%s",
                    (
                        "Failed to retrieve 'to' endpoint for bulk workspace relationship!"
                        if not to_workspace_id and not to_workspace_name
                        else ""
                    ),
                    (
                        "Failed to retrieve 'to' endpoint (workspace name -> {}) for bulk workspace relationship!".format(
                            to_workspace_name,
                        )
                        if not to_workspace_id and to_workspace_name
                        else ""
                    ),
                )
                result["skipped_counter"] += 1
                continue

            # Check if workspace relationship has been created before (either in this run
            # or in a former run of the customizer):
            if (  # processed in former run?
                from_workspace_id in existing_workspace_relationships
                and to_workspace_id in existing_workspace_relationships[from_workspace_id]
            ) or (  # processed in current run?
                from_workspace_id in result["relationships"]
                and to_workspace_id in result["relationships"][from_workspace_id]
            ):
                self.logger.info(
                    "Workspace relationship between workspace -> '%s' (%s) and related workspace -> '%s' (%s) has successfully been processed before. Skipping...",
                    from_workspace_name,
                    str(from_workspace_id),
                    to_workspace_name,
                    str(to_workspace_id),
                )
                result["skipped_counter"] += 1
                continue

            # Check if workspace relationship does already exist in Extended ECM
            # (this is an additional safety measure to avoid errors):
            response = self._otcs_frontend.get_workspace_relationships(
                workspace_id=from_workspace_id,
                relationship_type=relationship_type,
                related_workspace_name=to_workspace_name,
            )
            current_workspace_relationships = self._otcs_frontend.exist_result_item(
                response=response,
                key="id",
                value=to_workspace_id,
            )
            if current_workspace_relationships:
                self.logger.info(
                    "Workspace relationship between workspace -> '%s' (%s) and related workspace -> '%s' (%s) does already exist. Skipping...",
                    from_workspace_name,
                    str(from_workspace_id),
                    to_workspace_name,
                    str(to_workspace_id),
                )
                result["skipped_counter"] += 1
                continue

            self.logger.info(
                "Bulk create workspace relationship '%s' (%s) -> '%s' (%s)...",
                from_workspace_name,
                str(from_workspace_id),
                to_workspace_name,
                str(to_workspace_id),
            )

            response = self._otcs_frontend.create_workspace_relationship(
                workspace_id=from_workspace_id,
                related_workspace_id=to_workspace_id,
                relationship_type=relationship_type,
                show_error=False,
            )

            if response is None:
                # Potential race condition: see if the workspace-2-workspace relationship has been created by a concurrent thread.
                # So we better check if the relationship is there even if the create_workspace_relationship() call delivered
                # a 'None' response:
                response = self._otcs_frontend.get_workspace_relationships(
                    workspace_id=from_workspace_id,
                    relationship_type=relationship_type,
                    related_workspace_name=to_workspace_name,
                )
                current_workspace_relationships = self._otcs_frontend.exist_result_item(
                    response=response,
                    key="id",
                    value=to_workspace_id,
                )
                if current_workspace_relationships:
                    self.logger.info(
                        "Workspace relationship between workspace -> '%s' (%s) and related workspace -> '%s' (%s) has been created concurrently. Skipping...",
                        from_workspace_name,
                        str(from_workspace_id),
                        to_workspace_name,
                        str(to_workspace_id),
                    )
                    result["skipped_counter"] += 1
                    continue
                self.logger.error(
                    "Failed to bulk create workspace relationship (%s) from -> '%s' (%s) to -> '%s' (%s)!",
                    relationship_type,
                    from_workspace_name,
                    str(from_workspace_id),
                    to_workspace_name,
                    str(to_workspace_id),
                )
                result["success"] = False
                result["failure_counter"] += 1
            else:
                self.logger.info(
                    "Successfully created bulk workspace relationship (%s) from -> '%s' (%s) to -> '%s' (%s).",
                    relationship_type,
                    from_workspace_name,
                    str(from_workspace_id),
                    to_workspace_name,
                    str(to_workspace_id),
                )
                result["success_counter"] += 1
                # Record the workspace name and ID to allow to read it from failure file
                # and speedup the process.
                if from_workspace_id not in result["relationships"]:
                    # Initialize the "to" list:
                    result["relationships"][from_workspace_id] = [to_workspace_id]
                else:
                    result["relationships"][from_workspace_id].append(to_workspace_id)

        self.logger.info("End working...")

        results.append(result)

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="prepare_category_data")
    def prepare_category_data(
        self,
        categories_payload: dict,
        source_node_id: int,
        source_is_document: bool = False,
    ) -> dict | None:
        """Prepare the metadata structure for a new workspace or document.

        This translates the category information from the PAYLOAD into the structure
        required by the OTCS REST API for category updates.

        The difficulty is to get the category schema for the OTCS category. The method tries different
        approaches to get the schema:
        1. Check if the category schema is on a source node (e.g. inherited from a parent node)
        2. Find the category definition by nickname and then assign it to the source node (then retrying approach 1)
        3. Find the category definition by a unique name and then assign it to the source node (then retrying approach 1)

        Args:
            categories_payload (dict):
                The payload information for the workspace or document categories.
            source_node_id (int):
                The item to derive or inherit the category data from. We expect this to
                be a folder, workspace or document that has the category assigned.
            source_is_document (bool, optional):
                If the source node type is a document we need to handle category assignment
                a bit different. Default is False.

        Returns:
            dict | None:
                Category data structure required for subsequent document upload OTCS REST call.

        """

        # Determine the category names in the payload. For this we use a
        # comprehension to create a set (with the curly braces) of unique category
        # names used in the payload that then is converted to a list:
        category_names = list({category["name"] for category in categories_payload})

        # Determine the names of all categories inherited from the source_node_id:
        response = self._otcs_frontend.get_node_categories(node_id=source_node_id)
        if response and response["results"]:
            inherited_category_names = [
                next(iter(item["metadata"]["categories"].values()))["name"]
                for item in response["results"]
                if item.get("metadata")
            ]
        else:
            inherited_category_names = []

        # Calculate the category names we have in payload that are NOT inherited (set difference):
        missing_categories: set[str] = set(category_names) - set(inherited_category_names)

        # Are any of the payload categories NOT inherited from the source_node_id ?
        if missing_categories:
            self.logger.info(
                "Categories -> %s are in payload but not found on source node with ID -> %s. Try to find these categories by nicknames or unique names...",
                str(missing_categories),
                str(source_node_id),
            )
            for category_name in missing_categories:  # category_names:
                category_node_id = None
                # First try via nickname. We expect prefix "cat_" and snake case nicknames:
                category_nickname = (
                    "cat_"
                    + category_name.replace(" - ", "_")
                    .replace("-", "_")
                    .replace(" ", "_")
                    .replace("___", "_")
                    .replace("(", "")
                    .replace(")", "")
                    .lower()
                )
                category_node = self._otcs_frontend.get_node_from_nickname(
                    nickname=category_nickname,
                )
                category_node_id = self._otcs_frontend.get_result_value(
                    response=category_node,
                    key="id",
                )
                if category_node_id:
                    self.logger.info(
                        "Found category -> '%s' with ID -> %s via nickname -> '%s'.",
                        category_name,
                        category_node_id,
                        category_nickname,
                    )
                # Next try via unique names:
                else:
                    response = self._otcs_frontend.get_unique_names(
                        names=[category_name],
                        subtype=self._otcs_frontend.ITEM_TYPE_CATEGORY,
                    )
                    if response and "results" in response and response["results"]:
                        category_node_id = next(
                            (result["NodeId"] for result in response["results"] if result["NodeName"] == category_name),
                            None,
                        )
                        if category_node_id:
                            self.logger.info(
                                "Found category -> '%s' with ID -> %s via unique name lookup!",
                                category_name,
                                category_node_id,
                            )

                # if we found the payload category via nickname or unique name
                # we assign it to thew source node (typically the parent node):
                if category_node_id:
                    self._otcs_frontend.assign_category(
                        node_id=source_node_id,
                        category_id=category_node_id,
                        inheritance=None if source_is_document else True,
                        apply_to_sub_items=not source_is_document,
                    )
            # end for category_name in missing_categories

            # Now with the category assigned to the parent (source node id)
            # We retry getting the inherited category:
            response = self._otcs_frontend.get_node_categories(node_id=source_node_id)
        # end if missing_categories:

        # Initialize the result dict we will return at the end of the method
        # and the list of inherited categories:
        category_data = {}
        inherited_categories = []

        # Redo the test...
        if not response or not response["results"]:
            self.logger.warning(
                "Source node with ID -> %s does not inherit categories but we have category payload! Processing without assiging category...",
                source_node_id,
            )
        else:
            inherited_categories = response["results"]

        # we iterate over all parent categories that are inherited
        # to the new document and try to find matching payload values...
        for inherited_category in inherited_categories:
            # We use the "metadata_order" which is a list of typically one
            # element that includes the category ID:
            metadata_order = inherited_category["metadata_order"]

            # If it is not a list or empty we continue with the
            # next inherited category:
            if not metadata_order["categories"] or not isinstance(
                metadata_order["categories"],
                list,
            ):
                continue
            inherited_category_id = metadata_order["categories"][0]

            # We use the "metadata" dict to determine the category name
            # the keys of the dict are the category ID and attribute IDs
            # the first element in the dict is always the category itself.
            metadata = inherited_category["metadata"]["categories"]
            category_name = metadata[str(inherited_category_id)]["name"]

            self.logger.debug(
                "Source node ID -> %s has category -> '%s' (%s)",
                source_node_id,
                category_name,
                inherited_category_id,
            )

            # The following method returns two values: the category ID and
            # a dict of the attributes. If the category is not found
            # on the parent node it returns -1 for the category ID
            # and an empty dict for the attribute definitions:
            (
                category_id,
                attribute_definitions,
            ) = self._otcs_frontend.get_node_category_definition(
                node_id=source_node_id,
                category_name=category_name,
            )
            if category_id == -1:
                self.logger.error(
                    "The item with ID -> %s does not have the specified category -> '%s' assigned. Skipping...",
                    source_node_id,
                    category_name,
                )
                continue

            # We now initialize the substructure for this particular category:
            category_data[str(category_id)] = {}

            self.logger.debug(
                "Processing the attributes in payload to find values for the inherited category -> '%s' (%s)...",
                category_name,
                category_id,
            )
            # now we fill the prepared (but empty) category_data
            # with the actual attribute values from the payload.
            # For this we traverse all category dicts in the payload
            # and check if they include data for the currently processed
            # category:
            for attribute in categories_payload:
                attribute_name = attribute["attribute"]
                set_name = attribute.get("set", "")
                row = attribute.get("row", "")
                if attribute["name"] != category_name:
                    self.logger.debug(
                        "Attribute -> '%s' does not belong to category -> '%s'. Skipping...",
                        attribute_name,
                        category_name,
                    )
                    continue
                attribute_value = attribute["value"]

                # Set attributes are constructed with <set>:<attribute>
                # by method get_node_category_definition(). This is not
                # an OTCS REST syntax but specific for payload.py
                if set_name:
                    attribute_name = set_name + ":" + attribute_name

                if attribute_name not in attribute_definitions:
                    self.logger.error(
                        "Illegal attribute name -> '%s' in payload for category -> '%s'",
                        attribute_name,
                        category_name,
                    )
                    continue
                attribute_type = attribute_definitions[attribute_name]["type"]
                attribute_id = attribute_definitions[attribute_name]["id"]
                # For multi-line sets the "x" is the placeholder for the
                # row number. We need to replace it with the actual row number
                # given in the payload:
                if "_x_" in attribute_id:
                    if not row:
                        self.logger.error(
                            "Row number is not specified in payload for attribute -> '%s' (%s)",
                            attribute_name,
                            attribute_id,
                        )
                        continue
                    attribute_id = attribute_id.replace("_x_", "_" + str(row) + "_")

                # Special treatment for type user: determine the ID for the login name.
                # the ID is the actual value we have to put in the attribute:
                if attribute_type == "user":
                    user = self._otcs_frontend.get_user(
                        name=attribute_value,
                        show_error=True,
                    )
                    user_id = self._otcs_frontend.get_result_value(
                        response=user,
                        key="id",
                    )
                    if not user_id:
                        self.logger.error(
                            "Cannot find user with login name -> '%s'. Skipping...",
                            attribute_value,
                        )
                        continue
                    attribute_value = user_id
                category_data[str(category_id)][attribute_id] = attribute_value
            # end for attribute in categories_payload:

            # If for none of the attributes of the current category ID a value was found
            # in the payload we remove the dictionary entry to not cause problems
            # for later category updates:
            if not category_data[str(category_id)]:
                del category_data[str(category_id)]
        # end for inherited_category in inherited_categories:

        self.logger.debug("Resulting category data -> %s", str(category_data))

        return category_data

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_documents")
    def process_bulk_documents(self, section_name: str = "bulkDocuments") -> bool:
        """Process bulkDocuments in payload and bulk create them in Content Server (multi-threaded).

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise.

        """

        if not self._bulk_documents:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        # For efficient idem-potent operation we may want to see which workspaces
        # have already been processed successfully before:
        if self.check_status_file(
            payload_section_name=section_name,
            payload_specific=True,
            prefix="failure_",
        ):
            self.logger.info(
                "Found failure file. Trying to reprocess just the failed ones...",
            )
            # Read payload from where we left it last time
            self._bulk_documents = self.get_status_file(
                payload_section_name=section_name,
                prefix="failure_",
            )
            if not self._bulk_documents:
                self.logger.error(
                    "Cannot load existing bulkDocuments failure file. Bailing out!",
                )
                return False

        success: bool = True

        for bulk_document in self._bulk_documents:
            # Check if element has been disabled in payload (enabled = false).
            # In this case we skip the element:
            enabled = bulk_document.get("enabled", True)
            if not enabled:
                self.logger.info("Payload for Bulk Document is disabled. Skipping...")
                continue

            # The payload element must have a "data_source" key:
            data_source_name = bulk_document.get("data_source", None)
            if not data_source_name:
                self.logger.error("Missing data source name in Bulk Document!")
                success = False
                continue

            copy_data_source = bulk_document.get("copy_data_source", False)
            force_reload = bulk_document.get("force_reload", True)

            # Load and prepare the data source for the bulk processing:
            if copy_data_source:
                self.logger.info(
                    "Take a copy of data source -> %s to avoid side-effects for repeative usage of the data source...",
                    data_source_name,
                )
                data = Data(
                    self.process_bulk_datasource(
                        data_source_name=data_source_name,
                        force_reload=force_reload,
                    ),
                    logger=self.logger,
                )
            else:
                self.logger.info(
                    "Use original data source -> '%s' and not do a copy.",
                    bulk_document["data_source"],
                )
                data = self.process_bulk_datasource(
                    data_source_name=data_source_name,
                    force_reload=force_reload,
                )
            if (
                data is None
                or data.get_data_frame() is None  # the 2nd check is important for the "copy_data_source" case!
            ):  # important to use "is None" here!
                self.logger.error(
                    "Failed to load data source -> '%s' for bulk documents!",
                    data_source_name,
                )
                success = False
                continue

            # Check if fields with list substructures should be exploded.
            # We may want to do this outside the bulkDatasource to only
            # explode for bulkDocuments and not for bulkWorkspaces or
            # bulkWorkspaceRelationships:
            explosions = bulk_document.get("explosions", [])
            for explosion in explosions:
                # explode field can be a string or a list
                # exploding multiple fields at once avoids
                # combinatorial explosions - this is VERY
                # different from exploding columns one after the other!
                if "explode_field" not in explosion and "explode_fields" not in explosion:
                    self.logger.error("Missing explosion field(s)!")
                    continue
                # we want to be backwards compatible...
                explode_fields = (
                    explosion["explode_field"] if "explode_field" in explosion else explosion["explode_fields"]
                )
                flatten_fields = explosion.get("flatten_fields", [])
                split_string_to_list = explosion.get("split_string_to_list", False)
                list_splitter = explosion.get("list_splitter", None)
                self.logger.info(
                    "Starting explosion of bulk documents by field(s) -> %s (type -> %s). Size of data frame before explosion -> %s.",
                    explode_fields,
                    str(type(explode_fields)),
                    str(len(data)),
                )
                data.explode_and_flatten(
                    explode_fields=explode_fields,
                    flatten_fields=flatten_fields,
                    make_unique=False,
                    split_string_to_list=split_string_to_list,
                    separator=list_splitter,
                    reset_index=True,
                )
                self.logger.info(
                    "Size of data frame after explosion -> %s.",
                    str(len(data)),
                )
            # end for explosion in explosions

            # Keep only selected rows if filters are specified in bulkDocuments.
            # We have this _after_ "explosions" to allow access to subfields as well.
            # We have this _before_ "sorting" and "deduplication" as we may keep the wrong
            # rows otherwise (unique / deduplication always keeps the first matching row).
            # We may want to do this outside the bulkDatasource to only
            # filter for bulkDocuments and not for bulkWorkspaces or
            # bulkWorkspaceRelationships:
            filters = bulk_document.get("filters", [])
            if filters:
                data.filter(conditions=filters)

            # Sort the data frame if "sort" specified in payload. We may want to do this to have a
            # higher chance that rows with workspace names that may collapse into
            # one name are put into the same partition. This can avoid race conditions
            # between different Python threads.
            # We do this before deduplication (unique) below as sorting has an influence
            # on which duplicates are kept.
            sort_fields = bulk_document.get("sort", [])
            if sort_fields:
                self.logger.info(
                    "Start sorting of bulk document data frame based on fields (columns) -> %s...",
                    str(sort_fields),
                )
                data.sort(sort_fields=sort_fields, inplace=True)
                self.logger.info(
                    "Sorting of bulk document data frame based on fields (columns) -> %s completed!",
                    str(sort_fields),
                )

            # Check if duplicate rows for given fields should be removed. It is
            # important to do this after sorting as Pandas always keep the first occurance,
            # so ordering plays an important role in deduplication!
            unique_fields = bulk_document.get("unique", [])
            if unique_fields:
                self.logger.info(
                    "Starting deduplication of data frame for bulk documents with unique fields -> %s. Size of data frame before deduplication -> %s.",
                    str(unique_fields),
                    str(len(data)),
                )
                data.deduplicate(unique_fields=unique_fields, inplace=True)
                self.logger.info(
                    "Size of data frame after deduplication -> %s.",
                    str(len(data)),
                )

            # Read name field from payload:
            name_field = bulk_document.get("name", None)
            if not name_field:
                self.logger.error(
                    "Bulk document needs a name field! Skipping to next bulk document...",
                )
                success = False
                continue

            self._log_header_callback(
                text="Process bulk documents -> '{}' from data source -> '{}'".format(
                    name_field,
                    data_source_name,
                ),
                char="-",
            )

            # Read optional description field from payload:
            description_field = bulk_document.get("description", None)

            # Read the optional categories payload:
            categories = bulk_document.get("categories", None)
            if not categories:
                self.logger.info(
                    "Bulk document payload has no category data! Will leave category attributes empty...",
                )

            # Should existing documents be updated? False (= no) is the default.
            operations = bulk_document.get("operations", ["create"])

            self.logger.info(
                "Bulk create documents (name field -> '%s', operations -> %s)...",
                name_field,
                str(operations),
            )

            # See if bulkWorkspace definition has a specific thread number
            # otherwise it is read from a global environment variable
            bulk_thread_number = int(
                bulk_document.get("thread_number", BULK_THREAD_NUMBER),
            )

            partitions = data.partitionate(bulk_thread_number)

            # Create a list to hold the threads
            threads = []
            results = []

            # Define source OTCS object and authenticate once and pass it to all workers if needed
            if bulk_document.get("source_type", "URL").lower() == "contentserver":
                if bulk_document.get("cs_hostname") is None:
                    source_otcs = None
                    self.logger.error(
                        "Required information for source type ContentServer is not configured -> cs_hostname",
                    )
                    success = False
                    continue

                if bulk_document.get("cs_username") is None:
                    source_otcs = None
                    self.logger.error(
                        "Required information for source type ContentServer is not configured -> cs_username",
                    )
                    success = False
                    continue

                if bulk_document.get("cs_password") is None:
                    source_otcs = None
                    self.logger.error(
                        "Required information for source type ContentServer is not configured -> cs_password",
                    )
                    success = False
                    continue

                self.logger.info(
                    "Generating reusable OTCS instance for bulk processing...",
                )
                source_otcs = OTCS(
                    protocol=bulk_document.get("cs_protocol", "https"),
                    hostname=bulk_document.get("cs_hostname"),
                    port=bulk_document.get("cs_port", "443"),
                    base_path=bulk_document.get("cs_basepath", "/cs/cs"),
                    username=bulk_document.get("cs_username"),
                    password=bulk_document.get("cs_password"),
                    logger=self.logger,
                )
                source_otcs.authenticate()
            # end if bulk_document.get("source_type", "URL").lower() == "contentserver"
            else:
                source_otcs = None

            # Create and start a thread for each partition
            for index, partition in enumerate(partitions, start=1):
                # start a thread executing the process_bulk_documents_worker() method below:
                thread = threading.Thread(
                    name=f"{section_name}_{index:02}",
                    target=self.thread_wrapper,
                    args=(
                        self.process_bulk_documents_worker,
                        bulk_document,
                        partition,
                        name_field,
                        description_field,
                        categories,
                        operations,
                        results,
                        source_otcs,
                    ),
                )
                self.logger.info("Starting thread -> '%s'...", str(thread.name))
                thread.start()
                threads.append(thread)
                # Avoid that all threads start at the exact same time with
                # potentially expired cookies that cases race conditions:
                time.sleep(1)
            # end for index, partition in enumerate(partitions, start=1)

            # Wait for all threads to complete
            for thread in threads:
                self.logger.info(
                    "Waiting for thread -> '%s' to complete...",
                    str(thread.name),
                )
                thread.join()
                self.logger.info("Thread -> '%s' has completed.", str(thread.name))

            if "documents" not in bulk_document:
                bulk_document["documents"] = {}

            # Print statistics for each thread. In addition,
            # check if all threads have completed without error / failure.
            # If there's a single failure in on of the thread results we
            # set 'success' variable to False.
            results.sort(key=lambda x: x["thread_name"])
            for result in results:
                self.logger.info(
                    "Thread -> '%s' completed with overall status '%s'.",
                    str(result["thread_name"]),
                    "successful" if result["success"] else "failed",
                )
                self.logger.info(
                    "Thread -> '%s' processed %s data rows with %s successful, %s failed, and %s skipped.",
                    str(result["thread_name"]),
                    str(
                        result["success_counter"] + result["failure_counter"] + result["skipped_counter"],
                    ),
                    str(result["success_counter"]),
                    str(result["failure_counter"]),
                    str(result["skipped_counter"]),
                )
                self.logger.info(
                    "Thread -> '%s' created %s documents, updated %s documents, and deleted %s documents.",
                    str(result["thread_name"]),
                    str(result["create_counter"]),
                    str(result["update_counter"]),
                    str(result["delete_counter"]),
                )

                if not result["success"]:
                    success = False
                # Record all generated documents. This should allow us
                # to restart in case of failures and avoid trying to
                # uploading that have been successfully uploaded before.
                bulk_document["documents"].update(result["documents"])
            # end for result in results
            self._log_header_callback(
                text="Completed processing of bulk documents -> '{}' using data source -> '{}'".format(
                    name_field,
                    data_source_name,
                ),
                char="-",
            )

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._bulk_documents,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="construct_file_name")
    def construct_file_name(
        self,
        path: str,
        download_name: str,
        download_name_wildcards: bool = False,
        file_extension: str = "",
    ) -> str | None:
        """Construct the file name of the document.

        This considers the path given in the download_dir, potential wildcards in the
        download_name variable and the file extensions provided (if any).

        Args:
            path (str):
                The base base in the file system. Placeholders need to be resolved before
                calling this method.
            download_name (str):
                The filenname of document in the file system. This may include wildcards
                like '*.pdf' or 'en/*.txt' or '**/*.pfd'. Then the actual download
                file name is determined by a directory traversal using the 'path' parameter.
            download_name_wildcards (bool, optional):
                Defines whether or not wildcards should be replaced in the download name.
                If True this walks through the given path and try to find the file in the
                filesystem that matches the wildcard pattern.
            file_extension (str, optional):
                The file extension - typically 3 letters like 'pdf'. Defaults to "".

        Returns:
            str:
                The file name that is used to find the document in the filesystem (or if it does not yet exist)
                the name that should be used for download. This can be None

        """

        if not download_name:
            self.logger.error("Download name is missing. Cannot construct file name!")
            return None

        if path:
            # Normalize on Linux style path separators. Python is able to handle this also under Windows
            path = path.replace("\\", "/")
            if os.path.exists(path):
                # if we have a path specified it should be a directory and not a file!
                # The isfile() test is only working if the file already exist. It cannot
                # tell the difference between a folder and a file otherwise!
                if os.path.isfile(path):
                    self.logger.warning(
                        "Download directory -> '%s' is pointing to an existing file and not a directory - please check the 'download_dir' variable in payload! Stripping path...",
                        path,
                    )
                    path = os.path.dirname(path)
                    self.logger.info("Stripped path -> '%s'...", path)
            else:
                # if we have a path specified but it does not exist, then create it.
                try:
                    os.makedirs(path)
                except (OSError, PermissionError):
                    self.logger.error(
                        "Cannot create folder -> '%s' in file system!",
                        path,
                    )
                    return None
            if not path.endswith("/"):
                path += "/"  # slashes are save in Linux and Windows. Python handles this correctly

        # If we have a file extension and the given download does
        # not yet ends with it, we add it to the download name:
        if file_extension:
            if not file_extension.startswith("."):
                file_extension = "." + file_extension
            if not download_name.endswith(file_extension):
                download_name += file_extension

        # Also find files with wildcards in 'download_name' if requested
        # by download_name_wildcards == True.
        # IMPORTANT: this only matches ONE file as this method
        # also processes only ONE file at a time:
        if download_name_wildcards and any(char in download_name for char in "*?[]"):
            if not path:
                self.logger.error(
                    "Download name includes wildcards but no (base) path is specified in payload (download_dir missing)! Cannot construct file name!",
                )
                return None
            file_name = None
            # Traverse the directory tree starting at 'path', looking for all files and subdirectories
            for _, _, tmpfiles in os.walk(path):
                # For each file found in the current directory:
                for file_data in tmpfiles:
                    # Check if the current file name matches the 'download_name' pattern using wildcards (fnmatch)
                    if fnmatch.fnmatch(file_data, download_name):
                        self.logger.debug(
                            "Found file name -> '%s' that is matching download name pattern -> '%s'",
                            file_data,
                            download_name,
                        )
                        # Construct the full file path by concatenating the base 'path' and the file name
                        file_name = path + file_data
                        # If we found a match we stop the (inner loop)
                        break
        # end if download_name_wildcards and any(char in download_name for char in "*?[]")
        else:
            # We have a normal filename without wildcards:
            file_name = path + download_name

        return file_name

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="download_bulk_document")
    def download_bulk_document(
        self,
        bulk_document: dict,
        download_name: str,
        row: pd.Series,
        result: dict,
        source_otcs: OTCS | None = None,
    ) -> tuple[str, str]:
        """Download the bulk document and determine the file name (with path) and the mime type.

        Args:
            bulk_document (dict):
                The payload item for the bulk document.
            download_name (str):
                The download name of the document.
            row (pd.Series):
                The current data frame series (row).
            result (dict):
                The result dictionary (mutable).
            source_otcs (OTCS | None, optional):
                The Content Server OTCS object if documents should be loaded
                from Content Server. Defaults to None.

        Returns:
            tuple[str, str]:
                The file name (first) and the mime type (second).

        """

        #
        # 1. Construct the local file name to upload:
        #

        path = bulk_document.get("download_dir", BULK_DOCUMENT_PATH)
        path = self.replace_bulk_placeholders(input_string=path, row=row)
        self.logger.debug("Download path for bulk document -> '%s'", path)

        # Determine file extensions:
        file_extension = bulk_document.get("file_extension", "")
        if file_extension:
            file_extension = self.replace_bulk_placeholders(
                input_string=file_extension,
                row=row,
            )
        file_extension_alt = bulk_document.get("file_extension_alt", "")
        if file_extension_alt:
            file_extension_alt = self.replace_bulk_placeholders(
                input_string=file_extension_alt,
                row=row,
            )

        # Determine file names:
        file_name = self.construct_file_name(
            path=path,
            download_name=download_name,
            download_name_wildcards=bulk_document.get("download_name_wildcards", False),
            file_extension=file_extension,
        )
        self.logger.debug("Constructed file name -> '%s'", file_name)
        if file_extension_alt and file_extension_alt != file_extension:
            file_name_alt = self.construct_file_name(
                path=path,
                download_name=download_name,
                download_name_wildcards=bulk_document.get(
                    "download_name_wildcards",
                    False,
                ),
                file_extension=file_extension_alt,
            )
        else:
            file_name_alt = file_name

        # If we couldn't construct the file name nor an alternative file name, we bail out:
        if not file_name and not file_name_alt:
            self.logger.error(
                "Cannot determine file name for document download (path -> '%s', download name -> '%s')!",
                path,
                download_name,
            )
            return None, None

        #
        # 2. Construct the mime type of the file to upload:
        #

        mime_type = bulk_document.get("mime_type", "application/pdf")
        mime_type = self.replace_bulk_placeholders(input_string=mime_type, row=row)
        mime_type_alt = bulk_document.get("mime_type_alt", "text/html")
        if mime_type_alt != "text/html":
            # if it is not the default it may have placeholders:
            mime_type_alt = self.replace_bulk_placeholders(
                input_string=mime_type_alt,
                row=row,
            )

        #
        # 3. Check if the file name(s) do(es) already exist and
        #    if we want to delete existing files (to download fresh ones)
        #

        file_exists = os.path.exists(file_name) if file_name else False

        # make sure there's no name conflict with stale documents:
        delete_download = bulk_document.get("delete_download", True)
        if file_exists and delete_download:
            os.remove(file_name)
            file_exists = False

        # Careful: If the construction of file name and the alternative
        # file name leads to identical result then there's actually
        # no alternative file name!
        if file_name != file_name_alt:
            file_exists_alt = os.path.exists(file_name_alt) if file_name_alt else False
            # make sure there's no name conflict with stale documents:
            if file_exists_alt and delete_download:
                os.remove(file_name_alt)
                file_exists_alt = False
        else:
            file_exists_alt = False

        #
        # 4. if the file does not exist in the local file system
        #    we now go and download it...
        #

        if not file_exists and not file_exists_alt:
            self.logger.debug("File -> '%s' does not exist in filesystem. We need to download it.", file_name)
            # Read "download retry number" and "wait before retry" duration from payload
            # (if specified) otherwise set default values
            wait_time = bulk_document.get("download_wait_time", 30)
            retries = bulk_document.get("download_retries", 2)

            source_type = bulk_document.get("source_type", "URL").lower()
            match source_type:
                case "contentserver":
                    # Functionality to download the document from Content Server
                    cs_source_id = bulk_document.get("cs_source_id", "")
                    cs_source_id = self.replace_bulk_placeholders(
                        input_string=cs_source_id,
                        row=row,
                    )
                    if source_otcs is not None and source_otcs.otcs_ticket() is not None:
                        self.logger.info(
                            "Downloading document from source Content Server with ID -> %s...",
                            cs_source_id,
                        )

                        if source_otcs.download_document(
                            node_id=cs_source_id,
                            file_path=file_name,
                        ):
                            self.logger.debug(
                                "Successfully downloaded from Content Server using URL -> %s with ID -> %s to local file -> '%s'",
                                source_otcs.cs_public_url,
                                cs_source_id,
                                file_name,
                            )
                        else:
                            self.logger.error(
                                "Cannot download file from Content Server using URL -> %s with ID -> %s to local file -> '%s'. Skipping...",
                                source_otcs.cs_public_url(),
                                cs_source_id,
                                file_name,
                            )
                            result["skipped_counter"] += 1
                            return None, None
                    else:
                        self.logger.error(
                            "Cannot download file with ID -> %s from Content Server. OTCS object not configured. Skipping...",
                            cs_source_id,
                        )
                        result["skipped_counter"] += 1
                        return None, None

                case "url":
                    # Default case, download from accessible URL
                    download_url = bulk_document.get("download_url")
                    if download_url:
                        download_url = self.replace_bulk_placeholders(
                            input_string=download_url,
                            row=row,
                        )
                    if not download_url:
                        self.logger.error(
                            "Download URL missing and no existing file -> '%s' in file system!",
                            file_name,
                        )
                        result["skipped_counter"] += 1
                        return None, None
                    # Use the HHTP class to download the file:
                    if not self._http_object.download_file(
                        url=download_url,
                        filename=file_name,
                        retries=retries,
                        wait_time=wait_time,
                        wait_on_status=[403],
                        show_error=False,
                    ):
                        # Fetch alternative download URL (if avialable)
                        download_url_alt = bulk_document.get("download_url_alt")
                        if download_url_alt:
                            download_url_alt = self.replace_bulk_placeholders(
                                input_string=download_url_alt,
                                row=row,
                            )
                        # Check if we have an alternative download URL we try this now:
                        if download_url_alt:
                            self.logger.warning(
                                "Cannot download file from -> %s to local file -> '%s'. Trying alternative download -> %s to file -> '%s'...",
                                download_url,
                                file_name,
                                download_url_alt,
                                file_name_alt,
                            )
                            if self._http_object.download_file(
                                url=download_url_alt,
                                filename=file_name_alt,
                                retries=retries,
                                wait_time=wait_time,
                                wait_on_status=[403],
                                show_error=False,
                            ):
                                self.logger.debug(
                                    "Successfully downloaded file from alternative URL -> %s to local file -> '%s'. Using this file...",
                                    download_url_alt,
                                    file_name_alt,
                                )
                                mime_type = mime_type_alt
                                file_name = file_name_alt
                            else:
                                # as we cannot fully rely one data source we don't treat this
                                # as an error but a warning:
                                self.logger.warning(
                                    "Cannot download file from alternative URL -> %s to local file -> '%s'. Skipping...",
                                    download_url_alt,
                                    file_name_alt,
                                )
                                result["skipped_counter"] += 1
                                return None, None
                        # end if download_url_alt
                        else:
                            # as we cannot fully rely one data source we don't treat this
                            # as an error but a warning:
                            self.logger.warning(
                                "Cannot download file from URL -> %s to local file -> '%s'. Skipping...",
                                download_url,
                                file_name,
                            )
                            result["skipped_counter"] += 1
                            return None, None
                    else:
                        self.logger.debug(
                            "Successfully downloaded file from -> %s to local file -> '%s'",
                            download_url,
                            file_name,
                        )
        # end if not file_exists and not file_exists_alt
        else:  # we already have a local file to reuse
            # If we found the alternative file name in file system
            # but not the regular one, then we switch to alternative
            # file name:
            if not file_exists and file_exists_alt:
                file_name = file_name_alt
                mime_type = mime_type_alt
            self.logger.info(
                "Reusing existing file -> '%s' in local storage.",
                file_name,
            )

        return file_name, mime_type

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_bulk_document_location")
    def get_bulk_document_location(
        self,
        workspace: dict,
        row: pd.Series,
        index: int,
        replacements: dict,
    ) -> tuple[int | None, bool]:
        """Determine the workspace location to store a document in bulk processing.

        This method is used by bulk document and bulk item workers.

        Args:
            workspace (dict):
                The payload for the workspace inside bulk document.
            row (pd.Series):
                The current row in the data frame.
            index (int):
                The index of current row in the data frame.
            replacements (dict):
                The replacement configuration for placeholders.

        Returns:
            int | None:
                Parent ID of folder or workspace to upload the document to or None in case of
                an error or insufficient payload information causing the workspace to be skipped.
            bool:
                True, if success. False if a failure has happened.

        """

        success = True

        if "workspace_name" not in workspace:
            self.logger.error(
                "No workspace name field specified for document upload! Cannot upload document to this workspace...",
            )
            success = False
            return None, success
        workspace_name_field = workspace["workspace_name"]
        workspace_name_field_alt = workspace.get("workspace_name_alt")

        workspace_name = self.replace_bulk_placeholders(
            input_string=workspace_name_field,
            row=row,
            replacements=replacements,
        )
        # If we cannot get the workspace name from the
        # workspace_name_field we try the alternative name field
        # (if specified in payload via "workspace_name_alt"):
        if not workspace_name and workspace_name_field_alt:
            self.logger.debug(
                "Row -> %s does not have the data to resolve the placeholders in workspace name field -> %s! Trying alternative name field -> %s...",
                str(index),
                workspace_name_field,
                workspace_name_field_alt,
            )
            workspace_name = self.replace_bulk_placeholders(
                input_string=workspace_name_field_alt,
                row=row,
                replacements=replacements,
            )

        # it could be that the current data row does not have the
        # required fields to resolve the workspace name placeholders
        # then we skip uploading the document to this workspace
        # but still keep status as successful (don't set success = False)
        if not workspace_name:
            self.logger.warning(
                "Row -> %s does not have the required data to resolve workspace name field -> '%s' specified for document upload! Skipping document upload to this workspace...",
                str(index),
                workspace_name_field,
            )
            # We keep success = True as this is a data problem and not a config problem!
            return None, success

        # Cleanse the workspace name (allowed characters, maximum length):
        workspace_name = OTCS.cleanse_item_name(workspace_name)

        # Check if all data conditions to create the workspace are met
        conditions = workspace.get("conditions")
        if conditions:
            evaluated_condition = self.evaluate_conditions(
                conditions=conditions,
                row=row,
            )
            if not evaluated_condition:
                self.logger.info(
                    "Workspace condition for row -> %s not met. Skipping row for document upload to workspace -> '%s'...",
                    str(index),
                    workspace_name,
                )
                # We keep success = True as this is a data problem and not a config problem!
                return None, success

        if "workspace_type" not in workspace:
            self.logger.error(
                "No workspace type specified for document upload! Skipping workspace -> '%s'...",
                workspace_name,
            )
            success = False
            return None, success
        workspace_type = workspace["workspace_type"]
        workspace_type = self.replace_bulk_placeholders(
            input_string=workspace_type,
            row=row,
            replacements=replacements,
        )
        workspace_data_source_name = workspace.get("data_source")
        # Try to find the workspace by name/synonym and type:
        (workspace_id, workspace_name) = self.process_bulk_workspaces_lookup(
            workspace_name=workspace_name,
            workspace_type=workspace_type,
            data_source_name=workspace_data_source_name,
        )
        if not workspace_id:
            self.logger.warning(
                "Cannot find workspace with name/synonym -> '%s' and type -> '%s'.",
                workspace_name,
                workspace_type,
            )
            success = False
            return None, success

        # If the workspace payload element has a "data_source" key,
        # then add all columns from the given data source to the bulk
        # document row to also support the lookup of values from the workspace
        # data source. These fields get a "lookup_" prefix to avoid name clashes.
        # the values must be specified with this "lookup_" prefix in the payload.
        # We CANNOT do this at the very beginning of the workspace loop as we
        # need the workspace_name to be properly resolved (incl. synonyms):
        if workspace_data_source_name:
            self.logger.info(
                "Workspace for bulk documents has a data source -> '%s' with lookup values. Adding them as row columns...",
                workspace_data_source_name,
            )
            workspace_data_source = next(
                (item for item in self._bulk_datasources if item["name"] == workspace_data_source_name),
                None,
            )
            # Read the synonym column and the name column from the data source payload item:
            workspace_data_source_name_column = workspace_data_source.get(
                "name_column",
                None,  # e.g. "Name"
            )

            if workspace_data_source_name_column:
                # Get additional data from workspace data source
                # for lookups. Synonyms are already resolved at
                # this point in time (workspace_name has been updated above
                # in case the initial workspace name wasn't found)
                lookup_row = self.lookup_data_source_value(
                    data_source=workspace_data_source,
                    lookup_column=workspace_data_source_name_column,
                    lookup_value=workspace_name,
                )

                # Adding all values of the lookup row with the prefix lookup_ to the bulk documents row
                # for replacement of placeholders:
                if lookup_row is not None:
                    for k, value in lookup_row.items():
                        row["lookup_" + k] = value

        # "workspace_folder" can be used if the payload contains
        # the path as a comma-separated string (top down)
        # otherwise "workspace_path" has precedence over "workspace_folder"
        workspace_folder = workspace.get("workspace_folder", "")
        workspace_path = workspace.get("workspace_path")

        if workspace_folder and not workspace_path:
            workspace_folder = self.replace_bulk_placeholders(
                input_string=workspace_folder,
                row=row,
                replacements=replacements,
            )
            workspace_path = (
                [item.strip() for item in workspace_folder.split(",")]
                if "," in workspace_folder
                else [workspace_folder]
            )

        if workspace_path:
            if isinstance(workspace_path, str):
                # if the path is actually a list in a string
                # we need to convert it to a python list in a safe way:
                try:
                    workspace_path = self.replace_bulk_placeholders(
                        input_string=workspace_path,
                        index=None,  # None is VERY important as otherwise index=0 is the default and we only get the first element
                        row=row,
                        replacements=replacements,
                    )
                    workspace_path = literal_eval(workspace_path) if workspace_path else None
                except (SyntaxError, ValueError):
                    self.logger.error(
                        "Cannot parse list-type folder path wrapped in string -> '%s'!",
                        workspace_path,
                    )
                    workspace_path = None
            elif isinstance(workspace_path, list):
                # We create a copy list to not modify original payload
                workspace_path = list(workspace_path)
                # Replace placeholders in payload for the path elements:
                # Note: workspace_path is a mutable data type that is changed in place!
                result_placeholders = self.replace_bulk_placeholders_list(
                    input_list=workspace_path,
                    row=row,
                    replacements=replacements,
                )
                if not result_placeholders:
                    workspace_path = None
            else:
                self.logger.warning("Unsupported data type for workspace path!")
                workspace_path = None

            if not workspace_path:
                # we put the document into the root of the workspace
                # if we couldn't determine a path inside the workspace:
                self.logger.info(
                    "Workspace folder path for workspace -> '%s' of workspace type -> '%s' has no value. Using workspace root for document upload.",
                    workspace_name,
                    workspace_type,
                )
                parent_id = workspace_id
            else:
                # Check if the folder path does already exist and get the target folder at the end of the path:
                self.logger.info(
                    "Check if path -> %s does already exist in workspace -> '%s' (%s) or otherwise create it...",
                    str(workspace_path),
                    workspace_name,
                    workspace_id,
                )
                response = self._otcs_frontend.get_node_by_workspace_and_path(
                    workspace_id=workspace_id,
                    path=workspace_path,
                    create_path=True,  # we want the path to be created if it doesn't exist
                    show_error=False,
                )
                parent_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
                if not parent_id:
                    self.logger.error(
                        "Failed to create path -> %s in workspace -> '%s' (%s)!",
                        str(workspace_path),
                        workspace_name,
                        workspace_id,
                    )
                    success = False
                    return None, success
                else:
                    self.logger.info(
                        "Using path -> %s inside workspace -> '%s' (%s). Node ID for target folder -> %s.",
                        str(workspace_path),
                        workspace_name,
                        workspace_id,
                        str(parent_id),
                    )
        # end if workspace_path
        else:
            self.logger.info(
                "Workspace folder path for workspace -> '%s' of workspace type -> '%s' is not specified. Using workspace root for document upload.",
                workspace_name,
                workspace_type,
            )
            # we put the document into the root of the workspace:
            parent_id = workspace_id

        # Check if we have sub-workspaces configured.
        # If not, then we are done and can return the found
        # parent ID for the document:
        if "sub_workspace_type" not in workspace or "sub_workspace_name" not in workspace:
            self.logger.debug(
                "No sub workspace configured in payload. Return parent ID -> %s",
                parent_id,
            )
            return parent_id, success

        #
        # Determine (or create) the Sub workspace:
        #

        sub_workspace_type = workspace["sub_workspace_type"]
        sub_workspace_type = self.replace_bulk_placeholders(
            input_string=sub_workspace_type,
            row=row,
            replacements=replacements,
        )
        sub_workspace_name = workspace["sub_workspace_name"]
        sub_workspace_name = self.replace_bulk_placeholders(
            input_string=sub_workspace_name,
            row=row,
            replacements=replacements,
        )
        response = self._otcs_frontend.get_node_by_parent_and_name(
            name=sub_workspace_name,
            parent_id=parent_id,
        )
        # Check if the sub-workspaces does already exist:
        sub_workspace_id = self._otcs_frontend.get_result_value(
            response=response,
            key="id",
        )
        # Sub workspaces are dynamically created
        # during the processing of bulk documents...
        if not sub_workspace_id:
            self.logger.info(
                "Creating sub workspace -> '%s' of type -> '%s' and parent -> %s...",
                sub_workspace_name,
                sub_workspace_type,
                parent_id,
            )
            sub_workspace_template = workspace.get("sub_workspace_template", "")

            sub_workspace_template = self.replace_bulk_placeholders(
                input_string=sub_workspace_template,
                row=row,
                replacements=replacements,
            )
            # Now we try to determine the IDs for the sub-workspace type and template:
            (sub_workspace_type_id, sub_workspace_template_id) = self.determine_workspace_type_and_template_id(
                workspace_type_name=sub_workspace_type,
                workspace_template_name=sub_workspace_template,
            )
            # if either of the two couldn't be determined we cannot create the sub-workspace
            if not sub_workspace_type_id or not sub_workspace_template_id:
                self.logger.error(
                    "Couldn't dertermine workspace template ID and workspace type ID of sub-workspace!",
                )
                success = False
                return None, success

            # Check if we have categories for the sub-workspace:
            if "categories" not in workspace:
                self.logger.debug(
                    "Sub-Workspace payload has no category data! Will leave category attributes empty...",
                )
                sub_workspace_category_data = {}
            else:
                self.logger.debug(
                    "Sub-Workspace payload has category data! Will prepare category data for workspace creation...",
                )
                worker_categories = self.process_bulk_categories(
                    row=row,
                    index=index,
                    categories=workspace["categories"],
                    replacements=replacements,
                )
                self.logger.debug(
                    "Prepare category data for sub-workspace with template -> %s and parent -> %s",
                    sub_workspace_template_id,
                    parent_id,
                )
                sub_workspace_category_data = self.prepare_workspace_create_form(
                    categories=worker_categories,
                    template_id=sub_workspace_template_id,
                    parent_workspace_node_id=parent_id,
                )
                if not sub_workspace_category_data:
                    self.logger.error(
                        "Failed to prepare the category data for sub-workspace -> '%s'!",
                        sub_workspace_name,
                    )
                    success = False
                    return None, success
            # Now we create the sub-workspace:
            response = self._otcs_frontend.create_workspace(
                workspace_template_id=sub_workspace_template_id,
                workspace_name=sub_workspace_name,
                workspace_description="",
                workspace_type=sub_workspace_type_id,
                category_data=sub_workspace_category_data,
                parent_id=parent_id,
                show_error=False,
            )
            if response is None:
                # Potential race condition: see if the sub-workspace has been created by a concurrent thread.
                # So we better check if the workspace is there even if the create_workspace() call delivered
                # a 'None' response:
                response = self._otcs_frontend.get_node_by_parent_and_name(
                    parent_id=parent_id,
                    name=sub_workspace_name,
                )
            sub_workspace_id = self._otcs_frontend.get_result_value(
                response=response,
                key="id",
            )
            if not sub_workspace_id:
                self.logger.error(
                    "Failed to create sub-workspace -> '%s' with type ID -> %s under parent with ID -> %s!",
                    sub_workspace_name,
                    sub_workspace_type_id,
                    parent_id,
                )
                parent_id = None
                success = False
                return None, success
            else:
                self.logger.info(
                    "Successfully created sub-workspace -> '%s' (%s).",
                    sub_workspace_name,
                    sub_workspace_id,
                )

            # Create Business Relationship between workspace and sub-workspace:
            if workspace_id and sub_workspace_id:
                # Check if workspace relationship does already exist in OTCS
                # (this is an additional safety measure to avoid errors):
                response = self._otcs_frontend.get_workspace_relationships(
                    workspace_id=workspace_id,
                    related_workspace_name=sub_workspace_name,
                )
                current_workspace_relationships = self._otcs_frontend.exist_result_item(
                    response=response,
                    key="id",
                    value=sub_workspace_id,
                )
                if current_workspace_relationships:
                    self.logger.info(
                        "Workspace relationship between workspace -> '%s' (%s) and sub-workspace -> '%s' (%s) does already exist. Skipping...",
                        workspace_name,
                        workspace_id,
                        sub_workspace_name,
                        sub_workspace_id,
                    )
                else:
                    self.logger.info(
                        "Create workspace relationship %s -> %s...",
                        workspace_id,
                        sub_workspace_id,
                    )
                    response = self._otcs_frontend.create_workspace_relationship(
                        workspace_id=workspace_id,
                        related_workspace_id=sub_workspace_id,
                        show_error=False,  # we don't want to show an error because of race conditions handled below
                    )
                    if response:
                        self.logger.info(
                            "Successfully created workspace relationship between workspace ID -> %s and sub-workspace ID -> %s.",
                            workspace_id,
                            sub_workspace_id,
                        )
                    else:
                        # Potential race condition: see if the workspace-2-sub-workspace relationship has been created by a concurrent thread.
                        # So we better check if the relationship is there even if the create_workspace_relationship() call delivered
                        # a 'None' response:
                        response = self._otcs_frontend.get_workspace_relationships(
                            workspace_id=workspace_id,
                            related_workspace_name=sub_workspace_name,
                        )
                        current_workspace_relationships = self._otcs_frontend.exist_result_item(
                            response=response,
                            key="id",
                            value=sub_workspace_id,
                        )
                        if not current_workspace_relationships:
                            self.logger.error(
                                "Failed to create workspace relationship between workspace ID -> %s and sub-workspace ID -> %s!",
                                workspace_id,
                                sub_workspace_id,
                            )
                        else:
                            self.logger.info(
                                "Successfully created workspace relationship between workspace ID -> %s and sub-workspace ID -> %s.",
                                workspace_id,
                                sub_workspace_id,
                            )

        # end if sub-workspace does not exist
        else:
            self.logger.info(
                "Using existing sub workspace -> '%s' (%s) of type -> '%s'...",
                sub_workspace_name,
                str(sub_workspace_id),
                sub_workspace_type,
            )

        #
        # Get the target folder in the sub-workspace by the provided sub workspace path
        #

        # "sub_workspace_folder" can be used if the payload contains
        # the path as a comma-separated string (top down)
        sub_workspace_folder = workspace.get("sub_workspace_folder", "")
        sub_workspace_path = workspace.get("sub_workspace_path")

        if not sub_workspace_path and sub_workspace_folder:
            sub_workspace_folder = self.replace_bulk_placeholders(
                input_string=sub_workspace_folder,
                row=row,
                replacements=replacements,
            )
            sub_workspace_path = (
                [item.strip() for item in sub_workspace_folder.split(",")]
                if "," in sub_workspace_folder
                else [sub_workspace_folder]
            )

        if sub_workspace_path:
            if isinstance(sub_workspace_path, str):
                # if the path is actually a list in a string
                # we need to convert it to a python list in a safe way:
                try:
                    sub_workspace_path = self.replace_bulk_placeholders(
                        input_string=sub_workspace_path,
                        index=None,  # None is VERY important as otherwise index=0 is the default and we only get the first element
                        row=row,
                        replacements=replacements,
                    )
                    sub_workspace_path = literal_eval(sub_workspace_path) if sub_workspace_path else None
                except (SyntaxError, ValueError):
                    self.logger.error(
                        "Cannot parse list-type folder path wrapped in string -> '%s'!",
                        sub_workspace_path,
                    )
                    sub_workspace_path = None
            elif isinstance(sub_workspace_path, list):
                # We create a copy list to not modify original payload
                sub_workspace_path = list(sub_workspace_path)
                # Replace placeholders in payload for the path elements:
                # Note: sub_workspace_path is a mutable data type that is changed in place!
                result_placeholders = self.replace_bulk_placeholders_list(
                    input_list=sub_workspace_path,
                    row=row,
                    replacements=replacements,
                )
                if not result_placeholders:
                    sub_workspace_path = None
            else:
                self.logger.warning("Unsupported data type for sub workspace path!")
                sub_workspace_path = None

            if not sub_workspace_path:
                self.logger.warning(
                    "Sub-Workspace folder path for sub workspace -> '%s' of sub workspace type -> '%s' cannot be resolved (placeholder issue). Using sub workspace root for document upload.",
                    sub_workspace_name,
                    sub_workspace_type,
                )
                # we put the document into the root of the workspace:
                parent_id = sub_workspace_id
            else:
                # Check if the folder path does already exist and get the target folder at the end of the path:
                self.logger.info(
                    "Check if path -> %s does already exist in workspace -> '%s' (%s)... (otherwise create it)",
                    str(sub_workspace_path),
                    sub_workspace_name,
                    sub_workspace_id,
                )
                response = self._otcs_frontend.get_node_by_workspace_and_path(
                    workspace_id=sub_workspace_id,
                    path=sub_workspace_path,
                    create_path=True,  # we want the path to be created if it doesn't exist
                    show_error=False,
                )
                parent_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
                if not parent_id:
                    self.logger.error(
                        "Failed to create path -> %s in sub workspace -> '%s' (%s)!",
                        str(sub_workspace_path),
                        sub_workspace_name,
                        sub_workspace_id,
                    )
                    success = False
                    return None, success
                else:
                    self.logger.info(
                        "Successfully created path -> %s in sub-workspace -> '%s' (%s). Node ID for target folder -> %s.",
                        str(sub_workspace_path),
                        sub_workspace_name,
                        sub_workspace_id,
                        str(parent_id),
                    )
        else:
            self.logger.info(
                "Folder path inside sub-workspace -> '%s' with sub workspace type -> '%s' is not specified. Using root of sub-workspace for document upload.",
                sub_workspace_name,
                sub_workspace_type,
            )
            # we put the document into the root of the workspace:
            parent_id = sub_workspace_id

        return parent_id, success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_documents_worker")
    def process_bulk_documents_worker(
        self,
        bulk_document: dict,
        partition: pd.DataFrame,
        name_field: str,
        description_field: str,
        categories: list | None = None,
        operations: list | None = None,
        results: list | None = None,
        source_otcs: OTCS | None = None,
    ) -> None:
        """Thread worker to download + create documents in bulk.

        Each worker thread gets a partition of the rows that include
        the data required for the document creation.

        Args:
            bulk_document (dict):
                The bulkDocument payload element.
            partition (pd.DataFrame):
                Data partition with rows to process by this thread.
            name_field (str):
                The payload field where the document name is stored.
            description_field (str):
                The payload field where the document description is stored.
            categories (list, optional):
                A list of category dictionaries.
            operations (list, optional):
                A list of operations that should be applyed on workspaces.
                Possible values: "create", "update", "delete", "recreate".
            results (list, optional):
                A mutable list of thread results.
            source_otcs (OTCS | None, optional):
                Provide the OTCS object if a direct download from Content Server is done.

        """

        thread_id = threading.get_ident()
        thread_name = threading.current_thread().name

        self.logger.info(
            "Start working on data frame partition of size -> %s to bulk create documents...",
            str(len(partition)),
        )

        # Avoid linter warnings - so make parameter default None while we
        # actually want ["create"] to be the default:
        if operations is None:
            operations = ["create"]

        # get the specific update operations given in the payload
        # if not specified we do the following update operations.
        # The 'purge' operation needs to be specified explicitly.
        update_operations = bulk_document.get(
            "update_operations",
            ["name", "description", "categories", "nickname", "version"],
        )

        result = {}
        result["thread_id"] = thread_id
        result["thread_name"] = thread_name
        result["success_counter"] = 0
        result["failure_counter"] = 0
        result["skipped_counter"] = 0
        result["create_counter"] = 0
        result["update_counter"] = 0
        result["delete_counter"] = 0
        result["documents"] = {}
        result["success"] = True

        # Check if documents have been processed before, e.i. testing
        # if a "documents" key exists and if it is pointing to a non-empty list:
        # Additionally we check that workspace updates are not enforced:
        if (
            bulk_document.get("documents")
            and "update" not in operations
            and "delete" not in operations
            and "recreate" not in operations
        ):
            existing_documents = bulk_document["documents"]
            self.logger.info(
                "Found %s already processed documents. Try to complete the job...",
                str(len(existing_documents)),
            )
        else:
            existing_documents = {}

        # See if external creation and modification dates are in the payload data:
        external_modify_date_field = bulk_document.get("external_modify_date")
        external_create_date_field = bulk_document.get("external_create_date")

        # See if we have a key field to uniquely identify an existing document:
        key_field = bulk_document.get("key")

        # Get dictionary of replacements for bulk document creations
        # this we will be used of all places data is read from the
        # data frame. Each dictionary item has the field name as the
        # dictionary key and a list of regular expressions as dictionary value
        replacements = bulk_document.get("replacements")

        # In case the name cannot be resolved we allow to
        # specify an alternative name field in the payload.
        name_field_alt = bulk_document.get("name_alt")

        # If download_name field is not in payload we use name_field instead.
        # It can still be that download_name is "" as name_field is only
        # used if the entry for "download_name" is not in payload at all.
        download_name_field = bulk_document.get("download_name", name_field)

        # Determine if we want to automatically extract ZIP files and upload
        # them as folders / documents:
        extract_zip = bulk_document.get("extract_zip", False)

        # Document names are limited in terms of allowed characters.
        # E.g. if you don't want any path elements and "/" dividers
        # in the document name you could use name_regex = r".*/" in the payload.
        document_name_additional_regex_list = [bulk_document.get("name_regex", r"")]

        # Fetch the nickname field from the payload (if it is specified):
        nickname_field = bulk_document.get("nickname")

        # Nicknames are very limited in terms of allowed characters.
        # For nicknames we need additional regexp as we need to
        # replace all non-alphanumeric, non-space characters with ""
        # We also preserve hyphens in the first step to replace
        # them below with underscores. This is important to avoid
        # that different spellings of names produce different nicknames.
        # We want spellings with spaces match spellings with hyphens.
        # For this the workspace names have a regexp "-| " in the payload.
        nickname_additional_regex_list = [r"[^\w\s-]"]

        total_rows = len(partition)

        # Process all rows in the partition that was given to this thread:
        for index, row in partition.iterrows():
            # Calculate percentage of completion:
            percent_complete = ((partition.index.get_loc(index) + 1) / total_rows) * 100

            self.logger.info(
                "Processing data row -> %s for bulk document creation (%.2f %% complete)...",
                str(index),
                percent_complete,
            )

            # Clear variables to ensure clean state for each row:
            parent_id = None
            document_id = None

            # Create a copy of the mutable operations list as we may
            # want to modify it per row:
            row_operations = list(operations)

            # Check if all data conditions to create the document are met
            conditions = bulk_document.get("conditions")
            if conditions:
                evaluated_condition = self.evaluate_conditions(
                    conditions=conditions,
                    row=row,
                )
                if not evaluated_condition:
                    self.logger.info(
                        "Condition for bulk document row -> %s not met. Skipping row for document creation...",
                        str(index),
                    )
                    result["skipped_counter"] += 1
                    continue

            # Check if all data conditions to create the document are met:
            if "create" in row_operations or "recreate" in row_operations:
                conditions_create = bulk_document.get("conditions_create")
                if conditions_create:
                    evaluated_conditions_create = self.evaluate_conditions(
                        conditions=conditions_create,
                        row=row,
                        replacements=replacements,
                    )
                    if not evaluated_conditions_create:
                        self.logger.info(
                            "Create condition for bulk document row -> %s not met. Excluding create operation for current row...",
                            str(index),
                        )
                        if "create" in row_operations:
                            row_operations.remove("create")
                        if "recreate" in row_operations:
                            row_operations.remove("recreate")
                elif (
                    "recreate" in row_operations
                ):  # we still create and recreate without conditions_create. But give a warning for 'recreate' without condition.
                    self.logger.warning(
                        "No create condition provided but 'recreate' operation requested. This will recreate all existing documents!",
                    )

            # Check if all data conditions to delete the document are met:
            if "delete" in row_operations:
                conditions_delete = bulk_document.get("conditions_delete")
                if conditions_delete:
                    evaluated_conditions_delete = self.evaluate_conditions(
                        conditions=conditions_delete,
                        row=row,
                        replacements=replacements,
                    )
                    if not evaluated_conditions_delete:
                        self.logger.info(
                            "Delete condition for bulk document row -> %s not met. Excluding delete operation for current row...",
                            str(index),
                        )
                        row_operations.remove("delete")
                else:  # without delete_conditions we don't delete!!
                    self.logger.warning(
                        "Delete operation requested for bulk documents but conditions for deletion are missing! (specify 'conditions_delete'!)",
                    )
                    row_operations.remove("delete")

            # Check if all data conditions to update the document are met:
            if "update" in row_operations:
                conditions_update = bulk_document.get("conditions_update")
                if conditions_update:
                    evaluated_conditions_update = self.evaluate_conditions(
                        conditions=conditions_update,
                        row=row,
                        replacements=replacements,
                    )
                    if not evaluated_conditions_update:
                        self.logger.info(
                            "Update condition for bulk document row -> %s not met. Excluding update operation for current row...",
                            str(index),
                        )
                        row_operations.remove("update")

            # Determine the document name:
            document_name = self.replace_bulk_placeholders(
                input_string=name_field,
                row=row,
                replacements=replacements,
                additional_regex_list=document_name_additional_regex_list,
            )
            # If we cannot get the document_name from the
            # name_field we try the alternative name field
            # (if specified in payload via "name_alt"):
            if not document_name and name_field_alt:
                self.logger.debug(
                    "Row -> %s does not have the data to resolve the placeholders in document name -> %s! Trying alternative name field -> %s...",
                    str(index),
                    name_field,
                    name_field_alt,
                )
                # The additional regex list will make sure
                # that any path or URL element is stripped from
                # the value in the row element:
                document_name = self.replace_bulk_placeholders(
                    input_string=name_field_alt,
                    row=row,
                    replacements=replacements,
                    additional_regex_list=document_name_additional_regex_list,
                )
            if not document_name:
                self.logger.error(
                    "Row -> %s does not have the data to resolve the placeholders in document name -> %s%s!",
                    str(index),
                    name_field,
                    (" nor in alternative name field -> " + name_field_alt if name_field_alt else ""),
                )
                result["skipped_counter"] += 1
                continue

            # Cleanse the document name (allowed characters, maximum length):
            document_name = OTCS.cleanse_item_name(document_name)

            download_name = ""
            if download_name_field:
                # The additional regex list will make sure
                # that any path or URL element is stripped from
                # the value in the row element:
                download_name = self.replace_bulk_placeholders(
                    input_string=download_name_field,
                    row=row,
                    replacements=replacements,
                    additional_regex_list=document_name_additional_regex_list,
                )
            if not download_name:
                self.logger.warning(
                    "Download name is empty or row -> %s does not have the data to resolve the placeholders in document download name -> '%s'. Using -> '%s' instead!",
                    str(index),
                    download_name_field,
                    document_name,
                )
                # in this case we use the document_name also as the download_name:
                download_name = document_name

            # This is an optimization. We check if the document was created
            # in a former run. This helps if the customizer is re-run:
            if document_name and document_name in existing_documents:
                self.logger.info(
                    "Document -> '%s' does already exist and has ID -> %s. Skipping...",
                    document_name,
                    existing_documents[document_name],
                )
                result["skipped_counter"] += 1
                continue

            # Determine the description field:
            description = (
                self.replace_bulk_placeholders(
                    input_string=description_field,
                    row=row,
                )
                if description_field
                else ""
            )

            # Determine the external creation date (if any):
            if external_create_date_field:
                external_create_date = self.replace_bulk_placeholders(
                    input_string=external_create_date_field,
                    row=row,
                )
            else:
                external_create_date = None

            # Determine the external modification date (if any):
            if external_modify_date_field:
                external_modify_date = self.replace_bulk_placeholders(
                    input_string=external_modify_date_field,
                    row=row,
                )
            else:
                external_modify_date = None

            # Determine the key field (if any):
            key = self.replace_bulk_placeholders(input_string=key_field, row=row) if key_field else None

            # check if workspace with this nickname does already exist.
            # we also store the nickname to assign it to the new workspace:
            if nickname_field:
                nickname = self.replace_bulk_placeholders(
                    input_string=nickname_field,
                    row=row,
                    replacements=replacements,
                    additional_regex_list=nickname_additional_regex_list,
                )
            else:
                nickname = None

            # Based on the evaluation of conditions_create, conditions_delete,
            # and conditions_update we could end up with an empty operations list.
            # In such cases we skip the further processing of this row:
            if not row_operations:
                self.logger.info(
                    "Skip document -> '%s' as row operations is empty (no create, update, delete or recreate). This may be because conditions_create, conditions_delete, and conditions_update have been evaluated to false for this row.",
                    document_name,
                )
                result["skipped_counter"] += 1
                continue

            self.logger.info(
                "Bulk process document -> '%s' using effective operations -> %s...",
                document_name,
                str(row_operations),
            )

            # We only need to download the files if we want more than just
            # the "delete" operation:
            if any(operation in ["create", "update", "recreate"] for operation in row_operations):
                file_name, mime_type = self.download_bulk_document(
                    bulk_document=bulk_document,
                    download_name=download_name,
                    row=row,
                    source_otcs=source_otcs,
                    result=result,
                )

                if file_name is None:
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue

                self.logger.debug("Download name -> '%s'", download_name)
                self.logger.debug("Document name -> '%s'", document_name)
                self.logger.debug("File name -> '%s'", file_name)
                self.logger.debug("Mime type -> '%s'", mime_type)
            else:
                # if we only do a delete operation then there's no need for
                # processing local files for upload:
                file_name = None
                mime_type = None

            # Now we traverse a list of (multiple) workspaces
            # the document should be uploaded to:
            success = True
            workspaces = bulk_document.get("workspaces", [])
            for workspace in workspaces:
                # success will only be false if a config problem (failure)
                # and not just a data problem (skipped) has occured:
                parent_id, success = self.get_bulk_document_location(
                    workspace=workspace,
                    row=row,
                    index=index,
                    replacements=replacements,
                )

                if parent_id is None:
                    continue

                #
                # Create the document in the target folder specified by parent_id:
                #

                # Check if the document does already exist.
                # First we try to look up the document by a unique
                # key that does not change over time.
                # For this we expect a "key" value to be defined for the
                # bulkDocument and one of the category / attribute item
                # to be marked with "is_key" = True. If we don't find the
                # document via key we try via parent and name.
                self.logger.info(
                    "Check if document -> '%s' is already in target folder with ID -> %s%s...",
                    document_name,
                    parent_id,
                    " (using key -> {})".format(key) if key is not None else "",
                )
                # Initialize variables - this is important!
                document_old_name = None
                document_id = None
                handle_name_clash = False
                document_modify_date = None

                # We have 4 cases to differentiate:

                # 1. key given + key found = name irrelevant, item exist
                # 2. key given + key not found = if name exist it is a name clash
                # 3. no key given + name found = item exist
                # 4. no key given + name not found = item does not exist

                # We are NOT trying to compensate for a failed key lookup with a name lookup!
                # Names are only relevant if no key is in payload!

                if key:
                    key_attribute = next(
                        (cat_attr for cat_attr in categories if cat_attr.get("is_key", False) is True),
                        None,
                    )
                    if key_attribute:
                        cat_name = key_attribute.get("name", None)
                        att_name = key_attribute.get("attribute", None)
                        set_name = key_attribute.get("set", None)
                        # Try to find the node that has the given key attribute value:
                        response = self._otcs_frontend.lookup_nodes(
                            parent_node_id=parent_id,
                            category=cat_name,
                            attribute=att_name,
                            attribute_set=set_name,
                            value=key,
                        )
                        document_id = self._otcs_frontend.get_result_value(
                            response=response,
                            key="id",
                        )
                        if document_id:
                            # Case 1: key given + key found = name irrelevant, item exist
                            self.logger.info(
                                "Found existing document with the key value -> '%s' in category -> '%s' and attribute -> '%s' in folder with ID -> %s.",
                                key,
                                cat_name,
                                att_name,
                                parent_id,
                            )
                            document_old_name = self._otcs_frontend.get_result_value(
                                response=response,
                                key="name",
                            )
                            if (
                                document_old_name != document_name
                                and document_old_name != document_name + " (" + key + ")"
                            ):
                                self.logger.info(
                                    "Existing document with ID -> %s has the old name -> '%s' which is different from the new name -> '%s.'",
                                    document_id,
                                    document_old_name,
                                    document_name,
                                )
                            else:
                                # if there was a name clash before and got resolved
                                # then we need to stick to the resolved name also for updates:
                                if document_old_name == document_name + " (" + key + ")":
                                    self.logger.info(
                                        "Document name conflict was resolved before. Changing document name to -> '%s' to match former resolution.",
                                        document_old_name,
                                    )
                                    document_name = document_old_name
                                document_old_name = None
                            # We get the modify date of the existing document.
                            document_modify_date = self._otcs_frontend.get_result_value(
                                response=response,
                                key="modify_date",
                            )
                        else:
                            # Case 2: key given + key not found = if name exist it is a name clash
                            self.logger.info(
                                "Couldn't find existing document with the key value -> '%s' in category -> '%s' and attribute -> '%s' in folder with ID -> %s.",
                                key,
                                cat_name,
                                att_name,
                                parent_id,
                            )
                            # Keep track that we need to handle a name clash as based on the key
                            # the document should not exist.
                            handle_name_clash = True
                    else:
                        self.logger.error(
                            "Document has a key -> '%s' defined but none of the category attributes is marked as a key attribute ('is_key' is missing)!",
                            key,
                        )
                        success = False
                        continue
                # end if key
                else:
                    # If we haven't a key we try by parent + name
                    response = self._otcs_frontend.get_node_by_parent_and_name(
                        name=document_name,
                        parent_id=parent_id,
                    )
                    document_id = self._otcs_frontend.get_result_value(
                        response=response,
                        key="id",
                    )
                    if document_id:
                        # Case 3: no key given + name found = item exist
                        self.logger.info(
                            "Found existing document -> '%s' (%s) in parent with ID -> %s.",
                            document_name,
                            document_id,
                            parent_id,
                        )
                        # We get the modify date of the existing document.
                        document_modify_date = self._otcs_frontend.get_result_value(
                            response=response,
                            key="modify_date",
                        )
                    elif not extract_zip or mime_type != "application/zip":
                        # Case 4: no key given + name not found = item does not exist
                        self.logger.info(
                            "Cannot find document -> '%s' in parent with ID -> %s.",
                            document_name,
                            parent_id,
                        )
                    else:
                        # Edge case: no key given + name not found but it is a zip file that got extracted.
                        # So we don't really know if it exists yet.
                        self.logger.info(
                            "File -> '%s' is a zip file that potentially has been extracted into parent with ID -> %s before. Existence of extracted files will be determined later.",
                            document_name,
                            parent_id,
                        )

                # Check if we want to recreate an existing document
                # then we handle the "delete" part of "recreate" first:
                if document_id and "recreate" in row_operations:
                    response = self._otcs_frontend.delete_node(
                        node_id=document_id,
                        purge=True,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to recreate existing document -> '%s' (%s) under parent ID -> %s! Delete failed.",
                            (document_name if document_old_name is None else document_old_name),
                            document_id,
                            parent_id,
                        )
                        success = False
                        continue
                    self.logger.info(
                        "Deleted existing document -> '%s' (%s) as part of the recreate operation.",
                        (document_name if document_old_name is None else document_old_name),
                        document_id,
                    )
                    result["delete_counter"] += 1
                    document_id = None

                # Check if document does not exists - then we create a new document
                # if this is requested ("create" value in operations list in payload)
                if not document_id and ("create" in row_operations or "recreate" in row_operations):
                    # The document does not exist in Content Server - so we
                    # upload it now...

                    # for Case 2 (we looked up the document by key but could have name collisions)
                    # we need to see if we have name collisions:
                    if handle_name_clash:
                        response = self._otcs_frontend.check_node_name(
                            parent_id=parent_id,
                            node_name=document_name,
                        )
                        # result is a list of names that clash - if it is empty we have no clash
                        if response.get("results"):
                            # We need to handle a race condition here: it could be that
                            # the document has been created by another thread. This is because
                            # the key could be in multiple rows of the data frame depending how the data loader works.
                            # We can also not assume that this can be resolved by making the key unique
                            # in the data source as it could be that a data set with the same key should
                            # go to multiple workspaces that then are folded into one by synonym resolution.
                            conflicting_node_id = response["results"][0].get("id")
                            conflicting_key = self._otcs_frontend.get_category_value_by_name(
                                node_id=conflicting_node_id,
                                category_name=cat_name,
                                attribute_name=att_name,
                                set_name=set_name,
                            )
                            if key == conflicting_key:
                                # We have a race condition as the two documents don't really clash but are identical.
                                # Just skip uploading the same document once more.
                                self.logger.warning(
                                    "Detected a race condition in name clash handling! Skipping document upload!",
                                )
                                continue

                            # We add the suffix with the key which should be unique:
                            self.logger.warning(
                                "Document -> '%s' does already exist in workspace folder with ID -> %s and we need to handle the name clash and use name -> '%s'",
                                document_name,
                                parent_id,
                                document_name + " (" + key + ")",
                            )
                            document_name = document_name + " (" + key + ")"

                    # If category data is in payload we substitute
                    # the values with data from the current data row:
                    if categories:
                        # Make sure the threads are not changing data structures that
                        # are shared between threads. categories is a list of dicts.
                        # list and dicts are "mutable" data structures in Python!
                        worker_categories = self.process_bulk_categories(
                            row=row,
                            index=index,
                            categories=categories,
                            replacements=replacements,
                        )
                        document_category_data = self.prepare_category_data(
                            categories_payload=worker_categories,
                            source_node_id=parent_id,
                        )
                    # end if categories
                    else:
                        document_category_data = {}

                    response = self._otcs_frontend.upload_file_to_parent(
                        file_url=file_name,
                        file_name=document_name,
                        mime_type=mime_type,
                        parent_id=int(parent_id),
                        category_data=document_category_data,
                        description=description,
                        external_create_date=external_create_date,
                        external_modify_date=external_modify_date,
                        extract_zip=extract_zip,
                        replace_existing="update" in row_operations and "version" in update_operations,
                        show_error=False,
                    )
                    document_id = self._otcs_frontend.get_result_value(
                        response=response,
                        key="id",
                    )
                    if not document_id:
                        # We may have a race condition here. Double check the document does not exist
                        # before issuing an error:
                        response = self._otcs_frontend.get_node_by_parent_and_name(
                            parent_id=int(parent_id),
                            name=document_name,
                        )
                        document_id = self._otcs_frontend.get_result_value(
                            response=response,
                            key="id",
                        )
                    if not document_id:
                        self.logger.error(
                            "Cannot create document -> '%s' (download name -> '%s') in folder/workspace with ID -> %s!",
                            document_name,
                            download_name,
                            parent_id,
                        )
                        success = False
                        continue
                    self.logger.info(
                        "Created document -> '%s' (ID -> %s, file -> '%s', mime type -> '%s'%s%s) in parent with ID -> %s.",
                        document_name,
                        document_id,
                        file_name,
                        mime_type,
                        ", description -> {}".format(description) if description else "",
                        ", size -> {}".format(os.path.getsize(file_name))
                        if os.path.exists(file_name)
                        else "",  # this can happen for files in zip files that have been cleaned up already
                        parent_id,
                    )
                    result["create_counter"] += 1

                    # Check if metadata or images embeddings need to be updated for the document:
                    aviator_metadata = bulk_document.get("aviator_metadata", False)
                    aviator_images = bulk_document.get("aviator_images", False)
                    aviator_remove_existing = bulk_document.get("aviator_remove_existing", False)
                    if aviator_metadata or aviator_images or aviator_remove_existing:
                        self.logger.info(
                            "%s Content Aviator %s%s%s embedding via FEME for document or image -> '%s' (%s)...",
                            "Trigger" if not aviator_remove_existing else "Remove",
                            "metadata" if aviator_metadata else "",
                            " and " if aviator_metadata and aviator_images else "",
                            "image" if aviator_images else "",
                            document_name,
                            document_id,
                        )

                        self._otcs_frontend.aviator_embed_metadata(
                            node_id=document_id,
                            workspace_metadata=False,
                            document_metadata=aviator_metadata,
                            images=aviator_images,
                            wait_for_completion=False,
                            remove_existing=aviator_remove_existing,
                        )

                # end if not workspace_id and "create" in row_operations

                # If updates are requested we update the existing document with
                # a new document version and with fresh metadata from the payload.
                # Additionally we check the external modify date to support
                # incremental load for content that has really changed.
                # In addition we check that "delete" is not requested as otherwise it will
                # never go in elif "delete" ... below (and it does not make sense to update a document
                # that is deleted in the next step...)

                elif (
                    document_id
                    and "update" in row_operations
                    and "delete" not in row_operations  # note the NOT !
                    and OTCS.date_is_newer(
                        date_old=document_modify_date,
                        date_new=external_modify_date,
                    )
                ):
                    # If category data is in payload we substitute
                    # the values with data from the current data row.
                    # This is only done if "categories" update is not
                    # excluded from the update_operations:
                    if categories and "categories" in update_operations:
                        # Make sure the threads are not changing data structures that
                        # are shared between threads. categories is a list of dicts.
                        # list and dicts are "mutable" data structures in Python!
                        worker_categories = self.process_bulk_categories(
                            row=row,
                            index=index,
                            categories=categories,
                            replacements=replacements,
                        )
                        # document_category_data = self.prepare_item_create_form(
                        #     parent_id=parent_id,
                        #     categories=worker_categories,
                        #     subtype=self._otcs_frontend.ITEM_TYPE_DOCUMENT,
                        # )
                        # Transform the payload structure into the format
                        # the OTCS REST API requires:
                        document_category_data = self.prepare_category_data(
                            categories_payload=worker_categories,
                            source_node_id=document_id,
                            source_is_document=True,
                        )
                        if not document_category_data:
                            self.logger.error(
                                "Failed to prepare the updated category data for document -> '%s' (%s)!",
                                document_name,
                                str(document_id),
                            )
                            success = False
                            continue  # for index, row in partition.iterrows()
                    # end if categories
                    else:
                        document_category_data = {}

                    self.logger.info(
                        "Update existing document -> '%s' (%s) with operations -> %s...",
                        document_old_name if document_old_name else document_name,
                        document_id,
                        str(update_operations),
                    )
                    # check if adding a version is requested:
                    if "version" in update_operations:
                        response = self._otcs_frontend.add_document_version(
                            node_id=document_id,
                            file_url=file_name,
                            file_name=document_name,
                            mime_type=mime_type,
                            description=description,
                        )
                        if not response:
                            self.logger.error(
                                "Failed to add new version to existing document -> '%s' (%s)!",
                                (document_old_name if document_old_name else document_name),
                                document_id,
                            )
                            success = False
                            continue
                    if "purge" in update_operations:
                        max_versions = bulk_document.get("max_versions", 1)
                        response = self._otcs_frontend.purge_document_versions(
                            node_id=document_id, versions_to_keep=max_versions
                        )
                        if not response:
                            self.logger.error(
                                "Failed to purge versions of document -> '%s' (%s) to %d version%s!",
                                (document_old_name if document_old_name else document_name),
                                document_id,
                                max_versions,
                                "s" if max_versions > 1 else "",
                            )
                            success = False
                            continue
                    response = self._otcs_frontend.update_item(
                        node_id=document_id,
                        parent_id=None,  # None = do not move item
                        item_name=(document_name if "name" in update_operations else None),
                        item_description=(description if "description" in update_operations else None),
                        category_data=(document_category_data if "categories" in update_operations else None),
                        external_create_date=external_create_date,
                        external_modify_date=external_modify_date,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to update metadata of existing document -> '%s' (%s) with metadata -> %s!",
                            (document_old_name if document_old_name else document_name),
                            document_id,
                            str(document_category_data),
                        )
                        success = False
                        continue
                    self.logger.info(
                        "Updated existing document -> '%s' (ID -> %s, file -> '%s', mime type -> '%s', description -> '%s'%s).",
                        document_name,
                        document_id,
                        file_name,
                        mime_type,
                        description,
                        ", size -> {}".format(os.path.getsize(file_name))
                        if os.path.exists(file_name)
                        else "",  # this can happen for files in zip files that have been cleaned up already
                    )
                    result["update_counter"] += 1

                    # Check if metadata or image embeddings need to be updated for the document:
                    aviator_metadata = bulk_document.get("aviator_metadata", False)
                    aviator_images = bulk_document.get("aviator_images", False)
                    aviator_remove_existing = bulk_document.get("aviator_remove_existing", False)
                    if aviator_metadata or aviator_images or aviator_remove_existing:
                        self.logger.info(
                            "%s Content Aviator %s%s%s embedding via FEME for document or image -> '%s' (%s)...",
                            "Update" if not aviator_remove_existing else "Remove",
                            "metadata" if aviator_metadata else "",
                            " and " if aviator_metadata and aviator_images else "",
                            "image" if aviator_images else "",
                            document_name,
                            document_id,
                        )

                        self._otcs_frontend.aviator_embed_metadata(
                            node_id=document_id,
                            workspace_metadata=False,
                            document_metadata=aviator_metadata,
                            images=aviator_images,
                            wait_for_completion=False,
                            remove_existing=aviator_remove_existing,
                        )
                # end if workspace_id and "update" in row_operations
                elif document_id and "delete" in row_operations:
                    # We delete with immediate purging to keep recycle bin clean
                    # and to not run into issues with nicknames used in deleted items:
                    response = self._otcs_frontend.delete_node(
                        node_id=document_id,
                        purge=True,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to bulk delete existing document -> '%s' (%s)!",
                            (document_old_name if document_old_name else document_name),
                            document_id,
                        )
                        success = False
                        continue
                    self.logger.info(
                        "Successfully deleted existing document -> '%s' (%s).",
                        document_old_name if document_old_name else document_name,
                        document_id,
                    )
                    result["delete_counter"] += 1
                    document_id = None
                # end elif document_id and "delete" in row_operations

                # this is the plain old "if it does exist we just skip it" case:
                elif document_id:
                    self.logger.info(
                        "Skipped existing document -> '%s' (%s).",
                        document_old_name if document_old_name else document_name,
                        document_id,
                    )
                # this is the case where we just want to operate on existing documents (update or delete)
                # but they do not exist:
                elif not document_id and ("update" in row_operations or "delete" in row_operations):
                    result["skipped_counter"] += 1
                    self.logger.info(
                        "Skipped update/delete of non-existing document -> '%s'.",
                        document_old_name if document_old_name else document_name,
                    )

                # The following code is executed for all operations
                # (create, recreate, update, delete):

                # Check if we want to set / update the nickname of the document.
                # Precondition is we have determined a nickname, we have the document ID
                # and the update of the nickname is either required (create, recreate)
                # or requested (update_operations include "nickname"):
                if (
                    nickname
                    and document_id
                    and (
                        "create" in row_operations
                        or "recreate" in row_operations
                        or ("update" in row_operations and "nickname" in update_operations)
                    )
                ):
                    response = self._otcs_frontend.set_node_nickname(
                        node_id=document_id,
                        nickname=nickname,
                        show_error=True,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to assign nickname -> '%s' to document -> '%s'!",
                            nickname,
                            document_name,
                        )
                if document_id is not None:
                    # Record the document name and ID to allow to read it from failure file
                    # and speedup the process.
                    result["documents"][document_name] = document_id

            # end for workspaces

            # We want the success, failure and skip counter
            # to consider only complete data frame rows. In
            # case of bulk documents we can potentially create
            # update or delete more than 1 document per row.
            # So we use the "success" variable to accumulate
            # this for all documents per row:
            if not success:
                result["success"] = False
                result["failure_counter"] += 1
            elif document_name not in result["documents"]:
                self.logger.info(
                    "Bulk document -> '%s' was not uploaded to any workspace.",
                    document_name,
                )
                result["skipped_counter"] += 1
            else:
                result["success_counter"] += 1

            # Make sure no temp documents are piling up except
            # we want it (e.g. if using cloud document storage):
            if file_name and os.path.exists(file_name) and bulk_document.get("delete_download", True):
                os.remove(file_name)
        # end for index, row in partition.iterrows()

        self.logger.info("End working...")

        results.append(result)

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_items")
    def process_bulk_items(self, section_name: str = "bulkItems") -> bool:
        """Process bulkItems in payload and bulk create them in Content Server (multi-threaded).

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise.

        """

        if not self._bulk_items:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        # For efficient idem-potent operation we may want to see which workspaces
        # have already been processed before:
        if self.check_status_file(
            payload_section_name=section_name,
            payload_specific=True,
            prefix="failure_",
        ):
            self.logger.info(
                "Found failure file. Trying to reprocess just the failed ones...",
            )
            # Read payload from where we left it last time
            self._bulk_items = self.get_status_file(
                payload_section_name=section_name,
                prefix="failure_",
            )
            if not self._bulk_items:
                self.logger.error(
                    "Cannot load existing bulk items failure file. Bailing out!",
                )
                return False

        success: bool = True

        for bulk_item in self._bulk_items:
            # Check if element has been disabled in payload (enabled = false).
            # In this case we skip the element:
            enabled = bulk_item.get("enabled", True)
            if not enabled:
                self.logger.info("Payload for bulk items is disabled. Skipping...")
                continue

            # The payload element must have a "data_source" key:
            data_source_name = bulk_item.get("data_source", None)
            if not data_source_name:
                self.logger.error("Missing data source name in bulk items!")
                success = False
                continue

            copy_data_source = bulk_item.get("copy_data_source", False)
            force_reload = bulk_item.get("force_reload", True)

            # Load and prepare the data source for the bulk processing:
            if copy_data_source:
                self.logger.info(
                    "Take a copy of data source -> %s to avoid side-effects for repeative usage of the data source...",
                    data_source_name,
                )
                data = Data(
                    self.process_bulk_datasource(
                        data_source_name=data_source_name,
                        force_reload=force_reload,
                    ),
                    logger=self.logger,
                )
            else:
                self.logger.info(
                    "Use original data source -> '%s' and not do a copy.",
                    bulk_item["data_source"],
                )
                data = self.process_bulk_datasource(
                    data_source_name=data_source_name,
                    force_reload=force_reload,
                )
            if (
                data is None
                or data.get_data_frame() is None  # the 2nd check is important for the "copy_data_source" case!
            ):  # important to use "is None" here!
                self.logger.error(
                    "Failed to load data source -> '%s' for bulk items!",
                    data_source_name,
                )
                success = False
                continue

            # Check if fields with list substructures should be exploded.
            # We may want to do this outside the bulkDatasource to only
            # explode for bulkItems and not for bulkWorkspaces or
            # bulkWorkspaceRelationships:
            explosions = bulk_item.get("explosions", [])
            for explosion in explosions:
                # explode field can be a string or a list
                # exploding multiple fields at once avoids
                # combinatorial explosions - this is VERY
                # different from exploding columns one after the other!
                if "explode_field" not in explosion and "explode_fields" not in explosion:
                    self.logger.error("Missing explosion field(s)!")
                    continue
                # we want to be backwards compatible...
                explode_fields = (
                    explosion["explode_field"] if "explode_field" in explosion else explosion["explode_fields"]
                )
                flatten_fields = explosion.get("flatten_fields", [])
                split_string_to_list = explosion.get("split_string_to_list", False)
                list_splitter = explosion.get("list_splitter", None)
                self.logger.info(
                    "Starting explosion of bulk items by field(s) -> %s (type -> %s). Size of data frame before explosion -> %s.",
                    explode_fields,
                    str(type(explode_fields)),
                    str(len(data)),
                )
                data.explode_and_flatten(
                    explode_fields=explode_fields,
                    flatten_fields=flatten_fields,
                    make_unique=False,
                    split_string_to_list=split_string_to_list,
                    separator=list_splitter,
                    reset_index=True,
                )
                self.logger.info(
                    "Size of data frame after explosion -> %s.",
                    str(len(data)),
                )
            # end for explosion in explosions

            # Keep only selected rows if filters are specified in bulkDocuments.
            # We have this _after_ "explosions" to allow access to subfields as well.
            # We have this _before_ "sorting" and "deduplication" as we may keep the wrong
            # rows otherwise (unique / deduplication always keeps the first matching row).
            # We may want to do this outside the bulkDatasource to only
            # filter for bulkDocuments and not for bulkWorkspaces or
            # bulkWorkspaceRelationships:
            filters = bulk_item.get("filters", [])
            if filters:
                data.filter(conditions=filters)

            # Sort the data frame if "sort" specified in payload. We may want to do this to have a
            # higher chance that rows with workspace names that may collapse into
            # one name are put into the same partition. This can avoid race conditions
            # between different Python threads.
            # We do this before deduplication (unique) below as sorting has an influence
            # on which duplicates are kept.
            sort_fields = bulk_item.get("sort", [])
            if sort_fields:
                self.logger.info(
                    "Start sorting of bulk items data frame based on fields (columns) -> %s...",
                    str(sort_fields),
                )
                data.sort(sort_fields=sort_fields, inplace=True)
                self.logger.info(
                    "Sorting of bulk items data frame based on fields (columns) -> %s completed.",
                    str(sort_fields),
                )

            # Check if duplicate rows for given fields should be removed. It is
            # important to do this after sorting as Pandas always keep the first occurance,
            # so ordering plays an important role in deduplication!
            unique_fields = bulk_item.get("unique", [])
            if unique_fields:
                self.logger.info(
                    "Starting deduplication of data frame for bulk items with unique fields -> %s. Size of data frame before deduplication -> %s.",
                    str(unique_fields),
                    str(len(data)),
                )
                data.deduplicate(unique_fields=unique_fields, inplace=True)
                self.logger.info(
                    "Size of data frame after deduplication -> %s.",
                    str(len(data)),
                )

            # Read name field from payload:
            name_field = bulk_item.get("name", None)
            if not name_field:
                self.logger.error(
                    "Bulk item needs a name field! Skipping to next bulk item...",
                )
                success = False
                continue

            self._log_header_callback(
                text="Process bulk items -> '{}' from data source -> '{}'".format(
                    name_field,
                    data_source_name,
                ),
                char="-",
            )

            # Read optional description field from payload:
            description_field = bulk_item.get("description", None)

            # Read the optional categories payload:
            categories = bulk_item.get("categories", None)
            if not categories:
                self.logger.info(
                    "Bulk item payload has no category data! Will leave category attributes empty...",
                )

            # Should existing items be updated? False (= no) is the default.
            operations = bulk_item.get("operations", ["create"])

            self.logger.info(
                "Bulk create items (name field -> %s. Operations -> %s.)",
                name_field,
                str(operations),
            )

            # See if bulkWorkspace definition has a specific thread number
            # otherwise it is read from a global environment variable
            bulk_thread_number = int(
                bulk_item.get("thread_number", BULK_THREAD_NUMBER),
            )

            partitions = data.partitionate(bulk_thread_number)

            # Create a list to hold the threads
            threads = []
            results = []

            # Create and start a thread for each partition
            for index, partition in enumerate(partitions, start=1):
                # start a thread executing the process_bulk_items_worker() method below:
                thread = threading.Thread(
                    name=f"{section_name}_{index:02}",
                    target=self.thread_wrapper,
                    args=(
                        self.process_bulk_items_worker,
                        bulk_item,
                        partition,
                        name_field,
                        description_field,
                        categories,
                        operations,
                        results,
                    ),
                )
                self.logger.info("Starting thread -> '%s'...", str(thread.name))
                thread.start()
                threads.append(thread)
                # Avoid that all threads start at the exact same time with
                # potentially expired cookies that cases race conditions:
                time.sleep(1)
            # end for index, partition in enumerate(partitions, start=1)

            # Wait for all threads to complete
            for thread in threads:
                self.logger.info(
                    "Waiting for thread -> '%s' to complete...",
                    str(thread.name),
                )
                thread.join()
                self.logger.info("Thread -> '%s' has completed.", str(thread.name))

            if "items" not in bulk_item:
                bulk_item["items"] = {}

            # Print statistics for each thread. In addition,
            # check if all threads have completed without error / failure.
            # If there's a single failure in on of the thread results we
            # set 'success' variable to False.
            results.sort(key=lambda x: x["thread_name"])
            for result in results:
                self.logger.info(
                    "Thread -> '%s' completed with overall status '%s'.",
                    str(result["thread_name"]),
                    "successful" if result["success"] else "failed",
                )
                self.logger.info(
                    "Thread -> '%s' processed %s data rows with %s successful, %s failed, and %s skipped.",
                    str(result["thread_name"]),
                    str(
                        result["success_counter"] + result["failure_counter"] + result["skipped_counter"],
                    ),
                    str(result["success_counter"]),
                    str(result["failure_counter"]),
                    str(result["skipped_counter"]),
                )
                self.logger.info(
                    "Thread -> '%s' created %s items, updated %s items, and deleted %s items.",
                    str(result["thread_name"]),
                    str(result["create_counter"]),
                    str(result["update_counter"]),
                    str(result["delete_counter"]),
                )

                if not result["success"]:
                    success = False
                # Record all generated items. This should allow us
                # to restart in case of failures and avoid trying to
                # create items that have been successfully created before.
                bulk_item["items"].update(result["items"])
            # end for result in results
            self._log_header_callback(
                text="Completed processing of bulk items -> '{}' using data source -> '{}'".format(
                    name_field,
                    data_source_name,
                ),
                char="-",
            )

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._bulk_items,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_items_worker")
    def process_bulk_items_worker(
        self,
        bulk_item: dict,
        partition: pd.DataFrame,
        name_field: str,
        description_field: str,
        categories: list | None = None,
        operations: list | None = None,
        results: list | None = None,
    ) -> None:
        """Thread worker to create items in bulk.

        Each worker thread gets a partition of the rows that include
        the data required for the item creation.

        Args:
            bulk_item (dict):
                The bulkItem payload element.
            partition (pd.DataFrame):
                Data partition with rows to process by this thread.
            name_field (str):
                The payload field where the item name is stored.
            description_field (str):
                The payload field where the item description is stored.
            categories (list, optional):
                A list of category dictionaries.
            operations (list, optional):
                A list of operations that should be applyed on workspaces.
                Possible values: "create", "update", "delete", "recreate".
            results (list, optional):
                A mutable list of thread results.

        """

        thread_id = threading.get_ident()
        thread_name = threading.current_thread().name

        self.logger.info(
            "Start working on data frame partition of size -> %s to bulk create items...",
            str(len(partition)),
        )

        # Avoid linter warnings - so make parameter default None while we
        # actually want ["create"] to be the default:
        if operations is None:
            operations = ["create"]

        result = {}
        result["thread_id"] = thread_id
        result["thread_name"] = thread_name
        result["success_counter"] = 0
        result["failure_counter"] = 0
        result["skipped_counter"] = 0
        result["create_counter"] = 0
        result["update_counter"] = 0
        result["delete_counter"] = 0
        result["items"] = {}
        result["success"] = True

        # Check if items have been processed before, e.i. testing
        # if a "items" key exists and if it is pointing to a non-empty list:
        # Additionally we check that workspace updates are not enforced:
        if (
            bulk_item.get("items")
            and "update" not in operations
            and "delete" not in operations
            and "recreate" not in operations
        ):
            existing_items = bulk_item["items"]
            self.logger.info(
                "Found %s already processed items. Try to complete the job...",
                str(len(existing_items)),
            )
        else:
            existing_items = {}

        # See if external creation and modification dates are in the payload data:
        external_modify_date_field = bulk_item.get("external_modify_date")
        external_create_date_field = bulk_item.get("external_create_date")

        # See if we have a key field to uniquely identify an existing item:
        key_field = bulk_item.get("key")

        # Get dictionary of replacements for bulk item creations
        # this we will be used of all places data is read from the
        # data frame. Each dictionary item has the field name as the
        # dictionary key and a list of regular expressions as dictionary value
        replacements = bulk_item.get("replacements")

        # In case the name cannot be resolved we allow to
        # specify an alternative name field in the payload.
        name_field_alt = bulk_item.get("name_alt")

        # Determine the type of the item (OTCS subtype ID):
        type_field = bulk_item.get("type", self._otcs_frontend.ITEM_TYPE_URL)

        # In case the type is URL, determine the URL field:
        url_field = bulk_item.get("url")

        # In case the type is Shortcut, determine the origin field:
        original_nickname_field = bulk_item.get("original_nickname")
        original_path = bulk_item.get("original_path")

        # Document names are limited in terms of allowed characters.
        # E.g. if you don't want any path elements and "/" dividers
        # in the item name you could use name_regex = r".*/" in the payload.
        item_name_additional_regex_list = [bulk_item.get("name_regex", r"")]

        # Fetch the nickname field from the payload (if it is specified):
        nickname_field = bulk_item.get("nickname")

        # Nicknames are very limited in terms of allowed characters.
        # For nicknames we need additional regexp as we need to
        # replace all non-alphanumeric, non-space characters with ""
        # We also preserve hyphens in the first step to replace
        # them below with underscores. This is important to avoid
        # that different spellings of names produce different nicknames.
        # We want spellings with spaces match spellings with hyphens.
        # For this the workspace names have a regexp "-| " in the payload.
        nickname_additional_regex_list = [r"[^\w\s-]"]

        total_rows = len(partition)

        # Process all rows in the partition that was given to this thread:
        for index, row in partition.iterrows():
            # Calculate percentage of completion:
            percent_complete = ((partition.index.get_loc(index) + 1) / total_rows) * 100

            self.logger.info(
                "Processing data row -> %s for bulk item creation (%.2f %% complete)...",
                str(index),
                percent_complete,
            )

            # Clear variables to ensure clean state for each row:
            parent_id = None
            item_id = None

            # Create a copy of the mutable operations list as we may
            # want to modify it per row:
            row_operations = list(operations)

            # Check if all data conditions to create the item are met
            conditions = bulk_item.get("conditions")
            if conditions:
                evaluated_condition = self.evaluate_conditions(
                    conditions=conditions,
                    row=row,
                )
                if not evaluated_condition:
                    self.logger.info(
                        "Condition for bulk item row -> %s not met. Skipping row for item creation...",
                        str(index),
                    )
                    result["skipped_counter"] += 1
                    continue

            # Check if all data conditions to create the item are met:
            if "create" in row_operations or "recreate" in row_operations:
                conditions_create = bulk_item.get("conditions_create")
                if conditions_create:
                    evaluated_conditions_create = self.evaluate_conditions(
                        conditions=conditions_create,
                        row=row,
                        replacements=replacements,
                    )
                    if not evaluated_conditions_create:
                        self.logger.info(
                            "Create condition for bulk item row -> %s not met. Excluding create operation for current row...",
                            str(index),
                        )
                        if "create" in row_operations:
                            row_operations.remove("create")
                        if "recreate" in row_operations:
                            row_operations.remove("recreate")
                elif (
                    "recreate" in row_operations
                ):  # we still create and recreate without conditions_create. But give a warning for 'recreate' without condition.
                    self.logger.warning(
                        "No create condition provided but 'recreate' operation requested. This will recreate all existing items!",
                    )

            # Check if all data conditions to delete the item are met:
            if "delete" in row_operations:
                conditions_delete = bulk_item.get("conditions_delete")
                if conditions_delete:
                    evaluated_conditions_delete = self.evaluate_conditions(
                        conditions=conditions_delete,
                        row=row,
                        replacements=replacements,
                    )
                    if not evaluated_conditions_delete:
                        self.logger.info(
                            "Delete condition for bulk item row -> %s not met. Excluding delete operation for current row...",
                            str(index),
                        )
                        row_operations.remove("delete")
                else:  # without delete_conditions we don't delete!!
                    self.logger.warning(
                        "Delete operation requested for bulk items but conditions for deletion are missing! (specify 'conditions_delete'!)",
                    )
                    row_operations.remove("delete")

            # Check if all data conditions to update the item are met:
            if "update" in row_operations:
                conditions_update = bulk_item.get("conditions_update")
                if conditions_update:
                    evaluated_conditions_update = self.evaluate_conditions(
                        conditions=conditions_update,
                        row=row,
                        replacements=replacements,
                    )
                    if not evaluated_conditions_update:
                        self.logger.info(
                            "Update condition for bulk item row -> %s not met. Excluding update operation for current row...",
                            str(index),
                        )
                        row_operations.remove("update")

            # Determine the item name:
            item_name = self.replace_bulk_placeholders(
                input_string=name_field,
                row=row,
                replacements=replacements,
                additional_regex_list=item_name_additional_regex_list,
            )
            # If we cannot get the item_name from the
            # name_field we try the alternative name field
            # (if specified in payload via "name_alt"):
            if not item_name and name_field_alt:
                self.logger.debug(
                    "Row -> %s does not have the data to resolve the placeholders in item name -> %s! Trying alternative name field -> %s...",
                    str(index),
                    name_field,
                    name_field_alt,
                )
                # The additional regex list will make sure
                # that any path or URL element is stripped from
                # the value in the row element:
                item_name = self.replace_bulk_placeholders(
                    input_string=name_field_alt,
                    row=row,
                    replacements=replacements,
                    additional_regex_list=item_name_additional_regex_list,
                )
            if not item_name:
                self.logger.error(
                    "Row -> %s does not have the data to resolve the placeholders in item name -> %s%s!",
                    str(index),
                    name_field,
                    (" nor in alternative name field -> " + name_field_alt if name_field_alt else ""),
                )
                result["skipped_counter"] += 1
                continue

            # Cleanse the item name (allowed characters, maximum length):
            item_name = OTCS.cleanse_item_name(item_name)

            # This is an optimization. We check if the item was created
            # in a former run. This helps if the customizer is re-run:
            if item_name and item_name in existing_items:
                self.logger.info(
                    "Item -> '%s' does already exist and has ID -> %s. Skipping...",
                    item_name,
                    existing_items[item_name],
                )
                result["skipped_counter"] += 1
                continue

            # Determine the description field:
            description = (
                self.replace_bulk_placeholders(
                    input_string=description_field,
                    row=row,
                )
                if description_field
                else ""
            )

            # Determine the item type:
            item_type = self.replace_bulk_placeholders(
                input_string=type_field,
                row=row,
            )

            # Determine the item type:
            item_url = self.replace_bulk_placeholders(
                input_string=url_field,
                row=row,
            )

            # Determine the item origin:
            original_nickname = self.replace_bulk_placeholders(
                input_string=original_nickname_field,
                row=row,
            )

            # Determine the item origin:
            self.replace_bulk_placeholders_list(
                input_list=original_path,
                row=row,
            )

            item_original_node = None
            # First try to get the original item for the shortcut via the nickname:
            if original_nickname:
                item_original_node = self._otcs_frontend.get_node_from_nickname(nickname=original_nickname)
            # FIf that didn't resolve the original node we try a path if provided in the payload:
            if not item_original_node and original_path:
                item_original_node = self._otcs_frontend.get_node_by_volume_and_path(
                    volume_type=self._otcs_frontend.VOLUME_TYPE_ENTERPRISE_WORKSPACE,
                    path=original_path,
                )
            item_original_node_id = self._otcs_frontend.get_result_value(response=item_original_node, key="id")

            # Determine the external creation date (if any):
            if external_create_date_field:
                external_create_date = self.replace_bulk_placeholders(
                    input_string=external_create_date_field,
                    row=row,
                )
            else:
                external_create_date = None

            # Determine the external modification date (if any):
            if external_modify_date_field:
                external_modify_date = self.replace_bulk_placeholders(
                    input_string=external_modify_date_field,
                    row=row,
                )
            else:
                external_modify_date = None

            # Determine the key field (if any):
            key = self.replace_bulk_placeholders(input_string=key_field, row=row) if key_field else None

            # check if workspace with this nickname does already exist.
            # we also store the nickname to assign it to the new workspace:
            if nickname_field:
                nickname = self.replace_bulk_placeholders(
                    input_string=nickname_field,
                    row=row,
                    replacements=replacements,
                    additional_regex_list=nickname_additional_regex_list,
                )
            else:
                nickname = None

            # Based on the evaluation of conditions_create, conditions_delete,
            # and conditions_update we could end up with an empty operations list.
            # In such cases we skip the further processing of this row:
            if not row_operations:
                self.logger.info(
                    "Skip item -> '%s' as row operations is empty (no create, update, delete or recreate). This may be because conditions_create, conditions_delete, and conditions_update have been evaluated to false for this row.",
                    item_name,
                )
                result["skipped_counter"] += 1
                continue

            self.logger.info(
                "Bulk process item -> '%s' using effective operations -> %s...",
                item_name,
                str(row_operations),
            )

            # Now we traverse a list of (multiple) workspaces
            # the item should be added to:
            success = True
            workspaces = bulk_item.get("workspaces", [])
            for workspace in workspaces:
                # success will only be false if a config problem (failure)
                # and not just a data problem (skipped) has occured:
                parent_id, success = self.get_bulk_document_location(
                    workspace=workspace,
                    row=row,
                    index=index,
                    replacements=replacements,
                )

                if parent_id is None:
                    continue

                #
                # Create the item in the target folder specified by parent_id:
                #

                # Check if the item does already exist.
                # First we try to look up the item by a unique
                # key that does not change over time.
                # For this we expect a "key" value to be defined for the
                # bulkDocument and one of the category / attribute item
                # to be marked with "is_key" = True. If we don't find the
                # item via key we try via parent and name.
                self.logger.info(
                    "Check if item -> '%s' is already in target folder with ID -> %s%s...",
                    item_name,
                    parent_id,
                    " (using key -> {})".format(key) if key is not None else "",
                )
                # Initialize variables - this is important!
                item_old_name = None
                item_id = None
                handle_name_clash = False
                item_modify_date = None

                # We have 4 cases to differentiate:

                # 1. key given + key found = name irrelevant, item exist
                # 2. key given + key not found = if name exist it is a name clash
                # 3. no key given + name found = item exist
                # 4. no key given + name not found = item does not exist

                # We are NOT trying to compensate for a failed key lookup with a name lookup!
                # Names are only relevant if no key is in payload!

                if key:
                    key_attribute = next(
                        (cat_attr for cat_attr in categories if cat_attr.get("is_key", False) is True),
                        None,
                    )
                    if key_attribute:
                        cat_name = key_attribute.get("name", None)
                        att_name = key_attribute.get("attribute", None)
                        set_name = key_attribute.get("set", None)
                        # Try to find the node that has the given key attribute value:
                        response = self._otcs_frontend.lookup_nodes(
                            parent_node_id=parent_id,
                            category=cat_name,
                            attribute=att_name,
                            attribute_set=set_name,
                            value=key,
                        )
                        item_id = self._otcs_frontend.get_result_value(
                            response=response,
                            key="id",
                        )
                        if item_id:
                            # Case 1: key given + key found = name irrelevant, item exist
                            self.logger.info(
                                "Found existing item with the key value -> '%s' in category -> '%s' and attribute -> '%s' in folder with ID -> %s.",
                                key,
                                cat_name,
                                att_name,
                                parent_id,
                            )
                            item_old_name = self._otcs_frontend.get_result_value(
                                response=response,
                                key="name",
                            )
                            if item_old_name != item_name and item_old_name != item_name + " (" + key + ")":
                                self.logger.info(
                                    "Existing item with ID -> %s has the old name -> '%s' which is different from the new name -> '%s.'",
                                    item_id,
                                    item_old_name,
                                    item_name,
                                )
                            else:
                                # if there was a name clash before and got resolved
                                # then we need to stick to the resolved name also for updates:
                                if item_old_name == item_name + " (" + key + ")":
                                    self.logger.info(
                                        "Document name conflict was resolved before. Changing item name to -> '%s' to match former resolution.",
                                        item_old_name,
                                    )
                                    item_name = item_old_name
                                item_old_name = None
                            # We get the modify date of the existing item.
                            item_modify_date = self._otcs_frontend.get_result_value(
                                response=response,
                                key="modify_date",
                            )
                        else:
                            # Case 2: key given + key not found = if name exist it is a name clash
                            self.logger.info(
                                "Couldn't find existing item with the key value -> '%s' in category -> '%s' and attribute -> '%s' in folder with ID -> %s.",
                                key,
                                cat_name,
                                att_name,
                                parent_id,
                            )
                            handle_name_clash = True
                    else:
                        self.logger.error(
                            "Item has a key -> '%s' defined but none of the category attributes is marked as a key attribute ('is_key' is missing)!",
                            key,
                        )
                        success = False
                        continue
                    # end if key_attribute:
                # end if key:
                else:
                    # If we haven't a key we try by parent + name
                    response = self._otcs_frontend.get_node_by_parent_and_name(
                        name=item_name,
                        parent_id=parent_id,
                    )
                    item_id = self._otcs_frontend.get_result_value(
                        response=response,
                        key="id",
                    )
                    if item_id:
                        # Case 3: no key given + name found = item exist
                        self.logger.info(
                            "Found existing item -> '%s' (%s) in parent with ID -> %s.",
                            item_name,
                            item_id,
                            parent_id,
                        )
                        # We get the modify date of the existing item.
                        item_modify_date = self._otcs_frontend.get_result_value(
                            response=response,
                            key="modify_date",
                        )
                    else:
                        # Case 4: no key given + name not found = item does not exist
                        self.logger.info(
                            "No existing item -> '%s' in parent with ID -> %s.",
                            item_name,
                            parent_id,
                        )

                # Check if we want to recreate an existing item
                # then we handle the "delete" part of "recreate" first:
                if item_id and "recreate" in row_operations:
                    response = self._otcs_frontend.delete_node(
                        node_id=item_id,
                        purge=True,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to recreate existing item -> '%s' (%s) under parent ID -> %s! Delete failed.",
                            (item_name if item_old_name is None else item_old_name),
                            item_id,
                            parent_id,
                        )
                        success = False
                        continue
                    self.logger.info(
                        "Successfully deleted existing item -> '%s' (%s) as part of the recreate operation.",
                        (item_name if item_old_name is None else item_old_name),
                        item_id,
                    )
                    result["delete_counter"] += 1
                    item_id = None

                # Check if item does not exists - then we create a new item
                # if this is requested ("create" value in operations list in payload)
                if not item_id and ("create" in row_operations or "recreate" in row_operations):
                    # The item does not exist in Content Server - so we
                    # add it now...

                    # for Case 2 (we looked up the item by key but could have name collisions)
                    # we need to see if we have name collisions:
                    if handle_name_clash:
                        response = self._otcs_frontend.check_node_name(
                            parent_id=parent_id,
                            node_name=item_name,
                        )
                        # result is a list of names that clash - if it is empty we have no clash
                        if response.get("results"):
                            # We need to handle a race condition here: it could be that
                            # the right item has been created by another thread. This is because
                            # the key could be in multiple rows of the data frame depending how the data loader works.
                            # We can also not assume that this can be resolved by making the key unique
                            # in the data source as it could be that a data set with the same key should
                            # go to multiple workspaces that then are folded into one by synonym resolution.
                            conflicting_node_id = response["results"][0].get("id")
                            conflicting_key = self._otcs_frontend.get_category_value_by_name(
                                node_id=conflicting_node_id,
                                category_name=cat_name,
                                attribute_name=att_name,
                                set_name=set_name,
                            )
                            if key == conflicting_key:
                                # We have a race condition as the two items don't really clash but are identical.
                                # Just skip creating the same item once more.
                                self.logger.warning(
                                    "Detected a race condition in name clash handling! Skipping item creation!",
                                )
                                continue

                            # We add the suffix with the key which should be unique:
                            self.logger.warning(
                                "Item -> '%s' does already exist in workspace folder with ID -> %s and we need to handle the name clash and use name -> '%s'",
                                item_name,
                                parent_id,
                                item_name + " (" + key + ")",
                            )
                            item_name = item_name + " (" + key + ")"

                    # If category data is in payload we substitute
                    # the values with data from the current data row:
                    if categories:
                        # Make sure the threads are not changing data structures that
                        # are shared between threads. categories is a list of dicts.
                        # list and dicts are "mutable" data structures in Python!
                        worker_categories = self.process_bulk_categories(
                            row=row,
                            index=index,
                            categories=categories,
                            replacements=replacements,
                        )
                        item_category_data = self.prepare_category_data(
                            categories_payload=worker_categories,
                            source_node_id=parent_id,
                        )
                    # end if categories
                    else:
                        item_category_data = {}

                    response = self._otcs_frontend.create_item(
                        parent_id=int(parent_id),
                        item_type=int(item_type),
                        item_name=item_name,
                        item_description=description,
                        url=item_url,
                        original_id=int(item_original_node_id),
                        category_data=item_category_data,
                        external_create_date=external_create_date,
                        external_modify_date=external_modify_date,
                        show_error=False,
                    )
                    item_id = self._otcs_frontend.get_result_value(
                        response=response,
                        key="id",
                    )
                    if not item_id:
                        # We may have a race condition here. Double check the item does not yet exist:
                        response = self._otcs_frontend.get_node_by_parent_and_name(
                            parent_id=int(parent_id),
                            name=item_name,
                        )
                        item_id = self._otcs_frontend.get_result_value(
                            response=response,
                            key="id",
                        )
                    if not item_id:
                        self.logger.error(
                            "Cannot create item -> '%s' in folder/workspace with ID -> %s!",
                            item_name,
                            parent_id,
                        )
                        success = False
                        continue
                    self.logger.info(
                        "Created item -> '%s' (%s), description -> '%s' in parent with ID -> %s.",
                        item_name,
                        item_id,
                        description,
                        parent_id,
                    )
                    result["create_counter"] += 1

                # end if not workspace_id and "create" in row_operations

                # If updates are requested we update the existing item with
                # a new item version and with fresh metadata from the payload.
                # Additionally we check the external modify date to support
                # incremental load for content that has really changed.
                # In addition we check that "delete" is not requested as otherwise it will
                # never go in elif "delete" ... below (and it does not make sense to update a item
                # that is deleted in the next step...)
                elif (
                    item_id
                    and "update" in row_operations
                    and "delete" not in row_operations  # note the NOT !
                    and OTCS.date_is_newer(
                        date_old=item_modify_date,
                        date_new=external_modify_date,
                    )
                ):
                    # get the specific update operations given in the payload
                    # if not specified we do all 4 update operations (name, description, categories and version)
                    update_operations = bulk_item.get(
                        "update_operations",
                        ["name", "description", "categories", "nickname", "url"],
                    )

                    # If category data is in payload we substitute
                    # the values with data from the current data row.
                    # This is only done if "categories" update is not
                    # excluded from the update_operations:
                    if categories and "categories" in update_operations:
                        # Make sure the threads are not changing data structures that
                        # are shared between threads. categories is a list of dicts.
                        # list and dicts are "mutable" data structures in Python!
                        worker_categories = self.process_bulk_categories(
                            row=row,
                            index=index,
                            categories=categories,
                            replacements=replacements,
                        )
                        # Transform the payload structure into the format
                        # the OTCS REST API requires:
                        item_category_data = self.prepare_category_data(
                            categories_payload=worker_categories,
                            source_node_id=item_id,
                            source_is_document=True,
                        )
                        if not item_category_data:
                            self.logger.error(
                                "Failed to prepare the updated category data for item -> '%s' (%s)!",
                                item_name,
                                str(item_id),
                            )
                            success = False
                            continue  # for index, row in partition.iterrows()
                    # end if categories
                    else:
                        item_category_data = {}

                    self.logger.info(
                        "Update existing item -> '%s' (%s) with operations -> %s...",
                        item_old_name if item_old_name else item_name,
                        item_id,
                        str(update_operations),
                    )
                    response = self._otcs_frontend.update_item(
                        node_id=item_id,
                        parent_id=None,  # None = do not move item
                        item_name=(item_name if "name" in update_operations else None),
                        item_description=(description if "description" in update_operations else None),
                        category_data=(item_category_data if "categories" in update_operations else None),
                        url=(item_url if "url" in update_operations else None),
                        external_create_date=external_create_date,
                        external_modify_date=external_modify_date,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to update metadata of existing item -> '%s' (%s) with metadata -> %s!",
                            (item_old_name if item_old_name else item_name),
                            item_id,
                            str(item_category_data),
                        )
                        success = False
                        continue
                    self.logger.info(
                        "Updated existing item -> '%s' (%s, description -> '%s')",
                        item_name,
                        item_id,
                        description,
                    )
                    result["update_counter"] += 1
                # end if item_id and "update" in row_operations
                elif item_id and "delete" in row_operations:
                    # We delete with immediate purging to keep recycle bin clean
                    # and to not run into issues with nicknames used in deleted items:
                    response = self._otcs_frontend.delete_node(
                        node_id=item_id,
                        purge=True,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to bulk delete existing item -> '%s' (%s)!",
                            (item_old_name if item_old_name else item_name),
                            item_id,
                        )
                        success = False
                        continue
                    self.logger.info(
                        "Successfully deleted existing item -> '%s' (%s).",
                        item_old_name if item_old_name else item_name,
                        item_id,
                    )
                    result["delete_counter"] += 1
                    item_id = None
                # end elif item_id and "delete" in row_operations

                # this is the plain old "if it does exist we just skip it" case:
                elif item_id:
                    self.logger.info(
                        "Skipped existing item -> '%s' (%s).",
                        item_old_name if item_old_name else item_name,
                        item_id,
                    )
                # this is the case where we just want to operate on existing items (update or delete)
                # but they do not exist:
                elif not item_id and ("update" in row_operations or "delete" in row_operations):
                    result["skipped_counter"] += 1
                    self.logger.info(
                        "Skipped update/delete of non-existing item -> '%s'.",
                        item_old_name if item_old_name else item_name,
                    )

                # The following code is executed for all operations
                # (create, recreate, update, delete):

                # Check if we want to set / update the nickname of the item.
                # Precondition is we have determined a nickname, we have the item ID
                # and the update of the nickname is either required (create, recreate)
                # or requested (update_operations include "nickname"):
                if (
                    nickname
                    and item_id
                    and (
                        "create" in row_operations
                        or "recreate" in row_operations
                        or ("update" in row_operations and "nickname" in update_operations)
                    )
                ):
                    response = self._otcs_frontend.set_node_nickname(
                        node_id=item_id,
                        nickname=nickname,
                        show_error=True,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to assign nickname -> '%s' to item -> '%s'!",
                            nickname,
                            item_name,
                        )
                if item_id is not None:
                    # Record the item name and ID to allow to read it from failure file
                    # and speedup the process.
                    result["items"][item_name] = item_id

            # end for workspaces

            # We want the success, failure and skip counter
            # to consider only complete data frame rows. In
            # case of bulk items we can potentially create
            # update or delete more than 1 item per row.
            # So we use the "success" variable to accumulate
            # this for all items per row:
            if not success:
                result["success"] = False
                result["failure_counter"] += 1
            elif item_name not in result["items"]:
                self.logger.info(
                    "Bulk item -> '%s' was not added to any workspace.",
                    item_name,
                )
                result["skipped_counter"] += 1
            else:
                result["success_counter"] += 1

        # end for index, row in partition.iterrows()

        self.logger.info("End working...")

        results.append(result)

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_avts_repositories")
    def process_avts_repositories(self, section_name: str = "avtsRepositories") -> bool:
        """Process Aviator Search repositories.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._avts_repositories:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        token = self._avts.authenticate()
        if not token:
            self.logger.error("Cannot authenticate at Aviator Search!")
            success = False
        else:
            for payload_repo in self._avts_repositories:
                if not payload_repo.get("enabled", True):
                    continue

                repository = self._avts.get_repo_by_name(name=payload_repo["name"])

                if repository is None:
                    self.logger.info(
                        "Repository -> '%s' does not exist, creating it...",
                        payload_repo["name"],
                    )

                    if payload_repo.get("type", "Extended ECM") == "Extended ECM":
                        repository = self._avts.create_extended_ecm_repo(
                            name=payload_repo["name"],
                            username=payload_repo["username"],
                            password=payload_repo["password"],
                            otcs_url=payload_repo["otcs_url"],
                            otcs_api_url=payload_repo["otcs_api_url"],
                            node_id=int(payload_repo["node_id"]),
                        )

                    elif payload_repo["type"] == "Documentum":
                        self.logger.warning("Not yet implemented")
                    elif payload_repo["type"] == "MSTeams":
                        repository = self._avts.create_msteams_repo(
                            name=payload_repo["name"],
                            client_id=payload_repo["client_id"],
                            tenant_id=payload_repo["tenant_id"],
                            certificate_file=payload_repo["certificate_file"],
                            certificate_password=payload_repo["certificate_password"],
                            index_attachments=payload_repo.get("index_attachments", True),
                            index_call_recordings=payload_repo.get(
                                "index_call_recordings",
                                True,
                            ),
                            index_message_replies=payload_repo.get(
                                "index_message_replies",
                                True,
                            ),
                            index_user_chats=payload_repo.get("index_user_chats", True),
                        )
                    elif payload_repo["type"] == "SharePoint":
                        repository = self._avts.create_sharepoint_repo(
                            name=payload_repo["name"],
                            client_id=payload_repo["client_id"],
                            tenant_id=payload_repo["tenant_id"],
                            certificate_file=payload_repo["certificate_file"],
                            certificate_password=payload_repo["certificate_password"],
                            sharepoint_url=payload_repo["sharepoint_url"],
                            sharepoint_url_type=payload_repo["sharepoint_url_type"],
                            sharepoint_mysite_url=payload_repo["sharepoint_mysite_url"],
                            sharepoint_admin_url=payload_repo["sharepoint_admin_url"],
                            index_user_profiles=payload_repo.get(
                                "index_message_replies",
                                False,
                            ),
                        )
                    else:
                        self.logger.error(
                            "Invalid repository type -> '%s' specified! Valid values are: Extended ECM, Documentum, MSTeams, SharePoint.",
                            payload_repo["type"],
                        )
                        success = False
                        break

                    if repository is None:
                        self.logger.error(
                            "Creation of Aviator Search repository -> '%s' failed!",
                            payload_repo["name"],
                        )
                        success = False
                    else:
                        self.logger.info(
                            "Successfully created Aviator Search repository -> '%s'.",
                            payload_repo["name"],
                        )
                        self.logger.debug("%s", repository)

                else:
                    self.logger.info(
                        "Aviator Search repository -> '%s' already exists.",
                        payload_repo["name"],
                    )

                # Start Crawling
                start_crawling = payload_repo.get("start", False)

                if repository is not None and start_crawling:
                    response = self._avts.start_crawling(repo_name=payload_repo["name"])

                    if response is None:
                        self.logger.error(
                            "Aviator Search start crawling on repository failed -> '%s'!",
                            payload_repo["name"],
                        )
                        success = False
                    else:
                        self.logger.info(
                            "Aviator Search crawling started on repository -> '%s'",
                            payload_repo["name"],
                        )
                        self.logger.debug("%s", response)
            # end for payload_repo in self._avts_repositories:
        # end else:

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._avts_repositories,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_avts_questions")
    def process_avts_questions(self, section_name: str = "avtsQuestions") -> bool:
        """Process Aviator Search repositories.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._avts_questions:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        if not self._avts_questions.get("enabled", True):
            self.logger.info(
                "Payload section -> '%s' is not enabled. Skipping...",
                section_name,
            )
            return True
        questions = self._avts_questions.get("questions", [])
        self.logger.info("Sample questions -> %s.", str(questions))

        token = self._avts.authenticate()
        if not token:
            self.logger.error("Cannot authenticate at Aviator Search!")
            success = False
        else:
            response = self._avts.set_questions(questions=questions)

            if response is None:
                self.logger.error("Aviator Search setting questions failed!")
                success = False
            else:
                self.logger.info("Successfully configured Aviator Search questions.")
                self.logger.debug("%s", response)

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._avts_questions,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_embeddings")
    def process_embeddings(self, section_name: str = "embeddings") -> bool:
        """Process additional Aviator embeddings.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise.

        """

        if not self._embeddings:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for embedding in self._embeddings:
            # Check if item has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not embedding.get("enabled", True):
                self.logger.info(
                    "Payload for embedding -> '%s' is disabled for FEME. Skipping...",
                    str(embedding),
                )
                continue

            # Backwards-compatibility for old syntax "documents":
            if "document_metadata" not in embedding:
                document_metadata = bool(embedding.get("documents", False))
            else:
                document_metadata = bool(embedding.get("document_metadata", False))

            # Backwards-compatibility for old syntax "workspaces":
            if "workspace_metadata" not in embedding:
                workspace_metadata = bool(embedding.get("workspaces", False))
            else:
                workspace_metadata = bool(embedding.get("workspace_metadata", False))

            wait_for_completion = bool(embedding.get("wait_for_completion", True))
            crawl = bool(embedding.get("crawl", False))
            images = bool(embedding.get("images", False))

            # We support 3 ways to determine the node ID(s):
            # 1. Node ID is specified directly in the payload using 'id = "..."'
            # 2. Node is is specified via a nickname of the node using 'nickname = "..."'
            # 3. Nodes are specified via the name of a workspace type. In this
            #    case all nodes of workspace instances are considered.

            node_id = None

            if embedding.get("id", None) is None and "nickname" in embedding:
                response = self._otcs.get_node_from_nickname(
                    nickname=embedding.get("nickname"),
                )
                node_id = self._otcs.get_result_value(response=response, key="id")

            else:
                node_id = embedding.get("id", None)
                response = None

            if node_id:
                result = self._otcs.aviator_embed_metadata(
                    node_id=node_id,
                    node=response,
                    wait_for_completion=wait_for_completion,
                    crawl=crawl,
                    document_metadata=document_metadata,
                    workspace_metadata=workspace_metadata,
                    images=images,
                )
                if not result:
                    success = False
            elif "workspace_types" in embedding:
                workspace_types = embedding["workspace_types"]
                # Handle the case of a single workspace type name:
                if isinstance(workspace_types, str):
                    workspace_types = [workspace_types]
                for workspace_type_name in workspace_types:
                    self.logger.info(
                        "Embedding metadata of workspace instances with type -> '%s'...", workspace_type_name
                    )
                    workspace_instances = self._otcs.get_workspace_instances_iterator(type_name=workspace_type_name)
                    for workspace in workspace_instances:
                        properties = workspace.get("data").get("properties")
                        self.logger.info(
                            "Embedding metadata of workspace instance -> '%s' (%s)...",
                            properties["name"],
                            properties["id"],
                        )
                        result = self._otcs.aviator_embed_metadata(
                            node_id=None,
                            node=workspace,
                            wait_for_completion=wait_for_completion,
                            crawl=crawl,
                            document_metadata=document_metadata,
                            workspace_metadata=workspace_metadata,
                            images=images,
                        )
                        if not result:
                            success = False

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._embeddings,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="replace_bulk_placeholders_list")
    def replace_bulk_placeholders_list(
        self,
        input_list: list,
        row: pd.Series,
        index: int = 0,
        replacements: dict | None = None,
        additional_regex_list: list | None = None,
    ) -> bool:
        """Process list of payload strings and replace placeholders (see next method).

        Args:
            input_list (list):
                A list of strings that contain placeholders.
            row (pd.Series):
                The curent data frame row.
            index (int, optional):
                Index for use if we encounter a list value.
                Pass None here if you want alle elements of
                the list to be parsed and placeholders replaced.
            replacements (dict, optional):
                The replacements to apply to given fields (dictionary key = field name)
            additional_regex_list (list, optional):
                These are not coming from the payload but dynamically
                added for special needs like determining the nicknames.

        Returns:
            bool:
                True = all replacements worked, False = some replacements had lookup errors.

        """

        success = True

        if not input_list:
            return False

        for i, value in enumerate(input_list):
            input_list[i] = self.replace_bulk_placeholders(
                input_string=value,
                row=row,
                index=index,
                replacements=replacements,
                additional_regex_list=additional_regex_list,
            )
            if not input_list[i]:
                success = False

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="replace_bulk_placeholders")
    def replace_bulk_placeholders(
        self,
        input_string: str,
        row: pd.Series,
        index: int | None = 0,  # don't use None as the default here!
        replacements: dict | None = None,
        additional_regex_list: list | None = None,
    ) -> str:
        """Replace placeholders like "{variable.subvalue}" in payload of bulk processing.

        Args:
            input_string (str):
                The string to replace placeholders in.
            row (pd.Series):
                Curent row in the data frame.
            index (int | None, optional):
                The index for use if we encounter a list value.
                If index is "None" then we return the complete list as value.
                Otherwise we return the list item with the given index (0 = first element is the default).
                IMPORTANT: None and 0 have different effects!
            replacements (dict, optional):
                The replacements to apply to given fields (dictionary key = field name)
            additional_regex_list (list, optional):
                These are not coming from the payload but dynamically
                added for special needs like determining the nicknames.

        Returns:
            str:
                Updated string with replacements.

        """

        r"""
        XML data sources may include "@" in Pandas column names as well!
        This happens if the XML elements have attributes.
               pattern = r"\{([\w@]+(\.[\w@]+)*)\}"
               pattern = r"\{(\w+(\.\w+)*)\}"
        Adjust Pattern to allow any sequence of characters withint the {...}
           pattern = r"\{([^}]*)\}"
        """

        # non-greedy match of placeholders that are surrounded by curly braces:
        pattern = r"\{(.*?)\}"

        had_lookup_error = False

        # Define a function to replace placeholders. This
        # function is called by re.sub() for each pattern match below.
        def replace(match: re.Match) -> str:
            # we want to change the variable of the main method
            nonlocal had_lookup_error

            # Extract the first captured group from the regex match, which corresponds
            # to the placeholder name. In the regex pattern r"\{(.*?)\}", the parentheses
            # define a capturing group that matches any content inside curly braces `{}`.
            # For example, in the string "Hello {user.name}", the match would capture "user.name".
            # `match.group(1)` retrieves this captured text (the placeholder name),
            # which will later be used to look up corresponding values in the provided data frame
            # row or replacement dictionary.
            field_name = match.group(1)
            # split up the field name into keys at ".", e.g. cm_vehicles.make
            keys = field_name.split(".")  # Splitting the variable and sub-value
            # we initialize value with the data frame row (pd.Series):
            value = row
            # Walk through the list of keys:
            for key in keys:
                # first we access the field in the row and handle the
                # exception that key may not be a valid column (KeyError):
                try:
                    # read the value of the data frame column defined by key
                    value = value[key]
                except KeyError as e:
                    self.logger.warning(
                        "KeyError: Cannot replace field -> '%s'%s as the data frame row does not have a column called -> '%s': %s",
                        field_name,
                        " (sub-key -> '{}')".format(key) if key != field_name else "",
                        field_name,
                        str(e),
                    )
                    had_lookup_error = True
                    return ""
                except TypeError:
                    # If we get an error with (value type -> <class 'float'>)
                    # this is actually indicating we have a NaN value!
                    self.logger.error(
                        "TypeError: Cannot replace field -> '%s'%s (value type -> %s). Expecting a dictionary-like value.",
                        field_name,
                        " (sub-key -> '{}')".format(key) if key != field_name else "",
                        str(type(value)),
                    )
                    had_lookup_error = True
                    return ""

                # if the returned value is a list we use the index parameter
                # to select the item in the list according to the given index
                # We handle the exception that index may be out of range for
                # the list (IndexError).
                # If the given index is None we return the whole list. This
                # is required for multi-value attributes.
                if isinstance(value, list) and index is not None and len(value) > 0:
                    try:
                        value = value[index]
                    except IndexError:
                        self.logger.error(
                            "Index error in replacement of list field -> '%s' using index -> %s.",
                            field_name,
                            str(index),
                        )
                        had_lookup_error = True
                        return ""

                # Check if the column or sub-field value is NaN or None. Then we should not
                # continue to process the keys and return "" (it is important to do this
                # after the test for a list above otherwise we may get an error
                # "The truth value of an array with more than one element is ambiguous."):
                if not isinstance(value, list) and pd.isna(value):
                    self.logger.debug(
                        "Cannot replace field -> '%s' as %s has no value in the current row!",
                        field_name,
                        (
                            "sub-key -> '{}'".format(key) if key != keys[0] else "column -> '{}'".format(key)
                        ),  # the first key is the column
                    )
                    had_lookup_error = True
                    return ""

            # end for key in keys

            if isinstance(value, list):
                if value == []:
                    had_lookup_error = True
                    return ""
            else:
                if pd.isna(value):
                    had_lookup_error = True
                    return ""
                value = str(value)

            if replacements and field_name in replacements:
                # replacements is a dictionary that is defined
                # in the payload. Each item is a dictionary
                # that can be looked up by the field name
                field_replacements = replacements[field_name]
                upper = field_replacements.get("upper_case", False)
                lower = field_replacements.get("lower_case", False)
                regex_list = field_replacements.get("regex_list", [])
            else:
                regex_list = []
                upper = False
                lower = False

            if additional_regex_list:
                regex_list = (
                    regex_list + additional_regex_list
                )  # don't do an append here as it would modify the original list

            if regex_list or upper or lower:
                if not isinstance(value, list):
                    value = self.cleanup_value(
                        cleanup_value=value,
                        regex_list=regex_list,
                        upper=upper,
                        lower=lower,
                    )
                else:  # we have a list, so we need to iterate
                    for v in value:
                        v = self.cleanup_value(
                            cleanup_value=v,
                            regex_list=regex_list,
                            upper=upper,
                            lower=lower,
                        )

            value = str(value)

            return value

        # end sub-method replace()

        if not input_string:
            return ""

        # Use re.sub() to replace placeholders using the defined function
        # replace() - see above.
        result_string = re.sub(pattern, replace, input_string)

        if had_lookup_error:
            return ""

        return result_string

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="cleanup_value")
    def cleanup_value(
        self,
        cleanup_value: str,
        regex_list: list,
        upper: bool = False,
        lower: bool = False,
    ) -> str:
        """Cleanup field values based on regular expressions.

        Args:
            cleanup_value (str):
                The string to clean up.
            regex_list (list):
                A list of regular expressions to apply.
            upper (bool, optional):
                If True, convert name to upper case letters.
            lower (bool, optional):
                If true, convert name to lower case letters.

        Returns:
            str:
                The cleaned string.

        """

        cleaned_string = cleanup_value

        if upper:
            cleaned_string = cleaned_string.upper()
        if lower:
            cleaned_string = cleaned_string.lower()

        if regex_list:
            try:
                for regex in regex_list:
                    # We use the pipe symbol to divide patterns from replacements
                    # this is a short-hand syntax to keep it simple. If there's
                    # no pipe in regex string and than we remove the pattern
                    # from the string
                    parts = regex.split("|")
                    pattern = parts[0]
                    replacement = ""  # Treat replacement as empty if no pipe specified
                    if len(parts) > 1:
                        pattern = r"\b" + pattern + r"\b"  # Match whole words only
                        replacement = parts[1]
                    cleaned_string = re.sub(pattern, replacement, cleaned_string)
            except re.error:
                self.logger.error(
                    "Invalid regular expression pattern -> %s",
                    pattern,
                )
                return cleanup_value

        return cleaned_string

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="evaluate_conditions")
    def evaluate_conditions(
        self,
        conditions: list,
        row: pd.Series,
        replacements: dict | None = None,
    ) -> bool:
        """Evaluate given conditions for a DataFrame series (i.e. a row).

        Args:
            conditions (list):
                A list of dictionaries that have a "field" (mandatory)
                and an "value" (optional) element.
            row (pd.Series):
                The current data row to pull data from.
            replacements (dict):
                The replacements to apply to given fields (dictionary key = field name).

        Returns:
            bool:
                True if all given conditions evaluate to True. False otherwise.

        """

        evaluated_condition = True

        # We traverse a list of conditions. All conditions must evaluate to true
        # otherwise the current workspace or document (i.e. the data row for these objects)
        # will be skipped:
        for condition in conditions:
            field = condition.get("field", None)
            if not field:
                self.logger.error(
                    "Missing field in condition.",
                )
                evaluated_condition = False
                break
            field_value = self.replace_bulk_placeholders(
                input_string=field,
                row=row,
                replacements=replacements,
            )
            self.logger.debug(
                "Evaluated field name -> '%s' to value -> '%s'",
                field,
                field_value,
            )
            # we have 3 options for value:
            # a) it does not exist in payload - then just the existance of the field is tested
            # b) it is a string - then we compare it 1:1 with the field value
            # c) it is a list of string - then the condition is met if one or more
            #    of the list values is equal to the field value
            value = condition.get("value", None)
            #            if not value:
            if value is None:
                # if there's no "value" element in the payload
                # this means that we just check the existance of the field
                if field_value:
                    # field does exist and has any non-"" value ==> condition met!
                    continue
                # field does not exist ==> condition not met!
                evaluated_condition = False
                break
            #
            # we handle string, boolean, and list values:
            #
            should_be_equal = condition.get("equal", True)
            if isinstance(value, str):
                if (should_be_equal and (value != str(field_value))) or (
                    not should_be_equal and (value == str(field_value))
                ):
                    self.logger.debug(
                        "String value -> '%s' is %s to field value -> '%s' but it %s. Condition not met for field -> '%s'!",
                        value,
                        "not equal" if should_be_equal else "equal",
                        field_value,
                        "should" if should_be_equal else "shouldn't",
                        field,
                    )
                    evaluated_condition = False
                    break
            elif isinstance(value, bool):
                # We can't do a bool(field_value) as this would return True
                # for any non-empty string. So we explictly convert to a bool value
                # in a safe way:
                bool_field_value: bool = field_value.lower() == "true" if field_value else False
                if (should_be_equal and (value != bool_field_value)) or (
                    not should_be_equal and (value == bool_field_value)
                ):
                    self.logger.debug(
                        "Boolean value -> '%s' is %s to field value -> '%s' but it %s. Condition not met for field -> '%s'!",
                        value,
                        "not equal" if should_be_equal else "equal",
                        str(bool_field_value),
                        "should" if should_be_equal else "shouldn't",
                        field,
                    )
                    evaluated_condition = False
                    break
            elif isinstance(value, list):
                for value_item in value:
                    # Check if we found a matching element and then break.
                    # Break equals positive result if should_be_equal is True:
                    if should_be_equal and (value_item == field_value):
                        break
                else:  # just executed if the for loop is not interrupted by break
                    # not finding a match is only negative if should_be_equal is TRUE:
                    if should_be_equal:
                        self.logger.debug(
                            "Value list -> %s does not include field value -> '%s'. Condition not met!",
                            str(value),
                            field_value,
                        )
                        evaluated_condition = False

        return evaluated_condition

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_classifications")
    def process_bulk_classifications(self, section_name: str = "bulkClassifications") -> bool:
        """Process bulk classifications in payload and bulk create them in Content Server (multi-threaded).

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True if payload has been processed without errors, False otherwise.

        """

        if not self._bulk_classifications:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        # For efficient idem-potent operation we may want to see which
        # bulk classifications have already been processed before:
        if self.check_status_file(
            payload_section_name=section_name,
            payload_specific=True,
            prefix="failure_",
        ):
            # Read payload from where we left it last time
            self._bulk_classifications = self.get_status_file(
                payload_section_name=section_name,
                prefix="failure_",
            )
            if not self._bulk_classifications:
                self.logger.error(
                    "Cannot load existing bulkClassifications failure file. Bailing out!",
                )
                return False

        success: bool = True

        for bulk_classification in self._bulk_classifications:
            # Check if element has been disabled in payload (enabled = false).
            # In this case we skip the element:
            enabled = bulk_classification.get("enabled", True)
            if not enabled:
                self.logger.info("Payload for bulk classification is disabled. Skipping...")
                continue

            # The payload element must have a "data_source" key:
            data_source_name = bulk_classification.get("data_source", None)
            if not data_source_name:
                self.logger.error("Missing data source name in bulk classification!")
                success = False
                continue

            self._log_header_callback(
                text="Process bulk classifications using data source -> '{}'".format(
                    data_source_name,
                ),
                char="-",
            )

            copy_data_source = bulk_classification.get("copy_data_source", False)
            force_reload = bulk_classification.get("force_reload", True)

            # Load and prepare the data source for the bulk processing:
            if copy_data_source:
                self.logger.info(
                    "Take a copy of data source -> '%s' to avoid side-effects for repeative usage of the data source...",
                    data_source_name,
                )
                data = Data(
                    self.process_bulk_datasource(
                        data_source_name=data_source_name,
                        force_reload=force_reload,
                    ),
                    logger=self.logger,
                )
            else:
                self.logger.info(
                    "Use original data source -> '%s' and not do a copy.",
                    bulk_classification["data_source"],
                )
                data = self.process_bulk_datasource(
                    data_source_name=data_source_name,
                    force_reload=force_reload,
                )
            if data is None:  # important to use "is None" here!
                self.logger.error("Failed to load data source -> '%s' for bulk classification!", data_source_name)
                success = False
                continue
            if data.get_data_frame() is None:  # important to use "is None" here!
                self.logger.error("Data source -> '%s' for bulk classification is empty!", data_source_name)
                continue

            # Check if fields with list substructures should be exploded.
            # We may want to do this outside the bulkDatasource to only
            # explode for bulkClassifications and not other bulk elements
            # like bulkDocuments or bulkWorkspaces:
            explosions = bulk_classification.get("explosions", [])
            for explosion in explosions:
                # explode field can be a string or a list
                # exploding multiple fields at once avoids
                # combinatorial explosions - this is VERY
                # different from exploding columns one after the other!
                if "explode_field" not in explosion and "explode_fields" not in explosion:
                    self.logger.error("Missing explosion field(s)!")
                    continue
                # we want to be backwards compatible...
                explode_fields = (
                    explosion["explode_field"] if "explode_field" in explosion else explosion["explode_fields"]
                )
                flatten_fields = explosion.get("flatten_fields", [])
                split_string_to_list = explosion.get("split_string_to_list", False)
                list_splitter = explosion.get(
                    "list_splitter",
                    ",",
                )  # don't have None as default!
                self.logger.info(
                    "Starting explosion of bulk classifications by field(s) -> %s (type -> %s). Size of data frame before explosion -> %s.",
                    str(explode_fields),
                    type(explode_fields),
                    str(len(data)),
                )
                data.explode_and_flatten(
                    explode_fields=explode_fields,
                    flatten_fields=flatten_fields,
                    make_unique=False,
                    split_string_to_list=split_string_to_list,
                    separator=list_splitter,
                    reset_index=True,
                )
                self.logger.info(
                    "Size of data frame after explosion -> %s.",
                    str(len(data)),
                )

            # Keep only selected rows if filters are specified in bulkClassifications.
            # We have this _after_ "explosions" to allow access to subfields as well.
            # We have this _before_ "sorting" and "deduplication" as we may keep the wrong
            # rows otherwise (unique / deduplication always keeps the first matching row).
            # We may want to do this outside the bulkDatasource to only
            # filter for bulkClassifications and not for bulkWorkspaces or
            # bulkWorkspaceRelationships:
            filters = bulk_classification.get("filters", [])
            if filters:
                data.filter(conditions=filters)

            # Sort the data frame if "sort" specified in payload. We may want to do this to have a
            # higher chance that rows with classifications names that may collapse into
            # one name are put into the same partition. This can avoid race conditions
            # between different Python threads.
            sort_fields = bulk_classification.get("sort", [])
            if sort_fields:
                self.logger.info(
                    "Start sorting of data frame for bulk classifications based on fields (columns) -> %s...",
                    str(sort_fields),
                )
                data.sort(sort_fields=sort_fields, inplace=True)
                self.logger.info(
                    "Sorting of data frame for bulk classifications based on fields (columns) -> %s completed.",
                    str(sort_fields),
                )

            # Check if duplicate rows for given fields should be removed. It is
            # important to do this after sorting as Pandas always keep the first occurance,
            # so ordering plays an important role in deduplication!
            unique_fields = bulk_classification.get("unique", [])
            if unique_fields:
                self.logger.info(
                    "Starting deduplication of data frame for bulk classifications with unique fields -> %s. Size of data frame before deduplication -> %s.",
                    str(unique_fields),
                    str(len(data)),
                )
                data.deduplicate(unique_fields=unique_fields, inplace=True)
                self.logger.info(
                    "Size of data frame after deduplication -> %s.",
                    str(len(data)),
                )

            # Read the optional categories payload:
            categories = bulk_classification.get("categories", None)
            if not categories:
                self.logger.info(
                    "Bulk classification payload has no category data! Will leave category attributes empty...",
                )

            # Get the operations to carry out during bulk processing.
            # Just doing a create is the default. Other options are
            # "update", "delete", "recreate":
            operations = bulk_classification.get("operations", ["create"])

            self.logger.info(
                "Bulk create classifications. Operations -> %s.",
                str(operations),
            )

            # See if bulkClassification definition has a specific thread number
            # otherwise it is read from a global environment variable:
            bulk_thread_number = int(
                bulk_classification.get("thread_number", BULK_THREAD_NUMBER),
            )

            partitions = data.partitionate(bulk_thread_number)

            # Create a list to hold the threads
            threads = []
            results = []

            # Create and start a thread for each partition
            for index, partition in enumerate(partitions, start=1):
                thread = threading.Thread(
                    name=f"{section_name}_{index:02}",
                    target=self.thread_wrapper,
                    args=(
                        self.process_bulk_classifications_worker,
                        bulk_classification,
                        partition,
                        categories,
                        operations,
                        results,
                    ),
                )
                # start a thread executing the process_bulk_classifications_worker() method below:
                self.logger.info("Starting thread -> '%s'...", str(thread.name))
                thread.start()
                threads.append(thread)
                # Avoid that all threads start at the exact same time with
                # potentially expired cookies that cases race conditions:
                time.sleep(1)
            # end for index, partition in enumerate(partitions, start=1)

            # Wait for all threads to complete
            for thread in threads:
                self.logger.info(
                    "Waiting for thread -> '%s' to complete...",
                    str(thread.name),
                )
                thread.join()
                self.logger.info("Thread -> '%s' has completed.", str(thread.name))

            if "classifications" not in bulk_classification:
                bulk_classification["classifications"] = {}

            # Print statistics for each thread. In addition,
            # check if all threads have completed without error / failure.
            # If there's a single failure in on of the thread results we
            # set 'success' variable to False.
            results.sort(key=lambda x: x["thread_name"])
            for result in results:
                self.logger.info(
                    "Thread -> '%s' completed with overall status '%s'.",
                    str(result["thread_name"]),
                    "successful" if result["success"] else "failed",
                )
                self.logger.info(
                    "Thread -> '%s' processed %s data rows with %s successful, %s failed, and %s skipped.",
                    str(result["thread_name"]),
                    str(
                        result["success_counter"] + result["failure_counter"] + result["skipped_counter"],
                    ),
                    str(result["success_counter"]),
                    str(result["failure_counter"]),
                    str(result["skipped_counter"]),
                )
                self.logger.info(
                    "Thread -> '%s' created %s classifications, updated %s classifications, and deleted %s classifications.",
                    str(result["thread_name"]),
                    str(result["create_counter"]),
                    str(result["update_counter"]),
                    str(result["delete_counter"]),
                )

                if not result["success"]:
                    success = False
                # Record all generated classifications. This should allow us
                # to restart in case of failures and avoid trying to
                # create classifications that have been created before.
                bulk_classification["classifications"].update(result["classifications"])
            self._log_header_callback(
                text="Completed processing of bulk classifications using data source -> '{}'".format(data_source_name),
                char="-",
            )

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._bulk_classifications,
        )

        return success

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_classifications_worker")
    def process_bulk_classifications_worker(
        self,
        bulk_classification: dict,
        partition: pd.DataFrame,
        categories: list | None = None,
        operations: list | None = None,
        results: list | None = None,
    ) -> None:
        """Thread worker to create classificatins in bulk.

        Each worker thread gets a partition of the rows that include
        the data required for the classification creation.

        Args:
            bulk_classification (dict):
                The payload of the bulkClassification.
            partition (pd.DataFrame):
                Data partition with rows to process.
            categories (list):
                List of category dictionaries.
            operations (list):
                Defines which operations should be applyed on classifications.
                Possible values are "create", "update", "delete", "recreate".
            results (list):
                A mutable list of thread results.

        """

        thread_id = threading.get_ident()
        thread_name = threading.current_thread().name

        self.logger.info(
            "Start working on data frame partition of size -> %s to bulk create classifications...",
            str(len(partition)),
        )

        # Avoid linter warnings - so make parameter default None while we
        # actually want ["create"] to be the default:
        if operations is None:
            operations = ["create"]

        result = {}
        result["thread_id"] = thread_id
        result["thread_name"] = thread_name
        result["success_counter"] = 0
        result["failure_counter"] = 0
        result["skipped_counter"] = 0
        result["create_counter"] = 0
        result["update_counter"] = 0
        result["delete_counter"] = 0
        result["classifications"] = {}
        result["success"] = True

        # Check if classifications have been processed before, e.i. testing
        # if a "classifications" key exists and if it is pointing to a non-empty list.
        # Additionally we check that classification updates are not enforced:
        if (
            bulk_classification.get("classifications")
            and "update" not in operations
            and "delete" not in operations
            and "recreate" not in operations
        ):
            existing_classifications = bulk_classification["classifications"]
            self.logger.info(
                "Found %s already processed classifications. Try to complete the job...",
                str(len(existing_classifications)),
            )
        else:
            existing_classifications = {}

        # See if external creation and modification dates are in the payload data:
        external_modify_date_field = bulk_classification.get("external_modify_date")
        external_create_date_field = bulk_classification.get("external_create_date")

        # See if we have a key field to uniquely identify an existing document:
        key_field = bulk_classification.get("key")

        # Get dictionary of replacements for bulk classification creations
        # this we will be used of all places data is read from the
        # data frame. Each dictionary item has the field name as the
        # dictionary key and a list of regular expressions as dictionary value
        replacements = bulk_classification.get("replacements")

        # In case the name cannot be resolved we allow to
        # specify an alternative name field in the payload.
        name_field = bulk_classification.get("name")

        # In case the name cannot be resolved we allow to
        # specify an alternative name field in the payload.
        name_field_alt = bulk_classification.get("name_alt")

        # In case the description cannot be resolved we allow to
        # specify an alternative description field in the payload.
        description_field = bulk_classification.get("description")

        # In case the description cannot be resolved we allow to
        # specify an alternative description field in the payload.
        description_field_alt = bulk_classification.get("description_alt")

        # Fetch the nickname field from the payload (if it is specified):
        nickname_field = bulk_classification.get("nickname")

        # In case the nickname cannot be resolved we allow to
        # specify an alternative nickname field in the payload.
        nickname_field_alt = bulk_classification.get("nickname_alt")

        # Nicknames are very limited in terms of allowed characters.
        # For nicknames we need additional regexp as we need to
        # replace all non-alphanumeric, non-space characters with ""
        # We also preserve hyphens in the first step to replace
        # them below with underscores. This is important to avoid
        # that different spellings of names produce different nicknames.
        # We want spellings with spaces match spellings with hyphens.
        # For this the classification names have a regexp "-| " in the payload.
        nickname_additional_regex_list = [r"[^\w\s-]"]

        total_rows = len(partition)

        # Process all rows in the partition that was given to this thread:
        for index, row in partition.iterrows():
            # Calculate percentage of completion:
            percent_complete = ((partition.index.get_loc(index) + 1) / total_rows) * 100

            self.logger.info(
                "Processing data row -> %s for bulk classification creation (%.2f %% complete)...",
                str(index),
                percent_complete,
            )

            # Clear variables to esure clean state for each row:
            classification_id = None

            # Determine the classification name:
            classification_name = self.replace_bulk_placeholders(
                input_string=name_field,
                row=row,
                replacements=replacements,
            )
            # If we cannot get the classification name from the
            # name_field we try the alternative name field
            # (if specified in payload via "name_alt"):
            if not classification_name and name_field_alt:
                self.logger.debug(
                    "Row -> %s does not have the data to resolve the placeholders in classification name -> %s! Trying alternative name field -> %s...",
                    str(index),
                    name_field,
                    name_field_alt,
                )
                classification_name = self.replace_bulk_placeholders(
                    input_string=name_field_alt,
                    row=row,
                    replacements=replacements,
                )

            if not classification_name:
                self.logger.warning(
                    "Row -> %s does not have the required data to resolve -> %s for the classification name!%s",
                    str(index),
                    name_field,
                    " There's no alternative field name given in the payload (via 'name_alt')."
                    if not name_field_alt
                    else " The alternative field -> '{}' could not be resolved either!".format(
                        name_field_alt,
                    ),
                )
                result["skipped_counter"] += 1
                continue

            # Cleanse the classification name (allowed characters, maximum length):
            classification_name = OTCS.cleanse_item_name(classification_name)

            # Check if classification has been created before (either in this run
            # or in a former run of the customizer):
            if (
                classification_name in existing_classifications  # processed in former run?
                or classification_name in result["classifications"]  # processed in current run?
            ):
                self.logger.info(
                    "Classification -> '%s' does already exist. Skipping...",
                    classification_name,
                )
                result["skipped_counter"] += 1
                continue

            # Determine the description field:
            if description_field:
                description = self.replace_bulk_placeholders(
                    input_string=description_field,
                    row=row,
                )
                # If we cannot get the description from the
                # description_field we try the alternative description field
                # (if specified in payload via "description_alt"):
                if not description and description_field_alt:
                    self.logger.debug(
                        "Row -> %s does not have the data to resolve the placeholders in classification description -> %s! Trying alternative description field -> %s...",
                        str(index),
                        name_field,
                        description_field_alt,
                    )
                    description = self.replace_bulk_placeholders(
                        input_string=description_field_alt,
                        row=row,
                    )
            else:
                description = ""

            # Create a copy of the mutable operations list as we may
            # want to modify it per row:
            row_operations = list(operations)

            # Check if all data conditions to create the classification are met
            conditions = bulk_classification.get("conditions")
            if conditions:
                evaluated_condition = self.evaluate_conditions(
                    conditions=conditions,
                    row=row,
                    replacements=replacements,
                )
                if not evaluated_condition:
                    self.logger.info(
                        "Condition for bulk classification row -> %s not met. Skipping row for classification creation...",
                        str(index),
                    )
                    result["skipped_counter"] += 1
                    continue

            # Check if all data conditions to create or recreate the classification are met:
            if "create" in row_operations or "recreate" in row_operations:
                conditions_create = bulk_classification.get("conditions_create")
                if conditions_create:
                    evaluated_conditions_create = self.evaluate_conditions(
                        conditions=conditions_create,
                        row=row,
                        replacements=replacements,
                    )
                    if not evaluated_conditions_create:
                        self.logger.info(
                            "Create condition for bulk classification row -> %s not met. Excluding create operation...",
                            str(index),
                        )
                        if "create" in row_operations:
                            row_operations.remove("create")
                        if "recreate" in row_operations:
                            row_operations.remove("recreate")
                elif "recreate" in row_operations:
                    # we still create and recreate without conditions_create.
                    # But give a warning for 'recreate' without condition.
                    self.logger.warning(
                        "No create condition provided but 'recreate' operation requested. Recreating all existing classifications!",
                    )

            # Check if all data conditions to delete the classification are met:
            if "delete" in row_operations:
                conditions_delete = bulk_classification.get("conditions_delete")
                if conditions_delete:
                    evaluated_conditions_delete = self.evaluate_conditions(
                        conditions=conditions_delete,
                        row=row,
                        replacements=replacements,
                    )
                    if not evaluated_conditions_delete:
                        self.logger.info(
                            "Delete condition for bulk classification row -> %s not met. Excluding delete operation...",
                            str(index),
                        )
                        row_operations.remove("delete")
                else:  # without delete_conditions we don't delete!!
                    self.logger.warning(
                        "Delete operation requested for bulk classifications but conditions for deletion are missing! (specify 'conditions_delete')!",
                    )
                    row_operations.remove("delete")

            # Check if all data conditions to delete the classification are met:
            if "update" in row_operations:
                conditions_update = bulk_classification.get("conditions_update")
                if conditions_update:
                    evaluated_conditions_update = self.evaluate_conditions(
                        conditions=conditions_update,
                        row=row,
                        replacements=replacements,
                    )
                    if not evaluated_conditions_update:
                        self.logger.info(
                            "Update condition for bulk classification row -> %s not met. Excluding update operation...",
                            str(index),
                        )
                        row_operations.remove("update")

            # Determine the external modification date (if any):
            if external_modify_date_field:
                external_modify_date = self.replace_bulk_placeholders(
                    input_string=external_modify_date_field,
                    row=row,
                )
            else:
                external_modify_date = None

            # Determine the external creation date (if any):
            if external_create_date_field:
                external_create_date = self.replace_bulk_placeholders(
                    input_string=external_create_date_field,
                    row=row,
                )
            else:
                external_create_date = None

            # Determine the key field (if any):
            key = self.replace_bulk_placeholders(input_string=key_field, row=row) if key_field else None

            # check if classification with this nickname does already exist.
            # we also store the nickname to assign it to the new classification:
            if nickname_field:
                nickname = self.replace_bulk_placeholders(
                    input_string=nickname_field,
                    row=row,
                    replacements=replacements,
                    additional_regex_list=nickname_additional_regex_list,
                )
                # If we cannot get the nickname from the
                # nickname_field we try the alternative nickname field
                # (if specified in payload via "nickname_alt"):
                if not nickname and nickname_field_alt:
                    nickname = self.replace_bulk_placeholders(
                        input_string=nickname_field_alt,
                        row=row,
                        replacements=replacements,
                        additional_regex_list=nickname_additional_regex_list,
                    )

                # Nicknames for sure should not have leading or trailing spaces:
                nickname = nickname.strip()
                # Nicknames for sure are not allowed to include spaces:
                nickname = nickname.replace(" ", "_")
                # We also want to replace hyphens with underscores
                # to make sure that classification name spellings with
                # spaces and with hyphens are mapped to the same
                # classification nicknames (aligned with the classification names
                # that have a regexp rule for this in the payload):
                nickname = nickname.replace("-", "_")
                nickname = nickname.replace("___", "_")
                nickname = nickname.lower()
                response = self._otcs_frontend.get_node_from_nickname(nickname=nickname)
                if response:
                    found_classification_name = self._otcs_frontend.get_result_value(
                        response=response,
                        key="name",
                    )
                    found_classification_id = self._otcs_frontend.get_result_value(
                        response=response,
                        key="id",
                    )
                    if found_classification_name != classification_name:
                        self.logger.warning(
                            "Clash of nicknames for classification -> '%s' and classification -> '%s' (%s)!",
                            classification_name,
                            found_classification_name,
                            found_classification_id,
                        )
                    # Only skip, if classification 'update' or 'delete' is NOT requested:
                    elif "update" not in row_operations and "delete" not in row_operations:
                        self.logger.info(
                            "Classification -> '%s' with nickname -> '%s' does already exist (found -> '%s'). No update or delete operations requested or allowed. Skipping...",
                            classification_name,
                            nickname,
                            found_classification_name,
                        )
                        result["skipped_counter"] += 1
                        continue
            # end if nickname_field
            else:
                nickname = None

            # Based on the evaluation of conditions_create, conditions_delete,
            # and conditions_update we could end up with an empty operations list.
            # In such cases we skip the further processing of this row:
            if not row_operations:
                self.logger.info(
                    "Skip classification -> '%s' as row operations is empty (no create, update, delete or recreate). This may be because conditions_create, conditions_delete, and conditions_update have been evaluated to false for this row.",
                    classification_name,
                )
                result["skipped_counter"] += 1
                continue

            self.logger.info(
                "Bulk process classification -> '%s' using effective operations -> %s...",
                classification_name,
                str(row_operations),
            )

            self.logger.info(
                "Check if classification -> '%s' does already exist...",
                classification_name,
            )

            # Initialize variables - this is important!
            classification_old_name = None
            classification_id = None
            handle_name_clash = False
            classification_modify_date = None
            parent_id = None

            # We create a copy list to not modify original payload
            classification_path = list(bulk_classification.get("path", []))
            result_placeholders = self.replace_bulk_placeholders_list(
                input_list=classification_path,
                row=row,
                replacements=replacements,
            )
            if not result_placeholders:
                classification_path = None

            # We have 4 cases to differentiate:

            # 1. key given + key found = name irrelevant, item exist
            # 2. key given + key not found = if name exist it is a name clash
            # 3. no key given + name found = item exist
            # 4. no key given + name not found = item does not exist

            # We are NOT trying to compensate for a failed key lookup with a name lookup!
            # Names are only relevant if no key is in payload!

            if key:
                # if we have a key we need to also know which category attribute is
                # storing the value for the key:
                key_attribute = next(
                    (cat_attr for cat_attr in categories if cat_attr.get("is_key", False) is True),
                    None,
                )
                if not key_attribute:
                    self.logger.error(
                        "Bulk Classification has a key -> '%s' defined but none of the category attributes is marked as key ('is_key' is missing)!",
                        key,
                    )
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue
                cat_name = key_attribute.get("name", None)
                att_name = key_attribute.get("attribute", None)
                set_name = key_attribute.get("set", None)

                # Get the parent classification element:
                response = self._otcs_frontend.get_node_by_volume_and_path(
                    volume_type=self._otcs_frontend.VOLUME_TYPE_CLASSIFICATION_VOLUME,
                    path=classification_path,
                    create_path=True,
                )
                parent_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
                if parent_id:
                    # Try to find the node that has the given key attribute value:
                    response = self._otcs_frontend.lookup_nodes(
                        parent_node_id=parent_id,
                        category=cat_name,
                        attribute=att_name,
                        attribute_set=set_name,
                        value=key,
                    )
                    classification_id = self._otcs_frontend.get_result_value(
                        response=response,
                        key="id",
                    )
                else:
                    # if we couldn't determine the parent ID this means there are
                    # now classification instances for this classification type. Then we set
                    # classification_id = None and let the code go into the else case below:
                    classification_id = None

                if classification_id:
                    # Case 1: key given + key found = name irrelevant, item exist
                    classification_old_name = self._otcs_frontend.get_result_value(
                        response=response,
                        key="name",
                    )
                    self.logger.info(
                        "Found existing classification -> %s (%s) in classification with ID -> %s using key value -> '%s', category -> '%s', and attribute -> '%s'.",
                        classification_old_name,
                        classification_id,
                        parent_id,
                        key,
                        cat_name,
                        att_name,
                    )
                    if classification_old_name != classification_name:
                        self.logger.info(
                            "Existing classification has the old name -> '%s' which is different from the new name -> '%s.'",
                            classification_old_name,
                            classification_name,
                        )
                    else:
                        classification_old_name = None
                    # We get the modify date of the existing classification.
                    classification_modify_date = self._otcs_frontend.get_result_value(
                        response=response,
                        key="modify_date",
                    )
                else:
                    # Case 2: key given + key not found = if name exist it is a name clash
                    self.logger.info(
                        "Couldn't find existing classification with the key value -> '%s' in category -> '%s' and attribute -> '%s' in folder with ID -> %s.",
                        key,
                        cat_name,
                        att_name,
                        parent_id,
                    )
                    handle_name_clash = True
                # end if key_attribute
            # end if key
            else:
                # Get the parent classification element:
                response = self._otcs_frontend.get_node_by_volume_and_path(
                    volume_type=self._otcs_frontend.VOLUME_TYPE_CLASSIFICATION_VOLUME,
                    path=classification_path,
                )
                parent_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
                # If we haven't a key we try by parent + name
                response = self._otcs_frontend.get_node_by_parent_and_name(
                    parent_id=parent_id,
                    name=classification_name,
                )
                classification_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
                if classification_id:
                    # Case 3: no key given + name found = item exist
                    self.logger.info(
                        "Found existing classification -> '%s' (%s).",
                        classification_name,
                        classification_id,
                    )
                    # We get the modify date of the existing classification.
                    classification_modify_date = self._otcs_frontend.get_result_value(
                        response=response,
                        key="modify_date",
                    )
                else:
                    # Case 4: no key given + name not found = item does not exist
                    self.logger.info(
                        "No existing classification with name -> '%s' in path -> %s.",
                        classification_name,
                        classification_path,
                    )

            # Check if we want to recreate an existing classification
            # then we handle the "delete" part of "recreate" first:
            if classification_id and "recreate" in row_operations:
                response = self._otcs_frontend.delete_node(
                    node_id=classification_id,
                    purge=True,
                )
                if not response:
                    self.logger.error(
                        "Failed to recreate existing classification -> '%s' (%s)! Delete failed.",
                        (classification_name if classification_old_name is None else classification_old_name),
                        classification_id,
                    )
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue
                result["delete_counter"] += 1
                self.logger.info(
                    "Successfully deleted existing classification -> '%s' (%s) as part of the recreate operation.",
                    (classification_name if classification_old_name is None else classification_old_name),
                    classification_id,
                )
                classification_id = None

            # Check if classification does not exists - then we create a new classification
            # if this is requested ("create" or "recreate" value in operations list in payload)
            if not classification_id and ("create" in row_operations or "recreate" in row_operations):
                # for Case 2 (we looked up the classification by key but could have name collisions)
                # we need to see if we have a name collision:
                if handle_name_clash and parent_id:
                    response = self._otcs_frontend.check_node_name(
                        parent_id=parent_id,
                        node_name=classification_name,
                    )
                    # result is a list of names that clash - if it is empty we have no clash
                    if response and response["results"]:
                        # We add the suffix with the key which should be unique:
                        self.logger.warning(
                            "Classification -> '%s' does already exist in folder with ID -> %s and we need to handle the name clash by using name -> '%s'",
                            classification_name,
                            parent_id,
                            classification_name + " (" + key + ")",
                        )
                        classification_name = classification_name + " (" + key + ")"

                # If category data is in payload we substitute
                # the values with data from the current data row:
                if categories:
                    # Make sure the threads are not changing data structures that
                    # are shared between threads. categories is a list of dicts.
                    # list and dicts are "mutable" data structures in Python!
                    worker_categories = self.process_bulk_categories(
                        row=row,
                        index=index,
                        categories=categories,
                        replacements=replacements,
                    )
                    # classification_category_data = self.prepare_item_create_form(
                    #     parent_id=parent_id,
                    #     categories=worker_categories,
                    #     subtype=self._otcs_frontend.ITEM_TYPE_CLASSIFICATION,
                    # )
                    classification_category_data = self.prepare_category_data(
                        categories_payload=worker_categories,
                        source_node_id=parent_id,
                    )
                    if not classification_category_data:
                        self.logger.error(
                            "Failed to prepare the category data for new classification -> '%s'!",
                            classification_name,
                        )
                        result["success"] = False
                        result["failure_counter"] += 1
                        continue  # for index, row in partition.iterrows()
                else:
                    classification_category_data = {}

                self.logger.info(
                    "Bulk create classification -> '%s'...",
                    classification_name,
                )

                # Create the classification with all provided information:
                response = self._otcs_frontend.create_item(
                    parent_id=parent_id,
                    item_type=self._otcs_frontend.ITEM_TYPE_CLASSIFICATION,
                    item_name=classification_name,
                    item_description=description,
                    category_data=classification_category_data,
                    external_create_date=external_create_date,
                    external_modify_date=external_modify_date,
                    show_error=False,
                )
                if response is None:
                    # Potential race condition: see if the classification has been created by a concurrent thread.
                    # So we better check if the classification is there even if the create_item() call delivered
                    # a 'None' response:
                    response = self._otcs_frontend.get_node_by_parent_and_name(
                        parent_id=parent_id,
                        name=classification_name,
                    )
                classification_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
                if not classification_id:
                    self.logger.error(
                        "Failed to bulk create classification -> '%s' under parent with ID -> %s!",
                        classification_name,
                        parent_id,
                    )
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue

                # Set the categories separately:
                # if classification_category_data:
                #     response = self._otcs_frontend.update_item(
                #         node_id=classification_id,
                #         category_data=classification_category_data,
                #     )
                self.logger.info(
                    "Created classification -> '%s' with ID -> %s!",
                    classification_name,
                    classification_id,
                )
                result["create_counter"] += 1

            # end if not classification_id and "create" or "recreate" in row_operations

            # If "updates" are an requested row operation we update the existing classification with
            # fresh metadata from the payload. Additionally we check the external
            # modify date to support incremental load for content that has really
            # changed.
            # In addition we check that "delete" is not requested as otherwise it will
            # never go in elif "delete" ... below (and it does not make sense to update a classification
            # that is deleted in the next step...)
            elif (
                classification_id
                and "update" in row_operations
                and "delete" not in row_operations  # note the NOT !
                and OTCS.date_is_newer(
                    date_old=classification_modify_date,
                    date_new=external_modify_date,
                )
            ):
                # get the specific update operations given in the payload
                # if not specified we do all 4 update operations (name, description, categories and nickname)
                update_operations = bulk_classification.get(
                    "update_operations",
                    ["name", "description", "categories", "nickname"],
                )

                # If category data is in payload we substitute
                # the values with data from the current data row.
                # This is only done if "categories" update is not
                # excluded from the update_operations:
                if categories and "categories" in update_operations:
                    # Make sure the threads are not changing data structures that
                    # are shared between threads. categories is a list of dicts.
                    # list and dicts are "mutable" data structures in Python!
                    worker_categories = self.process_bulk_categories(
                        row=row,
                        index=index,
                        categories=categories,
                        replacements=replacements,
                    )
                    # classification_category_data = self.prepare_item_create_form(
                    #     parent_id=parent_id,
                    #     categories=worker_categories,
                    #     subtype=self._otcs_frontend.ITEM_TYPE_CLASSIFICATION,
                    # )
                    # Transform the payload structure into the format
                    # the OTCS REST API requires:
                    classification_category_data = self.prepare_category_data(
                        categories_payload=worker_categories,
                        source_node_id=classification_id,
                    )
                    if not classification_category_data:
                        self.logger.error(
                            "Failed to prepare the updated category data for classification -> '%s'!",
                            classification_name,
                        )
                        result["success"] = False
                        result["failure_counter"] += 1
                        continue  # for index, row in partition.iterrows()
                # end if categories
                else:
                    classification_category_data = {}

                self.logger.info(
                    "Update existing classification -> '%s' (%s) with operations -> %s...",
                    classification_old_name if classification_old_name else classification_name,
                    str(classification_id),
                    str(update_operations),
                )
                # Update the classification with all provided information:
                response = self._otcs_frontend.update_item(
                    classification_id=classification_id,
                    item_name=(classification_name if "name" in update_operations else None),
                    item_description=(description if "description" in update_operations else None),
                    category_data=(classification_category_data if "categories" in update_operations else None),
                    external_create_date=external_create_date,
                    external_modify_date=external_modify_date,
                    show_error=True,
                )
                if not response:
                    self.logger.error(
                        "Failed to update existing classification -> '%s' (%s)!",
                        (classification_old_name if classification_old_name else classification_name),
                        classification_id,
                    )
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue
                self.logger.info(
                    "Updated existing classification -> '%s' (%s).",
                    classification_name
                    if "name" in update_operations or not classification_old_name
                    else classification_old_name,
                    classification_id,
                )
                result["update_counter"] += 1

            # end elif "update" in row_operations...
            elif classification_id and "delete" in row_operations:
                # We delete with immediate purging to keep recycle bin clean
                # and to not run into issues with nicknames used in deleted items:
                response = self._otcs_frontend.delete_node(
                    node_id=classification_id,
                    purge=True,
                )
                if not response:
                    self.logger.error(
                        "Failed to delete existing classification -> '%s' (%s)!",
                        (classification_old_name if classification_old_name else classification_name),
                        classification_id,
                    )
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue
                self.logger.info(
                    "Successfully deleted existing classification -> '%s' (%s).",
                    classification_old_name if classification_old_name else classification_name,
                    classification_id,
                )
                result["delete_counter"] += 1
                classification_id = None
            # end elif classification_id and "delete" in row_operations

            # this is the plain old "it does exist and we just skip it case":
            elif classification_id:
                result["skipped_counter"] += 1
                self.logger.info(
                    "Skipped existing classification -> '%s' (%s).",
                    classification_old_name if classification_old_name else classification_name,
                    classification_id,
                )
            # this is the case where we just want to operate on existing classifications (update or delete)
            # but they do not exist:
            elif not classification_id and ("update" in row_operations or "delete" in row_operations):
                result["skipped_counter"] += 1
                self.logger.info(
                    "Skipped update/delete of non-existing classification -> '%s'.",
                    classification_old_name if classification_old_name else classification_name,
                )

            # The following code is executed for all operations
            # (create, recreate, update, delete):

            # Check if we want to set / update the nickname of the classification.
            # Precondition is we have determined a nickname, we have the classification ID
            # and the update of the nickname is either required (create, recreate)
            # or requested (update_operations include "nickname"):
            if (
                nickname
                and classification_id
                and (
                    "create" in row_operations
                    or "recreate" in row_operations
                    or ("update" in row_operations and "nickname" in update_operations)
                )
            ):
                response = self._otcs_frontend.set_node_nickname(
                    node_id=classification_id,
                    nickname=nickname,
                    show_error=True,
                )
                if not response:
                    self.logger.error(
                        "Failed to assign nickname -> '%s' to classification -> '%s'!",
                        nickname,
                        classification_name,
                    )

            # Depending on the bulk operations (create, update, delete)
            # and the related conditions it may well be that classification_id is None.
            # Only if classification ID is not none we want to count this row as success:
            if classification_id is not None:
                result["success_counter"] += 1
                # Record the classification name and ID to allow to read it from failure file
                # and speedup the process.
                result["classifications"][classification_name] = classification_id
        # end for index...

        self.logger.info("End working...")

        results.append(result)

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="start_impersonation")
    def start_impersonation(self, username: str, otcs_object: OTCS | None = None) -> bool:
        """Impersonate to a defined user.

        Args:
            username (str):
                The user to impersonate.
            otcs_object (OTCS | None, optional):
                The OTCS object to use. If not provided then the _otcs object
                in the Payload object is used.

        """

        if not otcs_object:
            otcs_object = self._otcs

        # Depending on the authentication type used with OTDS (token or ticket based)
        # the response structure is different:
        response = self._otds.impersonate_user(user_id=username)
        if not response:
            return False

        if "ticket" in response:
            otds_ticket = response.get("ticket", None)
            otcs_object.set_otds_ticket(ticket=otds_ticket)
        elif "access_token" in response:
            access_token = response.get("access_token", None)
            otcs_object.set_otds_token(token=access_token)
        else:
            self.logger.error("Impersonation response does not contain ticket or access_token!")
            return False

        otcs_object.invalidate_authentication_ticket()
        response = otcs_object.authenticate(revalidate=False)

        return bool(response)

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="stop_impersonation")
    def stop_impersonation(self, otcs_object: OTCS | None = None) -> bool:
        """Impersonate back to admin user.

        Args:
            username (str):
                The user to impersonate.
            otcs_object (OTCS | None, optional):
                The OTCS object to use. If not provided then the _otcs object
                in the Payload object is used.

        """

        if not otcs_object:
            otcs_object = self._otcs

        # Clear OTDS ticket and token:
        otcs_object.set_otds_ticket(None)
        otcs_object.set_otds_token(None)

        # Clear OTCS ticket:
        otcs_object.invalidate_authentication_ticket()

        # Reauthenticate as admin (using username / password of OTCS object):
        response = otcs_object.authenticate(revalidate=True)

        return bool(response)

    # end method definition

    @tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_nifi_flows")
    def process_nifi_flows(self, section_name: str = "nifi") -> bool:
        """Process Knowledge Discovery Nifi flows in payload and create them in Nifi.

        Args:
            section_name (str, optional):
                The name of the payload section. It can be overridden
                for cases where multiple sections of same type
                are used (e.g. the "Post" sections).
                This name is also used for the "success" status
                files written to the Admin Personal Workspace.

        Returns:
            bool:
                True, if payload has been processed without errors, False otherwise

        """

        if not self._nifi_flows:
            self.logger.info(
                "Payload section -> '%s' is empty. Skipping...",
                section_name,
            )
            return True

        # If this payload section has been processed successfully before we
        # can return True and skip processing it once more:
        if self.check_status_file(payload_section_name=section_name):
            return True

        success: bool = True

        for nifi_flow in self._nifi_flows:
            if "name" not in nifi_flow:
                self.logger.error(
                    "Knowledge Discovery Nifi flow needs a name! Skipping to next Nifi flow...",
                )
                success = False
                continue
            name = nifi_flow["name"]

            # Check if element has been disabled in payload (enabled = false).
            # In this case we skip the element:
            if not nifi_flow.get("enabled", True):
                self.logger.info(
                    "Payload for Knowledge Discovery Nifi flow -> '%s' is disabled. Skipping...",
                    name,
                )
                continue

            if "file" not in nifi_flow:
                self.logger.error(
                    "Knowledge Discovery Nifi flow -> '%s' needs a file! Skipping to next Nifi flow...", name
                )
                success = False
                continue
            filename = nifi_flow["file"]

            parameters = nifi_flow.get("parameters", [])

            if not self._otkd:
                self.logger.error("Knowledge Discovery is not initialized. Stop processing Nifi flows.")
                success = False
                break

            # Optional layout positions of the flow:
            position_x = nifi_flow.get("position_x", 0.0)
            position_y = nifi_flow.get("position_y", 0.0)
            start = nifi_flow.get("start", False)

            self.logger.info("Processing Knowledge Discovery Nifi flow -> '%s'...", name)

            existing = self._otkd.get_process_group_by_name(name=name)
            if existing:
                self.logger.warning("Nifi flow -> '%s' does already exist. Updating parameters only...", name)
                # We better don't start existing flows!? Otherwise this may produce errors.
                start = False
            else:
                response = self._otkd.upload_process_group(
                    file_path=filename, name=name, position_x=position_x, position_y=position_y
                )
                if not response:
                    self.logger.error("Failed to upload new Nifi flow -> '%s' for Knowledge Discovery!", name)
                    success = False
                    continue
                self.logger.info("Sucessfully uploaded new Nifi flow -> '%s' for Knowledge Discovery.", name)

            for parameter in parameters:
                component = parameter.get("component", None)
                if not component:
                    self.logger.error("Missing component in parameter of Nifi flow -> '%s'!", name)
                    success = False
                    continue
                parameter_name = parameter.get("name", None)
                if not parameter_name:
                    self.logger.error(
                        "Missing name in parameter of Nifi flow -> '%s', component -> '%s'!", name, component
                    )
                    success = False
                    continue
                parameter_description = parameter.get("description", "")
                parameter_value = parameter.get("value", None)
                if not parameter_value:
                    self.logger.error(
                        "Missing value in parameter of Nifi flow -> '%s', component -> '%s'!", name, component
                    )
                    success = False
                    continue
                parameter_sensitive = parameter.get("sensitive", False)

                response = self._otkd.update_parameter(
                    component=component,
                    parameter=parameter_name,
                    value=parameter_value,
                    sensitive=parameter_sensitive,
                    description=parameter_description,
                )
                if not response:
                    self.logger.error("Failed to update parameter -> '%s' of Nifi flow -> '%s'!", parameter_name, name)
                    success = False
                    continue
                self.logger.info(
                    "Successfully updated parameter -> '%s' of component -> '%s' in Nifi flow -> '%s' to value -> '%s'.",
                    parameter_name,
                    component,
                    name,
                    parameter_value if not parameter_sensitive else "<sensitive>",
                )
            # end for parameter in parameters:
            if start:
                response = self._otkd.start_all_processors(name=name)
                if response:
                    self.logger.info("Successfully started Nifi flow -> '%s'.", name)
                else:
                    self.logger.error("Failed to start Nifi flow -> '%s'!", name)
                    success = False

                response = self._otkd.set_controller_services_state(name=name, state="ENABLED")
                if response:
                    self.logger.info("Successfully enabled Nifi Controller Services for Nifi flow -> '%s'.", name)
                else:
                    self.logger.error("Failed to enable Nifi Controller Services for Nifi flow -> '%s'!", name)
                    success = False

            else:
                self.logger.info("Don't (re)start Nifi flow -> '%s'.", name)
        # end for nifi_flow in self._nifi_flows:

        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=self._nifi_flows,
        )

        return success

__init__(payload_source, custom_settings_dir, k8s_object, otds_object, otac_object, otcs_backend_object, otcs_frontend_object, otcs_restart_callback, otiv_object, otpd_object, m365_object, core_share_object, placeholder_values, log_header_callback, browser_headless=True, stop_on_error=False, aviator_enabled=False, upload_status_files=True, otawp_object=None, otca_object=None, otkd_object=None, avts_object=None, logger=default_logger)

Initialize the Payload object.

Parameters:

Name	Type	Description	Default
payload_source	str	The path or URL to payload source file.	required
custom_settings_dir	str	Provide a custom settings directory.	required
k8s_object	K8s | None	The Kubernetes object. Pass None if deployment is not running in Kubernetes.	required
otds_object	OTDS	The OTDS object. This is mandatory.	required
otac_object	OTAC | None	The optional OTAC object (Archive Center). Pass None if Archive Center is not part of the deployment.	required
otcs_backend_object	OTCS	The OTCS backend object.	required
otcs_frontend_object	OTCS	The OTCS frontend object.	required
otcs_restart_callback	Callable	A function to call if OTCS service needs a restart.	required
otiv_object	OTIV | None	The OTIV object (Intelligent Viewing). Pass None if Intelligent Viewing is not part of the deployment.	required
otpd_object	OTPD | None	The OTPD object (PowerDocs). Pass None if PowerDocs is not part of the deployment.	required
m365_object	object	The M365 object to talk to Microsoft Graph API.	required
core_share_object	CoreShare | None	The Core Share object.	required
placeholder_values	dict	A dictionary of placeholder values to be replaced in admin settings.	required
log_header_callback	Callable	Method to print a section break / header line into the log.	required
browser_headless	bool	If true, the Browser for the Automation will be started in Headless mode (default)	True
stop_on_error	bool	This flag controls if transport deployment should stop if a transport deployment in OTCS fails.	False
aviator_enabled	bool	Flag that indicates whether or not the Content Aviator is enabled.	False
upload_status_files	bool	Whether or not status file should be uploaded to the peronal workspace of the admin user in Content Server.	True
otawp_object	OTAWP	An optional AppWorks Platform object.	None
otca_object	OTCA	An optional Content Aviator object.	None
otkd_object	OTKD	An optional Knowledge Discovery object.	None
avts_object	AVTS	An optional Aviator Search object.	None
logger	Logger	The logging object to use for all log messages. Defaults to default_logger.	default_logger

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

def __init__(
    self,
    payload_source: str,
    custom_settings_dir: str,
    k8s_object: K8s | None,
    otds_object: OTDS,
    otac_object: OTAC | None,
    otcs_backend_object: OTCS,
    otcs_frontend_object: OTCS,
    otcs_restart_callback: Callable,
    otiv_object: OTIV | None,
    otpd_object: OTPD | None,
    m365_object: M365 | None,
    core_share_object: CoreShare | None,
    placeholder_values: dict,
    log_header_callback: Callable,
    browser_headless: bool = True,
    stop_on_error: bool = False,
    aviator_enabled: bool = False,
    upload_status_files: bool = True,
    otawp_object: OTAWP | None = None,
    otca_object: OTCA | None = None,
    otkd_object: OTKD | None = None,
    avts_object: AVTS | None = None,
    logger: logging.Logger = default_logger,
) -> None:
    """Initialize the Payload object.

    Args:
        payload_source (str):
            The path or URL to payload source file.
        custom_settings_dir (str):
            Provide a custom settings directory.
        k8s_object (K8s | None):
            The Kubernetes object.
            Pass None if deployment is not running in Kubernetes.
        otds_object (OTDS):
            The OTDS object. This is mandatory.
        otac_object (OTAC | None):
            The optional OTAC object (Archive Center).
            Pass None if Archive Center is not part of the deployment.
        otcs_backend_object (OTCS):
            The OTCS backend object.
        otcs_frontend_object (OTCS):
            The OTCS frontend object.
        otcs_restart_callback (Callable):
            A function to call if OTCS service needs a restart.
        otiv_object (OTIV | None):
            The OTIV object (Intelligent Viewing).
            Pass None if Intelligent Viewing is not part of the deployment.
        otpd_object (OTPD | None):
            The OTPD object (PowerDocs).
            Pass None if PowerDocs is not part of the deployment.
        m365_object (object):
            The M365 object to talk to Microsoft Graph API.
        core_share_object (CoreShare | None):
            The Core Share object.
        placeholder_values (dict):
            A dictionary of placeholder values to be replaced in admin settings.
        log_header_callback:
            Method to print a section break / header line into the log.
        browser_headless (bool):
            If true, the Browser for the Automation will be started in Headless mode (default)
        stop_on_error (bool):
            This flag controls if transport deployment should stop
            if a transport deployment in OTCS fails.
        aviator_enabled (bool):
            Flag that indicates whether or not the Content Aviator is enabled.
        upload_status_files (bool, optional):
            Whether or not status file should be uploaded to the peronal workspace
            of the admin user in Content Server.
        otawp_object (OTAWP):
            An optional AppWorks Platform object.
        otca_object (OTCA):
            An optional Content Aviator object.
        otkd_object (OTKD):
            An optional Knowledge Discovery object.
        avts_object (AVTS):
            An optional Aviator Search object.
        logger (logging.Logger, optional):
            The logging object to use for all log messages. Defaults to default_logger.

    """

    self.logger = logger.getChild("payload")
    for logfilter in logger.filters:
        self.logger.addFilter(logfilter)

    self._stop_on_error = stop_on_error
    self._payload_source = payload_source
    self._k8s = k8s_object
    self._otds = otds_object
    self._otac = otac_object
    self._otcs = otcs_backend_object
    self._otcs_backend = otcs_backend_object
    self._otcs_frontend = otcs_frontend_object
    self._otiv = otiv_object
    self._otpd = otpd_object
    self._m365 = m365_object
    self._core_share = core_share_object
    # The SAP, SuccessFactors and Salesforce objects only exists after external systems have been processed
    self._sap = None
    self._successfactors = None
    self._salesforce = None
    self._servicenow = None
    self._guidewire_policy_center = None
    self._guidewire_claims_center = None
    self._otmm = None
    self._otcs_source = None
    self._pht = None  # the OpenText prodcut hierarchy
    self._nhc = None  # National Hurricane Center
    self._otca = otca_object  # Content Aviator
    self._otkd = otkd_object  # Knowledge Discovery
    self._avts = avts_object  # Aviator Search
    self._browser_headless = browser_headless
    self._custom_settings_dir = custom_settings_dir
    self._placeholder_values = placeholder_values
    self._otcs_restart_callback = otcs_restart_callback
    self._log_header_callback = log_header_callback
    self._aviator_enabled = aviator_enabled
    self._http_object = HTTP(logger=self.logger)
    self.upload_status_files = upload_status_files
    self._otawp = otawp_object

add_transport_extractions(extractions)

Determine the number of extrations.

Safe them in a global list self._transport_extractions.

Parameters:

Name	Type	Description	Default
extractions	list	A list of extractions from a single transport package.	required

Returns:

Name	Type	Description
int	int	THE number of extractions that have actually extracted data.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="add_transport_extractions")
def add_transport_extractions(self, extractions: list) -> int:
    """Determine the number of extrations.

    Safe them in a global list self._transport_extractions.

    Args:
        extractions (list):
            A list of extractions from a single transport package.

    Returns:
        int:
            THE number of extractions that have actually extracted data.

    """

    counter = 0
    for extraction in extractions:
        if extraction.get("enabled", True) and "data" in extraction:
            self._transport_extractions.append(extraction)
            counter += 1
    self.logger.info("Added -> %s transport extractions.", str(counter))

    return counter

check_external_system(external_system)

Check if external system is reachable.

Parameters:

Name	Type	Description	Default
external_system	dict	The payload data structure of external system. We assume here that sanity check for valid data is already done before.	required

Returns:

Name	Type	Description
bool	bool	True = system is reachable, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="check_external_system")
def check_external_system(self, external_system: dict) -> bool:
    """Check if external system is reachable.

    Args:
        external_system (dict):
            The payload data structure of external system.
            We assume here that sanity check for valid data is already done before.

    Returns:
        bool:
            True = system is reachable, False otherwise.

    """

    as_url = external_system["as_url"]

    # Extract the hostname:
    external_system_hostname = urlparse(as_url).hostname
    # Write this information back into the data structure:
    external_system["external_system_hostname"] = external_system_hostname
    # Extract the port:
    external_system_port = urlparse(as_url).port if urlparse(as_url).port else 80
    # Write this information back into the data structure:
    external_system["external_system_port"] = external_system_port

    if self._http_object.check_host_reachable(
        hostname=external_system_hostname,
        port=external_system_port,
    ):
        self.logger.info(
            "Mark external system -> '%s' as reachable for later workspace creation and user synchronization...",
            external_system["external_system_name"],
        )
        external_system["reachable"] = True
        return True
    else:
        external_system["reachable"] = False
        return False

check_status_file(payload_section_name, payload_specific=True, prefix='success_')

Check if the payload section has been processed before.

This is done by checking the existance of a text file in the Admin Personal workspace in Content Server with the name of the payload section.

Parameters:

Name	Type	Description	Default
payload_section_name	str	The name of the payload section. This is used to construct the file name	required
payload_specific	bool	Whether or not the success should be specific for each payload file or if success is "global" - like for the deletion of the existing M365 teams (which we don't want to execute per payload file)	True
prefix	str	The prefix of the file. Typically, either "success_" or "failure_".	'success_'

Returns:

Name	Type	Description
bool	bool	True if the payload has been processed successfully before, False otherwise

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="check_status_file")
def check_status_file(
    self,
    payload_section_name: str,
    payload_specific: bool = True,
    prefix: str = "success_",
) -> bool:
    """Check if the payload section has been processed before.

    This is done by checking the existance of a text file in the Admin Personal
    workspace in Content Server with the name of the payload section.

    Args:
        payload_section_name (str):
            The name of the payload section. This
            is used to construct the file name
        payload_specific (bool, optional):
            Whether or not the success should be specific for
            each payload file or if success is "global" - like for the deletion
            of the existing M365 teams (which we don't want to execute per
            payload file)
        prefix (str, optional):
            The prefix of the file. Typically, either "success_" or "failure_".

    Returns:
        bool:
            True if the payload has been processed successfully before, False otherwise

    """

    message = "successfully" if prefix.startswith("success_") else "with failures"

    self.logger.info(
        "Check if payload section -> '%s' has been processed %s before...",
        payload_section_name,
        message,
    )

    while not self._otcs.is_ready():
        self.logger.info(
            "OTCS is not ready. Cannot check status file for -> '%s' in OTCS. Waiting 30 seconds and retry...",
            payload_section_name,
        )
        time.sleep(30)

    response = self._otcs.get_node_by_volume_and_path(
        volume_type=self._otcs.VOLUME_TYPE_PERSONAL_WORKSPACE,
    )  # write to Personal Workspace of Admin
    target_folder_id = self._otcs.get_result_value(response=response, key="id")
    if not target_folder_id:
        target_folder_id = 2004  # use Personal Workspace of Admin as fallback

    file_name = self.get_status_file_name(
        payload_section_name=payload_section_name,
        payload_specific=payload_specific,
        prefix=prefix,
    )

    status_document = self._otcs.get_node_by_parent_and_name(
        parent_id=int(target_folder_id),
        name=file_name,
        show_error=False,
    )
    if status_document and status_document.get("results", []):
        name = self._otcs.get_result_value(response=status_document, key="name")
        if name == file_name:
            self.logger.info(
                "Payload section -> '%s' has been processed %s before. %s...",
                payload_section_name,
                message,
                "Skipping" if prefix.startswith("success_") else "Retrying",
            )
            return True
    self.logger.info(
        "Payload section -> '%s' has not been processed %s before. Processing...",
        payload_section_name,
        message,
    )
    return False

cleanup_all_teams_m365(section_name='teamsM365Cleanup')

Delete Microsoft Teams that are left-overs from former deployments.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'teamsM365Cleanup'

Returns:

Name	Type	Description
bool	bool	True, if teams have been deleted, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="cleanup_all_teams_m365")
def cleanup_all_teams_m365(self, section_name: str = "teamsM365Cleanup") -> bool:
    """Delete Microsoft Teams that are left-overs from former deployments.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if teams have been deleted, False otherwise.

    """

    if not isinstance(self._m365, M365):
        self.logger.error(
            "Microsoft 365 connection not setup properly. Skipping payload section -> '%s'...",
            section_name,
        )
        return False

    # We want this cleanup to only run once even if we have
    # multiple payload files - so we pass payload_specific=False here:
    if self.check_status_file(
        payload_section_name=section_name,
        payload_specific=False,
    ):
        self.logger.info(
            "Payload section -> '%s' has been processed successfully before. Skip cleanup of M365 teams...",
            section_name,
        )
        return True

    self.logger.info("Processing payload section -> '%s'...", section_name)

    # We don't want to delete MS Teams that are matching the regular OTCS Group Names (like "Sales")
    exception_list = self.get_all_group_names()

    # These are the patterns that each MS Teams needs to match at least one of to be deleted
    # Pattern 1: all MS teams with a name that has a number in brackets, like "(1234)"
    # Pattern 2: all MS Teams with a name that starts with a number followed by a space,
    #            followed by a "-" and followed by another space
    # Pattern 3: all MS Teams with a name that starts with "WS" and a 1-4 digit number
    #            (these are the workspaces for Purchase Contracts generated for Intelligent Filing)
    # Pattern 4: all MS Teams with a name that ends with a 1-2 character + a number in brackets, like (US-1000)
    #            this is a specialization of pattern 1
    # Pattern 5: remove the teams that are created for the dummy copy&paste template for the
    #            Intelligent Filing workspaces
    pattern_list = [
        r"\(\d+\)",
        r"\d+\s-\s",
        r"^WS\d{1,4}$",
        r"^.+?\s\(.{1,2}-\d+\)$",
        r"Purchase\sContract\s\(Template\sfor\sIntelligent\sFiling\)",
        r"^OpenText.*$",
        r"^P-100.*$",
        r"^OILRIG.*$",
        r"^AGILUM.*$",
        r"^HD-102T.*$",
        r"^SG325A.*$",
        r"^[A-Za-z0-9]{18} - .*$",  # delete teams that start with the typical Salesforce IDs (e.g. opportunities)
        r".*\s\([A-Z]{3,4}\)$",  # delete stale Locations from NTSB scenario
    ]

    result = self._m365.delete_all_teams(
        exception_list=exception_list,
        pattern_list=pattern_list,
    )

    # We want this cleanup to only run once even if we have
    # multiple payload files - so we pass payload_specific=False here:
    self.write_status_file(
        success=True,
        payload_section_name=section_name,
        payload_section=exception_list + pattern_list,
        payload_specific=False,
    )

    return result

cleanup_stale_teams_m365(workspace_types)

Delete Microsoft Teams that are left-overs from former deployments.

This method is currently not used.

Parameters:

Name	Type	Description	Default
workspace_types	list	The list of all workspace types.	required

Returns:

Name	Type	Description
bool	bool	True if successful, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="cleanup_stale_teams_m365")
def cleanup_stale_teams_m365(self, workspace_types: list) -> bool:
    """Delete Microsoft Teams that are left-overs from former deployments.

    This method is currently not used.

    Args:
        workspace_types (list):
            The list of all workspace types.

    Returns:
        bool:
            True if successful, False otherwise.

    """

    if not isinstance(self._m365, M365):
        self.logger.error(
            "Microsoft 365 connection not setup properly. Skipping cleanup...",
        )
        return False

    if workspace_types == []:
        self.logger.error("Empty workspace type list!")
        return False

    for workspace_type in workspace_types:
        if "name" not in workspace_type:
            self.logger.error(
                "Workspace type -> '%s' does not have a name. Skipping...",
                workspace_type,
            )
            continue

        workspace_instances = self._otcs.get_workspace_instances_iterator(
            type_name=workspace_type["name"],
        )
        for workspace in workspace_instances:
            workspace = workspace.get("data").get("properties")

            workspace_name = workspace["name"]
            self.logger.info(
                "Check if stale Microsoft 365 Teams -> '%s' exist...",
                workspace_name,
            )
            self._m365.delete_teams(name=workspace_name)

    return True

cleanup_value(cleanup_value, regex_list, upper=False, lower=False)

Cleanup field values based on regular expressions.

Parameters:

Name	Type	Description	Default
cleanup_value	str	The string to clean up.	required
regex_list	list	A list of regular expressions to apply.	required
upper	bool	If True, convert name to upper case letters.	False
lower	bool	If true, convert name to lower case letters.	False

Returns:

Name	Type	Description
str	str	The cleaned string.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="cleanup_value")
def cleanup_value(
    self,
    cleanup_value: str,
    regex_list: list,
    upper: bool = False,
    lower: bool = False,
) -> str:
    """Cleanup field values based on regular expressions.

    Args:
        cleanup_value (str):
            The string to clean up.
        regex_list (list):
            A list of regular expressions to apply.
        upper (bool, optional):
            If True, convert name to upper case letters.
        lower (bool, optional):
            If true, convert name to lower case letters.

    Returns:
        str:
            The cleaned string.

    """

    cleaned_string = cleanup_value

    if upper:
        cleaned_string = cleaned_string.upper()
    if lower:
        cleaned_string = cleaned_string.lower()

    if regex_list:
        try:
            for regex in regex_list:
                # We use the pipe symbol to divide patterns from replacements
                # this is a short-hand syntax to keep it simple. If there's
                # no pipe in regex string and than we remove the pattern
                # from the string
                parts = regex.split("|")
                pattern = parts[0]
                replacement = ""  # Treat replacement as empty if no pipe specified
                if len(parts) > 1:
                    pattern = r"\b" + pattern + r"\b"  # Match whole words only
                    replacement = parts[1]
                cleaned_string = re.sub(pattern, replacement, cleaned_string)
        except re.error:
            self.logger.error(
                "Invalid regular expression pattern -> %s",
                pattern,
            )
            return cleanup_value

    return cleaned_string

construct_file_name(path, download_name, download_name_wildcards=False, file_extension='')

Construct the file name of the document.

This considers the path given in the download_dir, potential wildcards in the download_name variable and the file extensions provided (if any).

Parameters:

Name	Type	Description	Default
path	str	The base base in the file system. Placeholders need to be resolved before calling this method.	required
download_name	str	The filenname of document in the file system. This may include wildcards like '.pdf' or 'en/.txt' or '*/.pfd'. Then the actual download file name is determined by a directory traversal using the 'path' parameter.	required
download_name_wildcards	bool	Defines whether or not wildcards should be replaced in the download name. If True this walks through the given path and try to find the file in the filesystem that matches the wildcard pattern.	False
file_extension	str	The file extension - typically 3 letters like 'pdf'. Defaults to "".	''

Returns:

Name	Type	Description
str	str | None	The file name that is used to find the document in the filesystem (or if it does not yet exist) the name that should be used for download. This can be None

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="construct_file_name")
def construct_file_name(
    self,
    path: str,
    download_name: str,
    download_name_wildcards: bool = False,
    file_extension: str = "",
) -> str | None:
    """Construct the file name of the document.

    This considers the path given in the download_dir, potential wildcards in the
    download_name variable and the file extensions provided (if any).

    Args:
        path (str):
            The base base in the file system. Placeholders need to be resolved before
            calling this method.
        download_name (str):
            The filenname of document in the file system. This may include wildcards
            like '*.pdf' or 'en/*.txt' or '**/*.pfd'. Then the actual download
            file name is determined by a directory traversal using the 'path' parameter.
        download_name_wildcards (bool, optional):
            Defines whether or not wildcards should be replaced in the download name.
            If True this walks through the given path and try to find the file in the
            filesystem that matches the wildcard pattern.
        file_extension (str, optional):
            The file extension - typically 3 letters like 'pdf'. Defaults to "".

    Returns:
        str:
            The file name that is used to find the document in the filesystem (or if it does not yet exist)
            the name that should be used for download. This can be None

    """

    if not download_name:
        self.logger.error("Download name is missing. Cannot construct file name!")
        return None

    if path:
        # Normalize on Linux style path separators. Python is able to handle this also under Windows
        path = path.replace("\\", "/")
        if os.path.exists(path):
            # if we have a path specified it should be a directory and not a file!
            # The isfile() test is only working if the file already exist. It cannot
            # tell the difference between a folder and a file otherwise!
            if os.path.isfile(path):
                self.logger.warning(
                    "Download directory -> '%s' is pointing to an existing file and not a directory - please check the 'download_dir' variable in payload! Stripping path...",
                    path,
                )
                path = os.path.dirname(path)
                self.logger.info("Stripped path -> '%s'...", path)
        else:
            # if we have a path specified but it does not exist, then create it.
            try:
                os.makedirs(path)
            except (OSError, PermissionError):
                self.logger.error(
                    "Cannot create folder -> '%s' in file system!",
                    path,
                )
                return None
        if not path.endswith("/"):
            path += "/"  # slashes are save in Linux and Windows. Python handles this correctly

    # If we have a file extension and the given download does
    # not yet ends with it, we add it to the download name:
    if file_extension:
        if not file_extension.startswith("."):
            file_extension = "." + file_extension
        if not download_name.endswith(file_extension):
            download_name += file_extension

    # Also find files with wildcards in 'download_name' if requested
    # by download_name_wildcards == True.
    # IMPORTANT: this only matches ONE file as this method
    # also processes only ONE file at a time:
    if download_name_wildcards and any(char in download_name for char in "*?[]"):
        if not path:
            self.logger.error(
                "Download name includes wildcards but no (base) path is specified in payload (download_dir missing)! Cannot construct file name!",
            )
            return None
        file_name = None
        # Traverse the directory tree starting at 'path', looking for all files and subdirectories
        for _, _, tmpfiles in os.walk(path):
            # For each file found in the current directory:
            for file_data in tmpfiles:
                # Check if the current file name matches the 'download_name' pattern using wildcards (fnmatch)
                if fnmatch.fnmatch(file_data, download_name):
                    self.logger.debug(
                        "Found file name -> '%s' that is matching download name pattern -> '%s'",
                        file_data,
                        download_name,
                    )
                    # Construct the full file path by concatenating the base 'path' and the file name
                    file_name = path + file_data
                    # If we found a match we stop the (inner loop)
                    break
    # end if download_name_wildcards and any(char in download_name for char in "*?[]")
    else:
        # We have a normal filename without wildcards:
        file_name = path + download_name

    return file_name

determine_group_id(group)

Determine the id of a group - either from payload or from OTCS.

If the group is found in OTCS write back the ID into the payload.

Parameters:

Name	Type	Description	Default
group	dict	The group payload element.	required

Returns:

Name	Type	Description
int	int	group ID

Side Effects

the group items are modified by adding an "id" dict element that includes the technical ID of the group in Content Server.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_group_id")
def determine_group_id(self, group: dict) -> int:
    """Determine the id of a group - either from payload or from OTCS.

    If the group is found in OTCS write back the ID into the payload.

    Args:
        group (dict):
            The group payload element.

    Returns:
        int: group ID

    Side Effects:
        the group items are modified by adding an "id" dict element that
        includes the technical ID of the group in Content Server.

    """

    # Is the ID already known in payload? (if determined before)
    if "id" in group:
        return group["id"]

    if "name" not in group:
        self.logger.error("Group needs a name to lookup the ID!")
        return 0
    group_name = group["name"]

    existing_groups = self._otcs.get_group(name=group_name)
    # We use the lookup method here as get_group() could deliver more
    # then 1 result element (in edge cases):
    existing_group_id = self._otcs.lookup_result_value(
        response=existing_groups,
        key="name",
        value=group_name,
        return_key="id",
    )

    # Have we found an exact match?
    if existing_group_id:
        self.logger.debug(
            "Found existing group -> '%s' with ID -> %s. Update ID in payload...",
            group_name,
            existing_group_id,
        )
        # Write ID back into the payload:
        group["id"] = existing_group_id
        return group["id"]
    else:
        self.logger.debug(
            "Cannot find an existing group -> '%s'",
            group_name,
        )
        return 0

determine_group_id_core_share(group)

Determine the id of a Core Share group.

This can either be derived from payload or from Core Share directly.

Parameters:

Name	Type	Description	Default
group	dict	The payload dictionary of the group.	required

Returns:

Type	Description
str | None	str | None: Core Share Group ID or None.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_group_id_core_share")
def determine_group_id_core_share(self, group: dict) -> str | None:
    """Determine the id of a Core Share group.

    This can either be derived from payload or from Core Share directly.

    Args:
        group (dict):
            The payload dictionary of the group.

    Returns:
        str | None:
            Core Share Group ID or None.

    """

    # Is the ID already known in payload? (if determined before)
    if "core_share_id" in group:
        return group["core_share_id"]

    if "name" not in group:
        self.logger.error("Group needs a name to lookup the Core Share ID!")
        return None

    if not isinstance(self._core_share, CoreShare):
        self.logger.error(
            "Core Share connection not setup properly!",
        )
        return None

    core_share_group = self._core_share.get_group_by_name(name=group["name"])
    core_share_group_id = self._core_share.get_result_value(
        response=core_share_group,
        key="id",
    )

    # Have we found the group?
    if core_share_group_id:
        self.logger.debug(
            "Found existing Core Share group -> '%s' with ID -> %s. Update m365_id in payload...",
            group["name"],
            core_share_group_id,
        )
        # Write ID back into the payload:
        group["core_share_id"] = core_share_group_id
        return group["core_share_id"]
    else:
        self.logger.debug(
            "Cannot find an existing Core Share group -> '%s'",
            group["name"],
        )
        return None

determine_group_id_m365(group)

Determine the id of a M365 group - either from payload or from M365 via Graph API.

If the group is found in M365 write back the M365 group ID into the payload.

Parameters:

Name	Type	Description	Default
group	dict	The group payload element.	required

Returns:

Type	Description
str | None	str | None: M365 group ID or None if the group is not found.

Side Effects

The group items are modified by adding an "m365_id" dict item that includes the technical ID of the group in Microsoft 365.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_group_id_m365")
def determine_group_id_m365(self, group: dict) -> str | None:
    """Determine the id of a M365 group - either from payload or from M365 via Graph API.

    If the group is found in M365 write back the M365 group ID into the payload.

    Args:
        group (dict):
            The group payload element.

    Returns:
        str | None:
            M365 group ID or None if the group is not found.

    Side Effects:
        The group items are modified by adding an "m365_id" dict item that
        includes the technical ID of the group in Microsoft 365.

    """

    # is the payload already updated with the M365 group ID?
    if "m365_id" in group:
        return group["m365_id"]

    if "name" not in group:
        self.logger.error("Group needs a name to lookup the M365 group ID!")
        return None
    group_name = group["name"]

    existing_group = self._m365.get_group(group_name=group_name)
    existing_group_id = self._m365.get_result_value(
        response=existing_group,
        key="id",
    )
    if existing_group_id:
        self.logger.debug(
            "Found existing Microsoft 365 group -> '%s' with ID -> %s. Update m365_id in payload...",
            group_name,
            existing_group_id,
        )
        # write back the M365 user ID into the payload
        group["m365_id"] = existing_group_id
        return group["m365_id"]
    else:
        self.logger.debug(
            "Cannot find an existing M365 group -> '%s'",
            group_name,
        )
        return None

determine_user_id(user)

Determine the id of a user - either from payload or from OTCS.

If the user is found in OTCS write back the ID into the payload.

Parameters:

Name	Type	Description	Default
user	dict	The user payload element.	required

Returns:

Name	Type	Description
int	int	The user ID in OTCS.

Side Effects

The user items are modified by adding an "id" dict element that includes the technical ID of the user in Content Server

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_user_id")
def determine_user_id(self, user: dict) -> int:
    """Determine the id of a user - either from payload or from OTCS.

    If the user is found in OTCS write back the ID into the payload.

    Args:
        user (dict):
            The user payload element.

    Returns:
        int:
            The user ID in OTCS.

    Side Effects:
        The user items are modified by adding an "id" dict element that
        includes the technical ID of the user in Content Server

    """

    # Is the ID already known in payload? (if determined before)
    if "id" in user:
        return user["id"]

    user_name = user.get("name")
    if not user_name:
        self.logger.error("User needs a login name to lookup the ID!")
        return 0

    user_type = 17 if user.get("type", "User") == "ServiceUser" else 0

    response = self._otcs.get_user(name=user_name, user_type=user_type)

    # We use the lookup method here as get_user() could deliver more
    # then 1 result element (in edge cases):
    user_id = self._otcs.lookup_result_value(
        response=response,
        key="name",
        value=user_name,
        return_key="id",
    )

    # Have we found an exact match?
    if user_id:
        # Write ID back into the payload
        user["id"] = user_id
        return user["id"]
    else:
        self.logger.debug(
            "Cannot find an existing user -> '%s'!",
            user_name,
        )
        return 0

determine_user_id_core_share(user)

Determine the ID of a Core Share user.

The ID is either taken from payload or from Core Share directly.

Parameters:

Name	Type	Description	Default
user	dict	The payload dictionary of the user.	required

Returns:

Type	Description
str | None	str | None: Core Share User ID or None.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_user_id_core_share")
def determine_user_id_core_share(self, user: dict) -> str | None:
    """Determine the ID of a Core Share user.

    The ID is either taken from payload or from Core Share directly.

    Args:
        user (dict):
            The payload dictionary of the user.

    Returns:
        str | None:
            Core Share User ID or None.

    """

    # Is the ID already known in payload? (if determined before)
    if "core_share_id" in user:
        return user["core_share_id"]

    if not isinstance(self._core_share, CoreShare):
        self.logger.error(
            "Core Share connection not setup properly!",
        )
        return False

    core_share_user_id = None

    # Next try to lookup ID via the email address:
    if "email" in user:
        core_share_user = self._core_share.get_user_by_email(email=user["email"])
        core_share_user_id = self._core_share.get_result_value(
            response=core_share_user,
            key="id",
        )

    # Last resort is to lookup the ID via firstname + lastname.
    # This is handy in case the Email has changed:
    if not core_share_user_id and "lastname" in user and "firstname" in user:
        core_share_user = self._core_share.get_user_by_name(
            first_name=user["firstname"],
            last_name=user["lastname"],
        )
        core_share_user_id = self._core_share.get_result_value(
            response=core_share_user,
            key="id",
        )

    # Have we found the user?
    if core_share_user_id:
        # Write ID back into the payload:
        user["core_share_id"] = core_share_user_id
        return user["core_share_id"]
    else:
        if "email" in user:
            self.logger.debug(
                "Did not find an existing Core Share user with email -> '%s'",
                user["email"],
            )
        else:
            self.logger.debug(
                "Cannot find an existing Core Share user -> '%s %s'",
                user.get("firstname"),
                user.get("lastname"),
            )
        return None

determine_user_id_m365(user)

Determine the id of a M365 user - either from payload or from M365 via Graph API.

If the user is found in M365 write back the M365 user ID into the payload.

Parameters:

Name	Type	Description	Default
user	dict	The user payload element.	required

Returns:

Type	Description
str | None	str | None: The M365 user ID or None if the user is not found.

Side Effects

the user items are modified by adding an "m365_id" dict element that includes the technical ID of the user in Microsoft 365

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_user_id_m365")
def determine_user_id_m365(self, user: dict) -> str | None:
    """Determine the id of a M365 user - either from payload or from M365 via Graph API.

    If the user is found in M365 write back the M365 user ID into the payload.

    Args:
        user (dict):
            The user payload element.

    Returns:
        str | None:
            The M365 user ID or None if the user is not found.

    Side Effects:
        the user items are modified by adding an "m365_id" dict element that
        includes the technical ID of the user in Microsoft 365

    """

    # is the payload already updated with the M365 user ID?
    if "m365_id" in user:
        return user["m365_id"]

    if "name" not in user:
        self.logger.error("User needs a login name to lookup the M365 user ID!")
        return None
    user_name = user["name"]

    m365_user_name = user_name + "@" + self._m365.config()["domain"]
    existing_user = self._m365.get_user(user_email=m365_user_name)
    if existing_user:
        self.logger.debug(
            "Found existing Microsoft 365 user -> '%s' (%s) with ID -> %s. Update m365_id in payload...",
            existing_user["displayName"],
            existing_user["userPrincipalName"],
            existing_user["id"],
        )
        # write back the M365 user ID into the payload
        user["m365_id"] = existing_user["id"]
        return user["m365_id"]
    else:
        self.logger.debug(
            "Did not find an existing M365 user -> '%s'",
            user_name,
        )
        return None

determine_workspace_id(workspace)

Determine the node ID of a workspace - either from payload or from OTCS.

Parameters:

Name	Type	Description	Default
workspace	dict	The workspace payload element.	required

Returns:

Name	Type	Description
int	int	The workspace Node ID.

Side Effects

The workspace items are modified by adding an "nodeId" dict element that includes the node ID of the workspace in Content Server.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_workspace_id")
def determine_workspace_id(self, workspace: dict) -> int:
    """Determine the node ID of a workspace - either from payload or from OTCS.

    Args:
        workspace (dict):
            The workspace payload element.

    Returns:
        int:
            The workspace Node ID.

    Side Effects:
        The workspace items are modified by adding an "nodeId" dict element that
        includes the node ID of the workspace in Content Server.

    """

    if "nodeId" in workspace:
        return workspace["nodeId"]

    response = self._otcs.get_workspace_by_type_and_name(
        type_name=workspace["type_name"],
        name=workspace["name"],
    )
    workspace_id = self._otcs.get_result_value(response=response, key="id")
    if workspace_id:
        if not isinstance(workspace_id, int):
            self.logger.warning("Converting workspace ID -> %s to integer...", str(workspace_id))
            workspace_id = int(workspace_id)
        # Write nodeID back into the payload
        workspace["nodeId"] = workspace_id
        return workspace_id
    else:
        self.logger.info(
            "Workspace of type -> '%s' and name -> '%s' does not yet exist.",
            workspace["type_name"],
            workspace["name"],
        )
        return 0

determine_workspace_type_and_template_id(workspace_type_name, workspace_template_name='')

Determine the IDs of type and template based on the provided names.

This depends on the self._workspace_types list to be up to date (see process_workspace_types()).

Parameters:

Name	Type	Description	Default
workspace_type_name	str	Name of the workspace type.	required
workspace_template_name	str	Name of the workspace template. Defaults to "".	''

Returns:

Type	Description
tuple[int | None, int | None]	tuple[int, int]: IDs of the workspace type (first) and workspace template (second).

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="determine_workspace_type_and_template_id")
def determine_workspace_type_and_template_id(
    self,
    workspace_type_name: str,
    workspace_template_name: str = "",
) -> tuple[int | None, int | None]:
    """Determine the IDs of type and template based on the provided names.

    This depends on the self._workspace_types list to be up to date
    (see process_workspace_types()).

    Args:
        workspace_type_name (str):
            Name of the workspace type.
        workspace_template_name (str, optional):
            Name of the workspace template. Defaults to "".

    Returns:
        tuple[int, int]:
            IDs of the workspace type (first) and workspace template (second).

    """

    # Check if the customizer has initialized the workspace type list
    if not self._workspace_types:
        self.logger.error(
            "Workspace type list is not initialized! This should never happen!",
        )
        return (None, None)

    # Find the workspace type with the name given in the payload:
    workspace_type = next(
        (item for item in self._workspace_types if item["name"] == workspace_type_name),
        None,
    )
    if workspace_type is None:
        self.logger.error(
            "Workspace type -> '%s' not found!",
            workspace_type_name,
        )
        return (None, None)

    workspace_type_id = workspace_type["id"]

    if not workspace_type.get("templates", []):
        self.logger.warning(
            "Workspace type -> '%s' does not have templates!",
            workspace_type_name,
        )
        return (workspace_type_id, None)

    if workspace_template_name:
        workspace_template = next(
            (item for item in workspace_type["templates"] if item["name"] == workspace_template_name),
            None,
        )
        if workspace_template:  # does this template exist?
            self.logger.info(
                "Workspace template -> '%s' has been specified in payload and it does exist.",
                workspace_template_name,
            )
        else:
            self.logger.error(
                "Workspace template -> '%s' has been specified in payload but it doesn't exist!",
                workspace_template_name,
            )
            self.logger.error(
                "Workspace type -> '%s' has only these templates -> %s",
                workspace_type_name,
                workspace_type["templates"],
            )
            return (workspace_type_id, None)

    # template to be used is NOT specified in the payload - then we just take the first one:
    else:
        workspace_template = workspace_type["templates"][0]
        self.logger.info(
            "Workspace template has not been specified in payload - taking the first one (%s)...",
            workspace_template,
        )

    workspace_template_id = workspace_template["id"]

    return (workspace_type_id, workspace_template_id)

download_bulk_document(bulk_document, download_name, row, result, source_otcs=None)

Download the bulk document and determine the file name (with path) and the mime type.

Parameters:

Name	Type	Description	Default
bulk_document	dict	The payload item for the bulk document.	required
download_name	str	The download name of the document.	required
row	Series	The current data frame series (row).	required
result	dict	The result dictionary (mutable).	required
source_otcs	OTCS | None	The Content Server OTCS object if documents should be loaded from Content Server. Defaults to None.	None

Returns:

Type	Description
tuple[str, str]	tuple[str, str]: The file name (first) and the mime type (second).

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="download_bulk_document")
def download_bulk_document(
    self,
    bulk_document: dict,
    download_name: str,
    row: pd.Series,
    result: dict,
    source_otcs: OTCS | None = None,
) -> tuple[str, str]:
    """Download the bulk document and determine the file name (with path) and the mime type.

    Args:
        bulk_document (dict):
            The payload item for the bulk document.
        download_name (str):
            The download name of the document.
        row (pd.Series):
            The current data frame series (row).
        result (dict):
            The result dictionary (mutable).
        source_otcs (OTCS | None, optional):
            The Content Server OTCS object if documents should be loaded
            from Content Server. Defaults to None.

    Returns:
        tuple[str, str]:
            The file name (first) and the mime type (second).

    """

    #
    # 1. Construct the local file name to upload:
    #

    path = bulk_document.get("download_dir", BULK_DOCUMENT_PATH)
    path = self.replace_bulk_placeholders(input_string=path, row=row)
    self.logger.debug("Download path for bulk document -> '%s'", path)

    # Determine file extensions:
    file_extension = bulk_document.get("file_extension", "")
    if file_extension:
        file_extension = self.replace_bulk_placeholders(
            input_string=file_extension,
            row=row,
        )
    file_extension_alt = bulk_document.get("file_extension_alt", "")
    if file_extension_alt:
        file_extension_alt = self.replace_bulk_placeholders(
            input_string=file_extension_alt,
            row=row,
        )

    # Determine file names:
    file_name = self.construct_file_name(
        path=path,
        download_name=download_name,
        download_name_wildcards=bulk_document.get("download_name_wildcards", False),
        file_extension=file_extension,
    )
    self.logger.debug("Constructed file name -> '%s'", file_name)
    if file_extension_alt and file_extension_alt != file_extension:
        file_name_alt = self.construct_file_name(
            path=path,
            download_name=download_name,
            download_name_wildcards=bulk_document.get(
                "download_name_wildcards",
                False,
            ),
            file_extension=file_extension_alt,
        )
    else:
        file_name_alt = file_name

    # If we couldn't construct the file name nor an alternative file name, we bail out:
    if not file_name and not file_name_alt:
        self.logger.error(
            "Cannot determine file name for document download (path -> '%s', download name -> '%s')!",
            path,
            download_name,
        )
        return None, None

    #
    # 2. Construct the mime type of the file to upload:
    #

    mime_type = bulk_document.get("mime_type", "application/pdf")
    mime_type = self.replace_bulk_placeholders(input_string=mime_type, row=row)
    mime_type_alt = bulk_document.get("mime_type_alt", "text/html")
    if mime_type_alt != "text/html":
        # if it is not the default it may have placeholders:
        mime_type_alt = self.replace_bulk_placeholders(
            input_string=mime_type_alt,
            row=row,
        )

    #
    # 3. Check if the file name(s) do(es) already exist and
    #    if we want to delete existing files (to download fresh ones)
    #

    file_exists = os.path.exists(file_name) if file_name else False

    # make sure there's no name conflict with stale documents:
    delete_download = bulk_document.get("delete_download", True)
    if file_exists and delete_download:
        os.remove(file_name)
        file_exists = False

    # Careful: If the construction of file name and the alternative
    # file name leads to identical result then there's actually
    # no alternative file name!
    if file_name != file_name_alt:
        file_exists_alt = os.path.exists(file_name_alt) if file_name_alt else False
        # make sure there's no name conflict with stale documents:
        if file_exists_alt and delete_download:
            os.remove(file_name_alt)
            file_exists_alt = False
    else:
        file_exists_alt = False

    #
    # 4. if the file does not exist in the local file system
    #    we now go and download it...
    #

    if not file_exists and not file_exists_alt:
        self.logger.debug("File -> '%s' does not exist in filesystem. We need to download it.", file_name)
        # Read "download retry number" and "wait before retry" duration from payload
        # (if specified) otherwise set default values
        wait_time = bulk_document.get("download_wait_time", 30)
        retries = bulk_document.get("download_retries", 2)

        source_type = bulk_document.get("source_type", "URL").lower()
        match source_type:
            case "contentserver":
                # Functionality to download the document from Content Server
                cs_source_id = bulk_document.get("cs_source_id", "")
                cs_source_id = self.replace_bulk_placeholders(
                    input_string=cs_source_id,
                    row=row,
                )
                if source_otcs is not None and source_otcs.otcs_ticket() is not None:
                    self.logger.info(
                        "Downloading document from source Content Server with ID -> %s...",
                        cs_source_id,
                    )

                    if source_otcs.download_document(
                        node_id=cs_source_id,
                        file_path=file_name,
                    ):
                        self.logger.debug(
                            "Successfully downloaded from Content Server using URL -> %s with ID -> %s to local file -> '%s'",
                            source_otcs.cs_public_url,
                            cs_source_id,
                            file_name,
                        )
                    else:
                        self.logger.error(
                            "Cannot download file from Content Server using URL -> %s with ID -> %s to local file -> '%s'. Skipping...",
                            source_otcs.cs_public_url(),
                            cs_source_id,
                            file_name,
                        )
                        result["skipped_counter"] += 1
                        return None, None
                else:
                    self.logger.error(
                        "Cannot download file with ID -> %s from Content Server. OTCS object not configured. Skipping...",
                        cs_source_id,
                    )
                    result["skipped_counter"] += 1
                    return None, None

            case "url":
                # Default case, download from accessible URL
                download_url = bulk_document.get("download_url")
                if download_url:
                    download_url = self.replace_bulk_placeholders(
                        input_string=download_url,
                        row=row,
                    )
                if not download_url:
                    self.logger.error(
                        "Download URL missing and no existing file -> '%s' in file system!",
                        file_name,
                    )
                    result["skipped_counter"] += 1
                    return None, None
                # Use the HHTP class to download the file:
                if not self._http_object.download_file(
                    url=download_url,
                    filename=file_name,
                    retries=retries,
                    wait_time=wait_time,
                    wait_on_status=[403],
                    show_error=False,
                ):
                    # Fetch alternative download URL (if avialable)
                    download_url_alt = bulk_document.get("download_url_alt")
                    if download_url_alt:
                        download_url_alt = self.replace_bulk_placeholders(
                            input_string=download_url_alt,
                            row=row,
                        )
                    # Check if we have an alternative download URL we try this now:
                    if download_url_alt:
                        self.logger.warning(
                            "Cannot download file from -> %s to local file -> '%s'. Trying alternative download -> %s to file -> '%s'...",
                            download_url,
                            file_name,
                            download_url_alt,
                            file_name_alt,
                        )
                        if self._http_object.download_file(
                            url=download_url_alt,
                            filename=file_name_alt,
                            retries=retries,
                            wait_time=wait_time,
                            wait_on_status=[403],
                            show_error=False,
                        ):
                            self.logger.debug(
                                "Successfully downloaded file from alternative URL -> %s to local file -> '%s'. Using this file...",
                                download_url_alt,
                                file_name_alt,
                            )
                            mime_type = mime_type_alt
                            file_name = file_name_alt
                        else:
                            # as we cannot fully rely one data source we don't treat this
                            # as an error but a warning:
                            self.logger.warning(
                                "Cannot download file from alternative URL -> %s to local file -> '%s'. Skipping...",
                                download_url_alt,
                                file_name_alt,
                            )
                            result["skipped_counter"] += 1
                            return None, None
                    # end if download_url_alt
                    else:
                        # as we cannot fully rely one data source we don't treat this
                        # as an error but a warning:
                        self.logger.warning(
                            "Cannot download file from URL -> %s to local file -> '%s'. Skipping...",
                            download_url,
                            file_name,
                        )
                        result["skipped_counter"] += 1
                        return None, None
                else:
                    self.logger.debug(
                        "Successfully downloaded file from -> %s to local file -> '%s'",
                        download_url,
                        file_name,
                    )
    # end if not file_exists and not file_exists_alt
    else:  # we already have a local file to reuse
        # If we found the alternative file name in file system
        # but not the regular one, then we switch to alternative
        # file name:
        if not file_exists and file_exists_alt:
            file_name = file_name_alt
            mime_type = mime_type_alt
        self.logger.info(
            "Reusing existing file -> '%s' in local storage.",
            file_name,
        )

    return file_name, mime_type

download_transport_package(package_url, download_dir=None)

Download the transort package from the given URL.

Parameters:

Name	Type	Description	Default
package_url	str	The URL to the transport package.	required
package_name	str	The name of the transport package.	required
download_dir	str	The file system directory to download to. If None, a temporary directory is automatically determined.	None

Returns:

Type	Description
str | None	str | None: Path of the transport package in local file system or None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="download_transport_package")
def download_transport_package(
    self,
    package_url: str,
    download_dir: str | None = None,
) -> str | None:
    """Download the transort package from the given URL.

    Args:
        package_url (str):
            The URL to the transport package.
        package_name (str):
            The name of the transport package.
        download_dir (str, optional):
            The file system directory to download to. If None,
            a temporary directory is automatically determined.

    Returns:
        str | None:
            Path of the transport package in local file system or None in case of an error.

    """

    if not download_dir:
        download_dir = os.path.join(tempfile.gettempdir(), "transports/")

    if not self._http_object:
        self._http_object = HTTP(logger=self.logger)

    # Parse the URL
    parsed_url = urlparse(package_url)

    # Extract the path from the URL
    path = parsed_url.path

    # Get the file name from the path
    file_name = os.path.basename(path)

    download_name = os.path.join(download_dir, file_name)

    os.makedirs(download_dir, exist_ok=True)

    if not self._http_object.download_file(
        url=package_url,
        filename=download_name,
        show_error=True,
    ):
        return None

    self.logger.info(
        "Successfully downloaded transport package from -> '%s' to local file -> '%s'.",
        package_url,
        download_name,
    )

    return download_name

evaluate_conditions(conditions, row, replacements=None)

Evaluate given conditions for a DataFrame series (i.e. a row).

Parameters:

Name	Type	Description	Default
conditions	list	A list of dictionaries that have a "field" (mandatory) and an "value" (optional) element.	required
row	Series	The current data row to pull data from.	required
replacements	dict	The replacements to apply to given fields (dictionary key = field name).	None

Returns:

Name	Type	Description
bool	bool	True if all given conditions evaluate to True. False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="evaluate_conditions")
def evaluate_conditions(
    self,
    conditions: list,
    row: pd.Series,
    replacements: dict | None = None,
) -> bool:
    """Evaluate given conditions for a DataFrame series (i.e. a row).

    Args:
        conditions (list):
            A list of dictionaries that have a "field" (mandatory)
            and an "value" (optional) element.
        row (pd.Series):
            The current data row to pull data from.
        replacements (dict):
            The replacements to apply to given fields (dictionary key = field name).

    Returns:
        bool:
            True if all given conditions evaluate to True. False otherwise.

    """

    evaluated_condition = True

    # We traverse a list of conditions. All conditions must evaluate to true
    # otherwise the current workspace or document (i.e. the data row for these objects)
    # will be skipped:
    for condition in conditions:
        field = condition.get("field", None)
        if not field:
            self.logger.error(
                "Missing field in condition.",
            )
            evaluated_condition = False
            break
        field_value = self.replace_bulk_placeholders(
            input_string=field,
            row=row,
            replacements=replacements,
        )
        self.logger.debug(
            "Evaluated field name -> '%s' to value -> '%s'",
            field,
            field_value,
        )
        # we have 3 options for value:
        # a) it does not exist in payload - then just the existance of the field is tested
        # b) it is a string - then we compare it 1:1 with the field value
        # c) it is a list of string - then the condition is met if one or more
        #    of the list values is equal to the field value
        value = condition.get("value", None)
        #            if not value:
        if value is None:
            # if there's no "value" element in the payload
            # this means that we just check the existance of the field
            if field_value:
                # field does exist and has any non-"" value ==> condition met!
                continue
            # field does not exist ==> condition not met!
            evaluated_condition = False
            break
        #
        # we handle string, boolean, and list values:
        #
        should_be_equal = condition.get("equal", True)
        if isinstance(value, str):
            if (should_be_equal and (value != str(field_value))) or (
                not should_be_equal and (value == str(field_value))
            ):
                self.logger.debug(
                    "String value -> '%s' is %s to field value -> '%s' but it %s. Condition not met for field -> '%s'!",
                    value,
                    "not equal" if should_be_equal else "equal",
                    field_value,
                    "should" if should_be_equal else "shouldn't",
                    field,
                )
                evaluated_condition = False
                break
        elif isinstance(value, bool):
            # We can't do a bool(field_value) as this would return True
            # for any non-empty string. So we explictly convert to a bool value
            # in a safe way:
            bool_field_value: bool = field_value.lower() == "true" if field_value else False
            if (should_be_equal and (value != bool_field_value)) or (
                not should_be_equal and (value == bool_field_value)
            ):
                self.logger.debug(
                    "Boolean value -> '%s' is %s to field value -> '%s' but it %s. Condition not met for field -> '%s'!",
                    value,
                    "not equal" if should_be_equal else "equal",
                    str(bool_field_value),
                    "should" if should_be_equal else "shouldn't",
                    field,
                )
                evaluated_condition = False
                break
        elif isinstance(value, list):
            for value_item in value:
                # Check if we found a matching element and then break.
                # Break equals positive result if should_be_equal is True:
                if should_be_equal and (value_item == field_value):
                    break
            else:  # just executed if the for loop is not interrupted by break
                # not finding a match is only negative if should_be_equal is TRUE:
                if should_be_equal:
                    self.logger.debug(
                        "Value list -> %s does not include field value -> '%s'. Condition not met!",
                        str(value),
                        field_value,
                    )
                    evaluated_condition = False

    return evaluated_condition

extract_properties_from_transport_packages(business_object_type, bo_type_name, bo_type_id)

Extract properties from transport packages for a given Business Object Type.

Parameters:

Name	Type	Description	Default
business_object_type	dict	The Business Object Type dictionary to be enriched with properties.	required
bo_type_name	str	The name of the Business Object Type.	required
bo_type_id	int	The ID of the Business Object Type.	required

Returns:

Name	Type	Description
bool	bool	True if properties were successfully extracted and added, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="extract_properties_from_transport_packages")
def extract_properties_from_transport_packages(
    self, business_object_type: dict, bo_type_name: str, bo_type_id: int
) -> bool:
    """Extract properties from transport packages for a given Business Object Type.

    Args:
        business_object_type (dict):
            The Business Object Type dictionary to be enriched with properties.
        bo_type_name (str):
            The name of the Business Object Type.
        bo_type_id (int):
            The ID of the Business Object Type.

    Returns:
        bool:
            True if properties were successfully extracted and added, False otherwise.

    """

    business_object_type["business_properties"] = []
    business_object_type["business_property_groups"] = []

    # Now we complete the data with what we have extracted from the transport packages
    # for Business Object Types. This is a workaround for the insufficient REST API
    # implementation (see otcs.get_business_object_type)
    if self._transport_extractions:
        self.logger.info(
            "Enrich business object type -> '%s' (%d) with extractions from transport packages (found '%s' extractions)...",
            bo_type_name,
            bo_type_id,
            str(len(self._transport_extractions)),
        )
    else:
        self.logger.info(
            "No transport extractions are recorded. This may be because of customizer restart.",
        )
        extraction_status_file = "transportPackagesExtractions"
        if self.check_status_file(payload_section_name=extraction_status_file):
            self.logger.info(
                "Try to load extractions from success file -> '%s'...",
                extraction_status_file,
            )
            self._transport_extractions = self.get_status_file(
                payload_section_name=extraction_status_file,
            )

    for extraction in self._transport_extractions:
        xpath = extraction.get("data")
        data_list = extraction.get("data")
        if not data_list:
            self.logger.error(
                "Extraction -> '%s' is missing the data element. Skipping...",
                xpath,
            )
            return False
        if not isinstance(data_list, list):
            self.logger.warning(
                "Extracted data for -> '%s' is not a list. Cannot process it. Skipping...",
                xpath,
            )
            return False

        """
        The following loop processes a dictionary of this structure:

        llnode: {
            '@created': '2017-11-23T16:43:35',
            '@createdby': '1000',
            '@createdbyname': 'Terrarium Admin',
            '@description': '',
            '@id': '16013',
            '@modified': '2023-12-09T12:08:21',
            '@name': 'SFDC Order',
            '@objname': 'Business Object Type',
            '@objtype': '889',
            '@ownedby': '1000',
            '@ownedbyname': 'Terrarium Admin',
            '@parentguid': '95F96645-057D-4EAF-9083-BE9F24C0CB6C',
            '@parentid': '2898',
            '@parentname': 'Business Object Types',
            ...
            'Nickname': {'@domain': ''},
            'name': {'@xml:lang': 'en', '#text': 'SFDC Order'},
            'description': {'@xml:lang': 'en'},
            'businessObjectTypeInfo': {
                'basicInfo': {
                    '@businessObjectId': '9',
                    '@businessobjectType': 'Order',
                    '@deleted': 'false',
                    '@name': 'SFDC Order',
                    '@subtype': '889',
                    '@useBusWorkspace': 'true',
                    'displayUrl': {...}
                },
                'businessApplication': {
                    'businessObjectTypeReference': {...}},
                    'businessAttachmentInfo': {
                        '@automaticAddingOfBusinessObject': 'false',
                        '@canbeAddedAsBusinessObject': 'false',
                        '@enableBADIBeforeAddingBO': 'false',
                        '@enableBADIBeforeRemovingBO': 'false',
                        '@enableMetadataMapping': 'false'
                    },
                    'managedObjectTypes': {
                        'managedObjectType': [...]
                    },
                    'multilingualNames': {'language': [...]},
                    'callbacks': {'callback': [...]},
                    'workspaceTypeReference': {'@isDefaultDisplay': 'false', '@isDefaultSearch': 'false', 'businessObjectTypeReference': {...}},
                    'businessPropertyMappings': {
                        'propertyMapping': [...]
                    },
                    'businessPropertyGroupMappings': {
                        'propertyGroupMapping': [...]
                    },
                    'documentTypes': {
                        'documentType': [...]
                    },
                    'CustomBOTypeInfo': None
                }
        }
        """

        for data in data_list:
            #
            # Level 1: llnode
            #
            llnode = data.get("llnode")
            if not llnode:
                self.logger.error(
                    "Missing llnode structure in data. Skipping...",
                )
                return False

            #
            # Level 2: businessobjectTypeInfo
            #
            business_object_type_info = llnode.get(
                "businessobjectTypeInfo",
                None,
            )
            if not business_object_type_info:
                self.logger.error(
                    "Information is missing for business object type -> '%s'. Skipping...",
                    bo_type_name,
                )
                return False

            # Check if this extraction is for the current business object type:
            basic_info = business_object_type_info.get("basicInfo", None)
            if not basic_info:
                self.logger.error(
                    "Cannot find basic info of business object type -> '%s'. Skipping...",
                    bo_type_name,
                )
                return False
            name = basic_info.get("@businessobjectType", "")
            if not name:
                self.logger.error(
                    "Cannot find name of business object type -> '%s'. Skipping...",
                    bo_type_name,
                )
                return False
            obj_type = llnode.get("@objtype", None)
            # we need to compare bo_type and NOT bo_type_name here!
            # Otherwise we don't find the SAP and SuccessFactors data:
            if name != bo_type_name or obj_type != "889":
                continue

            #
            # Level 3: businessPropertyMappings - plain, non-grouped properties
            #
            business_property_mappings = business_object_type_info.get(
                "businessPropertyMappings",
                None,
            )
            if not business_property_mappings:
                self.logger.info(
                    "No property mapping for business object type -> '%s'. Skipping...",
                    bo_type_name,
                )
                continue
            property_mappings = business_property_mappings.get(
                "propertyMapping",
                [],
            )
            # This can happen if there's only 1 propertyMapping;
            if not isinstance(property_mappings, list):
                self.logger.debug(
                    "Found a single property mapping in a dictionary (not in a list). Package it into a list...",
                )
                property_mappings = [property_mappings]

            for property_mapping in property_mappings:
                property_name = property_mapping.get("@propertyName")
                attribute_name = property_mapping.get("@attributeName")
                category_id = property_mapping.get("@categoryId")
                mapping_type = property_mapping.get("@type")
                self.logger.debug(
                    "%s Property Mapping for Business Object -> '%s' property -> '%s' is mapped to attribute -> '%s' (category -> %s)",
                    mapping_type,
                    bo_type_name,
                    property_name,
                    attribute_name,
                    category_id,
                )
                business_object_type["business_properties"].append(
                    property_mapping,
                )

            #
            # Level 3: businessPropertyGroupMappings - grouped properties
            #
            business_property_group_mappings = business_object_type_info.get(
                "businessPropertyGroupMappings",
                None,
            )
            if not business_property_group_mappings:
                self.logger.info(
                    "No property group mapping for business object type -> '%s'. Skipping...",
                    bo_type_name,
                )
                continue

            property_group_mappings = business_property_group_mappings.get(
                "propertyGroupMapping",
                [],
            )
            # This can happen if there's only 1 propertyMapping;
            if isinstance(property_group_mappings, dict):
                self.logger.debug(
                    "Found a single property group mapping in a dictionary (not in a list). Pack it into a list...",
                )
                property_group_mappings = [property_group_mappings]

            for property_group_mapping in property_group_mappings:
                group_name = property_group_mapping.get("@groupName")
                set_name = property_group_mapping.get("@setName")
                category_id = property_group_mapping.get("@categoryId")
                mapping_type = property_group_mapping.get("@type")
                self.logger.debug(
                    "%s property group mapping for business object -> %s: group -> '%s' is mapped to set -> '%s' (category -> %s)",
                    mapping_type,
                    bo_type_name,
                    group_name,
                    set_name,
                    category_id,
                )

                property_mappings = property_group_mapping.get(
                    "propertyMapping",
                    [],
                )
                # This can happen if there's only 1 propertyMapping;
                if not isinstance(property_mappings, list):
                    self.logger.debug(
                        "Found a single property mapping in a dictionary (not in a list). Package it into a list...",
                    )
                    property_mappings = [property_mappings]

                for property_mapping in property_mappings:
                    # for nested mappings we only have 2 fields - the rest is on the group level - see above
                    property_name = property_mapping.get("@propertyName")
                    attribute_name = property_mapping.get("@attributeName")
                    self.logger.debug(
                        "%s Property Mapping inside group for Business Object -> '%s', group -> '%s', property -> '%s' is mapped to set -> %s, attribute -> '%s' (category -> %s)",
                        mapping_type,
                        bo_type_name,
                        group_name,
                        property_name,
                        set_name,
                        attribute_name,
                        category_id,
                    )
                    # we write the group / set information also in the property mapping
                    # tp have a plain list with all information:
                    property_mapping["@groupName"] = group_name
                    property_mapping["@setName"] = set_name
                    property_mapping["@type"] = mapping_type
                    business_object_type["business_property_groups"].append(
                        property_mapping,
                    )
            # end for property_group_mapping in property_group_mappings
        # end for data in data_list
    # end for extraction in self._transport_extractions

    return True

generate_password(length, use_special_chars=False, min_special=1, min_numerical=1, min_upper=1, min_lower=1, override_special=None)

Generate random passwords with a given specification.

Parameters:

Name	Type	Description	Default
length	int	Define password length.	required
use_special_chars	bool	Define if special characters should be used. Defaults to False.	False
min_special	int	Define min amount of special characters. Defaults to 1.	1
min_numerical	int	Define if numbers should be used. Defaults to 1.	1
min_upper	int	Define mininum number of upper case letters. Defaults to 1.	1
min_lower	int	Define minimum number of lower case letters. Defaults to 1.	1
override_special	string | None	Define special characters to be used, if not set: !@#$%^&*()_-+=<>?/{}[]. Defaults to None.	None

Returns:

Type	Description
str | None	str | None: The generated password. None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

def generate_password(
    self,
    length: int,
    use_special_chars: bool = False,
    min_special: int = 1,
    min_numerical: int = 1,
    min_upper: int = 1,
    min_lower: int = 1,
    override_special: str | None = None,
) -> str | None:
    """Generate random passwords with a given specification.

    Args:
        length (int):
            Define password length.
        use_special_chars (bool, optional):
            Define if special characters should be used. Defaults to False.
        min_special (int, optional):
            Define min amount of special characters. Defaults to 1.
        min_numerical (int, optional):
            Define if numbers should be used. Defaults to 1.
        min_upper (int, optional):
            Define mininum number of upper case letters. Defaults to 1.
        min_lower (int, optional):
            Define minimum number of lower case letters. Defaults to 1.
        override_special (string | None, optional):
            Define special characters to be used, if not set: !@#$%^&*()_-+=<>?/{}[]. Defaults to None.

    Returns:
        str | None:
            The generated password. None in case of an error.

    """
    # Define character sets
    lowercase_letters = string.ascii_lowercase
    uppercase_letters = string.ascii_uppercase
    numerical_digits = string.digits
    special_characters = "!@#$%^&*()_-+=<>?/{}[]"

    if override_special:
        special_characters = override_special
    # Ensure minimum requirements are met

    if min_special + min_numerical + min_upper + min_lower > length:
        self.logger.error("Minimum requirements exceed password length! Cannot generate password.")
        return None

    # Initialize the password
    password = []

    # Add required characters
    password.extend(random.sample(lowercase_letters, min_lower))
    password.extend(random.sample(uppercase_letters, min_upper))
    password.extend(random.sample(numerical_digits, min_numerical))

    if use_special_chars:
        password.extend(random.sample(special_characters, min_special))

    # Fill the rest of the password with random characters
    remaining_length = length - len(password)
    all_chars = lowercase_letters + uppercase_letters + numerical_digits

    if use_special_chars:
        all_chars += special_characters

    password.extend(random.choices(all_chars, k=remaining_length))

    # Shuffle the password to ensure randomness
    random.shuffle(password)

    # Convert the password list to a string
    final_password = "".join(password)

    return final_password

get_all_group_names()

Construct a list of all group name.

Returns:

Name	Type	Description
list	list	A list of all group names.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_all_group_names")
def get_all_group_names(self) -> list:
    """Construct a list of all group name.

    Returns:
        list:
            A list of all group names.

    """

    return [group.get("name") for group in self._groups]

get_bulk_document_location(workspace, row, index, replacements)

Determine the workspace location to store a document in bulk processing.

This method is used by bulk document and bulk item workers.

Parameters:

Name	Type	Description	Default
workspace	dict	The payload for the workspace inside bulk document.	required
row	Series	The current row in the data frame.	required
index	int	The index of current row in the data frame.	required
replacements	dict	The replacement configuration for placeholders.	required

Returns:

Name	Type	Description
	int | None	int | None: Parent ID of folder or workspace to upload the document to or None in case of an error or insufficient payload information causing the workspace to be skipped.
bool	bool	True, if success. False if a failure has happened.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_bulk_document_location")
def get_bulk_document_location(
    self,
    workspace: dict,
    row: pd.Series,
    index: int,
    replacements: dict,
) -> tuple[int | None, bool]:
    """Determine the workspace location to store a document in bulk processing.

    This method is used by bulk document and bulk item workers.

    Args:
        workspace (dict):
            The payload for the workspace inside bulk document.
        row (pd.Series):
            The current row in the data frame.
        index (int):
            The index of current row in the data frame.
        replacements (dict):
            The replacement configuration for placeholders.

    Returns:
        int | None:
            Parent ID of folder or workspace to upload the document to or None in case of
            an error or insufficient payload information causing the workspace to be skipped.
        bool:
            True, if success. False if a failure has happened.

    """

    success = True

    if "workspace_name" not in workspace:
        self.logger.error(
            "No workspace name field specified for document upload! Cannot upload document to this workspace...",
        )
        success = False
        return None, success
    workspace_name_field = workspace["workspace_name"]
    workspace_name_field_alt = workspace.get("workspace_name_alt")

    workspace_name = self.replace_bulk_placeholders(
        input_string=workspace_name_field,
        row=row,
        replacements=replacements,
    )
    # If we cannot get the workspace name from the
    # workspace_name_field we try the alternative name field
    # (if specified in payload via "workspace_name_alt"):
    if not workspace_name and workspace_name_field_alt:
        self.logger.debug(
            "Row -> %s does not have the data to resolve the placeholders in workspace name field -> %s! Trying alternative name field -> %s...",
            str(index),
            workspace_name_field,
            workspace_name_field_alt,
        )
        workspace_name = self.replace_bulk_placeholders(
            input_string=workspace_name_field_alt,
            row=row,
            replacements=replacements,
        )

    # it could be that the current data row does not have the
    # required fields to resolve the workspace name placeholders
    # then we skip uploading the document to this workspace
    # but still keep status as successful (don't set success = False)
    if not workspace_name:
        self.logger.warning(
            "Row -> %s does not have the required data to resolve workspace name field -> '%s' specified for document upload! Skipping document upload to this workspace...",
            str(index),
            workspace_name_field,
        )
        # We keep success = True as this is a data problem and not a config problem!
        return None, success

    # Cleanse the workspace name (allowed characters, maximum length):
    workspace_name = OTCS.cleanse_item_name(workspace_name)

    # Check if all data conditions to create the workspace are met
    conditions = workspace.get("conditions")
    if conditions:
        evaluated_condition = self.evaluate_conditions(
            conditions=conditions,
            row=row,
        )
        if not evaluated_condition:
            self.logger.info(
                "Workspace condition for row -> %s not met. Skipping row for document upload to workspace -> '%s'...",
                str(index),
                workspace_name,
            )
            # We keep success = True as this is a data problem and not a config problem!
            return None, success

    if "workspace_type" not in workspace:
        self.logger.error(
            "No workspace type specified for document upload! Skipping workspace -> '%s'...",
            workspace_name,
        )
        success = False
        return None, success
    workspace_type = workspace["workspace_type"]
    workspace_type = self.replace_bulk_placeholders(
        input_string=workspace_type,
        row=row,
        replacements=replacements,
    )
    workspace_data_source_name = workspace.get("data_source")
    # Try to find the workspace by name/synonym and type:
    (workspace_id, workspace_name) = self.process_bulk_workspaces_lookup(
        workspace_name=workspace_name,
        workspace_type=workspace_type,
        data_source_name=workspace_data_source_name,
    )
    if not workspace_id:
        self.logger.warning(
            "Cannot find workspace with name/synonym -> '%s' and type -> '%s'.",
            workspace_name,
            workspace_type,
        )
        success = False
        return None, success

    # If the workspace payload element has a "data_source" key,
    # then add all columns from the given data source to the bulk
    # document row to also support the lookup of values from the workspace
    # data source. These fields get a "lookup_" prefix to avoid name clashes.
    # the values must be specified with this "lookup_" prefix in the payload.
    # We CANNOT do this at the very beginning of the workspace loop as we
    # need the workspace_name to be properly resolved (incl. synonyms):
    if workspace_data_source_name:
        self.logger.info(
            "Workspace for bulk documents has a data source -> '%s' with lookup values. Adding them as row columns...",
            workspace_data_source_name,
        )
        workspace_data_source = next(
            (item for item in self._bulk_datasources if item["name"] == workspace_data_source_name),
            None,
        )
        # Read the synonym column and the name column from the data source payload item:
        workspace_data_source_name_column = workspace_data_source.get(
            "name_column",
            None,  # e.g. "Name"
        )

        if workspace_data_source_name_column:
            # Get additional data from workspace data source
            # for lookups. Synonyms are already resolved at
            # this point in time (workspace_name has been updated above
            # in case the initial workspace name wasn't found)
            lookup_row = self.lookup_data_source_value(
                data_source=workspace_data_source,
                lookup_column=workspace_data_source_name_column,
                lookup_value=workspace_name,
            )

            # Adding all values of the lookup row with the prefix lookup_ to the bulk documents row
            # for replacement of placeholders:
            if lookup_row is not None:
                for k, value in lookup_row.items():
                    row["lookup_" + k] = value

    # "workspace_folder" can be used if the payload contains
    # the path as a comma-separated string (top down)
    # otherwise "workspace_path" has precedence over "workspace_folder"
    workspace_folder = workspace.get("workspace_folder", "")
    workspace_path = workspace.get("workspace_path")

    if workspace_folder and not workspace_path:
        workspace_folder = self.replace_bulk_placeholders(
            input_string=workspace_folder,
            row=row,
            replacements=replacements,
        )
        workspace_path = (
            [item.strip() for item in workspace_folder.split(",")]
            if "," in workspace_folder
            else [workspace_folder]
        )

    if workspace_path:
        if isinstance(workspace_path, str):
            # if the path is actually a list in a string
            # we need to convert it to a python list in a safe way:
            try:
                workspace_path = self.replace_bulk_placeholders(
                    input_string=workspace_path,
                    index=None,  # None is VERY important as otherwise index=0 is the default and we only get the first element
                    row=row,
                    replacements=replacements,
                )
                workspace_path = literal_eval(workspace_path) if workspace_path else None
            except (SyntaxError, ValueError):
                self.logger.error(
                    "Cannot parse list-type folder path wrapped in string -> '%s'!",
                    workspace_path,
                )
                workspace_path = None
        elif isinstance(workspace_path, list):
            # We create a copy list to not modify original payload
            workspace_path = list(workspace_path)
            # Replace placeholders in payload for the path elements:
            # Note: workspace_path is a mutable data type that is changed in place!
            result_placeholders = self.replace_bulk_placeholders_list(
                input_list=workspace_path,
                row=row,
                replacements=replacements,
            )
            if not result_placeholders:
                workspace_path = None
        else:
            self.logger.warning("Unsupported data type for workspace path!")
            workspace_path = None

        if not workspace_path:
            # we put the document into the root of the workspace
            # if we couldn't determine a path inside the workspace:
            self.logger.info(
                "Workspace folder path for workspace -> '%s' of workspace type -> '%s' has no value. Using workspace root for document upload.",
                workspace_name,
                workspace_type,
            )
            parent_id = workspace_id
        else:
            # Check if the folder path does already exist and get the target folder at the end of the path:
            self.logger.info(
                "Check if path -> %s does already exist in workspace -> '%s' (%s) or otherwise create it...",
                str(workspace_path),
                workspace_name,
                workspace_id,
            )
            response = self._otcs_frontend.get_node_by_workspace_and_path(
                workspace_id=workspace_id,
                path=workspace_path,
                create_path=True,  # we want the path to be created if it doesn't exist
                show_error=False,
            )
            parent_id = self._otcs_frontend.get_result_value(
                response=response,
                key="id",
            )
            if not parent_id:
                self.logger.error(
                    "Failed to create path -> %s in workspace -> '%s' (%s)!",
                    str(workspace_path),
                    workspace_name,
                    workspace_id,
                )
                success = False
                return None, success
            else:
                self.logger.info(
                    "Using path -> %s inside workspace -> '%s' (%s). Node ID for target folder -> %s.",
                    str(workspace_path),
                    workspace_name,
                    workspace_id,
                    str(parent_id),
                )
    # end if workspace_path
    else:
        self.logger.info(
            "Workspace folder path for workspace -> '%s' of workspace type -> '%s' is not specified. Using workspace root for document upload.",
            workspace_name,
            workspace_type,
        )
        # we put the document into the root of the workspace:
        parent_id = workspace_id

    # Check if we have sub-workspaces configured.
    # If not, then we are done and can return the found
    # parent ID for the document:
    if "sub_workspace_type" not in workspace or "sub_workspace_name" not in workspace:
        self.logger.debug(
            "No sub workspace configured in payload. Return parent ID -> %s",
            parent_id,
        )
        return parent_id, success

    #
    # Determine (or create) the Sub workspace:
    #

    sub_workspace_type = workspace["sub_workspace_type"]
    sub_workspace_type = self.replace_bulk_placeholders(
        input_string=sub_workspace_type,
        row=row,
        replacements=replacements,
    )
    sub_workspace_name = workspace["sub_workspace_name"]
    sub_workspace_name = self.replace_bulk_placeholders(
        input_string=sub_workspace_name,
        row=row,
        replacements=replacements,
    )
    response = self._otcs_frontend.get_node_by_parent_and_name(
        name=sub_workspace_name,
        parent_id=parent_id,
    )
    # Check if the sub-workspaces does already exist:
    sub_workspace_id = self._otcs_frontend.get_result_value(
        response=response,
        key="id",
    )
    # Sub workspaces are dynamically created
    # during the processing of bulk documents...
    if not sub_workspace_id:
        self.logger.info(
            "Creating sub workspace -> '%s' of type -> '%s' and parent -> %s...",
            sub_workspace_name,
            sub_workspace_type,
            parent_id,
        )
        sub_workspace_template = workspace.get("sub_workspace_template", "")

        sub_workspace_template = self.replace_bulk_placeholders(
            input_string=sub_workspace_template,
            row=row,
            replacements=replacements,
        )
        # Now we try to determine the IDs for the sub-workspace type and template:
        (sub_workspace_type_id, sub_workspace_template_id) = self.determine_workspace_type_and_template_id(
            workspace_type_name=sub_workspace_type,
            workspace_template_name=sub_workspace_template,
        )
        # if either of the two couldn't be determined we cannot create the sub-workspace
        if not sub_workspace_type_id or not sub_workspace_template_id:
            self.logger.error(
                "Couldn't dertermine workspace template ID and workspace type ID of sub-workspace!",
            )
            success = False
            return None, success

        # Check if we have categories for the sub-workspace:
        if "categories" not in workspace:
            self.logger.debug(
                "Sub-Workspace payload has no category data! Will leave category attributes empty...",
            )
            sub_workspace_category_data = {}
        else:
            self.logger.debug(
                "Sub-Workspace payload has category data! Will prepare category data for workspace creation...",
            )
            worker_categories = self.process_bulk_categories(
                row=row,
                index=index,
                categories=workspace["categories"],
                replacements=replacements,
            )
            self.logger.debug(
                "Prepare category data for sub-workspace with template -> %s and parent -> %s",
                sub_workspace_template_id,
                parent_id,
            )
            sub_workspace_category_data = self.prepare_workspace_create_form(
                categories=worker_categories,
                template_id=sub_workspace_template_id,
                parent_workspace_node_id=parent_id,
            )
            if not sub_workspace_category_data:
                self.logger.error(
                    "Failed to prepare the category data for sub-workspace -> '%s'!",
                    sub_workspace_name,
                )
                success = False
                return None, success
        # Now we create the sub-workspace:
        response = self._otcs_frontend.create_workspace(
            workspace_template_id=sub_workspace_template_id,
            workspace_name=sub_workspace_name,
            workspace_description="",
            workspace_type=sub_workspace_type_id,
            category_data=sub_workspace_category_data,
            parent_id=parent_id,
            show_error=False,
        )
        if response is None:
            # Potential race condition: see if the sub-workspace has been created by a concurrent thread.
            # So we better check if the workspace is there even if the create_workspace() call delivered
            # a 'None' response:
            response = self._otcs_frontend.get_node_by_parent_and_name(
                parent_id=parent_id,
                name=sub_workspace_name,
            )
        sub_workspace_id = self._otcs_frontend.get_result_value(
            response=response,
            key="id",
        )
        if not sub_workspace_id:
            self.logger.error(
                "Failed to create sub-workspace -> '%s' with type ID -> %s under parent with ID -> %s!",
                sub_workspace_name,
                sub_workspace_type_id,
                parent_id,
            )
            parent_id = None
            success = False
            return None, success
        else:
            self.logger.info(
                "Successfully created sub-workspace -> '%s' (%s).",
                sub_workspace_name,
                sub_workspace_id,
            )

        # Create Business Relationship between workspace and sub-workspace:
        if workspace_id and sub_workspace_id:
            # Check if workspace relationship does already exist in OTCS
            # (this is an additional safety measure to avoid errors):
            response = self._otcs_frontend.get_workspace_relationships(
                workspace_id=workspace_id,
                related_workspace_name=sub_workspace_name,
            )
            current_workspace_relationships = self._otcs_frontend.exist_result_item(
                response=response,
                key="id",
                value=sub_workspace_id,
            )
            if current_workspace_relationships:
                self.logger.info(
                    "Workspace relationship between workspace -> '%s' (%s) and sub-workspace -> '%s' (%s) does already exist. Skipping...",
                    workspace_name,
                    workspace_id,
                    sub_workspace_name,
                    sub_workspace_id,
                )
            else:
                self.logger.info(
                    "Create workspace relationship %s -> %s...",
                    workspace_id,
                    sub_workspace_id,
                )
                response = self._otcs_frontend.create_workspace_relationship(
                    workspace_id=workspace_id,
                    related_workspace_id=sub_workspace_id,
                    show_error=False,  # we don't want to show an error because of race conditions handled below
                )
                if response:
                    self.logger.info(
                        "Successfully created workspace relationship between workspace ID -> %s and sub-workspace ID -> %s.",
                        workspace_id,
                        sub_workspace_id,
                    )
                else:
                    # Potential race condition: see if the workspace-2-sub-workspace relationship has been created by a concurrent thread.
                    # So we better check if the relationship is there even if the create_workspace_relationship() call delivered
                    # a 'None' response:
                    response = self._otcs_frontend.get_workspace_relationships(
                        workspace_id=workspace_id,
                        related_workspace_name=sub_workspace_name,
                    )
                    current_workspace_relationships = self._otcs_frontend.exist_result_item(
                        response=response,
                        key="id",
                        value=sub_workspace_id,
                    )
                    if not current_workspace_relationships:
                        self.logger.error(
                            "Failed to create workspace relationship between workspace ID -> %s and sub-workspace ID -> %s!",
                            workspace_id,
                            sub_workspace_id,
                        )
                    else:
                        self.logger.info(
                            "Successfully created workspace relationship between workspace ID -> %s and sub-workspace ID -> %s.",
                            workspace_id,
                            sub_workspace_id,
                        )

    # end if sub-workspace does not exist
    else:
        self.logger.info(
            "Using existing sub workspace -> '%s' (%s) of type -> '%s'...",
            sub_workspace_name,
            str(sub_workspace_id),
            sub_workspace_type,
        )

    #
    # Get the target folder in the sub-workspace by the provided sub workspace path
    #

    # "sub_workspace_folder" can be used if the payload contains
    # the path as a comma-separated string (top down)
    sub_workspace_folder = workspace.get("sub_workspace_folder", "")
    sub_workspace_path = workspace.get("sub_workspace_path")

    if not sub_workspace_path and sub_workspace_folder:
        sub_workspace_folder = self.replace_bulk_placeholders(
            input_string=sub_workspace_folder,
            row=row,
            replacements=replacements,
        )
        sub_workspace_path = (
            [item.strip() for item in sub_workspace_folder.split(",")]
            if "," in sub_workspace_folder
            else [sub_workspace_folder]
        )

    if sub_workspace_path:
        if isinstance(sub_workspace_path, str):
            # if the path is actually a list in a string
            # we need to convert it to a python list in a safe way:
            try:
                sub_workspace_path = self.replace_bulk_placeholders(
                    input_string=sub_workspace_path,
                    index=None,  # None is VERY important as otherwise index=0 is the default and we only get the first element
                    row=row,
                    replacements=replacements,
                )
                sub_workspace_path = literal_eval(sub_workspace_path) if sub_workspace_path else None
            except (SyntaxError, ValueError):
                self.logger.error(
                    "Cannot parse list-type folder path wrapped in string -> '%s'!",
                    sub_workspace_path,
                )
                sub_workspace_path = None
        elif isinstance(sub_workspace_path, list):
            # We create a copy list to not modify original payload
            sub_workspace_path = list(sub_workspace_path)
            # Replace placeholders in payload for the path elements:
            # Note: sub_workspace_path is a mutable data type that is changed in place!
            result_placeholders = self.replace_bulk_placeholders_list(
                input_list=sub_workspace_path,
                row=row,
                replacements=replacements,
            )
            if not result_placeholders:
                sub_workspace_path = None
        else:
            self.logger.warning("Unsupported data type for sub workspace path!")
            sub_workspace_path = None

        if not sub_workspace_path:
            self.logger.warning(
                "Sub-Workspace folder path for sub workspace -> '%s' of sub workspace type -> '%s' cannot be resolved (placeholder issue). Using sub workspace root for document upload.",
                sub_workspace_name,
                sub_workspace_type,
            )
            # we put the document into the root of the workspace:
            parent_id = sub_workspace_id
        else:
            # Check if the folder path does already exist and get the target folder at the end of the path:
            self.logger.info(
                "Check if path -> %s does already exist in workspace -> '%s' (%s)... (otherwise create it)",
                str(sub_workspace_path),
                sub_workspace_name,
                sub_workspace_id,
            )
            response = self._otcs_frontend.get_node_by_workspace_and_path(
                workspace_id=sub_workspace_id,
                path=sub_workspace_path,
                create_path=True,  # we want the path to be created if it doesn't exist
                show_error=False,
            )
            parent_id = self._otcs_frontend.get_result_value(
                response=response,
                key="id",
            )
            if not parent_id:
                self.logger.error(
                    "Failed to create path -> %s in sub workspace -> '%s' (%s)!",
                    str(sub_workspace_path),
                    sub_workspace_name,
                    sub_workspace_id,
                )
                success = False
                return None, success
            else:
                self.logger.info(
                    "Successfully created path -> %s in sub-workspace -> '%s' (%s). Node ID for target folder -> %s.",
                    str(sub_workspace_path),
                    sub_workspace_name,
                    sub_workspace_id,
                    str(parent_id),
                )
    else:
        self.logger.info(
            "Folder path inside sub-workspace -> '%s' with sub workspace type -> '%s' is not specified. Using root of sub-workspace for document upload.",
            sub_workspace_name,
            sub_workspace_type,
        )
        # we put the document into the root of the workspace:
        parent_id = sub_workspace_id

    return parent_id, success

get_bulk_workspace_relationship_endpoint(bulk_workspace_relationship, row, index, endpoint, replacements=None, nickname_additional_regex_list=None, show_error=True)

Determine the node ID of the workspace that is one of the endpoints of the workspace relationship (either 'from' or 'to').

Parameters:

Name	Type	Description	Default
bulk_workspace_relationship	dict	The payload element of the bulk workspace relationship	required
row	Series	The data frame row.	required
index	int	The index of the data frame row.	required
endpoint	str	The name of the endpoint - either "from" or "to".	required
replacements	dict | None	Replacements for placeholders. Defaults to None.	None
nickname_additional_regex_list	list | None	Additional regex replacements for nicknames. Defaults to None.	None
show_error	bool	Whether or not to log an error. If False just a warning is logged.	True

Returns:

Type	Description
tuple[int | None, str | None]	tuple[int | None, str | None]: The workspace ID and the looked up workspace name.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_bulk_workspace_relationship_endpoint")
def get_bulk_workspace_relationship_endpoint(
    self,
    bulk_workspace_relationship: dict,
    row: pd.Series,
    index: int,
    endpoint: str,
    replacements: dict | None = None,
    nickname_additional_regex_list: list | None = None,
    show_error: bool = True,
) -> tuple[int | None, str | None]:
    """Determine the node ID of the workspace that is one of the endpoints of the workspace relationship (either 'from' or 'to').

    Args:
        bulk_workspace_relationship (dict):
            The payload element of the bulk workspace relationship
        row (pd.Series):
            The data frame row.
        index (int):
            The index of the data frame row.
        endpoint (str):
            The name of the endpoint - either "from" or "to".
        replacements (dict | None, optional):
            Replacements for placeholders. Defaults to None.
        nickname_additional_regex_list (list | None, optional):
            Additional regex replacements for nicknames. Defaults to None.
        show_error (bool, optional):
            Whether or not to log an error. If False just a warning is logged.

    Returns:
        tuple[int | None, str | None]:
            The workspace ID and the looked up workspace name.

    """

    if endpoint not in ["from", "to"]:
        self.logger.error("The endpoint must be either 'from' or 'to'!")
        return (None, None)

    # Determine the workspace nickname field:
    workspace_nickname_field = bulk_workspace_relationship.get(
        "{}_workspace".format(endpoint),
        None,
    )
    workspace_nickname = self.replace_bulk_placeholders(
        input_string=workspace_nickname_field,
        row=row,
        replacements=replacements,
        additional_regex_list=nickname_additional_regex_list,
    )
    if not workspace_nickname:
        self.logger.warning(
            "Row -> %s does not have the required data to resolve -> '%s' for the workspace nickname (endpoint -> '%s')!",
            str(index),
            workspace_nickname_field,
            endpoint,
        )
        return (None, None)

    # Get the workspace type if specified:
    workspace_type = bulk_workspace_relationship.get(
        "{}_workspace_type".format(endpoint),
        None,
    )

    # Get the workspace name if specified:
    workspace_name_field = bulk_workspace_relationship.get(
        "{}_workspace_name".format(endpoint),
        None,
    )
    if workspace_name_field:
        workspace_name = self.replace_bulk_placeholders(
            input_string=workspace_name_field,
            row=row,
            replacements=replacements,
        )
        if not workspace_name:
            self.logger.warning(
                "Row -> %s does not have the required data to resolve -> '%s' for the workspace name (endpoint -> '%s')!",
                str(index),
                workspace_name_field,
                endpoint,
            )
            return (None, None)
    else:
        workspace_name = None

    # Get the workspace data source if specified:
    workspace_data_source = bulk_workspace_relationship.get(
        "{}_workspace_data_source".format(endpoint),
        None,
    )

    # Based on the given information, we now try to determine
    # the name and the ID of the workspace that is the endpoint
    # for the workspace relationship:
    (workspace_id, workspace_name) = self.process_bulk_workspaces_lookup(
        workspace_nickname=workspace_nickname,
        workspace_name=workspace_name,
        workspace_type=workspace_type,
        data_source_name=workspace_data_source,
        show_error=show_error,
    )

    if not workspace_id:
        self.logger.warning(
            "Cannot find workspace to establish relationship (endpoint -> '%s')%s%s%s%s",
            endpoint,
            (", nickname -> '{}'".format(workspace_nickname) if workspace_nickname else ""),
            (", workspace name -> '{}'".format(workspace_name) if workspace_name else ""),
            (", workspace type -> '{}'".format(workspace_type) if workspace_type else ""),
            (", data source -> '{}'".format(workspace_data_source) if workspace_data_source else ""),
        )
        return (None, None)

    # See if a sub-workspace is configured:
    sub_workspace_name_field = bulk_workspace_relationship.get(
        "{}_sub_workspace_name".format(endpoint),
        None,
    )
    # If no sub-workspace is configured we can already
    # return the resulting workspace ID and name here:
    if not sub_workspace_name_field:
        return (workspace_id, workspace_name)

    # Otherwise we are no processing the sub-workspaces to return
    # its ID instead:
    sub_workspace_name = self.replace_bulk_placeholders(
        input_string=sub_workspace_name_field,
        row=row,
        replacements=replacements,
    )
    if not sub_workspace_name:
        self.logger.warning(
            "Row -> %s does not have the required data to resolve -> %s for the sub-workspace name (endpoint -> '%s')!",
            str(index),
            sub_workspace_name_field,
            endpoint,
        )
        return (None, None)

    # See if a sub-workspace is in a sub-path of the main workspace:
    sub_workspace_path = bulk_workspace_relationship.get(
        "{}_sub_workspace_path".format(endpoint),
        None,
    )
    if sub_workspace_path:
        # sub_workspace_path is a mutable that is changed in place!
        result = self.replace_bulk_placeholders_list(
            input_list=sub_workspace_path,
            row=row,
            replacements=replacements,
        )
        if not result:
            self.logger.warning(
                "Row -> %s does not have the required data to resolve -> %s for the sub-workspace path (endpoint -> '%s')!",
                str(index),
                sub_workspace_path,
                endpoint,
            )
            return None

        self.logger.info(
            "Endpoint has a sub-workspace -> '%s' configured. Try to find the sub-workspace in workspace path -> %s...",
            sub_workspace_name,
            sub_workspace_path,
        )

        # We now want to retrieve the folder in the main workspace that
        # includes the sub-workspace:
        response = self._otcs_frontend.get_node_by_workspace_and_path(
            workspace_id=workspace_id,
            path=sub_workspace_path,
            create_path=False,  # we don't want the path to be created if it doesn't exist
            show_error=True,
        )
        parent_id = self._otcs_frontend.get_result_value(
            response=response,
            key="id",
        )
        if not parent_id:
            self.logger.error(
                "Failed to find path -> %s in workspace -> '%s' (%s)!",
                str(sub_workspace_path),
                workspace_name,
                workspace_id,
            )
            return (None, None)
    # end if sub_workspace_path_field
    else:
        # the sub-workspace is immediately under the main workspace:
        parent_id = workspace_id

    response = self._otcs_frontend.get_node_by_parent_and_name(
        parent_id=parent_id,
        name=sub_workspace_name,
        show_error=True,
    )
    sub_workspace_id = self._otcs_frontend.get_result_value(
        response=response,
        key="id",
    )

    return (sub_workspace_id, sub_workspace_name)

get_business_object_properties(bo_type_name)

Get a dictionary with all property mapping of a business object type.

We contruct this dictionary from the two lists for the given business object types (property mapping and property group mappings) These two lists have been created before by process_business_object_types()

This method is used for creation of business objects in Salesforce.

Parameters:

Name	Type	Description	Default
bo_type_name	str	The name of the business object type	required

Returns:

Type	Description
dict | None	dict | None: A dictionary with keys that are either the attribute name or a key that is contructed like this: set name + "-" + attribute name. This allows for an easy lookup in methods that have access to the category data of business workspaces.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_business_object_properties")
def get_business_object_properties(self, bo_type_name: str) -> dict | None:
    """Get a dictionary with all property mapping of a business object type.

    We contruct this dictionary from the two lists for the given
    business object types (property mapping and property group mappings)
    These two lists have been created before by process_business_object_types()

    This method is used for creation of business objects in Salesforce.

    Args:
        bo_type_name (str):
            The name of the business object type

    Returns:
        dict | None:
            A dictionary with keys that are either the attribute name or
            a key that is contructed like this: set name + "-" + attribute name.
            This allows for an easy lookup in methods that have access to
            the category data of business workspaces.

    """

    if not self._business_object_types:
        self.logger.warning(
            "List of business object types is empty / not initialized! Cannot lookup type -> '%s'!",
            bo_type_name,
        )
        return None

    # Find the matching business object type:
    business_object_type = next(
        (item for item in self._business_object_types if item["name"] == bo_type_name),
        None,
    )
    if not business_object_type:
        self.logger.warning(
            "Cannot find business object type -> '%s'!",
            bo_type_name,
        )
        return None

    business_properties = business_object_type.get("business_properties")
    business_property_groups = business_object_type.get("business_property_groups")

    lookup_dict = {}

    # 25.3 uses a new REST API to retreive the business object type details.
    # Pre 25.3 we use extractions from the transport package to get this information
    # See process_business_object_type() method.
    otcs_version = float(self._otcs.get_server_version())
    if otcs_version < 25.3:
        att_key = "@attributeName"
        set_key = "@setName"
    else:
        att_key = "att_name"
        set_key = "set_name"

    for mapping in business_properties:
        attribute_name = mapping.get(att_key)
        lookup_dict[attribute_name] = mapping

    for mapping in business_property_groups:
        set_name = mapping.get(set_key)
        attribute_name = mapping.get(att_key)
        lookup_dict[set_name + "-" + attribute_name] = mapping

    return lookup_dict

get_groups()

Get all groups in payload.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

def get_groups(self) -> list:
    """Get all groups in payload."""
    return self._groups

get_guidewire_business_object(external_system, object_type, search_field, search_value)

Get the Guidewire ID (str) of an Guidewire object by querying the Guidewire API.

Parameters:

Name	Type	Description	Default
external_system	dict	Payload of the external System. Required if Guidewire python object needs to be re-initialized.	required
object_type	str	The business object type (like "Claim", "Policy", "Account").	required
search_field	str	Search field to find business object in external system.	required
search_value	str	Search value to find business object in external system.	required

Returns:

Type	Description
str | None	str | None: The technical ID of the business object in Guidewire. None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_guidewire_business_object")
def get_guidewire_business_object(
    self,
    external_system: dict,
    object_type: str,
    search_field: str,
    search_value: str,
) -> str | None:
    """Get the Guidewire ID (str) of an Guidewire object by querying the Guidewire API.

    Args:
        external_system (dict):
            Payload of the external System. Required if Guidewire python object
            needs to be re-initialized.
        object_type (str):
            The business object type (like "Claim", "Policy", "Account").
        search_field (str):
            Search field to find business object in external system.
        search_value (str):
            Search value to find business object in external system.

    Returns:
        str | None:
            The technical ID of the business object in Guidewire. None in case of an error.

    """

    if object_type in ["Account", "Policy", "account", "policy", "gw.account", "gw.policy"]:
        if not self._guidewire_policy_center:
            self._guidewire_policy_center = self.init_guidewire(guidewire_external_system=external_system)
        guidewire_object = self._guidewire_policy_center
    elif object_type in ["Claim", "claim", "gw.claim"]:
        if not self._guidewire_claims_center:
            self._guidewire_claims_center = self.init_guidewire(guidewire_external_system=external_system)
        guidewire_object = self._guidewire_claims_center
    else:
        self.logger.error("Not supported Guidewire object type -> '%s'", object_type)
        return None

    if not guidewire_object:
        self.logger.error(
            "Guidewire connection not initialized! Cannot connect to Guidewire API!",
        )
        return None

    self.logger.debug(
        "Workspaces is connected to Guidewire and we need to lookup the business object ID...",
    )
    guidewire_token = guidewire_object.authenticate()
    if not guidewire_token:
        self.logger.error("Failed to authenticate with Guidewire!")
        return None

    match object_type:
        case "Account" | "account" | "gw.account":
            response = guidewire_object.search_account(
                attributes={search_field: search_value},
            )
            bo_id = guidewire_object.get_result_value(response=response, key="id")
        case "Policy" | "policy" | "gw.policy":
            response = guidewire_object.search_policy(
                attributes={search_field: search_value},
            )
            bo_id = guidewire_object.get_result_value(response=response, key="policyId")
        case _:
            self.logger.warning("Currently we only support lookup of 'Account' and 'Policy' objects in Guidewire!")
            return None

    num_of_bos = int(response.get("count", 0)) if (response is not None and "count" in response) else 0
    if num_of_bos > 1:
        self.logger.warning(
            "Guidewire lookup delivered %s business objects of type -> '%s'. Picking the first one with ID -> %s.",
            str(num_of_bos),
            str(object_type),
            bo_id,
        )
    if not bo_id:
        self.logger.warning(
            "Business object of type -> '%s' with '%s' = '%s' does not exist in Guidewire!",
            object_type,
            search_field,
            search_value,
        )
    else:  # BO found
        self.logger.info(
            "Retrieved ID -> %s for Guidewire object type -> '%s' (looking up -> '%s' in field -> '%s').",
            bo_id,
            object_type,
            search_value,
            search_field,
        )

    return bo_id

get_k8s()

Get the K8s object.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

def get_k8s(self) -> object:
    """Get the K8s object."""
    return self._k8s

get_m365()

Get the M365 object.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

def get_m365(self) -> object:
    """Get the M365 object."""
    return self._m365

get_otcs_backend()

Get OTCS Backend object.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

def get_otcs_backend(self) -> object:
    """Get OTCS Backend object."""
    return self._otcs_backend

get_otcs_frontend()

Get OTCS Frontend oject.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

def get_otcs_frontend(self) -> object:
    """Get OTCS Frontend oject."""
    return self._otcs_frontend

get_otds()

Get the OTDS object.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

def get_otds(self) -> object:
    """Get the OTDS object."""
    return self._otds

get_payload(drop_bulk_datasources_data=False)

Get the Payload as reference.

Optional a copy of the payload can be delivered the does not include the "data" value of "bulkDatasource" (its content can be HUGE and many times we don't want it).

Parameters:

Name	Type	Description	Default
drop_bulk_datasources_data	bool	If True, returns a !copy! of the payload without the bulkDatasources "data" in it. Defaults to False.	False

Returns:

Name	Type	Description
dict	dict	Returns the payload

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_payload")
def get_payload(self, drop_bulk_datasources_data: bool = False) -> dict:
    """Get the Payload as reference.

    Optional a copy of the payload can be delivered the does not include the
    "data" value of "bulkDatasource" (its content can be HUGE and many times we don't
    want it).

    Args:
        drop_bulk_datasources_data (bool, optional):
            If True, returns a !copy! of the payload without the bulkDatasources
            "data" in it. Defaults to False.

    Returns:
        dict: Returns the payload

    """

    if drop_bulk_datasources_data and "bulkDatasources" in self._payload:
        # Create a Copy of the payload, but without the bulkDatasources
        # using deepcopy would require significantly more memory, just to drop it directly after
        payload = {k: v for k, v in self._payload.items() if k != "bulkDatasources"}

        payload["bulkDatasources"] = [
            {k: v for k, v in data.items() if k != "data"} for data in self._payload.get("bulkDatasources", {})
        ]
        return payload

    return self._payload

get_payload_section(payload_section_name)

Get a specific section of the payload based on its name.

The section is delivered as a list of settings. It deliveres an empty list if this payload section is disabled by the corresponding payload switch (this is read from the payloadSections dictionary of the payload)

Parameters:

Name	Type	Description	Default
payload_section_name	str	The name of the dict element in the payload structure.	required

Returns:

Name	Type	Description
list	list	The section of the payload as a Python list. Empty list if section does not exist or section is disabled by the corresponding payload switch.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

def get_payload_section(self, payload_section_name: str) -> list:
    """Get a specific section of the payload based on its name.

    The section is delivered as a list of settings.
    It deliveres an empty list if this payload section is disabled by the corresponding
    payload switch (this is read from the payloadSections dictionary of the payload)

    Args:
        payload_section_name (str):
            The name of the dict element in the payload structure.

    Returns:
        list:
            The section of the payload as a Python list. Empty list if section does not exist
            or section is disabled by the corresponding payload switch.

    """

    if not isinstance(self._payload, dict):
        return []

    # if the secton is not in the payload we return an empty list:
    if not self._payload.get(payload_section_name):
        return []

    # Return an empty list if the payload section does not exist or is disabled:
    sections = self._payload.get("payloadSections")
    if sections:
        section = next(
            (item for item in sections if item["name"] == payload_section_name),
            None,
        )
        if not section or not section.get("enabled", False):
            return []

    return self._payload[payload_section_name]

get_salesforce_business_object(workspace, object_type, search_field, search_value)

Get the Salesforce ID (str) of an Salesforce object by querying the Salesforce API.

Parameters:

Name	Type	Description	Default
workspace	dict	The workspace payload element.	required
object_type	str	The business object type.	required
search_field	str	Search field to find business object in external system.	required
search_value	str	Search value to find business object in external system.	required

Returns:

Type	Description
str | None	str | None: The technical ID of the business object in Salesforce.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_salesforce_business_object")
def get_salesforce_business_object(
    self,
    workspace: dict,
    object_type: str,
    search_field: str,
    search_value: str,
) -> str | None:
    """Get the Salesforce ID (str) of an Salesforce object by querying the Salesforce API.

    Args:
        workspace (dict):
            The workspace payload element.
        object_type (str):
            The business object type.
        search_field (str):
            Search field to find business object in external system.
        search_value (str):
            Search value to find business object in external system.

    Returns:
        str | None:
            The technical ID of the business object in Salesforce.

    """

    if not self._salesforce:
        self.logger.error(
            "Salesforce connection not initialized! Cannot connect to Salesforce API!",
        )
        return None

    self.logger.debug(
        "Workspaces is connected to Salesforce and we need to lookup the business object ID...",
    )
    salesforce_token = self._salesforce.authenticate()
    if not salesforce_token:
        self.logger.error("Failed to authenticate with Salesforce!")
        return None

    response = self._salesforce.get_object(
        object_type=object_type,
        search_field=search_field,
        search_value=search_value,
        result_fields=["Id"],
    )
    bo_id = self._salesforce.get_result_value(response=response, key="Id")
    num_of_bos = int(response.get("totalSize", 0)) if (response is not None and "totalSize" in response) else 0
    if num_of_bos > 1:
        self.logger.warning(
            "Salesforce lookup delivered %s business objects of type -> '%s'! Picking the first one with ID -> %s...",
            str(num_of_bos),
            str(object_type),
            bo_id,
        )
    if not bo_id:
        self.logger.warning(
            "Business object of type -> '%s' with '%s' = '%s' does not exist in Salesforce!",
            object_type,
            search_field,
            search_value,
        )
        self.logger.info(
            "We try to create the Salesforce object of type -> '%s'...",
            object_type,
        )

        # Get a helper dict to quickly lookup Salesforce properties
        # for given set + attribute name:
        property_lookup = self.get_business_object_properties(
            bo_type_name=object_type,
        )
        # In case we couldn't find properties for the given Business Object Type
        # we bail out...
        if not property_lookup:
            self.logger.warning(
                "Cannot create Salesforce object - no business object properties found!",
            )
            return None

        categories = workspace.get("categories", [])
        parameter_dict = {}
        # We process all category entries in workspace payload
        # and see if we have a matching mapping to a business property
        # in the BO Type definition:
        for category in categories:
            # generate the lookup key:
            key = ""
            if "set" in category:
                key += category["set"] + "-"
            key += category.get("attribute")
            # get the attribute value:
            value = category.get("value")
            # lookup the mapping
            mapping = property_lookup.get(key, None)
            # Check if we have a mapping:
            if mapping:
                property_name = mapping.get("@propertyName", None)
                self.logger.debug(
                    "Found business property -> '%s' for attribute -> '%s'",
                    property_name,
                    category.get("attribute"),
                )
                parameter_dict[property_name] = value
            else:
                self.logger.debug(
                    "Attribute -> '%s' (key -> %s) does not have a mapped business property.",
                    category.get("attribute"),
                    key,
                )

        if not parameter_dict:
            self.logger.warning(
                "Cannot create Salesforce object of type -> '%s' - no parameters found!",
                object_type,
            )
            return None

        self.logger.info(
            "Create Salesforce object of type -> '%s' with parameters -> %s...",
            object_type,
            str(parameter_dict),
        )
        #
        # Now we try to create the Salesforce object
        #
        response = self._salesforce.add_object(
            object_type=object_type,
            **parameter_dict,
        )
        bo_id = self._salesforce.get_result_value(response=response, key="id")
        if bo_id:
            self.logger.info(
                "Successfully created Salesforce business object with ID -> %s of type -> '%s'.",
                bo_id,
                object_type,
            )
        else:
            self.logger.error(
                "Failed to create Salesforce business object of type -> '%s'!",
                object_type,
            )
    else:  # BO found
        self.logger.debug(
            "Retrieved ID -> %s for Salesforce object type -> '%s' (looking up -> '%s' in field -> '%s')",
            bo_id,
            object_type,
            search_value,
            search_field,
        )

    return bo_id

get_status_file(payload_section_name, payload_specific=True, prefix='success_')

Get the status file and read it into a list of dictionaries.

Parameters:

Name	Type	Description	Default
payload_section_name	str	The name of the payload section. This is used to construct the file name.	required
payload_specific	bool	Whether or not the success should be specific for each payload file or if success is "global" - like for the deletion of the existing M365 teams (which we don't want to execute per payload file)	True
prefix	str	The prefix of the file. Typically, either "success_" or "failure_".	'success_'

Returns:

Name	Type	Description
list	list | None	Content of the status file as a list of dictionaries or None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_status_file")
def get_status_file(
    self,
    payload_section_name: str,
    payload_specific: bool = True,
    prefix: str = "success_",
) -> list | None:
    """Get the status file and read it into a list of dictionaries.

    Args:
        payload_section_name (str):
            The name of the payload section. This
            is used to construct the file name.
        payload_specific (bool, optional):
            Whether or not the success should be specific for
            each payload file or if success is "global" - like for the deletion
            of the existing M365 teams (which we don't want to execute per
            payload file)
        prefix (str, optional):
            The prefix of the file. Typically, either "success_" or "failure_".

    Returns:
        list:
            Content of the status file as a list of dictionaries or None in case of an error.

    """

    self.logger.info(
        "Get the status file of the payload section -> '%s'...",
        payload_section_name,
    )

    while not self._otcs.is_ready():
        self.logger.info(
            "OTCS is not ready. Cannot read status file for -> '%s' from OTCS. Waiting 30 seconds and retry...",
            payload_section_name,
        )
        time.sleep(30)

    response = self._otcs.get_node_by_volume_and_path(
        volume_type=self._otcs.VOLUME_TYPE_PERSONAL_WORKSPACE,
    )  # read from Personal Workspace of Admin
    source_folder_id = self._otcs.get_result_value(response=response, key="id")
    if not source_folder_id:
        source_folder_id = 2004  # use Personal Workspace ID of Admin as fallback

    file_name = self.get_status_file_name(
        payload_section_name=payload_section_name,
        payload_specific=payload_specific,
        prefix=prefix,
    )

    status_document = self._otcs.get_node_by_parent_and_name(
        parent_id=int(source_folder_id),
        name=file_name,
        show_error=True,
    )
    status_file_id = self._otcs.get_result_value(response=status_document, key="id")
    if not status_file_id:
        self.logger.error("Cannot find status file -> '%s'!", file_name)
        return None

    return self._otcs.get_json_document(node_id=status_file_id)

get_status_file_name(payload_section_name, payload_specific=True, prefix='success_')

Construct the name of the status file.

Parameters:

Name	Type	Description	Default
payload_section_name	str	The name of the payload section. This is used to construct the file name	required
payload_specific	bool	Whether or not the success should be specific for each payload file or if success is "global" - like for the deletion of the existing M365 teams (which we don't want to execute per payload file)	True
prefix	str	The prefix of the file name. Typically, either "success_" or "failure_".	'success_'

Returns:

Name	Type	Description
str	str	The constructed name of the payload section file.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="get_status_file_name")
def get_status_file_name(
    self,
    payload_section_name: str,
    payload_specific: bool = True,
    prefix: str = "success_",
) -> str:
    """Construct the name of the status file.

    Args:
        payload_section_name (str):
            The name of the payload section. This
            is used to construct the file name
        payload_specific (bool, optional):
            Whether or not the success should be specific for
            each payload file or if success is "global" - like for the deletion
            of the existing M365 teams (which we don't want to execute per
            payload file)
        prefix (str, optional):
            The prefix of the file name. Typically, either "success_" or "failure_".

    Returns:
        str:
            The constructed name of the payload section file.

    """

    # Some sections are actually not payload specific like teamsM365Cleanup
    # we don't want external payload runs to re-apply this processing:
    if payload_specific:
        file_name = os.path.basename(self._payload_source)  # remove directories
        # Split once at the first occurance of a dot
        # as the _payload_source may have multiple suffixes
        # such as .yml.gz.b64:
        file_name = file_name.split(".", 1)[0]
        file_name = prefix + file_name + "_" + payload_section_name + ".json"
    else:
        file_name = prefix + payload_section_name + ".json"

    return file_name

get_users()

Get all users in payload.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

def get_users(self) -> list:
    """Get all users in payload."""
    return self._users

get_workspaces()

Get all workspaces in payload.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

def get_workspaces(self) -> list:
    """Get all workspaces in payload."""
    return self._workspaces

init_guidewire(guidewire_external_system)

Initialize Guidewire object for workspace creation.

This is needed to query Guidewire REST API to lookup IDs of Guidewire objects.

Parameters:

Name	Type	Description	Default
guidewire_external_system	dict	The payload of the Guidewire external system created before.	required

Returns:

Type	Description
Guidewire | None	Guidewire | None: Guidewire object or None in case an error occured.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="init_guidewire")
def init_guidewire(self, guidewire_external_system: dict) -> Guidewire | None:
    """Initialize Guidewire object for workspace creation.

    This is needed to query Guidewire REST API to lookup IDs of Guidewire objects.

    Args:
        guidewire_external_system (dict):
            The payload of the Guidewire external system created before.

    Returns:
        Guidewire | None:
            Guidewire object or None in case an error occured.

    """

    if not guidewire_external_system:
        return None

    system_name = guidewire_external_system["external_system_name"]
    username = guidewire_external_system["username"]
    password = guidewire_external_system["password"]
    base_url = guidewire_external_system["base_url"]
    as_url = guidewire_external_system["as_url"]
    auth_type = guidewire_external_system.get("auth_type", "basic").lower()

    self.logger.info("Connection parameters for %s:", system_name)
    self.logger.info("Guidewire base URL        = %s", base_url)
    self.logger.info("Guidewire application URL = %s", as_url)
    self.logger.info("Guidewire username        = %s", username)
    self.logger.debug("Guidewire password        = %s", password)
    guidewire_object = Guidewire(
        base_url=base_url,
        as_url=as_url,
        auth_type=auth_type,
        username=username,
        password=password,
        logger=self.logger,
    )

    return guidewire_object

init_payload()

Read the YAML or Terraform HCL payload file.

Returns:

Type	Description
dict | None	dict | None: The payload as a Python dict. Elements are the different payload sections. None in case the file couldn't be found or read.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="init_payload")
def init_payload(self) -> dict | None:
    """Read the YAML or Terraform HCL payload file.

    Args:
        None

    Returns:
        dict | None:
            The payload as a Python dict. Elements are the different payload sections.
            None in case the file couldn't be found or read.

    """

    self._payload = load_payload(self._payload_source, self.logger)
    if not self._payload:
        return None

    self._payload_sections = self._payload.get("payloadSections", None)
    self._payload_options = self._payload.get("payloadOptions", None)

    # Retrieve all the payload sections and store them in lists
    # (these are object variables that we want to initialize
    # even if payload is empty):
    self._webhooks = self.get_payload_section("webHooks")
    self._webhooks_post = self.get_payload_section("webHooksPost")
    self._resources = self.get_payload_section("resources")
    self._partitions = self.get_payload_section("partitions")
    self._licenses = self.get_payload_section("licenses")
    self._synchronized_partitions = self.get_payload_section(
        "synchronizedPartitions",
    )
    self._oauth_clients = self.get_payload_section("oauthClients")
    self._application_roles = self.get_payload_section("applicationRoles")
    self._auth_handlers = self.get_payload_section("authHandlers")
    self._trusted_sites = self.get_payload_section("trustedSites")
    self._system_attributes = self.get_payload_section("systemAttributes")
    self._docgen_settings = self.get_payload_section("docgenSettings")
    self._groups = self.get_payload_section("groups")
    self._users = self.get_payload_section("users")
    if self._users:
        # Check if multiple user instances should be created
        self.init_payload_user_instances()
    self._admin_settings = self.get_payload_section("adminSettings")
    self._admin_settings_post = self.get_payload_section("adminSettingsPost")
    self._exec_pod_commands = self.get_payload_section("execPodCommands")
    self._exec_commands = self.get_payload_section("execCommands")
    self._kubernetes = self.get_payload_section("kubernetes")
    self._exec_database_commands = self.get_payload_section("execDatabaseCommands")
    self._external_systems = self.get_payload_section("externalSystems")
    self._transport_packages = self.get_payload_section("transportPackages")
    self._content_transport_packages = self.get_payload_section(
        "contentTransportPackages",
    )
    self._transport_packages_post = self.get_payload_section(
        "transportPackagesPost",
    )
    self._workspace_templates = self.get_payload_section("workspaceTemplates")
    self._workspaces = self.get_payload_section("workspaces")
    self._bulk_datasources = self.get_payload_section("bulkDatasources")
    self._bulk_workspaces = self.get_payload_section("bulkWorkspaces")
    self._bulk_workspace_relationships = self.get_payload_section(
        "bulkWorkspaceRelationships",
    )
    self._bulk_documents = self.get_payload_section("bulkDocuments")
    self._bulk_items = self.get_payload_section("bulkItems")
    self._bulk_classifications = self.get_payload_section("bulkClassifications")
    self._sap_rfcs = self.get_payload_section("sapRFCs")
    self._sap_rfcs_post = self.get_payload_section("sapRFCsPost")
    self._web_reports = self.get_payload_section("webReports")
    self._web_reports_post = self.get_payload_section("webReportsPost")
    self._cs_applications = self.get_payload_section("csApplications")
    self._additional_group_members = self.get_payload_section(
        "additionalGroupMemberships",
    )
    self._additional_access_role_members = self.get_payload_section(
        "additionalAccessRoleMemberships",
    )
    self._additional_application_role_assignments = self.get_payload_section(
        "additionalApplicationRoleAssignments",
    )
    self._renamings = self.get_payload_section("renamings")
    self._items = self.get_payload_section("items")
    self._items_post = self.get_payload_section("itemsPost")
    self._permissions = self.get_payload_section("permissions")
    self._permissions_post = self.get_payload_section("permissionsPost")
    self._workspace_permissions = self.get_payload_section("workspacePermissions")
    self._assignments = self.get_payload_section("assignments")
    self._security_clearances = self.get_payload_section("securityClearances")
    self._supplemental_markings = self.get_payload_section("supplementalMarkings")
    self._records_management_settings = self.get_payload_section(
        "recordsManagementSettings",
    )
    self._holds = self.get_payload_section("holds")
    self._doc_generators = self.get_payload_section("documentGenerators")
    self._workflows = self.get_payload_section("workflows")
    self._browser_automations = self.get_payload_section("browserAutomations")
    self._browser_automations_post = self.get_payload_section(
        "browserAutomationsPost",
    )
    self._test_automations = self.get_payload_section("testAutomations")
    self._appworks_configurations = self.get_payload_section("appworks")
    self._avts_repositories = self.get_payload_section("avtsRepositories")
    self._avts_questions = self.get_payload_section("avtsQuestions")
    self._embeddings = self.get_payload_section("embeddings")
    self._nifi_flows = self.get_payload_section("nifi")

    return self._payload

init_payload_user_instances()

Read setting for Multiple User instances.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="init_payload_user_instances")
def init_payload_user_instances(self) -> None:
    """Read setting for Multiple User instances."""

    for dic in self._payload_sections:
        if dic.get("name") == "users":
            users_payload = dic
            break
    user_instances = users_payload.get("additional_instances", 0)

    if user_instances == 0:
        self.logger.info("No additional user instances configured (default = 0).")
        return

    i = 0

    original_users = copy.deepcopy(self._users)
    while i <= user_instances:
        for user in copy.deepcopy(original_users):
            user["name"] = user["name"] + "-" + str(i).zfill(2)
            user["lastname"] = user["lastname"] + " " + str(i).zfill(2)
            user["enable_sap"] = False
            user["enable_o365"] = False
            user["enable_core_share"] = False
            user["enable_salesforce"] = False
            user["enable_successfactors"] = False

            self.logger.info(
                "Creating additional user instance -> '%s'",
                user["name"],
            )
            self.logger.debug("Create user instance -> %s", user)
            self._users.append(user)

        i = i + 1

    return

init_salesforce(salesforce_external_system)

Initialize Salesforce object for workspace creation.

This is needed to query Salesforce API to lookup IDs of Salesforce objects.

Parameters:

Name	Type	Description	Default
salesforce_external_system	dict	The payload of the Salesforce external system created before.	required

Returns:

Type	Description
Salesforce | None	Salesforce | None: Salesforce object or None in case an error occured.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="init_salesforce")
def init_salesforce(self, salesforce_external_system: dict) -> Salesforce | None:
    """Initialize Salesforce object for workspace creation.

    This is needed to query Salesforce API to lookup IDs of Salesforce objects.

    Args:
        salesforce_external_system (dict):
            The payload of the Salesforce external system created before.

    Returns:
        Salesforce | None:
            Salesforce object or None in case an error occured.

    """

    if not salesforce_external_system:
        return None

    username = salesforce_external_system["username"]
    password = salesforce_external_system["password"]
    base_url = salesforce_external_system["base_url"]
    authorization_url = salesforce_external_system.get("token_endpoint", "")
    client_id = salesforce_external_system["oauth_client_id"]
    client_secret = salesforce_external_system["oauth_client_secret"]

    self.logger.info("Connection parameters Salesforce:")
    self.logger.info("Salesforce base URL          = %s", base_url)
    self.logger.info("Salesforce authorization URL = %s", base_url)
    self.logger.info("Salesforce username          = %s", username)
    self.logger.debug("Salesforce password          = %s", password)
    self.logger.info("Salesforce client ID         = %s", client_id)
    self.logger.debug("Salesforce client secret     = %s", client_secret)
    salesforce_object = Salesforce(
        base_url=base_url,
        client_id=client_id,
        client_secret=client_secret,
        username=username,
        password=password,
        authorization_url=authorization_url,
        logger=self.logger,
    )

    self._salesforce = salesforce_object

    return salesforce_object

init_sap(sap_external_system, direct_application_server_login=True)

Initialize SAP object for RFC communication with SAP S/4HANA.

Parameters:

Name	Type	Description	Default
sap_external_system	dict	SAP external system created before	required
direct_application_server_login	bool	Flag to control whether we comminicate directly with SAP application server or via a load balancer	True

Returns:

Type	Description
SAP | None	SAP | None: The SAP object or None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="init_sap")
def init_sap(
    self,
    sap_external_system: dict,
    direct_application_server_login: bool = True,
) -> SAP | None:
    """Initialize SAP object for RFC communication with SAP S/4HANA.

    Args:
        sap_external_system (dict):
            SAP external system created before
        direct_application_server_login (bool, optional):
            Flag to control whether we comminicate directly with
            SAP application server or via a load balancer

    Returns:
        SAP | None:
            The SAP object or None in case of an error.

    """

    if not sap_external_system:
        return None

    username = sap_external_system["username"]
    password = sap_external_system["password"]
    # "external_system_hostname" is extracted from as_url in process_external_systems()
    host = sap_external_system["external_system_hostname"]
    client = sap_external_system.get("client", "100")
    system_number = sap_external_system.get("external_system_number", "00")
    system_id = sap_external_system["external_system_name"]
    group = sap_external_system.get("group", "PUBLIC")
    destination = sap_external_system.get("destination", "")

    self.logger.info("Connection parameters SAP:")
    self.logger.info("SAP Hostname             = %s", host)
    self.logger.info("SAP Client               = %s", client)
    self.logger.info("SAP System Number        = %s", system_number)
    self.logger.info("SAP System ID            = %s", system_id)
    self.logger.info("SAP User Name            = %s", username)
    if not direct_application_server_login:
        self.logger.info("SAP Group Name (for RFC) = %s", group)
    if destination:
        self.logger.info("SAP Destination          = %s", destination)

    if direct_application_server_login:
        self.logger.info(
            "SAP Login                = Direct Application Server (ashost)",
        )
        sap_object = SAP(
            username=username,
            password=password,
            ashost=host,
            client=client,
            system_number=system_number,
            system_id=system_id,
            destination=destination,
            logger=self.logger,
        )
    else:
        self.logger.info(
            "SAP Login                = Logon with load balancing (mshost)",
        )
        sap_object = SAP(
            username=username,
            password=password,
            mshost=host,
            group=group,
            client=client,
            system_number=system_number,
            system_id=system_id,
            destination=destination,
            logger=self.logger,
        )

    self._sap = sap_object

    if (
        "archive_logical_name" in sap_external_system
        and "archive_certificate_file" in sap_external_system
        and self._otac
    ):
        self.logger.info(
            "Put certificate file -> '%s' for logical archive -> '%s' into Archive Center...",
            sap_external_system["archive_certificate_file"],
            sap_external_system["archive_logical_name"],
        )
        response = self._otac.put_cert(
            sap_external_system["external_system_name"],
            sap_external_system["archive_logical_name"],
            sap_external_system["archive_certificate_file"],
        )
        if not response:
            self.logger.error("Failed to install Archive Center certificate!")
        else:
            self.logger.info(
                "Enable certificate file -> '%s' for logical archive -> '%s'...",
                sap_external_system["archive_certificate_file"],
                sap_external_system["archive_logical_name"],
            )
            response = self._otac.enable_cert(
                auth_id=sap_external_system["external_system_name"],
                logical_archive=sap_external_system["archive_logical_name"],
                enable=True,
            )
            if not response:
                self.logger.debug("Failed to enable Archive Center certificate!")

    return sap_object

init_successfactors(sucessfactors_external_system)

Initialize SuccessFactors object for workspace creation.

This is needed synchronize user passwords and emails with SuccessFactors.

Parameters:

Name	Type	Description	Default
sucessfactors_external_system	dict	The payload of the SuccessFactors external system created before.	required

Returns:

Name	Type	Description
SuccessFactors	SuccessFactors | None	The SuccessFactors object.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="init_successfactors")
def init_successfactors(
    self,
    sucessfactors_external_system: dict,
) -> SuccessFactors | None:
    """Initialize SuccessFactors object for workspace creation.

    This is needed synchronize user passwords and emails with SuccessFactors.

    Args:
        sucessfactors_external_system (dict):
            The payload of the SuccessFactors external system created before.

    Returns:
        SuccessFactors: The SuccessFactors object.

    """

    def extract_company_from_url(url: str) -> str:
        parsed_url = urlparse(url)
        query_params = parse_qs(parsed_url.query)
        company_value = query_params.get("company", [""])[0]
        return company_value

    if not sucessfactors_external_system:
        return None

    username = sucessfactors_external_system["username"]
    password = sucessfactors_external_system["password"]
    base_url = sucessfactors_external_system["base_url"]
    as_url = sucessfactors_external_system["as_url"]
    saml_url = sucessfactors_external_system.get("saml_url", "")
    company_id = extract_company_from_url(saml_url)
    client_id = sucessfactors_external_system["oauth_client_id"]
    client_secret = sucessfactors_external_system["oauth_client_secret"]

    self.logger.info("Connection parameters SuccessFactors:")
    self.logger.info("SuccessFactors base URL            = %s", base_url)
    self.logger.info("SuccessFactors application URL     = %s", as_url)
    self.logger.info("SuccessFactors username            = %s", username)
    self.logger.debug("SuccessFactors password            = %s", password)
    self.logger.info("SuccessFactors client ID           = %s", client_id)
    self.logger.debug("SuccessFactors client secret       = %s", client_secret)
    self.logger.info("SuccessFactors company ID (tenant) = %s", company_id)
    successfactors_object = SuccessFactors(
        base_url=base_url,
        as_url=as_url,
        client_id=client_id,
        client_secret=client_secret,
        username=username,
        password=password,
        company_id=company_id,
        logger=self.logger,
    )

    self._successfactors = successfactors_object

    return successfactors_object

lookup_data_source_value(data_source, lookup_column, lookup_value, separator='|')

Lookup a value in a given data source (specified by payload dict).

If the data source has not been loaded before then load the data source. As this runs in a multi-threading environment we need to protect the data source update from multiple threads doing it at the same time. A global data_load_lock variable acts as a mutex.

Parameters:

Name	Type	Description	Default
data_source	dict	The payload dictionary of the data source definition.	required
lookup_column	str	The name of the column in the data frame (see Data class).	required
lookup_value	str	The value to lookup - selection criteria for the result row.	required
separator	str	The string list delimiter / separator. The pipe symbol | is the default as it is unlikely to appear in a normal string (other than a plain comma). The separator is NOT looked for in the lookup_value but in the column that is given by lookup_column!	'|'

Returns:

Type	Description
Series | None	pd.Series | None: Row that matches the lookup value in the lookup column.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="lookup_data_source_value")
def lookup_data_source_value(
    self,
    data_source: dict,
    lookup_column: str,
    lookup_value: str,
    separator: str = "|",
) -> pd.Series | None:
    """Lookup a value in a given data source (specified by payload dict).

    If the data source has not been loaded before then load the data source.
    As this runs in a multi-threading environment we need to protect
    the data source update from multiple threads doing it at the same time.
    A global data_load_lock variable acts as a mutex.

    Args:
        data_source (dict):
            The payload dictionary of the data source definition.
        lookup_column (str):
            The name of the column in the data frame (see Data class).
        lookup_value (str):
            The value to lookup - selection criteria for the result row.
        separator (str, optional):
            The string list delimiter / separator. The pipe symbol | is the default
            as it is unlikely to appear in a normal string (other than a plain comma).
            The separator is NOT looked for in the lookup_value but in the column that
            is given by lookup_column!

    Returns:
        pd.Series | None:
            Row that matches the lookup value in the lookup column.

    """

    data_source_name = data_source.get("name")
    if not data_source_name:
        self.logger.error("Data source has no name!")
        return None

    # We don't want multiple threads to trigger a data source load at the same time,
    # so we use a lock (mutex) to avoid this:
    data_load_lock.acquire()
    try:
        # First we check if the data source has been loaded already.
        # If not, we load the data source on the fly:
        data_source_data: Data = data_source.get("data")
        if not data_source_data:
            self.logger.info(
                "Lookup data source -> '%s' has no data yet. Reloading...",
                data_source_name,
            )
            data_source_data = self.process_bulk_datasource(
                data_source_name=data_source_name,
                force_reload=True,
            )
    finally:
        # Ensure the lock is released even if an error occurs
        data_load_lock.release()

    # If we still don't have data from this data source we bail out:
    if data_source_data is None:  # important to use "is None" here!
        self.logger.error(
            "Lookup data source -> '%s' has no data and reload did not work. Cannot lookup value -> '%s' in column -> '%s'!",
            data_source_name,
            lookup_value,
            lookup_column,
        )
        return None

    # Lookup the data frame row (pd.Series) in the given
    # column with the given lookup value:
    lookup_row: pd.Series = data_source_data.lookup_value(
        lookup_column=lookup_column,
        lookup_value=lookup_value,
        separator=separator,
    )

    return lookup_row

lookup_external_system(ext_system_id, prefix='success_payload_')

Lookup an external system.

Considers the current payload but also payload processed before (which is stored as JSON file in the Admin Personal Workspace).

TODO: This is a workaround as the REST API for external systems is too limited and does not return the actual configuration settings of external systems. Check for newer OTCS versions.

Parameters:

Name	Type	Description	Default
ext_system_id	str	The external system name to lookup.	required
prefix	str	The prefix of the success file in the Admin personal workspace.	'success_payload_'

Returns:

Type	Description
dict | None	dict | None: The configuration data of the external system.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="lookup_external_system")
def lookup_external_system(self, ext_system_id: str, prefix: str = "success_payload_") -> dict | None:
    """Lookup an external system.

    Considers the current payload but also payload processed before (which is stored
    as JSON file in the Admin Personal Workspace).

    TODO: This is a workaround as the REST API for external systems is too limited
    and does not return the actual configuration settings of external systems.
    Check for newer OTCS versions.

    Args:
        ext_system_id (str):
            The external system name to lookup.
        prefix (str):
            The prefix of the success file in the Admin personal workspace.

    Returns:
        dict | None:
            The configuration data of the external system.

    """

    # First check if external system is in current payload:
    external_system = next(
        (item for item in self._external_systems if item["external_system_name"] == ext_system_id), None
    )
    if external_system:
        self.logger.info("Found external system -> '%s' declared in current payload.", ext_system_id)
        return external_system

    # Now we try to load it from previous processed payloads:

    if not self.check_status_file(payload_section_name="externalSystems", payload_specific=False, prefix=prefix):
        self.logger.error("Cannot find external system -> '%s'", ext_system_id)
        return None

    additional_systems = self.get_status_file(
        payload_section_name="externalSystems", payload_specific=False, prefix=prefix
    )

    # Merge avoiding duplicates and only enabled entries. existing_names is a set:
    existing_names = {item["external_system_name"] for item in self._external_systems}
    for sys in additional_systems:
        if sys["enabled"] and sys["external_system_name"] not in existing_names:
            self._external_systems.append(sys)
            existing_names.add(sys["external_system_name"])

    # Try finding the external system payload again after merging:
    external_system = next(
        (item for item in self._external_systems if item["external_system_name"] == ext_system_id), None
    )

    if external_system:
        self.logger.info("Found external system -> '%s' in previously processed payload file.", ext_system_id)
    else:
        self.logger.error(
            "Cannot find external system -> '%s' in list of external systems -> %s!",
            ext_system_id,
            str(existing_names),
        )

    return external_system

prepare_category_data(categories_payload, source_node_id, source_is_document=False)

Prepare the metadata structure for a new workspace or document.

This translates the category information from the PAYLOAD into the structure required by the OTCS REST API for category updates.

The difficulty is to get the category schema for the OTCS category. The method tries different approaches to get the schema: 1. Check if the category schema is on a source node (e.g. inherited from a parent node) 2. Find the category definition by nickname and then assign it to the source node (then retrying approach 1) 3. Find the category definition by a unique name and then assign it to the source node (then retrying approach 1)

Parameters:

Name	Type	Description	Default
categories_payload	dict	The payload information for the workspace or document categories.	required
source_node_id	int	The item to derive or inherit the category data from. We expect this to be a folder, workspace or document that has the category assigned.	required
source_is_document	bool	If the source node type is a document we need to handle category assignment a bit different. Default is False.	False

Returns:

Type	Description
dict | None	dict | None: Category data structure required for subsequent document upload OTCS REST call.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="prepare_category_data")
def prepare_category_data(
    self,
    categories_payload: dict,
    source_node_id: int,
    source_is_document: bool = False,
) -> dict | None:
    """Prepare the metadata structure for a new workspace or document.

    This translates the category information from the PAYLOAD into the structure
    required by the OTCS REST API for category updates.

    The difficulty is to get the category schema for the OTCS category. The method tries different
    approaches to get the schema:
    1. Check if the category schema is on a source node (e.g. inherited from a parent node)
    2. Find the category definition by nickname and then assign it to the source node (then retrying approach 1)
    3. Find the category definition by a unique name and then assign it to the source node (then retrying approach 1)

    Args:
        categories_payload (dict):
            The payload information for the workspace or document categories.
        source_node_id (int):
            The item to derive or inherit the category data from. We expect this to
            be a folder, workspace or document that has the category assigned.
        source_is_document (bool, optional):
            If the source node type is a document we need to handle category assignment
            a bit different. Default is False.

    Returns:
        dict | None:
            Category data structure required for subsequent document upload OTCS REST call.

    """

    # Determine the category names in the payload. For this we use a
    # comprehension to create a set (with the curly braces) of unique category
    # names used in the payload that then is converted to a list:
    category_names = list({category["name"] for category in categories_payload})

    # Determine the names of all categories inherited from the source_node_id:
    response = self._otcs_frontend.get_node_categories(node_id=source_node_id)
    if response and response["results"]:
        inherited_category_names = [
            next(iter(item["metadata"]["categories"].values()))["name"]
            for item in response["results"]
            if item.get("metadata")
        ]
    else:
        inherited_category_names = []

    # Calculate the category names we have in payload that are NOT inherited (set difference):
    missing_categories: set[str] = set(category_names) - set(inherited_category_names)

    # Are any of the payload categories NOT inherited from the source_node_id ?
    if missing_categories:
        self.logger.info(
            "Categories -> %s are in payload but not found on source node with ID -> %s. Try to find these categories by nicknames or unique names...",
            str(missing_categories),
            str(source_node_id),
        )
        for category_name in missing_categories:  # category_names:
            category_node_id = None
            # First try via nickname. We expect prefix "cat_" and snake case nicknames:
            category_nickname = (
                "cat_"
                + category_name.replace(" - ", "_")
                .replace("-", "_")
                .replace(" ", "_")
                .replace("___", "_")
                .replace("(", "")
                .replace(")", "")
                .lower()
            )
            category_node = self._otcs_frontend.get_node_from_nickname(
                nickname=category_nickname,
            )
            category_node_id = self._otcs_frontend.get_result_value(
                response=category_node,
                key="id",
            )
            if category_node_id:
                self.logger.info(
                    "Found category -> '%s' with ID -> %s via nickname -> '%s'.",
                    category_name,
                    category_node_id,
                    category_nickname,
                )
            # Next try via unique names:
            else:
                response = self._otcs_frontend.get_unique_names(
                    names=[category_name],
                    subtype=self._otcs_frontend.ITEM_TYPE_CATEGORY,
                )
                if response and "results" in response and response["results"]:
                    category_node_id = next(
                        (result["NodeId"] for result in response["results"] if result["NodeName"] == category_name),
                        None,
                    )
                    if category_node_id:
                        self.logger.info(
                            "Found category -> '%s' with ID -> %s via unique name lookup!",
                            category_name,
                            category_node_id,
                        )

            # if we found the payload category via nickname or unique name
            # we assign it to thew source node (typically the parent node):
            if category_node_id:
                self._otcs_frontend.assign_category(
                    node_id=source_node_id,
                    category_id=category_node_id,
                    inheritance=None if source_is_document else True,
                    apply_to_sub_items=not source_is_document,
                )
        # end for category_name in missing_categories

        # Now with the category assigned to the parent (source node id)
        # We retry getting the inherited category:
        response = self._otcs_frontend.get_node_categories(node_id=source_node_id)
    # end if missing_categories:

    # Initialize the result dict we will return at the end of the method
    # and the list of inherited categories:
    category_data = {}
    inherited_categories = []

    # Redo the test...
    if not response or not response["results"]:
        self.logger.warning(
            "Source node with ID -> %s does not inherit categories but we have category payload! Processing without assiging category...",
            source_node_id,
        )
    else:
        inherited_categories = response["results"]

    # we iterate over all parent categories that are inherited
    # to the new document and try to find matching payload values...
    for inherited_category in inherited_categories:
        # We use the "metadata_order" which is a list of typically one
        # element that includes the category ID:
        metadata_order = inherited_category["metadata_order"]

        # If it is not a list or empty we continue with the
        # next inherited category:
        if not metadata_order["categories"] or not isinstance(
            metadata_order["categories"],
            list,
        ):
            continue
        inherited_category_id = metadata_order["categories"][0]

        # We use the "metadata" dict to determine the category name
        # the keys of the dict are the category ID and attribute IDs
        # the first element in the dict is always the category itself.
        metadata = inherited_category["metadata"]["categories"]
        category_name = metadata[str(inherited_category_id)]["name"]

        self.logger.debug(
            "Source node ID -> %s has category -> '%s' (%s)",
            source_node_id,
            category_name,
            inherited_category_id,
        )

        # The following method returns two values: the category ID and
        # a dict of the attributes. If the category is not found
        # on the parent node it returns -1 for the category ID
        # and an empty dict for the attribute definitions:
        (
            category_id,
            attribute_definitions,
        ) = self._otcs_frontend.get_node_category_definition(
            node_id=source_node_id,
            category_name=category_name,
        )
        if category_id == -1:
            self.logger.error(
                "The item with ID -> %s does not have the specified category -> '%s' assigned. Skipping...",
                source_node_id,
                category_name,
            )
            continue

        # We now initialize the substructure for this particular category:
        category_data[str(category_id)] = {}

        self.logger.debug(
            "Processing the attributes in payload to find values for the inherited category -> '%s' (%s)...",
            category_name,
            category_id,
        )
        # now we fill the prepared (but empty) category_data
        # with the actual attribute values from the payload.
        # For this we traverse all category dicts in the payload
        # and check if they include data for the currently processed
        # category:
        for attribute in categories_payload:
            attribute_name = attribute["attribute"]
            set_name = attribute.get("set", "")
            row = attribute.get("row", "")
            if attribute["name"] != category_name:
                self.logger.debug(
                    "Attribute -> '%s' does not belong to category -> '%s'. Skipping...",
                    attribute_name,
                    category_name,
                )
                continue
            attribute_value = attribute["value"]

            # Set attributes are constructed with <set>:<attribute>
            # by method get_node_category_definition(). This is not
            # an OTCS REST syntax but specific for payload.py
            if set_name:
                attribute_name = set_name + ":" + attribute_name

            if attribute_name not in attribute_definitions:
                self.logger.error(
                    "Illegal attribute name -> '%s' in payload for category -> '%s'",
                    attribute_name,
                    category_name,
                )
                continue
            attribute_type = attribute_definitions[attribute_name]["type"]
            attribute_id = attribute_definitions[attribute_name]["id"]
            # For multi-line sets the "x" is the placeholder for the
            # row number. We need to replace it with the actual row number
            # given in the payload:
            if "_x_" in attribute_id:
                if not row:
                    self.logger.error(
                        "Row number is not specified in payload for attribute -> '%s' (%s)",
                        attribute_name,
                        attribute_id,
                    )
                    continue
                attribute_id = attribute_id.replace("_x_", "_" + str(row) + "_")

            # Special treatment for type user: determine the ID for the login name.
            # the ID is the actual value we have to put in the attribute:
            if attribute_type == "user":
                user = self._otcs_frontend.get_user(
                    name=attribute_value,
                    show_error=True,
                )
                user_id = self._otcs_frontend.get_result_value(
                    response=user,
                    key="id",
                )
                if not user_id:
                    self.logger.error(
                        "Cannot find user with login name -> '%s'. Skipping...",
                        attribute_value,
                    )
                    continue
                attribute_value = user_id
            category_data[str(category_id)][attribute_id] = attribute_value
        # end for attribute in categories_payload:

        # If for none of the attributes of the current category ID a value was found
        # in the payload we remove the dictionary entry to not cause problems
        # for later category updates:
        if not category_data[str(category_id)]:
            del category_data[str(category_id)]
    # end for inherited_category in inherited_categories:

    self.logger.debug("Resulting category data -> %s", str(category_data))

    return category_data

prepare_datasource_file(data_source, filename)

Download data source from HTTP/HTTPS link and store it in a local temp file.

Parameters:

Name	Type	Description	Default
data_source	dict	The data source object from payload.	required
filename	str	The filename to check if it is a reference to an external file.	required

Returns:

Name	Type	Description
str	str	Filename or Filename of the local temp file.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="prepare_datasource_file")
def prepare_datasource_file(self, data_source: dict, filename: str) -> str:
    """Download data source from HTTP/HTTPS link and store it in a local temp file.

    Args:
        data_source (dict):
            The data source object from payload.
        filename (str):
            The filename to check if it is a reference to an external file.

    Returns:
        str:
            Filename or Filename of the local temp file.

    """

    if not filename.startswith(("http://", "https://")):
        return filename

    name = data_source.get("name", "")
    tmp_filename = os.path.join(tempfile.gettempdir(), "{}_{}".format(name, os.path.basename(filename)))

    if os.path.isfile(tmp_filename):
        self.logger.info("Reusing previously downloaded file -> '%s'.", tmp_filename)
        return tmp_filename

    try:
        self.logger.info(
            "Downloading data source from -> %s to -> %s...",
            filename,
            tmp_filename,
        )
        response = requests.get(filename, stream=True, timeout=10)

        with open(tmp_filename, "wb") as f:
            for chunk in response.iter_content(chunk_size=1024):
                if chunk:
                    f.write(chunk)

    except Exception:
        self.logger.error("Error downloading JSON data source -> '%s'!", filename)

    return tmp_filename

prepare_item_create_form(parent_id, categories, subtype=OTCS.ITEM_TYPE_DOCUMENT)

Prepare the category structure for the item creation.

Parameters:

Name	Type	Description	Default
parent_id	int	The node the category should be applied to.	required
categories	list	The categories list from the document payload.	required
subtype	int	The subtype of the new node. Default is document.	ITEM_TYPE_DOCUMENT

Returns:

Type	Description
dict | None	dict | None: Category structure for workspace creation or None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="prepare_item_create_form")
def prepare_item_create_form(
    self,
    parent_id: int,
    categories: list,
    subtype: int = OTCS.ITEM_TYPE_DOCUMENT,
) -> dict | None:
    """Prepare the category structure for the item creation.

    Args:
        parent_id (int):
            The node the category should be applied to.
        categories (list):
            The categories list from the document payload.
        subtype (int):
            The subtype of the new node. Default is document.

    Returns:
        dict | None:
            Category structure for workspace creation or None
            in case of an error.

    """

    category_create_data = {}

    category_ids = []

    # Get unique combination of volume, path, name to get a list of all categories
    unique_categories = {
        (entry.get("volume", OTCS.VOLUME_TYPE_CATEGORIES_VOLUME), entry.get("path", None), entry["name"])
        for entry in categories
    }

    # Determine the names of all categories inherited from the parent_id:
    response = self._otcs_frontend.get_node_categories(node_id=parent_id)
    if response and response["results"]:
        category_ids = [
            next(iter(item["metadata_order"]["categories"])) for item in response["results"] if item.get("metadata")
        ]

    for volume, path, name in unique_categories:
        self.logger.debug("%s : %s", name, path)

        if isinstance(path, str):
            path = [path, name]
        elif isinstance(path, list):
            path.append(name)
        else:
            continue

        response = self._otcs_frontend.get_node_by_volume_and_path(
            volume_type=volume,
            path=path,
        )
        cat_id = self._otcs_frontend.get_result_value(response=response, key="id")
        if cat_id and cat_id not in category_ids:
            category_ids.append(cat_id)
    # end for volume, path, name in unique_categories:

    response = self._otcs_frontend.get_node_create_form(
        parent_id=parent_id,
        subtype=subtype,
        category_ids=category_ids,
    )
    if response is None or "forms" not in response:
        self.logger.error(
            "Failed to retrieve create information for subtype -> %s in parent with ID -> %d!",
            subtype,
            parent_id,
        )
        return None

    self.logger.debug(
        "Successfully retrieved create information for subtype -> %s in parent with ID -> %d.",
        subtype,
        parent_id,
    )

    # Process category information
    forms = response["forms"]

    categories_form = {}

    # Typically the the create item form delivers 4-5 forms:
    # 1. Form for System Attributes (has no role name)
    # 2. Form for Category Data (role name = "categories")
    # 3. Form for Classifications (role name = "classifications")
    # 4. Form for importing team members from another existing team
    # 5. Form for Microsoft Teams integration (role name = "microsoftteams")
    for form in forms:
        if "role_name" not in form:
            self.logger.debug("Found 'system attributes' form -> %s", str(form))
            continue
        match form["role_name"]:
            case "categories":
                categories_form = form
                self.logger.debug("Found 'categories' form -> %s", form)
                continue
            case "classifications":
                self.logger.debug("Found 'classifications' form -> %s", form)
                continue
            case "rmclassifications":
                self.logger.debug("Found 'RM classifications' form -> %s", form)
                continue
            case "importteam":
                self.logger.debug("Found 'import team participants' form -> %s", form)
                continue
            case "microsoftteams":
                self.logger.debug("Found Import Team participants form -> %s", form)
                continue
            case _:
                # the remaining option is that this form is the system attributes form:
                self.logger.warning("Unknown form -> %s", str(form))
        # end match form["role_name"]:
    # end for form in forms:

    # We are just interested in the single category data set (role_name = "categories"):
    data = categories_form.get("data", None)

    if not data:
        self.logger.debug("No categories data found.")
        return category_create_data

    self.logger.debug("Categories data found -> %s", data)
    schema = categories_form["schema"]["properties"]
    self.logger.debug("Categories schema found -> %s", schema)
    # Loop over categories:
    for cat_id in data:  # these are the category IDs (dict keys):
        self.logger.debug("Category ID -> %s", cat_id)
        data_attributes = data[cat_id]
        self.logger.debug("Data attributes -> %s", data_attributes)
        schema_attributes = schema[cat_id]["properties"]
        self.logger.debug("Schema attributes -> %s", schema_attributes)
        cat_name = schema[cat_id]["title"]
        self.logger.debug("Category name -> %s", cat_name)
        # Loop over attributes:
        # * Sets with one (fixed) row have type = object
        # * Multi-value Sets with (multiple) rows have type = array and "properties" in "items" schema
        # * Multi-value attributes have also type = array but NO "properties" in "items" schema
        for attr_id in data_attributes:  # these a attribute IDs (dict keys)
            self.logger.debug("Attribute ID -> %s", attr_id)
            self.logger.debug("Attribute data -> %s", data_attributes[attr_id])
            self.logger.debug(
                "Attribute schema -> %s",
                schema_attributes[attr_id],
            )
            attr_type = schema_attributes[attr_id]["type"]
            self.logger.debug("Attribute type -> %s", attr_type)
            if "title" not in schema_attributes[attr_id]:
                self.logger.debug("Attribute has no title. Skipping...")
                continue

            #
            # Handle multi-line set:
            #
            if attr_type == "array" and ("properties" in schema_attributes[attr_id]["items"]):
                set_name = schema_attributes[attr_id]["title"]
                self.logger.debug("Multi-line set -> %s", set_name)
                set_data_attributes = data_attributes[attr_id]  # this is a list []
                if set_data_attributes is None:
                    self.logger.debug("Attribute has no value in data. Skipping...")
                    continue
                self.logger.debug("Set data attributes -> %s", set_data_attributes)
                set_schema_attributes = schema_attributes[attr_id]["items"]["properties"]
                self.logger.debug(
                    "Set schema attributes -> %s",
                    set_schema_attributes,
                )
                set_schema_max_rows = schema_attributes[attr_id]["items"]["maxItems"]
                self.logger.debug("Set schema max rows -> %s", set_schema_max_rows)
                set_data_max_rows = len(set_data_attributes)
                self.logger.debug("Set data xax rows -> %s", set_data_max_rows)
                row = 1
                # it can happen that the payload contains more rows than the
                # initial rows in the set data structure. In this case we use
                # a copy of the data structure from row 0 as template...
                first_row = dict(set_data_attributes[0])
                # We don't know upfront how many rows of data we will find in payload
                # but we at max process the maxItems specified in the schema:
                while row <= set_schema_max_rows:
                    # Test if we have any payload for this row:
                    attribute = next(
                        (
                            item
                            for item in categories
                            if (
                                item["name"] == cat_name
                                and "set" in item  # not all items may have a "set" key
                                and item["set"] == set_name
                                and "row" in item  # not all items may have a "row" key
                                and item["row"] == row
                            )
                        ),
                        None,
                    )
                    # stop if there's no payload for the row:
                    if attribute is None:
                        self.logger.debug(
                            "No payload found for set -> %s, row -> %s",
                            set_name,
                            row,
                        )
                        # we assume that if there's no payload for row n there will be no payload for rows > n
                        # and break the while loop:
                        break
                    # do we need to create a new row in the data frame?
                    if row > set_data_max_rows:
                        # use the row we stored above to create a new empty row:
                        self.logger.debug(
                            "Found payload for row -> %s, we need a new data row for it",
                            row,
                        )
                        self.logger.debug(
                            "Adding an additional row -> %s to set data -> '%s'",
                            row,
                            set_name,
                        )
                        # add the empty dict to the list:
                        set_data_attributes.append(dict(first_row))
                        set_data_max_rows += 1
                    else:
                        self.logger.debug(
                            "Found payload for row -> %s %s we can store in existing data row",
                            row,
                            set_name,
                        )
                    # traverse all attributes in a single row:
                    for set_attr_id in set_schema_attributes:
                        self.logger.debug(
                            "Set attribute ID -> %s (row -> %s)",
                            set_attr_id,
                            row,
                        )
                        self.logger.debug(
                            "Set attribute schema -> %s (row -> %s)",
                            set_schema_attributes[set_attr_id],
                            row,
                        )
                        set_attr_type = set_schema_attributes[set_attr_id]["type"]
                        self.logger.debug(
                            "Set attribute type -> %s (row -> %s)",
                            set_attr_type,
                            row,
                        )
                        set_attr_name = set_schema_attributes[set_attr_id]["title"]
                        self.logger.debug(
                            "Set attribute name -> %s (row -> %s)",
                            set_attr_name,
                            row,
                        )
                        # Lookup the attribute with the right category, set, attribute name, and row number in payload:
                        attribute = next(
                            (
                                item
                                for item in categories
                                if (
                                    item["name"] == cat_name
                                    and "set" in item  # not all items may have a "set" key
                                    and item["set"] == set_name
                                    and item["attribute"] == set_attr_name
                                    and "row" in item  # not all items may have a "row" key
                                    and item["row"] == row
                                )
                            ),
                            None,
                        )
                        if attribute is None:
                            self.logger.debug(
                                "Set -> '%s', attribute -> '%s', row -> %s not found in payload.",
                                set_name,
                                set_attr_name,
                                row,
                            )

                            # need to use row - 1 as index starts with 0 but payload rows start with 1
                            set_data_attributes[row - 1][set_attr_id] = ""
                        else:
                            if set_attr_type == "array" and "items" in set_schema_attributes[set_attr_id]:
                                set_attr_type = set_schema_attributes[set_attr_id]["items"].get(
                                    "type",
                                    set_attr_type,
                                )

                            self.logger.debug(
                                "Set -> '%s', attribute -> '%s', row -> %s found in payload, value -> '%s'",
                                set_name,
                                set_attr_name,
                                row,
                                attribute["value"],
                            )
                            set_data_attributes[row - 1][set_attr_id] = self.resolve_attribute_values(
                                attribute_name=set_attr_name,
                                attribute_type=set_attr_type,
                                attribute_values=attribute["value"],
                            )
                    row += 1  # continue the while loop with the next row
                # end while row <= set_schema_max_rows:
            # end if attr_type == "array" and ("properties" in schema_attributes[attr_id]["items"]):

            #
            # Handle single-line set:
            #
            elif attr_type == "object":
                set_name = schema_attributes[attr_id]["title"]
                self.logger.debug("Single-line set -> '%s'", set_name)
                set_data_attributes = data_attributes[attr_id]
                self.logger.debug("Set data attributes -> %s", set_data_attributes)

                set_schema_attributes = schema_attributes[attr_id]["properties"]
                self.logger.debug(
                    "Set schema attributes -> %s",
                    str(set_schema_attributes),
                )
                # Loop over set attributes:
                for set_attr_id in set_data_attributes:
                    self.logger.debug("Set attribute ID -> %s", set_attr_id)
                    self.logger.debug(
                        "Set attribute data -> %s",
                        str(set_data_attributes[set_attr_id]),
                    )
                    self.logger.debug(
                        "Set attribute schema -> %s",
                        str(set_schema_attributes[set_attr_id]),
                    )
                    set_attr_type = set_schema_attributes[set_attr_id]["type"]
                    self.logger.debug("Set attribute type -> %s", set_attr_type)
                    set_attr_name = set_schema_attributes[set_attr_id]["title"]
                    self.logger.debug("Set attribute name -> '%s'", set_attr_name)
                    # Lookup the attribute with the right category, set and attribute name in payload:
                    attribute = next(
                        (
                            item
                            for item in categories
                            if (
                                item["name"] == cat_name
                                and "set" in item  # not all items may have a "set" key
                                and item["set"] == set_name
                                and item["attribute"] == set_attr_name
                            )
                        ),
                        None,
                    )
                    if attribute is None:
                        self.logger.debug(
                            "Category -> '%s', set -> %s, attribute -> '%s' not found in payload.",
                            cat_name,
                            set_name,
                            set_attr_name,
                        )
                        set_data_attributes[set_attr_id] = ""
                    else:
                        self.logger.debug(
                            "Category -> '%s', set -> %s, attribute -> '%s' found in payload, value -> %s",
                            cat_name,
                            set_name,
                            set_attr_name,
                            attribute["value"],
                        )
                        # Resolve any special cases (e.g. user picker, group picker):
                        set_data_attributes[set_attr_id] = self.resolve_attribute_values(
                            attribute_name=set_attr_name,
                            attribute_type=set_attr_type,
                            attribute_values=attribute["value"],
                        )
                # end for set_attr_id in set_data_attributes:
            # end elif attr_type == "object":

            #
            # Handle plain attribute (not inside a set):
            #
            else:
                attr_name = schema_attributes[attr_id]["title"]
                self.logger.debug("Attribute name -> '%s'", attr_name)
                # Lookup the attribute with the right category and attribute name in payload:
                attribute = next(
                    (item for item in categories if (item["name"] == cat_name and item["attribute"] == attr_name)),
                    None,
                )

                if attr_type == "array" and "items" in schema_attributes[attr_id]:
                    attr_type = schema_attributes[attr_id]["items"].get("type", attr_type)

                if attribute is None:
                    self.logger.debug(
                        "Category -> '%s', attribute -> '%s' not found in payload.",
                        cat_name,
                        attr_name,
                    )
                    data_attributes[attr_id] = ""
                else:
                    self.logger.debug(
                        "Category -> '%s', attribute -> '%s' found in payload, value -> %s",
                        cat_name,
                        attr_name,
                        attribute["value"],
                    )
                    # Resolve any special cases (e.g. user picker, group picker):
                    data_attributes[attr_id] = self.resolve_attribute_values(
                        attribute_name=attr_name,
                        attribute_type=attr_type,
                        attribute_values=attribute["value"],
                    )
            # end else (plain attribute)
        # end for attr_data, attr_schema
        category_create_data[cat_id] = data_attributes
    # end for cat_data, cat_schema

    self.logger.debug("Category Create Data -> %s", category_create_data)

    return category_create_data

prepare_workspace_business_objects(workspace, business_objects)

Prepare the business object data for the workspace creation.

This supports multiple external system connections. This methods also checks if the external system is reachable and tries to create missing business objects in the leading system if they are missing.

Parameters:

Name	Type	Description	Default
workspace	dict	The payload data for the workspace.	required
business_objects	list	The payload data for the business object connections.	required

Returns:

Type	Description
list | None	list | None: A list of business object data connections (list elements are dictionaries).

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="prepare_workspace_business_objects")
def prepare_workspace_business_objects(
    self,
    workspace: dict,
    business_objects: list,
) -> list | None:
    """Prepare the business object data for the workspace creation.

    This supports multiple external system connections. This methods
    also checks if the external system is reachable and tries to create
    missing business objects in the leading system if they are missing.

    Args:
        workspace (dict):
            The payload data for the workspace.
        business_objects (list):
            The payload data for the business object connections.

    Returns:
        list | None:
            A list of business object data connections
            (list elements are dictionaries).

    """

    business_object_list = []

    for business_object_data in business_objects:
        business_object = {}

        name = workspace.get("name")

        # Read business object data from workspace payload.
        # business_object_data is a dict with 3-5 elements:
        if "external_system" in business_object_data:
            ext_system_id = business_object_data["external_system"]
        else:
            self.logger.error(
                "Missing External System in Business Object payload for workspace -> '%s'.",
                name,
            )
            continue
        if "bo_type" in business_object_data:
            bo_type = business_object_data["bo_type"]
        else:
            self.logger.error(
                "Missing type in Business Object payload for workspace -> '%s'.",
                name,
            )
            continue

        if "bo_id" in business_object_data:
            bo_id = business_object_data["bo_id"]
            bo_search_field = None
            bo_search_value = None
        elif "bo_search_field" not in business_object_data or "bo_search_value" not in business_object_data:
            self.logger.error(
                "Missing BO search fields (bo_search_field, bo_search_value) in Business Object payload without BO ID for workspace -> '%s'.",
                name,
            )
            continue
        else:
            bo_search_field = business_object_data["bo_search_field"]
            bo_search_value = business_object_data["bo_search_value"]
            bo_id = None

        # Lookup external system in this payload or a former processed payload:
        external_system = self.lookup_external_system(ext_system_id=ext_system_id)

        # If the external system is not in the current payload but in the system
        # we try to avoid errors in the following code.
        # TODO: review REST APIs in OTCS 26.1 version to see if we can improve this.
        if not external_system and self._otcs.get_external_system_connection(connection_name=ext_system_id):
            # As the REST API for reading external system data is pretty much limited
            # we try to do the bare minimum here:
            if "Guidewire" in business_object_data["external_system"]:
                external_system_type = "Guidewire"
            elif "Salesforce" in business_object_data["external_system"]:
                external_system_type = "Salesforce"
            external_system = {"enabled": True, "reachable": True, "external_system_type": external_system_type}
            self.logger.info(
                "External system -> '%s' is not found in current payload but it exists in the system. Construct minimum external system information -> %s.",
                ext_system_id,
                str(external_system),
            )

        if not external_system:
            self.logger.warning(
                "External System -> '%s' does not exist. Cannot connect workspace -> '%s' to -> %s. Create workspace without connection.",
                ext_system_id,
                name,
                ext_system_id,
            )
            continue
        if not external_system.get("enabled", True):
            self.logger.info(
                "External System -> '%s' is disabled in payload. Cannot connect workspace -> '%s' to -> (%s, %s, %s, %s, %s). Create workspace without connection...",
                ext_system_id,
                name,
                ext_system_id,
                bo_type,
                bo_id,
                bo_search_field,
                bo_search_value,
            )
            continue
        if not external_system.get("reachable"):
            self.logger.warning(
                "External System -> '%s' is not reachable. Cannot connect workspace -> '%s' to -> (%s, %s, %s, %s, %s). Create workspace without connection...",
                ext_system_id,
                name,
                ext_system_id,
                bo_type,
                bo_id,
                bo_search_field,
                bo_search_value,
            )
            continue
        external_system_type = external_system.get("external_system_type", "")

        # For Salesforce we try to determine the actual business object ID (technical ID) if it is
        # not specified directly (but instead a search field and search value):
        if external_system_type == "Salesforce" and not bo_id:
            # If Salesforce external system is used across payloads it may not be initialized here:
            if not self._salesforce:
                self._salesforce = self.init_salesforce(external_system)
            bo_id = self.get_salesforce_business_object(
                workspace,
                object_type=bo_type,
                search_field=bo_search_field,
                search_value=bo_search_value,
            )
            if not bo_id:
                self.logger.warning(
                    "Workspace -> '%s' will not be connected to Salesforce as the Business Object ID couldn't be determined (type -> '%s', search_field -> '%s', search_value -> '%s')!",
                    name,
                    bo_type,
                    bo_search_field,
                    bo_search_value,
                )
                continue
        # end if external_system_type == "Salesforce" and not bo_id

        # For Guidewire we try to determine the actual business object ID (technical ID) if it is
        # not specified in the payload (but instead a search field and search value):
        if external_system_type == "Guidewire" and not bo_id:
            bo_id = self.get_guidewire_business_object(
                external_system=external_system,
                object_type=bo_type,
                search_field=bo_search_field,
                search_value=bo_search_value,
            )
            if not bo_id:
                self.logger.warning(
                    "Workspace -> '%s' will not be connected to Guidewire as the business object ID couldn't be determined (type -> '%s', search_field -> '%s', search_value -> '%s')!",
                    name,
                    bo_type,
                    bo_search_field,
                    bo_search_value,
                )
                continue
        # end if external_system_type == "Guidewire" and not bo_id

        self.logger.info(
            "Workspace -> '%s' will be connected with external system -> '%s' (%s) with (type -> '%s', ID -> '%s').",
            name,
            external_system_type,
            ext_system_id,
            bo_type,
            bo_id,
        )

        business_object["ext_system_id"] = ext_system_id
        business_object["bo_type"] = bo_type
        business_object["bo_id"] = bo_id

        business_object_list.append(business_object)

    return business_object_list

prepare_workspace_create_form(categories, template_id, ext_system_id=None, bo_type=None, bo_id=None, parent_workspace_node_id=None)

Prepare the category structure for the workspace creation.

Parameters:

Name	Type	Description	Default
categories	list	The categories list from the workspace payload.	required
template_id	int	The workspace template ID.	required
ext_system_id	str	The optional external system ID.	None
bo_type	str	The optional business object type ID.	None
bo_id	str	the business object ID.	None
parent_workspace_node_id	int	The optional parent Workspace ID.	None

Returns:

Type	Description
dict | None	dict | None: Category structure for workspace creation or None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="prepare_workspace_create_form")
def prepare_workspace_create_form(
    self,
    categories: list,
    template_id: int,
    ext_system_id: str | None = None,
    bo_type: str | None = None,
    bo_id: str | None = None,
    parent_workspace_node_id: int | None = None,
) -> dict | None:
    """Prepare the category structure for the workspace creation.

    Args:
        categories (list):
            The categories list from the workspace payload.
        template_id (int):
            The workspace template ID.
        ext_system_id (str, optional):
            The optional external system ID.
        bo_type (str, optional):
            The optional business object type ID.
        bo_id (str, optional):
            the business object ID.
        parent_workspace_node_id (int, optional):
            The optional parent Workspace ID.

    Returns:
        dict | None:
            Category structure for workspace creation or None
            in case of an error.

    """

    category_create_data = {}

    response = self._otcs_frontend.get_workspace_create_form(
        template_id=template_id,
        external_system_id=ext_system_id,
        bo_type=bo_type,
        bo_id=bo_id,
        parent_id=parent_workspace_node_id,
    )
    if response is None or "forms" not in response:
        self.logger.error(
            "Failed to retrieve create information for template ID -> %s!",
            template_id,
        )
        return None

    self.logger.debug(
        "Successfully retrieved create information for template ID -> %d!",
        template_id,
    )

    # Process category information
    forms = response["forms"]

    categories_form = {}

    # Typically the the create workspace form delivers 4-5 forms:
    # 1. Form for System Attributes (has no role name)
    # 2. Form for Category Data (role name = "categories")
    # 3. Form for Classifications (role name = "classifications")
    # 4. Form for importing team members from another existing team
    # 5. Form for Microsoft Teams integration (role name = "microsoftteams")
    for form in forms:
        if "role_name" not in form:
            self.logger.debug("Found 'system attributes' form -> %s", str(form))
            continue
        match form["role_name"]:
            case "categories":
                categories_form = form
                self.logger.debug("Found 'categories' form -> %s", str(form))
                continue
            case "classifications":
                self.logger.debug("Found 'classifications' form -> %s", str(form))
                continue
            case "rmclassifications":
                self.logger.debug("Found 'RM classifications' form -> %s", str(form))
                continue
            case "importteam":
                self.logger.debug("Found 'import team participants' form -> %s", str(form))
                continue
            case "microsoftteams":
                self.logger.debug("Found 'microsoft teams' form -> %s", str(form))
                continue
            case _:
                # the remaining option is that this form is the system attributes form:
                self.logger.warning("Unknown form -> %s", str(form))
        # end match form["role_name"]:
    # end for form in forms:

    # We are just interested in the single category data set (role_name = "categories"):
    data = categories_form.get("data", None)

    if not data:
        self.logger.debug("No categories data found.")
        return category_create_data

    self.logger.debug("Categories data found -> %s", data)
    schema = categories_form["schema"]["properties"]
    self.logger.debug("Categories schema found -> %s", schema)
    # Loop over categories:
    for cat_id in data:  # these are the category IDs (dict keys):
        self.logger.debug("Category ID -> %s", cat_id)
        data_attributes = data[cat_id]
        self.logger.debug("Data attributes -> %s", data_attributes)
        schema_attributes = schema[cat_id]["properties"]
        self.logger.debug("Schema attributes -> %s", schema_attributes)
        cat_name = schema[cat_id]["title"]
        self.logger.debug("Category name -> %s", cat_name)
        # Loop over attributes:
        # * Sets with one (fixed) row have type = object
        # * Multi-value Sets with (multiple) rows have type = array and "properties" in "items" schema
        # * Multi-value attributes have also type = array but NO "properties" in "items" schema
        for attr_id in data_attributes:  # these a attribute IDs (dict keys)
            self.logger.debug("Attribute ID -> %s", attr_id)
            self.logger.debug("Attribute data -> %s", data_attributes[attr_id])
            self.logger.debug(
                "Attribute schema -> %s",
                str(schema_attributes[attr_id]),
            )
            attr_type = schema_attributes[attr_id]["type"]
            self.logger.debug("Attribute type -> %s", attr_type)
            if "title" not in schema_attributes[attr_id]:
                self.logger.debug("Attribute has no title in schema. Skipping...")
                continue

            #
            # Handle multi-line set:
            #
            if attr_type == "array" and ("properties" in schema_attributes[attr_id]["items"]):
                set_name = schema_attributes[attr_id]["title"]
                self.logger.debug("Multi-line set -> '%s'", set_name)
                set_data_attributes = data_attributes[attr_id]  # this is a list []
                if set_data_attributes is None:
                    self.logger.debug("Attribute has no value in data. Skipping...")
                    continue
                self.logger.debug("Set data attributes -> %s", set_data_attributes)
                set_schema_attributes = schema_attributes[attr_id]["items"]["properties"]
                self.logger.debug(
                    "Set schema attributes -> %s",
                    set_schema_attributes,
                )
                set_schema_max_rows = schema_attributes[attr_id]["items"]["maxItems"]
                self.logger.debug("Set schema max rows -> %s", set_schema_max_rows)
                set_data_max_rows = len(set_data_attributes)
                self.logger.debug("Set data max rows -> %s", set_data_max_rows)
                row = 1
                # it can happen that the payload contains more rows than the
                # initial rows in the set data structure. In this case we use
                # a copy of the data structure from row 0 as template...
                first_row = dict(set_data_attributes[0])
                # We don't know upfront how many rows of data we will find in payload
                # but we at max process the maxItems specified in the schema:
                while row <= set_schema_max_rows:
                    # Test if we have any payload for this row:
                    attribute = next(
                        (
                            item
                            for item in categories
                            if (
                                item["name"] == cat_name
                                and "set" in item  # not all items may have a "set" key
                                and item["set"] == set_name
                                and "row" in item  # not all items may have a "row" key
                                and item["row"] == row
                            )
                        ),
                        None,
                    )
                    # stop if there's no payload for the row:
                    if attribute is None:
                        self.logger.debug(
                            "No payload found for set -> %s, row -> %s",
                            set_name,
                            row,
                        )
                        # we assume that if there's no payload for row n there will be no payload for rows > n
                        # and break the while loop:
                        break
                    # do we need to create a new row in the data frame?
                    if row > set_data_max_rows:
                        # use the row we stored above to create a new empty row:
                        self.logger.debug(
                            "Found payload for row -> %s, we need a new data row for it",
                            row,
                        )
                        self.logger.debug(
                            "Adding an additional row -> %s to set data -> '%s'",
                            row,
                            set_name,
                        )
                        # add the empty dict to the list:
                        set_data_attributes.append(dict(first_row))
                        set_data_max_rows += 1
                    else:
                        self.logger.debug(
                            "Found payload for row -> %s %s we can store in existing data row",
                            row,
                            set_name,
                        )
                    # traverse all attributes in a single set row:
                    for set_attr_id in set_schema_attributes:
                        self.logger.debug(
                            "Set attribute ID -> %s (row -> %s)",
                            set_attr_id,
                            row,
                        )
                        self.logger.debug(
                            "Set attribute schema -> %s (row -> %s)",
                            set_schema_attributes[set_attr_id],
                            row,
                        )
                        set_attr_type = set_schema_attributes[set_attr_id]["type"]
                        self.logger.debug(
                            "Set attribute type -> %s (row -> %s)",
                            set_attr_type,
                            row,
                        )
                        set_attr_name = set_schema_attributes[set_attr_id]["title"]
                        self.logger.debug(
                            "Set attribute name -> %s (row -> %s)",
                            set_attr_name,
                            row,
                        )
                        # Lookup the attribute with the right category, set, attribute name, and row number in payload:
                        attribute = next(
                            (
                                item
                                for item in categories
                                if (
                                    item["name"] == cat_name
                                    and "set" in item  # not all items may have a "set" key
                                    and item["set"] == set_name
                                    and item["attribute"] == set_attr_name
                                    and "row" in item  # not all items may have a "row" key
                                    and item["row"] == row
                                )
                            ),
                            None,
                        )
                        if attribute is None:
                            self.logger.debug(
                                "Set -> '%s', attribute -> '%s', row -> %s not found in payload.",
                                set_name,
                                set_attr_name,
                                row,
                            )

                            # need to use row - 1 as index starts with 0 but payload rows start with 1
                            set_data_attributes[row - 1][set_attr_id] = ""
                        else:
                            if set_attr_type == "array" and "items" in set_schema_attributes[set_attr_id]:
                                set_attr_type = set_schema_attributes[set_attr_id]["items"].get(
                                    "type",
                                    set_attr_type,
                                )

                            self.logger.debug(
                                "Set -> '%s', attribute -> '%s', row -> %s found in payload, value -> '%s'",
                                set_name,
                                set_attr_name,
                                row,
                                attribute["value"],
                            )
                            set_data_attributes[row - 1][set_attr_id] = self.resolve_attribute_values(
                                attribute_name=set_attr_name,
                                attribute_type=set_attr_type,
                                attribute_values=attribute["value"],
                            )
                    row += 1  # continue the while loop with the next row
                # end while row <= set_schema_max_rows:
            # end if attr_type == "array" and ("properties" in schema_attributes[attr_id]["items"]):

            #
            # Handle single-line set:
            #
            elif attr_type == "object":
                set_name = schema_attributes[attr_id]["title"]
                self.logger.debug("Single-line set -> '%s'", set_name)
                set_data_attributes = data_attributes[attr_id]
                self.logger.debug("Set data attributes -> %s", set_data_attributes)

                set_schema_attributes = schema_attributes[attr_id]["properties"]
                self.logger.debug(
                    "Set schema attributes -> %s",
                    str(set_schema_attributes),
                )
                # Loop over set attributes:
                for set_attr_id in set_data_attributes:
                    self.logger.debug("Set attribute ID -> %s", set_attr_id)
                    self.logger.debug(
                        "Set attribute data -> %s",
                        str(set_data_attributes[set_attr_id]),
                    )
                    self.logger.debug(
                        "Set attribute schema -> %s",
                        str(set_schema_attributes[set_attr_id]),
                    )
                    set_attr_type = set_schema_attributes[set_attr_id]["type"]
                    self.logger.debug("Set attribute type -> %s", set_attr_type)
                    set_attr_name = set_schema_attributes[set_attr_id]["title"]
                    self.logger.debug("Set attribute name -> '%s'", set_attr_name)
                    # Lookup the attribute with the right category, set and attribute name in payload:
                    attribute = next(
                        (
                            item
                            for item in categories
                            if (
                                item["name"] == cat_name
                                and "set" in item  # not all items may have a "set" key
                                and item["set"] == set_name
                                and item["attribute"] == set_attr_name
                            )
                        ),
                        None,
                    )
                    if attribute is None:
                        self.logger.debug(
                            "Category -> '%s', set -> %s, attribute -> '%s' not found in payload.",
                            cat_name,
                            set_name,
                            set_attr_name,
                        )
                        set_data_attributes[set_attr_id] = ""
                    else:
                        self.logger.debug(
                            "Category -> '%s', set -> %s, attribute -> '%s' found in payload, value -> %s",
                            cat_name,
                            set_name,
                            set_attr_name,
                            attribute["value"],
                        )
                        # Resolve any special cases (e.g. user picker, group picker):
                        set_data_attributes[set_attr_id] = self.resolve_attribute_values(
                            attribute_name=set_attr_name,
                            attribute_type=set_attr_type,
                            attribute_values=attribute["value"],
                        )
                # end for set_attr_id in set_data_attributes:
            # end elif attr_type == "object":

            #
            # Handle plain attribute (not inside a set):
            #
            else:
                attr_name = schema_attributes[attr_id]["title"]
                self.logger.debug("Attribute name -> '%s'", attr_name)
                # Lookup the attribute with the right category and attribute name in payload:
                attribute = next(
                    (item for item in categories if (item["name"] == cat_name and item["attribute"] == attr_name)),
                    None,
                )

                if attr_type == "array" and "items" in schema_attributes[attr_id]:
                    attr_type = schema_attributes[attr_id]["items"].get("type", attr_type)

                if attribute is None:
                    self.logger.debug(
                        "Category -> '%s', attribute -> '%s' not found in payload.",
                        cat_name,
                        attr_name,
                    )
                    data_attributes[attr_id] = ""
                else:
                    self.logger.debug(
                        "Category -> '%s', attribute -> '%s' found in payload, value -> %s",
                        cat_name,
                        attr_name,
                        attribute["value"],
                    )
                    # Resolve any special cases (e.g. user picker, group picker):
                    data_attributes[attr_id] = self.resolve_attribute_values(
                        attribute_name=attr_name,
                        attribute_type=attr_type,
                        attribute_values=attribute["value"],
                    )
            # end else (plain attribute)
        # end for attr_data, attr_schema
        category_create_data[cat_id] = data_attributes
    # end for cat_data, cat_schema

    self.logger.debug("Category create data -> %s", category_create_data)

    return category_create_data

process_additional_access_role_members(section_name='additionalAccessRoleMemberships')

Process additional access role memberships we want to have in OTDS.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'additionalAccessRoleMemberships'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_additional_access_role_members")
def process_additional_access_role_members(
    self,
    section_name: str = "additionalAccessRoleMemberships",
) -> bool:
    """Process additional access role memberships we want to have in OTDS.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._additional_access_role_members:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for additional_access_role_member in self._additional_access_role_members:
        access_role = additional_access_role_member.get("access_role")
        if not access_role:
            self.logger.error(
                "Missing access role! Skipping additional role members...",
            )
            continue

        # Check if additional access role member has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not additional_access_role_member.get("enabled", True):
            self.logger.info(
                "Payload for additional member for access role -> '%s' is disabled. Skipping...",
                access_role,
            )
            continue

        if (
            ("user_name" not in additional_access_role_member)
            and ("group_name" not in additional_access_role_member)
            and ("partition_name" not in additional_access_role_member)
        ):
            self.logger.error(
                "Either group_name or user_name need to be specified! Skipping...",
            )
            success = False
            continue
        if "group_name" in additional_access_role_member:
            group_name = additional_access_role_member["group_name"]
            self.logger.info(
                "Adding group -> '%s' to access role -> '%s' in OTDS.",
                group_name,
                access_role,
            )
            response = self._otds.add_group_to_access_role(
                access_role=access_role,
                group=group_name,
            )
            if not response:
                self.logger.error(
                    "Failed to add group -> '%s' to access role -> '%s' in OTDS!",
                    group_name,
                    access_role,
                )
                success = False
        elif "user_name" in additional_access_role_member:
            user_name = additional_access_role_member["user_name"]
            self.logger.info(
                "Adding user -> '%s' to access role -> '%s' in OTDS.",
                user_name,
                access_role,
            )
            response = self._otds.add_user_to_access_role(
                access_role=access_role,
                user_id=user_name,
            )
            if not response:
                self.logger.error(
                    "Failed to add user -> '%s' to access role -> '%s' in OTDS!",
                    user_name,
                    access_role,
                )
                success = False
        elif "partition_name" in additional_access_role_member:
            partition_name = additional_access_role_member["partition_name"]
            self.logger.info(
                "Adding partition -> '%s' to access role -> '%s' in OTDS.",
                partition_name,
                access_role,
            )
            response = self._otds.add_partition_to_access_role(
                access_role=access_role,
                partition=partition_name,
            )
            if not response:
                self.logger.error(
                    "Failed to add partition -> '%s' to access role -> '%s' in OTDS!",
                    partition_name,
                    access_role,
                )
                success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._additional_access_role_members,
    )

    return success

process_additional_application_role_assignments(section_name='additionalApplicationRoleAssignments')

Process additional application role assignments we want to have in OTDS.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'additionalApplicationRoleAssignments'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(
    attributes=OTEL_TRACING_ATTRIBUTES, name="process_additional_application_role_assignments"
)
def process_additional_application_role_assignments(
    self,
    section_name: str = "additionalApplicationRoleAssignments",
) -> bool:
    """Process additional application role assignments we want to have in OTDS.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._additional_application_role_assignments:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for additional_role_assignment in self._additional_application_role_assignments:
        role = additional_role_assignment.get("role_name")
        role_parts = role.split("@", 1)
        role_name = role_parts[0]
        role_partition = role_parts[1] if len(role_parts) > 1 else "OAuthClients"

        if not role_name:
            self.logger.error(
                "Missing application role! Skipping additional role members...",
            )
            continue

        # Check if additional access role member has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not additional_role_assignment.get("enabled", True):
            self.logger.info(
                "Payload for additional assignment for application role -> '%s' is disabled. Skipping...",
                role_name,
            )
            continue

        if ("user_name" not in additional_role_assignment) and ("group_name" not in additional_role_assignment):
            self.logger.error(
                "Either group_name or user_name need to be specified for application role assignment! Skipping...",
            )
            success = False
            continue

        if "group_name" in additional_role_assignment:
            group = additional_role_assignment["group_name"]
            group_parts = group.split("@", 1)
            group_name = group_parts[0]
            group_partition = group_parts[1] if len(group_parts) > 1 else self._otcs.config()["partition"]

            self.logger.info(
                "Adding group -> '%s' in partition -> '%s' to application role -> '%s' in partition -> '%s'.",
                group_name,
                group_partition,
                role_name,
                role_partition,
            )

            response = self._otds.assign_group_to_application_role(
                group_id=group_name,
                group_partition=group_partition,
                role_name=role_name,
                role_partition=role_partition,
            )

            if not response:
                self.logger.error(
                    "Failed to assign group -> '%s' in partition -> '%s' to application role -> '%s' in partition -> '%s'!",
                    group_name,
                    group_partition,
                    role_name,
                    role_partition,
                )
                success = False

        elif "user_name" in additional_role_assignment:
            user = additional_role_assignment["user_name"]
            user_parts = user.split("@", 1)
            user_name = user_parts[0]
            user_partition = user_parts[1] if len(user_parts) > 1 else self._otcs.config()["partition"]

            self.logger.info(
                "Adding user -> '%s' in partition -> '%s' to application role -> '%s' in partition -> '%s'!",
                user_name,
                user_partition,
                role_name,
                role_partition,
            )

            response = self._otds.assign_user_to_application_role(
                user_id=user_name,
                user_partition=user_partition,
                role_name=role_name,
                role_partition=role_partition,
            )
            if not response:
                self.logger.error(
                    "Failed to assign user -> '%s' in partition -> '%s' to application role -> '%s' in partition -> '%s'!",
                    user_name,
                    user_partition,
                    role_name,
                    role_partition,
                )
                success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._additional_application_role_assignments,
    )

    return success

process_additional_group_members(section_name='additionalGroupMemberships')

Process additional groups memberships we want to have in OTDS.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'additionalGroupMemberships'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_additional_group_members")
def process_additional_group_members(
    self,
    section_name: str = "additionalGroupMemberships",
) -> bool:
    """Process additional groups memberships we want to have in OTDS.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._additional_group_members:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for additional_group_member in self._additional_group_members:
        parent_group = additional_group_member.get("parent_group")
        if not parent_group:
            self.logger.error("Missing parent group! Skipping...")
            continue

        # Check if additional group member has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not additional_group_member.get("enabled", True):
            self.logger.info(
                "Payload for additional group member with parent group -> '%s' is disabled. Skipping...",
                parent_group,
            )
            continue

        if ("user_name" not in additional_group_member) and ("group_name" not in additional_group_member):
            self.logger.error(
                "Either group name or user name need to be specified! Skipping additional group members...",
            )
            success = False
            continue
        if "group_name" in additional_group_member:
            group_name = additional_group_member["group_name"]
            self.logger.info(
                "Adding group -> '%s' to parent group -> '%s' in OTDS.",
                group_name,
                parent_group,
            )
            response = self._otds.add_group_to_parent_group(
                group=group_name,
                parent_group=parent_group,
            )
            if not response:
                self.logger.error(
                    "Failed to add group -> '%s' to parent group -> '%s' in OTDS!",
                    group_name,
                    parent_group,
                )
                success = False
        elif "user_name" in additional_group_member:
            user_name = additional_group_member["user_name"]
            self.logger.info(
                "Adding user -> '%s' to group -> '%s' in OTDS.",
                user_name,
                parent_group,
            )
            response = self._otds.add_user_to_group(
                user=user_name,
                group=parent_group,
            )
            if not response:
                self.logger.error(
                    "Failed to add user -> '%s' to group -> '%s' in OTDS!",
                    user_name,
                    parent_group,
                )
                success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._additional_group_members,
    )

    return success

process_admin_settings(admin_settings, section_name='adminSettings')

Process admin settings in payload and import them to Content Server.

The payload section is a list of dicts with these items:
{
    enabled: True or False to enable or disable the payload item
    filename: The filename of the XML file with admin settings.
              It needs to be the plain filename like "admin.xml".
              The files reside inside the container in /settings root
              directory. They are placed there by the Terraform automation
              and are taken from the ./settings/payload directory.
    description: Some description about the purpose of the settings.
                 Just for information and optional.
}

Parameters:

Name	Type	Description	Default
admin_settings	list	List of admin settings. We need this parameter as we process two different lists.	required
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'adminSettings'

Returns:

Name	Type	Description
bool	bool	True, if a restart of the OTCS pods is required. False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_admin_settings")
def process_admin_settings(
    self,
    admin_settings: list,
    section_name: str = "adminSettings",
) -> bool:
    """Process admin settings in payload and import them to Content Server.

        The payload section is a list of dicts with these items:
        {
            enabled: True or False to enable or disable the payload item
            filename: The filename of the XML file with admin settings.
                      It needs to be the plain filename like "admin.xml".
                      The files reside inside the container in /settings root
                      directory. They are placed there by the Terraform automation
                      and are taken from the ./settings/payload directory.
            description: Some description about the purpose of the settings.
                         Just for information and optional.
        }

    Args:
        admin_settings (list):
            List of admin settings. We need this parameter
            as we process two different lists.
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if a restart of the OTCS pods is required. False otherwise.

    """

    if not admin_settings:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return False  # important to return False here as otherwise we are triggering a restart of services!!

    # If this payload section has been processed successfully before we
    # can return False and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return False  # important to return False here as otherwise we are triggering a restart of services!!

    restart_required: bool = False
    success: bool = True

    for admin_setting in admin_settings:
        # Sanity checks:
        if "filename" not in admin_setting:
            self.logger.error(
                "Filename is missing. Skipping to next admin setting...",
            )
            continue
        filename = admin_setting["filename"]

        # Check if admin setting has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not admin_setting.get("enabled", True):
            self.logger.info(
                "Payload for setting file -> '%s' is disabled. Skipping...",
                filename,
            )
            continue

        settings_file = self._custom_settings_dir + filename
        if os.path.exists(settings_file):
            description = admin_setting.get("description")
            self.logger.info(
                "Processing admin settings from file -> '%s'%s...",
                filename,
                " ({})".format(description) if description else "",
            )

            # Read the config file:
            with open(settings_file, encoding="utf-8") as file:
                file_content = file.read()

            self.logger.debug(
                "Replace Placeholder -> '%s' in file -> %s",
                self._placeholder_values,
                file_content,
            )

            file_content = self.replace_placeholders(content=file_content)

            # Write the updated config file:
            tmpfile = os.path.join(tempfile.gettempdir(), os.path.basename(settings_file))
            with open(tmpfile, "w", encoding="utf-8") as file:
                file.write(file_content)

            response = self._otcs.apply_config(xml_file_path=tmpfile)
            if response and response["results"]["data"]["restart"]:
                self.logger.info(
                    "A restart of the Content Server service is required.",
                )
                restart_required = True

                if admin_setting.get("restart", False):
                    self.logger.info(
                        "Immediate restart requested - restart of OTCS services...",
                    )
                    # Restart OTCS frontend and backend pods:
                    self._otcs_restart_callback(
                        backend=self._otcs_backend,
                        frontend=self._otcs_frontend,
                    )

                    restart_required = False

        else:
            self.logger.error(
                "Admin settings file -> '%s' not found!",
                settings_file,
            )
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=admin_settings,
    )

    return restart_required

process_application_roles(section_name='applicationRoles')

Process OTDS application rolesin payload and create them in OTDS.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'applicationRoles'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_application_roles")
def process_application_roles(self, section_name: str = "applicationRoles") -> bool:
    """Process OTDS application rolesin payload and create them in OTDS.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._application_roles:
        self.logger.info("Payload section -> %s is empty. Skipping...", section_name)
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for role in self._application_roles:
        role_name = role.get("name")
        if not role_name:
            self.logger.error("Application role does not have a name in payload! Skipping...")
            success = False
            continue

        # Check if application role has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not role.get("enabled", True):
            self.logger.info(
                "Payload for application roles -> '%s' is disabled. Skipping...",
                role_name,
            )
            continue

        role_description = role.get("description")
        role_partition = role.get("partition", "OAuthClients")

        # Check if application role does already exist
        # (in an attempt to make the code idem-potent)
        self.logger.info(
            "Check if application role -> '%s' does already exist in partition -> '%s'...",
            role_name,
            role_partition,
        )
        response = self._otds.get_application_role(
            name=role,
            partition=role_partition,
            show_error=False,
        )
        if response:
            self.logger.info(
                "Application role -> '%s' does already exist in partition -> '%s'. Skipping...",
                role_name,
                role_partition,
            )
            continue
        self.logger.info(
            "Application role -> '%s' does not exist in partition -> '%s'. Creating...", role_name, role_partition
        )

        response = self._otds.add_application_role(
            name=role_name,
            partition_id=role_partition,
            description=role_description,
            values=role.get("values", None),
            custom_attributes=role.get("custom_attributes", None),
        )
        if response:
            self.logger.info(
                "Successfully added OTDS Application role -> '%s' to partition -> '%s'.", role_name, role_partition
            )
        else:
            self.logger.error(
                "Failed to add OTDS Application role -> '%s' to partition -> '%s'!", role_name, role_partition
            )
            success = False
            continue

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._application_roles,
    )

    return success

process_appworks_configurations(section_name='appworks')

Process the configurations for AppWorks projects.

This method is responsible for setting up the necessary configurations for AppWorks projects. If the payload contains a appworks section, it will execute the corresponding actions to process and apply the custom configuration.

This includes: * Resource configuration - add the OTCS user partition to the OTDS access role for the AppWorks organization - add the OTCS admin partition to the OTDS access role for the AppWorks organization - add an OTAWP license * Solution configuration - create the AppWorks artifacts

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'appworks'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_appworks_configurations")
def process_appworks_configurations(self, section_name: str = "appworks") -> bool:
    """Process the configurations for AppWorks projects.

    This method is responsible for setting up the necessary configurations for AppWorks projects.
    If the payload contains a `appworks` section, it will execute the corresponding actions
    to process and apply the custom configuration.

    This includes:
    * Resource configuration
        - add the OTCS user partition to the OTDS access role for the AppWorks organization
        - add the OTCS admin partition to the OTDS access role for the AppWorks organization
        - add an OTAWP license
    * Solution configuration
        - create the AppWorks artifacts

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._appworks_configurations:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    #
    # 1: Loop to create the required OTDS resources for the AppWorks organization:
    #
    for appworks_configuration in self._appworks_configurations:
        organization = appworks_configuration.get("organization", None)
        if not organization:
            self.logger.error("AppWorks configuration is missing the organization name! Skipping...")
            success = False
            continue

        self._log_header_callback(
            text="Process AppWorks resource configuration for '{}'".format(organization), char="-"
        )

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not appworks_configuration.get("enabled", True):
            self.logger.info(
                "Payload for AppWorks configuration -> '%s' is disabled. Skipping...",
                organization,
            )
            continue

        if not appworks_configuration.get("resource_config", False):
            self.logger.info("AppWorks resource configuration is disabled for -> '%s'. Skipping...", organization)
            continue

        access_role_name = "Access to " + organization

        # make sure code is idempotent and only try to add ressource if it doesn't exist already:
        awp_resource = self._otds.get_resource(organization)
        if not awp_resource:
            self.logger.info(
                "OTDS resource -> '%s' for AppWorks Platform does not yet exist. Creating...",
                organization,
            )
            awp_resource = self._otds.add_resource(
                name=organization,
                description="AppWorks Platform - {}".format(organization),
                display_name="AppWorks Platform - {}".format(organization),
                additional_payload=OTAWP.resource_payload(
                    org_name=organization,
                    username=self._otawp.username(),
                    password=self._otawp.password(),
                ),
            )
        else:
            self.logger.info(
                "OTDS resource -> '%s' for AppWorks Platform does already exist.",
                organization,
            )

        awp_resource_id = awp_resource["resourceID"]

        self.logger.info(
            "AppWorks Platform organization -> '%s' got OTDS resource ID -> %s",
            organization,
            awp_resource_id,
        )

        # Loop to wait for OTCS to create its OTDS user partition:
        otcs_partition = self._otds.get_partition(
            self._otcs.partition_name(),
            show_error=False,
        )
        while otcs_partition is None:
            self.logger.warning(
                "OTDS user partition -> '%s' for Content Server does not exist yet. Waiting...",
                self._otcs.partition_name(),
            )

            time.sleep(30)
            otcs_partition = self._otds.get_partition(
                name=self._otcs.partition_name(),
                show_error=False,
            )

        # Add the OTDS user partition for OTCS to the AppWorks Platform Access Role in OTDS.
        # This will effectvely sync all OTCS users with AppWorks Platform:
        self._otds.add_partition_to_access_role(
            access_role=access_role_name,
            partition=self._otcs.partition_name(),
        )

        # Add the OTDS admin partition to the AppWorks Platform Access Role in OTDS.
        self._otds.add_partition_to_access_role(
            access_role=access_role_name,
            partition=self._otds.admin_partition_name(),
        )

        # Set Group inclusion for Access Role for OTAWP to "True":
        self._otds.update_access_role_attributes(
            name=access_role_name,
            attribute_list=[{"name": "pushAllGroups", "values": ["True"]}],
        )

        # Add ResourceID User to 'otdsadmins' group to allow push
        self._otds.add_user_to_group(
            user=str(awp_resource_id) + "@otds.admin",
            group="otdsadmins@otds.admin",
        )

        # Allow impersonation for all users:
        self._otds.impersonate_resource(resource_name=organization)

        # Editing configmap
        config_map = self._k8s.get_config_map(config_map_name=self._otawp.config_map_name())
        if not config_map:
            self.logger.error(
                "Failed to retrieve AppWorks Kubernetes config map -> '%s'!",
                self._otawp.config_map_name(),
            )
            success = False
        else:
            self.logger.info(
                "Update Kubernetes config map for AppWorks organization -> '%s' with OTDS resource IDs...",
                organization,
            )
            solution = yaml.safe_load(
                config_map.data[organization + ".yaml"],
            )

            solution["platform"]["organizations"][organization]["otds"]["resourceId"] = awp_resource_id
            solution["platform"]["organizations"][organization]["database"]["connections"]["sysConnection"][
                "connectionString"
            ] = "jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_NAME}"

            solution["platform"]["organizations"][organization]["database"]["connections"]["sysConnection"][
                "database"
            ] = "${DATABASE_NAME}"

            solution["platform"]["organizations"][organization]["database"]["connections"]["sysConnection"][
                "password"
            ] = "${DATABASE_PASSWORD}"

            solution["platform"]["organizations"][organization]["content"]["ContentServer"]["contentServerUrl"] = (
                self._otcs.cs_url()
            )
            solution["platform"]["organizations"][organization]["content"]["ContentServer"][
                "contentServerSupportDirectoryUrl"
            ] = self._otcs.cs_support_url()
            solution["platform"]["organizations"][organization]["content"]["ContentServer"]["otdsResourceId"] = (
                self._otcs.resource_id()
            )
            solution["platform"]["organizations"][organization]["authenticators"]["OTDSAuth"]["publicLoginUrl"] = (
                self._otds.base_url() + "/otdsws/login"
            )

            config_map.data[organization + ".yaml"] = yaml.dump(solution)
            result = self._k8s.replace_config_map(
                config_map_name=self._otawp.config_map_name(),
                config_map_data=config_map.data,
            )
            if result:
                self.logger.info(
                    "Successfully updated AppWorks solution YAML for organization -> '%s'.",
                    organization,
                )
            else:
                self.logger.error(
                    "Failed to update AppWorks solution YAML for organization -> '%s'!",
                    organization,
                )
            self.logger.debug(
                "Solution YAML for Appworks organization -> '%s': %s",
                organization,
                solution,
            )
        # Add SPS license for OTAWP
        license_name = self._otawp.product_name()
        product_name = self._otawp.product_name() + "_" + organization.upper()
        product_description = self._otawp.product_name() + organization
        if os.path.isfile(self._otawp.license_file()):
            self.logger.info(
                "Found OTAWP license file -> '%s', assiging it to OTDS resource -> '%s'...",
                self._otawp.license_file(),
                organization,
            )

            otawp_license = self._otds.add_license_to_resource(
                path_to_license_file=self._otawp.license_file(),
                product_name=product_name,
                product_description=product_description,
                resource_id=awp_resource["resourceID"],
            )
            if not otawp_license:
                self.logger.error(
                    "Couldn't apply license -> '%s' for product -> '%s' to OTDS resource -> '%s'!",
                    self._otawp.license_file(),
                    product_name,
                    awp_resource["resourceID"],
                )
            else:
                self.logger.info(
                    "Successfully applied license -> '%s' for product -> '%s' to OTDS resource -> '%s'.",
                    self._otawp.license_file(),
                    product_name,
                    awp_resource["resourceID"],
                )

            # Assign AppWorks license to Content Server Members Partiton and otds.admin:
            for partition_name in ["otds.admin", self._otcs.partition_name()]:
                if self._otds.is_partition_licensed(
                    partition_name=partition_name,
                    resource_id=awp_resource["resourceID"],
                    license_feature="USERS",
                    license_name=license_name,
                ):
                    self.logger.info(
                        "Partition -> '%s' is already licensed for -> '%s' (%s).",
                        partition_name,
                        product_name,
                        "USERS",
                    )
                else:
                    assigned_license = self._otds.assign_partition_to_license(
                        partition_name=partition_name,
                        resource_id=awp_resource["resourceID"],
                        license_feature="USERS",
                        license_name=license_name,
                    )
                    if not assigned_license:
                        self.logger.error(
                            "Partition -> '%s' could not be assigned to license -> '%s' (%s)!",
                            partition_name,
                            product_name,
                            "USERS",
                        )
                        success = False
                    else:
                        self.logger.info(
                            "Partition -> '%s' successfully assigned to license -> '%s' (%s).",
                            partition_name,
                            product_name,
                            "USERS",
                        )
            # end for partition_name in ["otds.admin", self._otcs.partition_name()]:
        # end if os.path.isfile(self._otawp.license_file()):

        self.logger.info("Restart AppWorks Kubernetes stateful set -> '%s'...", self._otawp.hostname())

        self._k8s.restart_stateful_set(sts_name=self._otawp.hostname(), force=True, wait=True)

        self._otawp.set_organization(organization)
        otawp_cookie = self._otawp.authenticate(revalidate=True)
        if not otawp_cookie:
            self.logger.error(
                "Authentication at AppWorks failed! Cannot proceed with processing of AppWorks configuration -> '%s'!",
                organization,
            )
            success = False
            continue
        self._otawp.create_cws_config(
            partition=self._otcs.partition_name(), resource_name=organization, otcs_url=self._otcs.cs_url()
        )
        self._otawp.assign_role_to_user(
            organization=organization, user_name=self._otawp.username(), role_name="Developer"
        )
    # end for appworks_configuration in self._appworks_configurations:

    #
    # 2: Loop to create the AppWorks workspaces, projects, and entities:
    #
    for appworks_configuration in self._appworks_configurations:
        organization = appworks_configuration.get("organization", None)
        if not organization:
            self.logger.error("AppWorks configuration is missing the organization name! Skipping...")
            success = False
            continue

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not appworks_configuration.get("enabled", True):
            self.logger.info(
                "Payload for AppWorks configuration -> '%s' is disabled. Skipping...",
                organization,
            )
            continue

        self._log_header_callback(
            text="Process AppWorks workspaces, project, entities for '{}'".format(organization), char="-"
        )

        self._otawp.set_organization(organization)
        otawp_cookie = self._otawp.authenticate(revalidate=True)
        if not otawp_cookie:
            self.logger.error(
                "Authentication at AppWorks failed. Cannot proceed with processing of AppWorks configutation -> '%s'!",
                organization,
            )
            success = False
            continue

        if "workspaces" not in appworks_configuration:
            self.logger.warning(
                "No workspace information in AppWorks configuration -> '%s'. Skipping...", organization
            )
            continue

        for workspace in appworks_configuration["workspaces"]:
            workspace_id = workspace.get("workspace_id", None)
            workspace_name = workspace.get("name", None)
            workspace_path = workspace.get("path", None)
            if not workspace_id or not workspace_name or not workspace_path:
                self.logger.error(
                    "AppWorks workspace configuration for -> '%s'%s requires 'workspace_id', 'name', and 'path' settings! Skipping...",
                    organization,
                    " (workspace name -> {})".format(workspace_name) if workspace_name else "",
                )
                success = False
                continue

            response, created = self._otawp.create_workspace(
                workspace_name=workspace_name, workspace_id=workspace_id
            )
            if not response:
                self.logger.info("Failed to create workspace -> '%s' (%s)!", workspace_name, workspace_id)
                success = False
                continue

            if created:
                self.logger.info("Setup new workspace -> '%s' (%s)...", workspace_name, workspace_id)
                response = self._otawp.sync_workspace(
                    workspace_name=workspace_name,
                    workspace_id=workspace_id,
                )
                if not response:
                    self.logger.error("Failed to synchronize workspace -> '%s' (%s)!", workspace_name, workspace_id)
                    success = False
                    continue
                self.logger.info(
                    "Copy projects artifacts to workspace -> '%s' (%s) in AppWorks pod -> '%s'...",
                    workspace_name,
                    workspace_id,
                    "appworks-0",
                )
                self._k8s.exec_pod_command(
                    pod_name="appworks-0",
                    command=[
                        "/bin/sh",
                        "-c",
                        f'cp -r "{workspace_path}/"* "/opt/appworks/cluster/shared/cws/sync/{organization}/{workspace_name}"',
                    ],
                    timeout=600,
                )
                self.logger.info(
                    "Copying of projects artifacts to workspace -> '%s' (%s) completed.",
                    workspace_name,
                    workspace_id,
                )
            self.logger.info("Re-sync existing workspace -> '%s' (%s)...", workspace_name, workspace_id)
            response = self._otawp.sync_workspace(
                workspace_name=workspace_name,
                workspace_id=workspace_id,
            )
            if not response:
                self.logger.error("Failed to synchronize workspace -> '%s' (%s)!", workspace_name, workspace_id)
                success = False
                continue

            if "projects" in workspace:
                for project in workspace["projects"]:
                    if project.get("name") and project.get("documentId"):
                        if not self._otawp.publish_project(
                            workspace_name=workspace_name,
                            workspace_id=workspace_id,
                            project_name=project.get("name"),
                            project_id=project.get("documentId"),
                        ):
                            success = False
                            continue
                    else:
                        self.logger.error(
                            "Skipping project -> '%s' due to missing required project fields 'name' or 'documentId'!",
                            project.get("name"),
                        )
                        success = False
                        continue
                # end for project in workspace["projects"]:
            # end if "projects" in workspace
        # end for workspace in value["cws"]["workspaces"]

        # Process the entities in the payload:
        entities = appworks_configuration.get("entities", [])
        if entities:
            self._log_header_callback(
                text="Process AppWorks entities for organization -> '{}'".format(organization), char="-"
            )
            for entity in entities:
                if not self.process_appworks_entity(entity=entity):
                    success = False
            self.logger.info("Entity processing completed for organization -> '%s'.", organization)
    # end for appworks_configuration in self._appworks_configurations:

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._appworks_configurations,
    )

    return success

process_appworks_entity(entity)

Process a single AppWorks entity payload.

Parameters:

Name	Type	Description	Default
entity	dict	Entity payload. Should have a selection of the following keys: * type * name * description * status * prefix * category * priority * customer * case_type * ...	required

Returns:

Name	Type	Description
bool	bool	True = success, False = failure.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_appworks_entity")
def process_appworks_entity(self, entity: dict) -> bool:
    """Process a single AppWorks entity payload.

    Args:
        entity (dict):
            Entity payload.
            Should have a selection of the following keys:
            * type
            * name
            * description
            * status
            * prefix
            * category
            * priority
            * customer
            * case_type
            * ...

    Returns:
        bool: True = success, False = failure.

    """

    entity_type = entity.get("type")
    if not entity_type:
        return False

    match entity_type:
        case "category":
            cat = self._otawp.get_category_by_name(name=entity.get("name"))
            if cat:
                cat_id = self._otawp.get_entity_value(entity=cat, key="id")
                self.logger.info(
                    "Category -> '%s' (%s) does already exist. Skipping...", entity.get("name"), str(cat_id)
                )
                self.logger.debug("Category -> %s", str(cat))
            else:
                response = self._otawp.create_category(
                    case_prefix=entity.get("prefix"),
                    name=entity.get("name"),
                    description=entity.get("description", ""),
                    status=entity.get("status", 1),
                )
                if not response or "Identity" not in response:
                    self.logger.error("Failed to create category -> '%s'!", entity.get("name"))
                    return False
                cat_id = response["Identity"].get("Id")
                self.logger.info(
                    "Successfully created category -> '%s' (%s).",
                    entity.get("name"),
                    cat_id,
                )
                self.logger.debug("Response -> %s", str(response))
            if "sub_entities" in entity:
                for sub_entity in entity["sub_entities"]:
                    if sub_entity["type"] != "subCategory":
                        self.logger.warning(
                            "Found a category sub-entities with wrong type -> '%s'!", sub_entity["type"]
                        )
                        continue
                    response = self._otawp.create_sub_category(
                        parent_id=cat_id,
                        name=sub_entity.get("name"),
                        description=sub_entity.get("description", ""),
                        status=sub_entity.get("status", 1),
                    )
                    if not response or "Identity" not in response:
                        self.logger.error("Failed to create sub-category -> '%s'!", sub_entity.get("name"))
                        return False
                    self.logger.info(
                        "Successfully created sub-category -> '%s' (%s) in parent category -> '%s' (%s).",
                        sub_entity.get("name"),
                        response["Identity"].get("Id"),
                        entity.get("name"),
                        cat_id,
                    )
                    self.logger.debug("Response -> %s", str(response))
            return True
        case "priority":
            priority = self._otawp.get_priority_by_name(name=entity.get("name"))
            if priority:
                priority_id = self._otawp.get_entity_value(entity=priority, key="id")
                (
                    self.logger.info(
                        "Priority -> '%s' (%s) does already exist. Skipping...",
                        entity.get("name"),
                        str(priority_id),
                    )
                )
                return True
            response = self._otawp.create_priority(
                name=entity.get("name"),
                description=entity.get("description", ""),
                status=entity.get("status", 1),
            )
            if not response or "Identity" not in response:
                self.logger.error("Failed to create priority -> '%s'!", entity.get("name"))
                return False
            self.logger.info(
                "Successfully created priority -> '%s' (%s).", entity.get("name"), response["Identity"].get("Id")
            )
            self.logger.debug("Response -> %s", str(response))
            return True
        case "caseType":
            case_type = self._otawp.get_case_type_by_name(name=entity.get("name"))
            if case_type:
                case_type_id = self._otawp.get_entity_value(entity=case_type, key="id")
                self.logger.info(
                    "Case type -> '%s' (%s) does already exist. Skipping...", entity.get("name"), str(case_type_id)
                )
                return True
            response = self._otawp.create_case_type(
                name=entity.get("name"),
                description=entity.get("description", ""),
                status=entity.get("status", 1),
            )
            if not response or "Identity" not in response:
                self.logger.error("Failed to case type -> '%s'!", entity.get("name"))
                return False
            self.logger.info(
                "Successfully created case type -> '%s' (%s).", entity.get("name"), response["Identity"].get("Id")
            )
            self.logger.debug("Response -> %s", str(response))
            return True
        case "customer":
            customer = self._otawp.get_customer_by_name(name=entity.get("name"))
            if customer:
                customer_id = self._otawp.get_entity_value(entity=customer, key="id")
                self.logger.info(
                    "Customer -> '%s' (%s) does already exist. Skipping...", entity.get("name"), str(customer_id)
                )
                return True
            response = self._otawp.create_customer(
                customer_name=entity.get("name"),
                legal_business_name=entity.get("legal_business_name", ""),
                trading_name=entity.get("trading_name", ""),
            )
            if not response or "Identity" not in response:
                self.logger.error("Failed to create customer -> '%s'!", entity.get("name"))
                return False
            self.logger.info(
                "Successfully created customer -> '%s' (%s).", entity.get("name"), response["Identity"].get("Id")
            )
            self.logger.debug("Response -> %s", str(response))
            return True
        case "case":
            if "subject" not in entity:
                self.logger.error("Cannot create a case without a subject!")
                return False

            category_name = entity.get("category")
            if category_name:
                category = self._otawp.get_category_by_name(name=category_name)
                category_id = self._otawp.get_entity_value(entity=category, key="id")
                if not category_id:
                    self.logger.error(
                        "Cannot find case category -> '%s' to create case -> '%s'!",
                        category_name,
                        entity["subject"],
                    )
                    return False
            else:
                self.logger.warning(
                    "Case entity -> '%s' does not have a category specified in its payload!", entity["subject"]
                )
                category_id = None

            sub_category_name = entity.get("sub_category")
            if category_id and sub_category_name:
                sub_category_id = self._otawp.get_sub_category_id(name=sub_category_name, parent_id=category_id)
            else:
                sub_category_id = None

            priority_name = entity.get("priority")
            if priority_name:
                priority = self._otawp.get_priority_by_name(name=priority_name)
                priority_id = self._otawp.get_entity_value(entity=priority, key="id")
                if not priority_id:
                    self.logger.error(
                        "Cannot find case priority -> '%s' to create case -> '%s'!",
                        priority_name,
                        entity["subject"],
                    )
                    return False
            else:
                self.logger.warning(
                    "Case entity -> '%s' does not have a priority specified in its payload!", entity["subject"]
                )
                priority_id = None

            case_type_name = entity.get("case_type")
            if case_type_name:
                case_type = self._otawp.get_case_type_by_name(name=case_type_name)
                case_type_id = self._otawp.get_entity_value(entity=case_type, key="id")
                if not case_type_id:
                    self.logger.error(
                        "Cannot find case type -> '%s' to create case -> '%s'!",
                        case_type_name,
                        entity["subject"],
                    )
                    return False
            else:
                self.logger.warning(
                    "Case entity -> '%s' does not have a case type specified in its payload!", entity["subject"]
                )
                case_type_id = None

            customer_name = entity.get("customer")
            if customer_name:
                customer = self._otawp.get_customer_by_name(name=customer_name)
                customer_id = self._otawp.get_entity_value(entity=customer, key="id")
                if not customer_id:
                    self.logger.error(
                        "Cannot find customer -> '%s' to create case -> '%s'!",
                        customer_name,
                        entity["subject"],
                    )
                    return False
            else:
                self.logger.warning(
                    "Case entity -> '%s' does not have a customer specified in its payload!", entity["subject"]
                )
                customer_id = None

            response = self._otawp.create_case(
                subject=entity.get("subject"),
                description=entity.get("description", ""),
                loan_amount=entity.get("loan_amount", 1),
                loan_duration_in_months=entity.get("loan_duration_in_month", 2),
                category_id=category_id,
                sub_category_id=sub_category_id,
                priority_id=priority_id,
                case_type_id=case_type_id,
                customer_id=customer_id,
            )
            if not response:
                self.logger.error("Failed to create case with subject -> '%s'!", entity.get("subject"))
                return False
            self.logger.info(
                "Successfully created case with subject -> '%s'%s.",
                entity.get("subject"),
                " for customer with ID -> '{}'".format(customer_id) if customer_id else "",
            )
            self.logger.debug("Response -> %s", str(response))
            return True
        case _:
            self.logger.error("Illegal entity type -> '%s'!", entity_type)
            return False

    return False

process_assignments(section_name='assignments')

Process assignments and assign items (such as workspaces and items with nicknames) to users or groups.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'assignments'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_assignments")
def process_assignments(self, section_name: str = "assignments") -> bool:
    """Process assignments and assign items (such as workspaces and items with nicknames) to users or groups.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise

    """

    if not self._assignments:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for assignment in self._assignments:
        # Sanity check: we need a subject - it's mandatory:
        subject = assignment.get("subject")
        if not subject:
            self.logger.error("Assignment needs a subject! Skipping assignment...")
            success = False
            continue

        # Check if assignment has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not assignment.get("enabled", True):
            self.logger.info(
                "Payload for assignment -> '%s' is disabled. Skipping...",
                subject,
            )
            continue

        # instruction is optional but we give a warning if they are missing:
        instruction = assignment.get("instruction", "")
        if not instruction:
            self.logger.warning(
                "Assignment -> '%s' should have an instruction!",
                subject,
            )

        # Sanity check: we either need users or groups (or both):
        if "groups" not in assignment and "users" not in assignment:
            self.logger.error(
                "Assignment -> '%s' needs groups or users! Skipping assignment...",
                subject,
            )
            success = False
            continue

        # Check if a workspace is specified for the assignment and check it does exist:
        if assignment.get("workspace"):
            workspace = next(
                (item for item in self._workspaces if item["id"] == assignment["workspace"]),
                None,
            )
            if not workspace:
                self.logger.error(
                    "Assignment -> '%s' has specified a not existing workspace -> %s! Skipping assignment...",
                    subject,
                    assignment["workspace"],
                )
                success = False
                continue
            node_id = self.determine_workspace_id(workspace=workspace)
            if not node_id:
                self.logger.error(
                    "Assignment -> '%s' has specified a not existing workspace -> %s! Skipping assignment...",
                    subject,
                    assignment["workspace"],
                )
                success = False
                continue
        # If we don't have a workspace then check if a nickname is specified for the assignment:
        elif "nickname" in assignment:
            response = self._otcs.get_node_from_nickname(
                nickname=assignment["nickname"],
            )
            node_id = self._otcs.get_result_value(response=response, key="id")
            if not node_id:
                # if response == None:
                self.logger.error(
                    "Assignment item with nickname -> '%s' not found",
                    assignment["nickname"],
                )
                success = False
                continue
        else:
            self.logger.error(
                "Assignment -> '%s' needs a workspace or nickname! Skipping assignment...",
                subject,
            )
            success = False
            continue

        assignees = []

        if "groups" in assignment:
            group_assignees = assignment["groups"]
            for group_assignee in group_assignees:
                # find the group in the group list
                group = next(
                    (item for item in self._groups if item["name"] == group_assignee),
                    None,
                )
                if not group:
                    self.logger.error(
                        "Assignment group -> '%s' does not exist! Skipping group...",
                        group_assignee,
                    )
                    success = False
                    continue
                if "id" not in group:
                    self.logger.error(
                        "Assignment group -> '%s' does not have an ID. Skipping group...",
                        group_assignee,
                    )
                    success = False
                    continue
                group_id = group["id"]
                # add the group ID to the assignee list:
                assignees.append(group_id)

        if "users" in assignment:
            user_assignees = assignment["users"]
            for user_assignee in user_assignees:
                # find the user in the user list
                user = next(
                    (item for item in self._users if item["name"] == user_assignee),
                    None,
                )
                if not user:
                    self.logger.error(
                        "Assignment user -> '%s' does not exist! Skipping user...",
                        user_assignee,
                    )
                    success = False
                    continue
                if "id" not in user:
                    self.logger.error(
                        "Assignment user -> '%s' does not have an ID. Skipping user...",
                        user_assignee,
                    )
                    success = False
                    continue
                user_id = user["id"]
                # add the group ID to the assignee list:
                assignees.append(user_id)

        if not assignees:
            self.logger.error(
                "Cannot add assignment -> '%s' for node ID -> %s because no assignee was found.",
                subject,
                node_id,
            )
            success = False
            continue

        response = self._otcs.assign_item_to_user_group(
            node_id=int(node_id),
            subject=subject,
            instruction=instruction,
            assignees=assignees,
        )
        if not response:
            self.logger.error(
                "Failed to add assignment -> '%s' for node ID -> %s with assignees -> %s!",
                subject,
                node_id,
                assignees,
            )
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._assignments,
    )

    return success

process_auth_handlers(section_name='authHandlers')

Process OTDS authorization handlers in payload and create them in OTDS.

An authorization handler defined the connection to an Identity Provider (IdP).

The payload section is a list of dicts with these items: { enabled (bool): True or False to enable or disable the payload item name (str): Name of the authorization handler. This is shown in the first column of the Auth Handler list in OTDS. description (str): Description of the handler. This is shown in the second column of the Auth Handler type (str): Type of the handler. Possible values are SALM, SAP, OAUTH priority (int): A numeric value to order different handlers in OTDS by priority active_by_default (bool): Whether to activate this handler for any request to the OTDS login page. If True, any login request to the OTDS login page will be redirected to the IdP. If false, the user has to select the provider on the login page. provider_name: The name of the identity provider. This should be a single word since it will be part of the metadata URL. This is what is shown as a button on the OTDS login page. auth_principal_attributes: Authentication principal attributes (list) nameid_format: Specifies which NameID format supported by the identity provider contains the desired user identifier. The value in this identifier must correspond to the value of the user attribute specified for the authentication principal attribute. saml_url: Required for SAML Authentication Handler. The URL for the IdP's federation metadata. otds_sp_endpoint: Used for SAML Authentication Handler. Specifies the service provider URL that will be used to identify OTDS to the identity provider. certificate_file: Required for SAP Authentication Handler (SAPSSOEXT). Fully qualified file name (with path) to the certificate file (URI) certificate_password: Required for SAP Authentication Handler (SAPSSOEXT). Password of the certificate file. client_id: Client ID. Required for OAUTH authentication handler. client_secret: Client Secret. Required for OAUTH authentication handler. authorization_endpoint: Required for OAUTH authentication handler. The URL to redirect the browser to for authentication. It is used to retrieve the authorization code or an OIDC id_token. token_endpoint: Used for OAUTH authentication handler. The URL from which to retrieve the access token. Not strictly required with OpenID Connect if using the implicit flow. scope_string: Used for OAUTH authentication handler. Space delimited scope values to send. Include 'openid' to use OpenID Connect. }

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'authHandlers'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_auth_handlers")
def process_auth_handlers(self, section_name: str = "authHandlers") -> bool:
    """Process OTDS authorization handlers in payload and create them in OTDS.

    An authorization handler defined the connection to an Identity Provider (IdP).

    The payload section is a list of dicts with these items:
    {
        enabled (bool):
            True or False to enable or disable the payload item
        name (str):
            Name of the authorization handler. This is shown in the first
            column of the Auth Handler list in OTDS.
        description (str):
            Description of the handler. This is shown in the second
            column of the Auth Handler
        type (str):
            Type of the handler. Possible values are SALM, SAP, OAUTH
        priority (int):
            A numeric value to order different handlers in OTDS by priority
        active_by_default (bool):
            Whether to activate this handler for any request to the
            OTDS login page. If True, any login request to the OTDS
            login page will be redirected to the IdP. If false, the
            user has to select the provider on the login page.
        provider_name:
            The name of the identity provider. This should be a single word
            since it will be part of the metadata URL. This is what is
            shown as a button on the OTDS login page.
        auth_principal_attributes:
            Authentication principal attributes (list)
        nameid_format:
            Specifies which NameID format supported by the identity provider
            contains the desired user identifier. The value in this identifier
            must correspond to the value of the user attribute specified for the
            authentication principal attribute.
        saml_url:
            Required for SAML Authentication Handler. The URL for the IdP's federation metadata.
        otds_sp_endpoint:
            Used for SAML Authentication Handler. Specifies the service provider URL that will
            be used to identify OTDS to the identity provider.
        certificate_file:
            Required for SAP Authentication Handler (SAPSSOEXT).
            Fully qualified file name (with path) to the certificate file (URI)
        certificate_password:
            Required for SAP Authentication Handler (SAPSSOEXT).
            Password of the certificate file.
        client_id:
            Client ID. Required for OAUTH authentication handler.
        client_secret:
            Client Secret. Required for OAUTH authentication handler.
        authorization_endpoint:
            Required for OAUTH authentication handler.
            The URL to redirect the browser to for authentication.
            It is used to retrieve the authorization code or an OIDC id_token.
        token_endpoint:
            Used for OAUTH authentication handler. The URL from which to retrieve the access token.
            Not strictly required with OpenID Connect if using the implicit flow.
        scope_string:
            Used for OAUTH authentication handler. Space delimited scope values to send.
            Include 'openid' to use OpenID Connect.
    }

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._auth_handlers:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for auth_handler in self._auth_handlers:
        handler_name = auth_handler.get("name")

        if not handler_name:
            self.logger.error("Auth handler does not have a name in payload. Skipping...")
            success = False
            continue

        # Check if Auth Handler does already exist (e.g. after a restart of
        # the customizer pod):
        if self._otds.get_auth_handler(name=handler_name, show_error=False):
            self.logger.info(
                "Auth handler -> '%s' does already exist. Skipping...",
                handler_name,
            )
            continue

        handler_description = auth_handler.get("description")

        # Check if auth handler has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not auth_handler.get("enabled", True):
            self.logger.info(
                "Payload for OTDS authorization handler -> '%s' is disabled. Skipping...",
                handler_name,
            )
            continue

        handler_scope = auth_handler.get("scope")
        if not handler_scope:
            # Make sure to pass None also if scope is empty string
            handler_scope = None

        handler_type = auth_handler.get("type")
        if not handler_type:
            self.logger.error(
                "OTDS authorization handler does not have a type in payload! Skipping...",
            )
            success = False
            continue

        priority = auth_handler.get("priority")
        active_by_default = auth_handler.get("active_by_default")
        if not active_by_default:
            active_by_default = False

        match handler_type:
            case "SAML":
                provider_name = auth_handler.get("provider_name")
                if not provider_name:
                    self.logger.error(
                        "SAML authorization handler needs a provider name in payload! Skipping...",
                    )
                    success = False
                    continue
                saml_url = auth_handler.get("saml_url")
                if not saml_url:
                    self.logger.error(
                        "SAML authorization handler needs a SAML URL in payload! Skipping...",
                    )
                    success = False
                    continue
                otds_sp_endpoint = auth_handler.get("otds_sp_endpoint")
                if not otds_sp_endpoint:
                    self.logger.error(
                        "SAML authorization handler needs a OTDS SP endpoint in payload! Skipping...",
                    )
                    success = False
                    continue
                auth_principal_attributes = auth_handler.get(
                    "auth_principal_attributes",
                )
                nameid_format = auth_handler.get("nameid_format")
                response = self._otds.add_auth_handler_saml(
                    name=handler_name,
                    description=handler_description,
                    scope=handler_scope,
                    provider_name=provider_name,
                    saml_url=saml_url,
                    otds_sp_endpoint=otds_sp_endpoint,
                    priority=priority,
                    active_by_default=active_by_default,
                    auth_principal_attributes=auth_principal_attributes,
                    nameid_format=nameid_format,
                )
            case "SAP":
                certificate_file = auth_handler.get("certificate_file")
                if not certificate_file:
                    self.logger.error(
                        "SAP Authorization handler -> '%s' (%s) requires a certificate file name in payload! Skipping...",
                        handler_name,
                        handler_type,
                    )
                    success = False
                    continue
                certificate_password = auth_handler.get("certificate_password")
                if not certificate_password:
                    # This is not an error - we canhave this key with empty string!
                    self.logger.info(
                        "SAP Authorization handler -> '%s' (%s) does not have a certificate password - this can be OK.",
                        handler_name,
                        handler_type,
                    )
                response = self._otds.add_auth_handler_sap(
                    name=handler_name,
                    description=handler_description,
                    scope=handler_scope,
                    certificate_file=certificate_file,
                    certificate_password=certificate_password,
                    priority=priority,
                )
            case "OAUTH":
                provider_name = auth_handler.get("provider_name")
                if not provider_name:
                    self.logger.error(
                        "OAUTH Authorization handler -> '%s' (%s) requires a provider name in payload! Skipping...",
                        handler_name,
                        handler_type,
                    )
                    success = False
                    continue
                client_id = auth_handler.get("client_id")
                if not client_id:
                    self.logger.error(
                        "OAUTH Authorization handler -> '%s' (%s) requires a client ID in payload! Skipping...",
                        handler_name,
                        handler_type,
                    )
                    success = False
                    continue
                client_secret = auth_handler.get("client_secret")
                if not client_secret:
                    self.logger.error(
                        "OAUTH Authorization handler -> '%s' (%s) requires a client secret in payload! Skipping...",
                        handler_name,
                        handler_type,
                    )
                    success = False
                    continue
                authorization_endpoint = auth_handler.get("authorization_endpoint")
                if not authorization_endpoint:
                    self.logger.error(
                        "OAUTH Authorization handler -> '%s' (%s) requires a authorization endpoint in payload! Skipping...",
                        handler_name,
                        handler_type,
                    )
                    success = False
                    continue
                token_endpoint = auth_handler.get("token_endpoint")
                if not token_endpoint:
                    self.logger.warning(
                        "OAUTH Authorization handler -> '%s' (%s) does not have a token endpoint!",
                        handler_name,
                        handler_type,
                    )
                scope_string = auth_handler.get("scope_string")
                response = self._otds.add_auth_handler_oauth(
                    name=handler_name,
                    description=handler_description,
                    scope=handler_scope,
                    provider_name=provider_name,
                    client_id=client_id,
                    client_secret=client_secret,
                    priority=priority,
                    active_by_default=active_by_default,
                    authorization_endpoint=authorization_endpoint,
                    token_endpoint=token_endpoint,
                    scope_string=scope_string,
                )
            case _:
                self.logger.error(
                    "Unsupported authorization handler type -> '%s'!",
                    handler_type,
                )
                return False

        if response:
            self.logger.info(
                "Successfully added OTDS authorization handler -> '%s' (%s).",
                handler_name,
                handler_type,
            )
        else:
            self.logger.error(
                "Failed to add OTDS authorization handler -> '%s' (%s)!",
                handler_name,
                handler_type,
            )
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._auth_handlers,
    )

    return success

process_avts_questions(section_name='avtsQuestions')

Process Aviator Search repositories.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'avtsQuestions'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_avts_questions")
def process_avts_questions(self, section_name: str = "avtsQuestions") -> bool:
    """Process Aviator Search repositories.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._avts_questions:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    if not self._avts_questions.get("enabled", True):
        self.logger.info(
            "Payload section -> '%s' is not enabled. Skipping...",
            section_name,
        )
        return True
    questions = self._avts_questions.get("questions", [])
    self.logger.info("Sample questions -> %s.", str(questions))

    token = self._avts.authenticate()
    if not token:
        self.logger.error("Cannot authenticate at Aviator Search!")
        success = False
    else:
        response = self._avts.set_questions(questions=questions)

        if response is None:
            self.logger.error("Aviator Search setting questions failed!")
            success = False
        else:
            self.logger.info("Successfully configured Aviator Search questions.")
            self.logger.debug("%s", response)

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._avts_questions,
    )

    return success

process_avts_repositories(section_name='avtsRepositories')

Process Aviator Search repositories.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'avtsRepositories'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_avts_repositories")
def process_avts_repositories(self, section_name: str = "avtsRepositories") -> bool:
    """Process Aviator Search repositories.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._avts_repositories:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    token = self._avts.authenticate()
    if not token:
        self.logger.error("Cannot authenticate at Aviator Search!")
        success = False
    else:
        for payload_repo in self._avts_repositories:
            if not payload_repo.get("enabled", True):
                continue

            repository = self._avts.get_repo_by_name(name=payload_repo["name"])

            if repository is None:
                self.logger.info(
                    "Repository -> '%s' does not exist, creating it...",
                    payload_repo["name"],
                )

                if payload_repo.get("type", "Extended ECM") == "Extended ECM":
                    repository = self._avts.create_extended_ecm_repo(
                        name=payload_repo["name"],
                        username=payload_repo["username"],
                        password=payload_repo["password"],
                        otcs_url=payload_repo["otcs_url"],
                        otcs_api_url=payload_repo["otcs_api_url"],
                        node_id=int(payload_repo["node_id"]),
                    )

                elif payload_repo["type"] == "Documentum":
                    self.logger.warning("Not yet implemented")
                elif payload_repo["type"] == "MSTeams":
                    repository = self._avts.create_msteams_repo(
                        name=payload_repo["name"],
                        client_id=payload_repo["client_id"],
                        tenant_id=payload_repo["tenant_id"],
                        certificate_file=payload_repo["certificate_file"],
                        certificate_password=payload_repo["certificate_password"],
                        index_attachments=payload_repo.get("index_attachments", True),
                        index_call_recordings=payload_repo.get(
                            "index_call_recordings",
                            True,
                        ),
                        index_message_replies=payload_repo.get(
                            "index_message_replies",
                            True,
                        ),
                        index_user_chats=payload_repo.get("index_user_chats", True),
                    )
                elif payload_repo["type"] == "SharePoint":
                    repository = self._avts.create_sharepoint_repo(
                        name=payload_repo["name"],
                        client_id=payload_repo["client_id"],
                        tenant_id=payload_repo["tenant_id"],
                        certificate_file=payload_repo["certificate_file"],
                        certificate_password=payload_repo["certificate_password"],
                        sharepoint_url=payload_repo["sharepoint_url"],
                        sharepoint_url_type=payload_repo["sharepoint_url_type"],
                        sharepoint_mysite_url=payload_repo["sharepoint_mysite_url"],
                        sharepoint_admin_url=payload_repo["sharepoint_admin_url"],
                        index_user_profiles=payload_repo.get(
                            "index_message_replies",
                            False,
                        ),
                    )
                else:
                    self.logger.error(
                        "Invalid repository type -> '%s' specified! Valid values are: Extended ECM, Documentum, MSTeams, SharePoint.",
                        payload_repo["type"],
                    )
                    success = False
                    break

                if repository is None:
                    self.logger.error(
                        "Creation of Aviator Search repository -> '%s' failed!",
                        payload_repo["name"],
                    )
                    success = False
                else:
                    self.logger.info(
                        "Successfully created Aviator Search repository -> '%s'.",
                        payload_repo["name"],
                    )
                    self.logger.debug("%s", repository)

            else:
                self.logger.info(
                    "Aviator Search repository -> '%s' already exists.",
                    payload_repo["name"],
                )

            # Start Crawling
            start_crawling = payload_repo.get("start", False)

            if repository is not None and start_crawling:
                response = self._avts.start_crawling(repo_name=payload_repo["name"])

                if response is None:
                    self.logger.error(
                        "Aviator Search start crawling on repository failed -> '%s'!",
                        payload_repo["name"],
                    )
                    success = False
                else:
                    self.logger.info(
                        "Aviator Search crawling started on repository -> '%s'",
                        payload_repo["name"],
                    )
                    self.logger.debug("%s", response)
        # end for payload_repo in self._avts_repositories:
    # end else:

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._avts_repositories,
    )

    return success

process_browser_automations(browser_automations, section_name='browserAutomations', check_status=True)

Process Playwright-based browser automations and tests.

Parameters:

Name	Type	Description	Default
browser_automations	list	A list of browser automations (need this as parameter as we have multiple lists).	required
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'browserAutomations'
check_status	bool	Defines whether or not this needs to re-run for each customizer run (even if it has been successful before). If check_status is True (default) then it is only re-run if it has NOT been successfully before.	True

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_browser_automations")
def process_browser_automations(
    self,
    browser_automations: list,
    section_name: str = "browserAutomations",
    check_status: bool = True,
) -> bool:
    """Process Playwright-based browser automations and tests.

    Args:
        browser_automations (list):
            A list of browser automations (need this as parameter as we
            have multiple lists).
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.
        check_status (bool, optional):
            Defines whether or not this needs to re-run
            for each customizer run (even if it has been successful before).
            If check_status is True (default) then it is only re-run
            if it has NOT been successfully before.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise.

    """

    if not browser_automations:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if check_status and self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    automation_type = "Browser" if "browser" in section_name else "Test"

    for browser_automation in browser_automations:
        name = browser_automation.get("name")
        if not name:
            self.logger.error("%s automation is missing a unique name. Skipping...", automation_type)
            success = False
            continue

        self._log_header_callback(
            text="Process {} Automation -> '{}'".format(automation_type, name),
            char="-",
        )

        description = browser_automation.get("description", "")
        if description:
            self.logger.info(
                "%s automation description -> '%s'.",
                automation_type,
                description,
            )

        # Check if browser automation has been explicitly disabled in payload
        # (enabled = false). In this case we skip this payload element:
        if not browser_automation.get("enabled", True):
            self.logger.info(
                "Payload for %s automation -> '%s' is disabled. Skipping...",
                automation_type.lower(),
                name,
            )
            continue

        base_url = browser_automation.get("base_url")
        if not base_url:
            self.logger.error(
                "%s automation -> '%s' is missing 'base_url' parameter. Skipping...", automation_type, name
            )
            browser_automation["result"] = "failure"
            success = False
            continue

        user_name = browser_automation.get("user_name", "")
        if not user_name:
            self.logger.info("%s automation -> '%s' is not having user name.", automation_type, name)

        password = browser_automation.get("password", "")
        if not password:
            self.logger.info("%s automation -> '%s' is not having password.", automation_type, name)

        automation_steps = browser_automation.get("automations", [])
        if not automation_steps:
            self.logger.error(
                "%s automation -> '%s' is missing list of automations. Skipping...",
                automation_type,
                name,
            )
            browser_automation["result"] = "failure"
            success = False
            continue

        debug_automation: bool = browser_automation.get("debug", False)

        # Create Selenium Browser Automation:
        self.logger.info("%s automation base URL -> %s.", automation_type, base_url)
        self.logger.info("%s automation user -> '%s'.", automation_type, user_name)
        wait_until = browser_automation.get("wait_until")  # it is OK to be None
        if "wait_until" in browser_automation:
            # Only log the "wait until" value if it is specified in the payload:
            self.logger.info(
                "%s Automation page navigation strategy is to wait until -> '%s'.",
                automation_type,
                wait_until,
            )
        browser_automation_object = BrowserAutomation(
            base_url=base_url,
            user_name=user_name,
            user_password=password,
            automation_name=name,
            take_screenshots=debug_automation,
            headless=browser_automation.get("headless", self._browser_headless),
            logger=self.logger,
            wait_until=wait_until,
            browser=browser_automation.get("browser"),  # None is acceptable
        )
        if not browser_automation_object:
            self.logger.error(
                "Cannot execute browser automation -> '%s'. Initialization of browser automation object failed!",
                name,
            )
            browser_automation["result"] = "failure"
            success = False
            continue
        # Wait time is a global setting (for whole browser session)
        # This makes sure a page is fully loaded and elements are present
        # before accessing them. We set 30.0 seconds as default if not
        # otherwise specified by "wait_time" in the payload.
        wait_time = float(browser_automation.get("wait_time", 30.0))
        browser_automation_object.set_timeout(wait_time=wait_time)
        if "wait_time" in browser_automation:
            self.logger.info(
                "%s automation wait time -> '%s' configured.",
                automation_type,
                str(wait_time),
            )

        # Initialize overall result status:
        result = True
        first_step = True

        for automation_step in automation_steps:
            automation_step_type = automation_step.get("type", "")
            if not automation_step_type:
                self.logger.error(
                    "%s automation step -> %s in browser automation -> '%s' is missing 'type' parameter. Stopping automation -> '%s'.",
                    automation_type,
                    str(automation_step),
                    name,
                    name,
                )
                success = False
                break
            # Check if browser automation step has been explicitly disabled in payload
            # (enabled = false). In this case we skip this automation step:
            if not automation_step.get("enabled", True):
                self.logger.info(
                    "Automation step -> '%s' in automation -> '%s' is disabled. Skipping...",
                    automation_step_type,
                    name,
                )
                continue
            dependent = automation_step.get("dependent", True)
            if not dependent and not result:
                self.logger.warning(
                    "Ignore result of previous step as current step -> '%s' is NOT dependent on it.",
                    automation_step_type,
                )
                result = True
            elif not result:
                # In this case a proceeding automation step has failed
                # and this step is marked as dependent. Then it does not make sense
                # to continue with this automation step after the proceeding step failed.
                self.logger.warning(
                    "Step -> '%s' is dependent on a proceeding step that failed. Skipping this step...",
                    automation_step_type,
                )
                continue
            elif not first_step:
                self.logger.debug(
                    "Current step -> '%s' is %s on proceeding step.",
                    automation_step_type,
                    "dependent" if dependent else "not dependent",
                )

            match automation_step_type:
                case "login":
                    page = automation_step.get("page", "")
                    user_field = automation_step.get("user_field", "otds_username")
                    password_field = automation_step.get(
                        "password_field",
                        "otds_password",
                    )
                    login_button = automation_step.get("login_button", "loginbutton")
                    # Do we have a step-specific wait mechanism? If not, we pass None
                    # then the browser automation will take the default configured for
                    # the whole browser automation (see BrowserAutomation() constructor above):
                    wait_until = automation_step.get("wait_until", None)
                    self.logger.info(
                        "Login to -> %s as user -> '%s' (%s page navigation strategy is to wait until -> '%s')...",
                        base_url + page,
                        user_name,
                        "specific" if wait_until is not None else "default",
                        wait_until if wait_until is not None else browser_automation_object.wait_until,
                    )
                    result = browser_automation_object.run_login(
                        page=page,
                        user_field=user_field,
                        password_field=password_field,
                        login_button=login_button,
                        wait_until=wait_until,
                    )
                    if not result:
                        self.logger.error(
                            "Cannot log into -> %s. Skipping to next automation step...",
                            base_url + page,
                        )
                        automation_step["result"] = "failure"
                        success = False
                        continue
                    self.logger.info(
                        "Successfully logged into page -> %s. Page title -> '%s'.",
                        base_url + page,
                        browser_automation_object.get_title(),
                    )
                case "get_page":
                    page = automation_step.get("page", "")
                    if not page:
                        self.logger.error(
                            "Automation step type -> '%s' requires 'page' parameter. Stopping automation.",
                            automation_step_type,
                        )
                        automation_step["result"] = "failure"
                        success = False
                        break
                    volume = automation_step.get("volume", OTCS.VOLUME_TYPE_ENTERPRISE_WORKSPACE)
                    path = automation_step.get("path", [])
                    if path and volume:
                        page_node = self._otcs.get_node_by_volume_and_path(
                            volume_type=volume,
                            path=path,
                            create_path=False,
                        )
                        page_id = self._otcs.get_result_value(response=page_node, key="id")
                        if not page_id:
                            # if not parent_node:
                            self.logger.error(
                                "%s automation -> '%s' has a page path that does not exist. Skipping...",
                                automation_type,
                                name,
                            )
                            automation_step["result"] = "failure"
                            success = False
                            continue
                        self.logger.info(
                            "Resolved volume -> %s and page path -> %s to node ID -> %s.",
                            str(volume),
                            str(path),
                            str(page_id),
                        )
                    else:
                        page_id = None
                    if "{}" in page and page_id:
                        page = page.format(page_id)
                    # Do we have a step-specific wait mechanism? If not, we pass None
                    # then the browser automation will take the default configured for
                    # the whole browser automation (see BrowserAutomation() constructor called above):
                    wait_until = automation_step.get("wait_until", None)
                    self.logger.info(
                        "Load page -> %s (%s page navigation strategy is to wait until -> '%s')...",
                        base_url + page,
                        "specific" if wait_until is not None else "default",
                        wait_until if wait_until is not None else browser_automation_object.wait_until,
                    )
                    result = browser_automation_object.get_page(url=page, wait_until=wait_until)
                    if not result:
                        self.logger.error(
                            "Cannot load page -> %s! Skipping this step...",
                            page,
                        )
                        automation_step["result"] = "failure"
                        success = False
                        continue
                    self.logger.info(
                        "Successfully loaded page -> %s. Page title -> '%s'.",
                        base_url + page,
                        browser_automation_object.get_title(),
                    )
                case "click_elem":
                    # We keep the deprecated "elem" syntax supported (for now)
                    selector = automation_step.get("selector", automation_step.get("elem", ""))
                    if not selector:
                        self.logger.error(
                            "Automation step type -> '%s' requires 'selector' parameter. Stopping automation.",
                            automation_step_type,
                        )
                        automation_step["result"] = "failure"
                        success = False
                        break
                    # We keep the deprecated "find" syntax supported (for now)
                    selector_type = automation_step.get("selector_type", automation_step.get("find", "id"))
                    show_error = automation_step.get("show_error", True)
                    # Do we navigate away from the current page with the click?
                    navigation = automation_step.get("navigation", False)
                    # Do we open a new browser (popup) window with the click?
                    popup_window = automation_step.get("popup_window", False)
                    # De we close the current (popup) window with the click?
                    close_window = automation_step.get("close_window", False)
                    # Do we have a 'desired' state for clicking a checkbox?
                    checkbox_state = automation_step.get("checkbox_state", None)
                    wait_until = automation_step.get("wait_until", None)
                    wait_time = automation_step.get("wait_time", 0.0)
                    role_type = automation_step.get("role_type", None)
                    occurrence = automation_step.get("occurrence", 1)
                    scroll_to_element = automation_step.get("scroll_to_element", True)
                    exact_match = automation_step.get("exact_match", None)
                    regex = automation_step.get("regex", None)
                    hover_only = automation_step.get("hover_only", False)
                    iframe = automation_step.get("iframe", None)
                    force = automation_step.get("force", None)
                    click_button = automation_step.get("click_button", None)
                    click_count = automation_step.get("click_count", None)
                    click_modifiers = automation_step.get("click_modifiers", None)
                    repeat_reload = automation_step.get("repeat_reload", None)
                    repeat_reload_delay = automation_step.get("repeat_reload_delay", None)
                    result = browser_automation_object.find_elem_and_click(
                        selector=selector,
                        selector_type=selector_type,
                        role_type=role_type,
                        occurrence=occurrence,
                        scroll_to_element=scroll_to_element,
                        desired_checkbox_state=checkbox_state,
                        is_navigation_trigger=navigation,
                        is_popup_trigger=popup_window,
                        is_page_close_trigger=close_window,
                        wait_until=wait_until,
                        wait_time=wait_time,
                        exact_match=exact_match,
                        regex=regex,
                        hover_only=hover_only,
                        iframe=iframe,
                        force=force,
                        click_button=click_button,
                        click_count=click_count,
                        click_modifiers=click_modifiers,
                        repeat_reload=repeat_reload,
                        repeat_reload_delay=repeat_reload_delay,
                        show_error=show_error,
                    )
                    if not result:
                        message = "Cannot find clickable element with selector -> '{}' ({}) on current page. Skipping this step...".format(
                            selector, selector_type
                        )
                        if show_error:
                            self.logger.error(message)
                            automation_step["result"] = "failure"
                            success = False
                        else:
                            self.logger.warning(message)
                        continue
                    self.logger.info(
                        "Successfully %s %s element selected by -> '%s' (%s%s).",
                        "clicked" if not hover_only else "hovered over",
                        "navigational" if navigation else "non-navigational",
                        selector,
                        "selector type -> '{}'".format(selector_type),
                        ", role type -> '{}'".format(role_type) if role_type else "",
                    )
                case "set_elem":
                    # We keep the deprecated "elem" syntax supported (for now)
                    selector = automation_step.get("selector", automation_step.get("elem", ""))
                    if not selector:
                        self.logger.error(
                            "Automation step type -> '%s' requires 'selector' parameter. Stopping automation.",
                            automation_step_type,
                        )
                        automation_step["result"] = "failure"
                        success = False
                        break
                    # We keep the deprecated "find" syntax supported (for now)
                    selector_type = automation_step.get("selector_type", automation_step.get("find", "id"))
                    role_type = automation_step.get("role_type", None)
                    occurrence = automation_step.get("occurrence", 1)
                    exact_match = automation_step.get("exact_match", None)
                    regex = automation_step.get("regex", None)
                    iframe = automation_step.get("iframe", None)
                    value = automation_step.get("value", "")
                    typing = automation_step.get("typing", False)
                    press_enter = automation_step.get("press_enter", False)
                    if not value:
                        self.logger.error(
                            "Automation step type -> '%s' for element selected by -> '%s' (%s) requires 'value' parameter. Stopping automation.",
                            automation_step_type,
                            selector,
                            selector_type,
                        )
                        automation_step["result"] = "failure"
                        success = False
                        break
                    # we also support replacing placeholders that are
                    # enclosed in double % characters like %%OTCS_RESOURCE_ID%%:
                    if isinstance(value, str):
                        value = self.replace_placeholders(value)
                    show_error = automation_step.get("show_error", True)
                    result = browser_automation_object.find_elem_and_set(
                        selector=selector,
                        value=value,
                        selector_type=selector_type,
                        role_type=role_type,
                        occurrence=occurrence,
                        press_enter=press_enter,
                        exact_match=exact_match,
                        regex=regex,
                        iframe=iframe,
                        typing=typing,
                        show_error=show_error,
                    )
                    if not result:
                        message = "Cannot set element{} selected by -> '{}' ({}{}) to value -> '{}'. Skipping this step...".format(
                            " (occurrence -> {})".format(occurrence) if occurrence > 1 else "",
                            selector,
                            "selector type -> '{}'".format(selector_type),
                            ", role type -> '{}'".format(role_type) if role_type else "",
                            value,
                        )
                        if show_error:
                            self.logger.error(message)
                            automation_step["result"] = "failure"
                            success = False
                        else:
                            self.logger.warning(message)
                        continue
                    self.logger.info(
                        "Successfully set element%s selected by -> '%s' (%s%s) to value -> '%s'.",
                        " (occurrence -> {})".format(occurrence) if occurrence > 1 else "",
                        selector,
                        "selector type -> '{}'".format(selector_type),
                        ", role type -> '{}'".format(role_type) if role_type else "",
                        value,
                    )
                case "check_elem":
                    # We keep the deprecated "elem" syntax supported (for now)
                    selector = automation_step.get("selector", automation_step.get("elem", ""))
                    if not selector:
                        self.logger.error(
                            "Automation step type -> '%s' requires 'selector' parameter. Stopping automation.",
                            automation_step_type,
                        )
                        automation_step["result"] = "failure"
                        success = False
                        break
                    # We keep the deprecated "find" syntax supported (for now)
                    selector_type = automation_step.get("selector_type", automation_step.get("find", "id"))
                    role_type = automation_step.get("role_type", None)
                    value = automation_step.get("value", None)
                    attribute = automation_step.get("attribute", "")
                    substring = automation_step.get("substring", False)
                    iframe = automation_step.get("iframe", None)
                    min_count = automation_step.get("min_count", 1)
                    want_exist = automation_step.get("want_exist", True)
                    wait_time = automation_step.get("wait_time", 0.0)
                    if value:
                        # we also support replacing placeholders that are
                        # enclosed in double % characters like %%OTCS_RESOURCE_ID%%:
                        value = self.replace_placeholders(value)
                    (result, count) = browser_automation_object.check_elems_exist(
                        selector=selector,
                        selector_type=selector_type,
                        role_type=role_type,
                        value=value,
                        attribute=attribute,
                        substring=substring,
                        iframe=iframe,
                        min_count=min_count,
                        wait_time=wait_time,  # time to wait before the check is actually done
                        show_error=not want_exist,  # if element is not found that we do not want to find it is not an error
                    )
                    # Check if we didn't get what we want:
                    if (not result and want_exist) or (result and not want_exist):
                        self.logger.error(
                            "%s %s%s%s on current page. Test failed.%s",
                            "Cannot find" if not result and want_exist else "Found",
                            "{} elements with selector -> '{}' ({}{})".format(
                                min_count if want_exist else count,
                                selector,
                                "selector type -> '{}'".format(selector_type),
                                ", role type -> '{}'".format(role_type) if role_type else "",
                            )
                            if (min_count > 1 and want_exist) or (count > 1 and not want_exist)
                            else "an element with selector -> '{}' ({}{})".format(
                                selector,
                                "selector type -> '{}'".format(selector_type),
                                ", role type -> '{}'".format(role_type) if role_type else "",
                            ),
                            " with {}value -> '{}'".format("substring-" if substring else "", value)
                            if value
                            else "",
                            " in attribute -> '{}'".format(attribute) if attribute else "",
                            " Found {}{} occurences.".format(
                                count,
                                " undesirable" if not want_exist else " from a minimum of {}".format(min_count),
                            ),
                        )
                        success = False
                        continue
                        # Don't break here! We want to do all existance tests!
                    self.logger.info(
                        "Successfully passed %sexistence test for %s%s%s on current page.",
                        "non-" if not want_exist else "",
                        "{} elements with selector -> '{}' ({})".format(min_count, selector, selector_type)
                        if min_count > 1
                        else "an element with selector -> '{}' ({})".format(selector, selector_type),
                        " with {}value -> '{}'".format("substring-" if substring else "", value) if value else "",
                        " in attribute -> '{}'".format(attribute) if attribute else "",
                    )
                case _:
                    self.logger.error(
                        "Illegal automation step type -> '%s' in %s automation!",
                        automation_step_type,
                        automation_type.lower(),
                    )
                    automation_step["result"] = "failure"
                    success = False
                    break
            # end match automation_step_type:
            first_step = False
        # end for automation_step in automation_steps:

        # Cleanup session and and remove reference to the object:
        browser_automation_object.end_session()
        browser_automation_object = None

        browser_automation["result"] = (
            "failure" if any(step.get("result", "success") == "failure" for step in automation_steps) else "success"
        )
    # end for browser_automation in browser_automations:

    if check_status:
        self.write_status_file(
            success=success,
            payload_section_name=section_name,
            payload_section=browser_automations,
        )

    return success

process_bulk_categories(row, index, categories, replacements)

Replace the value placeholders of the bulk category structures.

The payload placeholders are replaced with values from the Pandas Series (row). It also processes value mappings + lookups and special treatment of list and table values.

Parameters:

Name	Type	Description	Default
row	Series	The current row in the Pandas data frame.	required
index	str	The index of the Pandas data frame. Just used here for logging.	required
categories	list	List of payload categories.	required
replacements	list	List of replacements.	required

Returns:

Name	Type	Description
list	list	List of categories.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_categories")
def process_bulk_categories(
    self,
    row: pd.Series,
    index: str,
    categories: list,
    replacements: list,
) -> list:
    """Replace the value placeholders of the bulk category structures.

    The payload placeholders are replaced with values from the Pandas Series (row).
    It also processes value mappings + lookups and special treatment of list and table values.

    Args:
        row (pd.Series):
            The current row in the Pandas data frame.
        index (str):
            The index of the Pandas data frame. Just used here for logging.
        categories (list):
            List of payload categories.
        replacements (list):
            List of replacements.

    Returns:
        list:
            List of categories.

    """

    # Make sure the threads are not changing data structures that
    # are shared between threads. categories is a list of dicts.
    # list and dicts are "mutable" data structures in Python!
    worker_categories = copy.deepcopy(categories)

    # In this first loop we expand table-value attributes into a new
    # list of category / attribute payload. The value of table-value attributes
    # is a list of dictionaries (in a string we evaluate into a Python
    # datastructure)
    worker_categories_expanded = []
    for category_item in worker_categories:
        if "value_type" in category_item and category_item["value_type"] == "table":
            value_field = category_item["value_field"]

            # The following method always returns a string even if the value is actually a list.
            # This is required as we want to support multiple placeholders in one string...
            value = self.replace_bulk_placeholders(
                input_string=value_field,
                row=row,
                index=None,
                replacements=replacements,
            )
            # If value is empty (probably because placeholders could not be resolved)
            # and if the payload provides an alternative value field we try this:
            if not value and "value_field_alt" in category_item:
                value = self.replace_bulk_placeholders(
                    input_string=category_item["value_field_alt"],
                    row=row,
                    index=None,
                    replacements=replacements,
                )
            if not value:
                self.logger.info(
                    "Table-type attribute -> '%s' is empty (value field -> '%s'). Cannot parse table. Skipping...",
                    category_item.get("name", ""),
                    value_field,
                )
                continue

            try:
                value_table = literal_eval(value)
            except (SyntaxError, ValueError):
                self.logger.error(
                    "Cannot parse table-type attribute -> '%s'; value field -> '%s'",
                    category_item.get("name", ""),
                    value_field,
                )
                continue

            if not isinstance(value_table, list):
                self.logger.error(
                    "Table-type value -> '%s' requires a list of dictionaries!",
                    value_field,
                )
                continue

            """
            Get the mapping of the loader generated columns in the Data Frame to the
            attribute names in the target OTCS category. If no mapping
            is in the payload, then it is assumed that the category
            attribute names are identical to the column names in the Data Frame

            Example mapping:

            attribute_mapping = {
              "Application": "u_product_model",
              "Version": "u_version_name"
            }
            """

            attribute_mapping = category_item.get("attribute_mapping", None)

            row_index = 1
            for value_row in value_table:
                if not isinstance(value_row, dict):
                    self.logger.error(
                        "Illegal data type for table row -> %s. Expected 'dict', got -> '%s' with value -> %s",
                        str(row_index),
                        type(value_row),
                        value_row,
                    )
                    continue
                for key, value in value_row.items():
                    attribute = {}
                    attribute["name"] = category_item.get("name", "")
                    attribute["set"] = category_item.get("set", "")
                    attribute["row"] = row_index
                    # check if we have a mapping for this attribute in the payload:
                    if attribute_mapping and key in attribute_mapping:
                        attribute["attribute"] = attribute_mapping[key]
                    else:
                        attribute["attribute"] = key
                    # For tables values can be None if the number of
                    # list items in the source columns are not equal.
                    # To avoid the warning below we set the value to empty string
                    # if it is None:
                    attribute["value"] = value if value is not None else ""
                    worker_categories_expanded.append(attribute)
                row_index += 1
        # end if "value_type" in category_item and category_item["value_type"] == "table"
        else:
            worker_categories_expanded.append(category_item)
    # end for category_item in worker_categories

    # this loop generates a "value" for each
    # "value_field". "value_field" may also contain lists
    # that are either delimited by [...] or by a "value_type" with value "list"
    for category_item in worker_categories_expanded:
        if "attribute" not in category_item:
            self.logger.error(
                "Category item -> %s is missing the attribute field!",
                category_item,
            )
            continue

        # per default the value is in the "value" item:
        value = category_item.get("value", None)

        # is there a value replacement for the current attribute?
        if "value_field" in category_item:
            value_field = category_item["value_field"]
            # If no row is specified we set the set index to None and not 0 to handle
            # cases where we have multi-value attributes that take a list as parameter
            set_index = int(category_item["row"]) - 1 if "row" in category_item else None

            # this method always returns a string even if the value is
            # actually a list.
            value = self.replace_bulk_placeholders(
                input_string=value_field,
                row=row,
                index=set_index,
                replacements=replacements,
            )
            # If value is empty (probably because placeholders could not be resolved)
            # and if the payload provides an alternative value field we try this:
            if not value and "value_field_alt" in category_item:
                value = self.replace_bulk_placeholders(
                    input_string=category_item["value_field_alt"],
                    row=row,
                    index=set_index,
                    replacements=replacements,
                )
        else:
            value_field = None

        # if we don't have a value now, then there's an issue:
        if value is None:
            value = ""
            self.logger.warning(
                "Category item needs either a value or value_field! Skipping attribute -> '%s'",
                category_item["attribute"],
            )

        # We have an empty string value (this is different from None!)
        if value == "":
            category_item["value"] = value
            # We can continue as any further processing (below) does not make sense for an empty string value:
            continue

        # This variable should only be true if we don't have
        # a native python string but a delimiter separated
        # value list in a string, e.g. "a, b, c" or "a | b | c" or "x;y;z":
        is_list_in_string = False

        # The data source loader may have written a real python list into the value
        # In this case the string value includes square brackets [...]
        # We only do  this if the actual type of the value is string and the
        # proposed value (value_type) is not string:
        if (
            isinstance(value, str)  # it is a string
            and value.startswith("[")  # it starts with a list indicator character
            and value.endswith("]")  # it ends with a list indicator character
            and category_item.get("value_type", "")
            != "string"  # it is NOT explicitly stated it should remain a string
        ):
            # Remove the square brackets and declare it is a list!
            try:
                value = literal_eval(value)
            except (SyntaxError, ValueError) as e:
                self.logger.warning(
                    "Cannot directly parse list-type attribute; value field -> %s; error -> %s. Try string processing...",
                    value_field,
                    str(e),
                )
                self.logger.warning(
                    "Value string -> %s has [...] - remove brackets and interpret as delimiter separated values in a string...",
                    value,
                )
                # In this failure case we try to remove the square brackets and hope the inner part
                # can be treated as a string of values delimited with a delimiter (e.g. comma or semicolon)
                value = value.strip("[]")
                is_list_in_string = True

        # Handle this special case where we get a string that actually represents a date time format and convert it.
        # We only do this if the attribute has a "value_type" of "datetime" in the payload:
        if category_item.get("value_type", "string") == "datetime":
            old_value = value
            try:
                date_obj = parse(value)
                value = datetime.strftime(date_obj, "%Y-%m-%dT%H:%M:%SZ")
            except (OverflowError, ValueError):
                self.logger.error(
                    "Cannot convert value -> '%s' into a date for attribute -> '%s'. Value will be empty.",
                    old_value,
                    category_item["attribute"],
                )
                value = ""
            else:
                self.logger.debug(
                    "Attribute -> %s is declared in payload to be a datetime (convert format). Converted it from -> %s to -> %s",
                    category_item.get("attribute"),
                    old_value,
                    value,
                )

        # Handle special case where we get a string that actually represents a list but is
        # not yet a Python list but a string. This requires that value_type == "list". The list splitter
        # can be specified via "list_splitter" in the payload:
        # Now we try to convert the string to a Python list:
        if (category_item.get("value_type", "string") == "list" or is_list_in_string) and isinstance(value, str):
            # we split the string after splitter characters:
            list_splitter = category_item.get("list_splitter", ";,")
            self.logger.info(
                "Value -> '%s' is declared in payload to be a list but it is provided as a string. Splitting it after these characters -> '%s'.",
                value,
                list_splitter,
            )

            # Escape the split characters to ensure they are treated literally in the regex pattern
            escaped_splitter = re.escape(list_splitter)

            # Construct the regex pattern dynamically
            pattern = rf"[{escaped_splitter}]\s*"

            # Split the value string at the defined splitter characters:
            elements = re.split(pattern, value)

            # Remove the quotes around each element
            elements = [element.strip("'") for element in elements]
            value = elements
            self.logger.info(
                "Found list for a multi-value category attribute -> '%s' from field -> '%s' in data row -> %s. Value -> '%s'.",
                category_item["attribute"],
                value_field,
                index,
                str(value),
            )

        # now we check if there's a data lookup configured in the payload:
        lookup_data_source = category_item.get("lookup_data_source", None)
        # Do we want to drop / clear values that fail to lookup?
        drop_value = category_item.get("lookup_data_failure_drop", False)

        if lookup_data_source:
            self.logger.info(
                "Found lookup data source -> '%s' for attribute -> '%s'. Processing...",
                lookup_data_source,
                category_item["attribute"],
            )
            if not isinstance(value, list):
                # value is a single value and not a list:
                (_, synonym) = self.process_bulk_workspaces_synonym_lookup(
                    data_source_name=lookup_data_source,
                    workspace_name_synonym=value,
                )
                if synonym:
                    self.logger.info(
                        "Found synonym -> '%s' for attribute -> '%s' and value -> '%s' in data source -> '%s'.",
                        synonym,
                        category_item["attribute"],
                        value,
                        lookup_data_source,
                    )
                    value = synonym
                elif drop_value:
                    self.logger.warning(
                        "Cannot lookup the value for attribute -> '%s' and value -> '%s' in data source -> '%s'. Clear existing value.",
                        category_item["attribute"],
                        value,
                        lookup_data_source,
                    )
                    # Clear the value:
                    value = ""
                else:
                    self.logger.warning(
                        "Cannot lookup the value for attribute -> '%s' and value -> '%s' in data source -> '%s'. Keep existing value.",
                        category_item["attribute"],
                        value,
                        lookup_data_source,
                    )
            # end if not isinstance(value, list)
            else:
                # value is a list - so we apply the lookup to each item:
                # Iterate backwards to avoid index issues while popping items:
                for i in range(len(value) - 1, -1, -1):
                    # Make sure the value does not have leading or trailing spaces:
                    value[i] = value[i].strip()
                    (_, synonym) = self.process_bulk_workspaces_synonym_lookup(
                        data_source_name=lookup_data_source,
                        workspace_name_synonym=value[i],
                        workspace_type=None,  # we don't need the workspace ID, just the workspace name
                    )
                    if synonym:
                        self.logger.info(
                            "Found synonym -> '%s' for attribute -> '%s' and value -> '%s' in data source -> '%s'.",
                            synonym,
                            category_item["attribute"],
                            value[i],
                            lookup_data_source,
                        )
                        value[i] = synonym
                    elif drop_value:
                        self.logger.warning(
                            "Cannot lookup the value -> '%s' for attribute -> '%s' in data source -> '%s'! Drop existing value from list.",
                            value[i],
                            category_item["attribute"],
                            lookup_data_source,
                        )
                        # Remove the list item we couldn't lookup as drop_value is True:
                        value.pop(i)
                    else:
                        self.logger.warning(
                            "Cannot lookup the value -> '%s' for attribute -> '%s' in data source -> '%s'! Keep existing value.",
                            value[i],
                            category_item["attribute"],
                            lookup_data_source,
                        )
        # end if lookup_data_source

        # If value is a list then we have a multi-value attribute.
        # We now want to make sure that we don't have duplicates in
        # the value list:
        if isinstance(value, list) and len(value) != len(set(value)):
            self.logger.info(
                "The multi-value attribute -> '%s' has duplicates in its values -> %s.",
                category_item["attribute"],
                value,
            )
            # Remove duplicates from the list while preserving order.
            # Uses `dict.fromkeys()` since dictionaries maintain insertion order (Python 3.7+).
            # This ensures that only the first occurrence of each element is kept.
            value = list(dict.fromkeys(value))
            self.logger.info("The attribute values got de-duplicated to -> %s.", value)

        # now we check if there's a values mapping configured in the payload.
        # This is a dictionary for keys being the original values and
        # values being the mapped values. This makes most sense for
        # values with a limited / fixed domain of values. Example
        # `value_mapping = { "TS": "Tropical Storm", "HU": "Hurricane"}`
        value_mapping = category_item.get("value_mapping", None)
        if value_mapping:
            if not isinstance(value, list):
                # value is a single value and not a list:
                if value in value_mapping:
                    self.logger.info(
                        "Found value mapping for attribute -> '%s' from value -> '%s' to value -> '%s'.",
                        category_item["attribute"],
                        value,
                        value_mapping[value],
                    )
                    value = value_mapping[value]
            else:
                # value is a list - so we apply the lookup to each item:
                # Iterate backwards to avoid index issues while popping items:
                for i in range(len(value) - 1, -1, -1):
                    # Make sure the value does not have leading or trailing spaces:
                    value[i] = value[i].strip()

                    if value[i] in value_mapping:
                        self.logger.info(
                            "Found value mapping for attribute -> '%s' from value -> '%s' to value -> '%s'",
                            category_item["attribute"],
                            value[i],
                            value_mapping[value[i]],
                        )
                        value[i] = value_mapping[value[i]]

        # If value is a list then we have a multi-value attribute.
        # If "sort_multi_values" is specified in payload and evaluate to True
        # we sort the list alphabetically (note that numbers are not necessarily sorted numerically):
        if isinstance(value, list) and len(value) > 1 and category_item.get("sort_multi_values", False):
            value.sort(key=str)
            self.logger.info(
                "Sorting the values of multi-value attribute -> '%s' to -> %s.",
                category_item["attribute"],
                value,
            )

        if value_field:
            self.logger.debug(
                "Reading category -> '%s', attribute -> '%s' from field -> '%s' in data row -> %s. Value -> '%s'.",
                category_item["name"],
                category_item["attribute"],
                value_field,
                index,
                str(value),
            )
        else:
            self.logger.debug(
                "Setting category -> '%s', attribute -> '%s' to value -> '%s'.",
                category_item["name"],
                category_item["attribute"],
                str(value),
            )
        category_item["value"] = value
    # end for category_item in worker_categories_expanded

    # Cleanup categories_payload to remove empty rows of sets:
    rows_to_remove = {}
    for attribute in worker_categories_expanded:
        if attribute.get("set") is not None and attribute.get("row") is not None:
            set_name = attribute["set"]
            row_number = attribute["row"]
            value = attribute["value"]

            # If value is empty, mark it for removal:
            if not value or value == [""]:  # Treat empty strings or None as empty
                # The following if statement is important to not mark a set row
                # as True = removable that has been set to False in the else case below!
                if (set_name, row_number) not in rows_to_remove:
                    rows_to_remove[(set_name, row_number)] = True
            else:
                # If any value in the row is not empty, mark the row as not removable.
                # This may change the marking that have been applied in the if case
                # above!
                rows_to_remove[(set_name, row_number)] = False

    # Keep only the rows marked as True (empty rows to be removed)
    rows_to_remove = {k: v for k, v in rows_to_remove.items() if v is True}

    if rows_to_remove:
        self.logger.debug("Empty rows to remove from sets -> %s.", rows_to_remove)
    else:
        self.logger.debug("No empty rows to remove from sets.")

    cleaned_categories = [
        item
        for item in worker_categories_expanded
        if "set" not in item or "row" not in item or (item["set"], item["row"]) not in rows_to_remove
    ]

    return cleaned_categories

process_bulk_classification_assignments(row, index, classifications, replacements)

Replace the value placeholders of the bulk classification structures.

The payload placeholders are replaced with values from the Pandas Series (row).

Parameters:

Name	Type	Description	Default
row	Series	The current row in the Pandas data frame.	required
index	str	The index of the Pandas data frame. Just used here for logging.	required
classifications	list	List of payload classifications.	required
replacements	list	List of replacements.	required

Returns:

Name	Type	Description
list	list	List of classification IDs.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_classification_assignments")
def process_bulk_classification_assignments(
    self,
    row: pd.Series,
    index: str,
    classifications: list,
    replacements: list,
) -> list:
    """Replace the value placeholders of the bulk classification structures.

    The payload placeholders are replaced with values from the Pandas Series (row).

    Args:
        row (pd.Series):
            The current row in the Pandas data frame.
        index (str):
            The index of the Pandas data frame. Just used here for logging.
        classifications (list):
            List of payload classifications.
        replacements (list):
            List of replacements.

    Returns:
        list:
            List of classification IDs.

    """

    result_list = []

    for classification in classifications:
        # Do we have a classification path?
        if isinstance(classification, list):
            # Replace placeholders in the list. As list is a mutable data type,
            # the replacement happens in-place:
            self.replace_bulk_placeholders_list(input_list=classification, row=row, replacements=replacements)
            class_node = self._otcs_frontend.get_node_by_volume_and_path(
                volume_type=self._otcs.VOLUME_TYPE_CLASSIFICATION_VOLUME,
                path=classification,
            )
        elif isinstance(classification, string):
            nickname = self.replace_bulk_placeholders(
                input_string=classification,
                row=row,
                replacements=replacements,
            )
            class_node = self._otcs_frontend.get_node_from_nickname(nickname=nickname)

        class_node_id = self._otcs.get_result_value(
            response=class_node,
            key="id",
        )
        if class_node_id:
            result_list.append(class_node_id)

    if result_list:
        self.logger.debug(
            "Found classifications for data row -> %s. Value -> %s",
            index,
            str(result_list),
        )

    return result_list

process_bulk_classifications(section_name='bulkClassifications')

Process bulk classifications in payload and bulk create them in Content Server (multi-threaded).

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'bulkClassifications'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_classifications")
def process_bulk_classifications(self, section_name: str = "bulkClassifications") -> bool:
    """Process bulk classifications in payload and bulk create them in Content Server (multi-threaded).

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise.

    """

    if not self._bulk_classifications:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    # For efficient idem-potent operation we may want to see which
    # bulk classifications have already been processed before:
    if self.check_status_file(
        payload_section_name=section_name,
        payload_specific=True,
        prefix="failure_",
    ):
        # Read payload from where we left it last time
        self._bulk_classifications = self.get_status_file(
            payload_section_name=section_name,
            prefix="failure_",
        )
        if not self._bulk_classifications:
            self.logger.error(
                "Cannot load existing bulkClassifications failure file. Bailing out!",
            )
            return False

    success: bool = True

    for bulk_classification in self._bulk_classifications:
        # Check if element has been disabled in payload (enabled = false).
        # In this case we skip the element:
        enabled = bulk_classification.get("enabled", True)
        if not enabled:
            self.logger.info("Payload for bulk classification is disabled. Skipping...")
            continue

        # The payload element must have a "data_source" key:
        data_source_name = bulk_classification.get("data_source", None)
        if not data_source_name:
            self.logger.error("Missing data source name in bulk classification!")
            success = False
            continue

        self._log_header_callback(
            text="Process bulk classifications using data source -> '{}'".format(
                data_source_name,
            ),
            char="-",
        )

        copy_data_source = bulk_classification.get("copy_data_source", False)
        force_reload = bulk_classification.get("force_reload", True)

        # Load and prepare the data source for the bulk processing:
        if copy_data_source:
            self.logger.info(
                "Take a copy of data source -> '%s' to avoid side-effects for repeative usage of the data source...",
                data_source_name,
            )
            data = Data(
                self.process_bulk_datasource(
                    data_source_name=data_source_name,
                    force_reload=force_reload,
                ),
                logger=self.logger,
            )
        else:
            self.logger.info(
                "Use original data source -> '%s' and not do a copy.",
                bulk_classification["data_source"],
            )
            data = self.process_bulk_datasource(
                data_source_name=data_source_name,
                force_reload=force_reload,
            )
        if data is None:  # important to use "is None" here!
            self.logger.error("Failed to load data source -> '%s' for bulk classification!", data_source_name)
            success = False
            continue
        if data.get_data_frame() is None:  # important to use "is None" here!
            self.logger.error("Data source -> '%s' for bulk classification is empty!", data_source_name)
            continue

        # Check if fields with list substructures should be exploded.
        # We may want to do this outside the bulkDatasource to only
        # explode for bulkClassifications and not other bulk elements
        # like bulkDocuments or bulkWorkspaces:
        explosions = bulk_classification.get("explosions", [])
        for explosion in explosions:
            # explode field can be a string or a list
            # exploding multiple fields at once avoids
            # combinatorial explosions - this is VERY
            # different from exploding columns one after the other!
            if "explode_field" not in explosion and "explode_fields" not in explosion:
                self.logger.error("Missing explosion field(s)!")
                continue
            # we want to be backwards compatible...
            explode_fields = (
                explosion["explode_field"] if "explode_field" in explosion else explosion["explode_fields"]
            )
            flatten_fields = explosion.get("flatten_fields", [])
            split_string_to_list = explosion.get("split_string_to_list", False)
            list_splitter = explosion.get(
                "list_splitter",
                ",",
            )  # don't have None as default!
            self.logger.info(
                "Starting explosion of bulk classifications by field(s) -> %s (type -> %s). Size of data frame before explosion -> %s.",
                str(explode_fields),
                type(explode_fields),
                str(len(data)),
            )
            data.explode_and_flatten(
                explode_fields=explode_fields,
                flatten_fields=flatten_fields,
                make_unique=False,
                split_string_to_list=split_string_to_list,
                separator=list_splitter,
                reset_index=True,
            )
            self.logger.info(
                "Size of data frame after explosion -> %s.",
                str(len(data)),
            )

        # Keep only selected rows if filters are specified in bulkClassifications.
        # We have this _after_ "explosions" to allow access to subfields as well.
        # We have this _before_ "sorting" and "deduplication" as we may keep the wrong
        # rows otherwise (unique / deduplication always keeps the first matching row).
        # We may want to do this outside the bulkDatasource to only
        # filter for bulkClassifications and not for bulkWorkspaces or
        # bulkWorkspaceRelationships:
        filters = bulk_classification.get("filters", [])
        if filters:
            data.filter(conditions=filters)

        # Sort the data frame if "sort" specified in payload. We may want to do this to have a
        # higher chance that rows with classifications names that may collapse into
        # one name are put into the same partition. This can avoid race conditions
        # between different Python threads.
        sort_fields = bulk_classification.get("sort", [])
        if sort_fields:
            self.logger.info(
                "Start sorting of data frame for bulk classifications based on fields (columns) -> %s...",
                str(sort_fields),
            )
            data.sort(sort_fields=sort_fields, inplace=True)
            self.logger.info(
                "Sorting of data frame for bulk classifications based on fields (columns) -> %s completed.",
                str(sort_fields),
            )

        # Check if duplicate rows for given fields should be removed. It is
        # important to do this after sorting as Pandas always keep the first occurance,
        # so ordering plays an important role in deduplication!
        unique_fields = bulk_classification.get("unique", [])
        if unique_fields:
            self.logger.info(
                "Starting deduplication of data frame for bulk classifications with unique fields -> %s. Size of data frame before deduplication -> %s.",
                str(unique_fields),
                str(len(data)),
            )
            data.deduplicate(unique_fields=unique_fields, inplace=True)
            self.logger.info(
                "Size of data frame after deduplication -> %s.",
                str(len(data)),
            )

        # Read the optional categories payload:
        categories = bulk_classification.get("categories", None)
        if not categories:
            self.logger.info(
                "Bulk classification payload has no category data! Will leave category attributes empty...",
            )

        # Get the operations to carry out during bulk processing.
        # Just doing a create is the default. Other options are
        # "update", "delete", "recreate":
        operations = bulk_classification.get("operations", ["create"])

        self.logger.info(
            "Bulk create classifications. Operations -> %s.",
            str(operations),
        )

        # See if bulkClassification definition has a specific thread number
        # otherwise it is read from a global environment variable:
        bulk_thread_number = int(
            bulk_classification.get("thread_number", BULK_THREAD_NUMBER),
        )

        partitions = data.partitionate(bulk_thread_number)

        # Create a list to hold the threads
        threads = []
        results = []

        # Create and start a thread for each partition
        for index, partition in enumerate(partitions, start=1):
            thread = threading.Thread(
                name=f"{section_name}_{index:02}",
                target=self.thread_wrapper,
                args=(
                    self.process_bulk_classifications_worker,
                    bulk_classification,
                    partition,
                    categories,
                    operations,
                    results,
                ),
            )
            # start a thread executing the process_bulk_classifications_worker() method below:
            self.logger.info("Starting thread -> '%s'...", str(thread.name))
            thread.start()
            threads.append(thread)
            # Avoid that all threads start at the exact same time with
            # potentially expired cookies that cases race conditions:
            time.sleep(1)
        # end for index, partition in enumerate(partitions, start=1)

        # Wait for all threads to complete
        for thread in threads:
            self.logger.info(
                "Waiting for thread -> '%s' to complete...",
                str(thread.name),
            )
            thread.join()
            self.logger.info("Thread -> '%s' has completed.", str(thread.name))

        if "classifications" not in bulk_classification:
            bulk_classification["classifications"] = {}

        # Print statistics for each thread. In addition,
        # check if all threads have completed without error / failure.
        # If there's a single failure in on of the thread results we
        # set 'success' variable to False.
        results.sort(key=lambda x: x["thread_name"])
        for result in results:
            self.logger.info(
                "Thread -> '%s' completed with overall status '%s'.",
                str(result["thread_name"]),
                "successful" if result["success"] else "failed",
            )
            self.logger.info(
                "Thread -> '%s' processed %s data rows with %s successful, %s failed, and %s skipped.",
                str(result["thread_name"]),
                str(
                    result["success_counter"] + result["failure_counter"] + result["skipped_counter"],
                ),
                str(result["success_counter"]),
                str(result["failure_counter"]),
                str(result["skipped_counter"]),
            )
            self.logger.info(
                "Thread -> '%s' created %s classifications, updated %s classifications, and deleted %s classifications.",
                str(result["thread_name"]),
                str(result["create_counter"]),
                str(result["update_counter"]),
                str(result["delete_counter"]),
            )

            if not result["success"]:
                success = False
            # Record all generated classifications. This should allow us
            # to restart in case of failures and avoid trying to
            # create classifications that have been created before.
            bulk_classification["classifications"].update(result["classifications"])
        self._log_header_callback(
            text="Completed processing of bulk classifications using data source -> '{}'".format(data_source_name),
            char="-",
        )

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._bulk_classifications,
    )

    return success

process_bulk_classifications_worker(bulk_classification, partition, categories=None, operations=None, results=None)

Thread worker to create classificatins in bulk.

Each worker thread gets a partition of the rows that include the data required for the classification creation.

Parameters:

Name	Type	Description	Default
bulk_classification	dict	The payload of the bulkClassification.	required
partition	DataFrame	Data partition with rows to process.	required
categories	list	List of category dictionaries.	None
operations	list	Defines which operations should be applyed on classifications. Possible values are "create", "update", "delete", "recreate".	None
results	list	A mutable list of thread results.	None

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_classifications_worker")
def process_bulk_classifications_worker(
    self,
    bulk_classification: dict,
    partition: pd.DataFrame,
    categories: list | None = None,
    operations: list | None = None,
    results: list | None = None,
) -> None:
    """Thread worker to create classificatins in bulk.

    Each worker thread gets a partition of the rows that include
    the data required for the classification creation.

    Args:
        bulk_classification (dict):
            The payload of the bulkClassification.
        partition (pd.DataFrame):
            Data partition with rows to process.
        categories (list):
            List of category dictionaries.
        operations (list):
            Defines which operations should be applyed on classifications.
            Possible values are "create", "update", "delete", "recreate".
        results (list):
            A mutable list of thread results.

    """

    thread_id = threading.get_ident()
    thread_name = threading.current_thread().name

    self.logger.info(
        "Start working on data frame partition of size -> %s to bulk create classifications...",
        str(len(partition)),
    )

    # Avoid linter warnings - so make parameter default None while we
    # actually want ["create"] to be the default:
    if operations is None:
        operations = ["create"]

    result = {}
    result["thread_id"] = thread_id
    result["thread_name"] = thread_name
    result["success_counter"] = 0
    result["failure_counter"] = 0
    result["skipped_counter"] = 0
    result["create_counter"] = 0
    result["update_counter"] = 0
    result["delete_counter"] = 0
    result["classifications"] = {}
    result["success"] = True

    # Check if classifications have been processed before, e.i. testing
    # if a "classifications" key exists and if it is pointing to a non-empty list.
    # Additionally we check that classification updates are not enforced:
    if (
        bulk_classification.get("classifications")
        and "update" not in operations
        and "delete" not in operations
        and "recreate" not in operations
    ):
        existing_classifications = bulk_classification["classifications"]
        self.logger.info(
            "Found %s already processed classifications. Try to complete the job...",
            str(len(existing_classifications)),
        )
    else:
        existing_classifications = {}

    # See if external creation and modification dates are in the payload data:
    external_modify_date_field = bulk_classification.get("external_modify_date")
    external_create_date_field = bulk_classification.get("external_create_date")

    # See if we have a key field to uniquely identify an existing document:
    key_field = bulk_classification.get("key")

    # Get dictionary of replacements for bulk classification creations
    # this we will be used of all places data is read from the
    # data frame. Each dictionary item has the field name as the
    # dictionary key and a list of regular expressions as dictionary value
    replacements = bulk_classification.get("replacements")

    # In case the name cannot be resolved we allow to
    # specify an alternative name field in the payload.
    name_field = bulk_classification.get("name")

    # In case the name cannot be resolved we allow to
    # specify an alternative name field in the payload.
    name_field_alt = bulk_classification.get("name_alt")

    # In case the description cannot be resolved we allow to
    # specify an alternative description field in the payload.
    description_field = bulk_classification.get("description")

    # In case the description cannot be resolved we allow to
    # specify an alternative description field in the payload.
    description_field_alt = bulk_classification.get("description_alt")

    # Fetch the nickname field from the payload (if it is specified):
    nickname_field = bulk_classification.get("nickname")

    # In case the nickname cannot be resolved we allow to
    # specify an alternative nickname field in the payload.
    nickname_field_alt = bulk_classification.get("nickname_alt")

    # Nicknames are very limited in terms of allowed characters.
    # For nicknames we need additional regexp as we need to
    # replace all non-alphanumeric, non-space characters with ""
    # We also preserve hyphens in the first step to replace
    # them below with underscores. This is important to avoid
    # that different spellings of names produce different nicknames.
    # We want spellings with spaces match spellings with hyphens.
    # For this the classification names have a regexp "-| " in the payload.
    nickname_additional_regex_list = [r"[^\w\s-]"]

    total_rows = len(partition)

    # Process all rows in the partition that was given to this thread:
    for index, row in partition.iterrows():
        # Calculate percentage of completion:
        percent_complete = ((partition.index.get_loc(index) + 1) / total_rows) * 100

        self.logger.info(
            "Processing data row -> %s for bulk classification creation (%.2f %% complete)...",
            str(index),
            percent_complete,
        )

        # Clear variables to esure clean state for each row:
        classification_id = None

        # Determine the classification name:
        classification_name = self.replace_bulk_placeholders(
            input_string=name_field,
            row=row,
            replacements=replacements,
        )
        # If we cannot get the classification name from the
        # name_field we try the alternative name field
        # (if specified in payload via "name_alt"):
        if not classification_name and name_field_alt:
            self.logger.debug(
                "Row -> %s does not have the data to resolve the placeholders in classification name -> %s! Trying alternative name field -> %s...",
                str(index),
                name_field,
                name_field_alt,
            )
            classification_name = self.replace_bulk_placeholders(
                input_string=name_field_alt,
                row=row,
                replacements=replacements,
            )

        if not classification_name:
            self.logger.warning(
                "Row -> %s does not have the required data to resolve -> %s for the classification name!%s",
                str(index),
                name_field,
                " There's no alternative field name given in the payload (via 'name_alt')."
                if not name_field_alt
                else " The alternative field -> '{}' could not be resolved either!".format(
                    name_field_alt,
                ),
            )
            result["skipped_counter"] += 1
            continue

        # Cleanse the classification name (allowed characters, maximum length):
        classification_name = OTCS.cleanse_item_name(classification_name)

        # Check if classification has been created before (either in this run
        # or in a former run of the customizer):
        if (
            classification_name in existing_classifications  # processed in former run?
            or classification_name in result["classifications"]  # processed in current run?
        ):
            self.logger.info(
                "Classification -> '%s' does already exist. Skipping...",
                classification_name,
            )
            result["skipped_counter"] += 1
            continue

        # Determine the description field:
        if description_field:
            description = self.replace_bulk_placeholders(
                input_string=description_field,
                row=row,
            )
            # If we cannot get the description from the
            # description_field we try the alternative description field
            # (if specified in payload via "description_alt"):
            if not description and description_field_alt:
                self.logger.debug(
                    "Row -> %s does not have the data to resolve the placeholders in classification description -> %s! Trying alternative description field -> %s...",
                    str(index),
                    name_field,
                    description_field_alt,
                )
                description = self.replace_bulk_placeholders(
                    input_string=description_field_alt,
                    row=row,
                )
        else:
            description = ""

        # Create a copy of the mutable operations list as we may
        # want to modify it per row:
        row_operations = list(operations)

        # Check if all data conditions to create the classification are met
        conditions = bulk_classification.get("conditions")
        if conditions:
            evaluated_condition = self.evaluate_conditions(
                conditions=conditions,
                row=row,
                replacements=replacements,
            )
            if not evaluated_condition:
                self.logger.info(
                    "Condition for bulk classification row -> %s not met. Skipping row for classification creation...",
                    str(index),
                )
                result["skipped_counter"] += 1
                continue

        # Check if all data conditions to create or recreate the classification are met:
        if "create" in row_operations or "recreate" in row_operations:
            conditions_create = bulk_classification.get("conditions_create")
            if conditions_create:
                evaluated_conditions_create = self.evaluate_conditions(
                    conditions=conditions_create,
                    row=row,
                    replacements=replacements,
                )
                if not evaluated_conditions_create:
                    self.logger.info(
                        "Create condition for bulk classification row -> %s not met. Excluding create operation...",
                        str(index),
                    )
                    if "create" in row_operations:
                        row_operations.remove("create")
                    if "recreate" in row_operations:
                        row_operations.remove("recreate")
            elif "recreate" in row_operations:
                # we still create and recreate without conditions_create.
                # But give a warning for 'recreate' without condition.
                self.logger.warning(
                    "No create condition provided but 'recreate' operation requested. Recreating all existing classifications!",
                )

        # Check if all data conditions to delete the classification are met:
        if "delete" in row_operations:
            conditions_delete = bulk_classification.get("conditions_delete")
            if conditions_delete:
                evaluated_conditions_delete = self.evaluate_conditions(
                    conditions=conditions_delete,
                    row=row,
                    replacements=replacements,
                )
                if not evaluated_conditions_delete:
                    self.logger.info(
                        "Delete condition for bulk classification row -> %s not met. Excluding delete operation...",
                        str(index),
                    )
                    row_operations.remove("delete")
            else:  # without delete_conditions we don't delete!!
                self.logger.warning(
                    "Delete operation requested for bulk classifications but conditions for deletion are missing! (specify 'conditions_delete')!",
                )
                row_operations.remove("delete")

        # Check if all data conditions to delete the classification are met:
        if "update" in row_operations:
            conditions_update = bulk_classification.get("conditions_update")
            if conditions_update:
                evaluated_conditions_update = self.evaluate_conditions(
                    conditions=conditions_update,
                    row=row,
                    replacements=replacements,
                )
                if not evaluated_conditions_update:
                    self.logger.info(
                        "Update condition for bulk classification row -> %s not met. Excluding update operation...",
                        str(index),
                    )
                    row_operations.remove("update")

        # Determine the external modification date (if any):
        if external_modify_date_field:
            external_modify_date = self.replace_bulk_placeholders(
                input_string=external_modify_date_field,
                row=row,
            )
        else:
            external_modify_date = None

        # Determine the external creation date (if any):
        if external_create_date_field:
            external_create_date = self.replace_bulk_placeholders(
                input_string=external_create_date_field,
                row=row,
            )
        else:
            external_create_date = None

        # Determine the key field (if any):
        key = self.replace_bulk_placeholders(input_string=key_field, row=row) if key_field else None

        # check if classification with this nickname does already exist.
        # we also store the nickname to assign it to the new classification:
        if nickname_field:
            nickname = self.replace_bulk_placeholders(
                input_string=nickname_field,
                row=row,
                replacements=replacements,
                additional_regex_list=nickname_additional_regex_list,
            )
            # If we cannot get the nickname from the
            # nickname_field we try the alternative nickname field
            # (if specified in payload via "nickname_alt"):
            if not nickname and nickname_field_alt:
                nickname = self.replace_bulk_placeholders(
                    input_string=nickname_field_alt,
                    row=row,
                    replacements=replacements,
                    additional_regex_list=nickname_additional_regex_list,
                )

            # Nicknames for sure should not have leading or trailing spaces:
            nickname = nickname.strip()
            # Nicknames for sure are not allowed to include spaces:
            nickname = nickname.replace(" ", "_")
            # We also want to replace hyphens with underscores
            # to make sure that classification name spellings with
            # spaces and with hyphens are mapped to the same
            # classification nicknames (aligned with the classification names
            # that have a regexp rule for this in the payload):
            nickname = nickname.replace("-", "_")
            nickname = nickname.replace("___", "_")
            nickname = nickname.lower()
            response = self._otcs_frontend.get_node_from_nickname(nickname=nickname)
            if response:
                found_classification_name = self._otcs_frontend.get_result_value(
                    response=response,
                    key="name",
                )
                found_classification_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
                if found_classification_name != classification_name:
                    self.logger.warning(
                        "Clash of nicknames for classification -> '%s' and classification -> '%s' (%s)!",
                        classification_name,
                        found_classification_name,
                        found_classification_id,
                    )
                # Only skip, if classification 'update' or 'delete' is NOT requested:
                elif "update" not in row_operations and "delete" not in row_operations:
                    self.logger.info(
                        "Classification -> '%s' with nickname -> '%s' does already exist (found -> '%s'). No update or delete operations requested or allowed. Skipping...",
                        classification_name,
                        nickname,
                        found_classification_name,
                    )
                    result["skipped_counter"] += 1
                    continue
        # end if nickname_field
        else:
            nickname = None

        # Based on the evaluation of conditions_create, conditions_delete,
        # and conditions_update we could end up with an empty operations list.
        # In such cases we skip the further processing of this row:
        if not row_operations:
            self.logger.info(
                "Skip classification -> '%s' as row operations is empty (no create, update, delete or recreate). This may be because conditions_create, conditions_delete, and conditions_update have been evaluated to false for this row.",
                classification_name,
            )
            result["skipped_counter"] += 1
            continue

        self.logger.info(
            "Bulk process classification -> '%s' using effective operations -> %s...",
            classification_name,
            str(row_operations),
        )

        self.logger.info(
            "Check if classification -> '%s' does already exist...",
            classification_name,
        )

        # Initialize variables - this is important!
        classification_old_name = None
        classification_id = None
        handle_name_clash = False
        classification_modify_date = None
        parent_id = None

        # We create a copy list to not modify original payload
        classification_path = list(bulk_classification.get("path", []))
        result_placeholders = self.replace_bulk_placeholders_list(
            input_list=classification_path,
            row=row,
            replacements=replacements,
        )
        if not result_placeholders:
            classification_path = None

        # We have 4 cases to differentiate:

        # 1. key given + key found = name irrelevant, item exist
        # 2. key given + key not found = if name exist it is a name clash
        # 3. no key given + name found = item exist
        # 4. no key given + name not found = item does not exist

        # We are NOT trying to compensate for a failed key lookup with a name lookup!
        # Names are only relevant if no key is in payload!

        if key:
            # if we have a key we need to also know which category attribute is
            # storing the value for the key:
            key_attribute = next(
                (cat_attr for cat_attr in categories if cat_attr.get("is_key", False) is True),
                None,
            )
            if not key_attribute:
                self.logger.error(
                    "Bulk Classification has a key -> '%s' defined but none of the category attributes is marked as key ('is_key' is missing)!",
                    key,
                )
                result["success"] = False
                result["failure_counter"] += 1
                continue
            cat_name = key_attribute.get("name", None)
            att_name = key_attribute.get("attribute", None)
            set_name = key_attribute.get("set", None)

            # Get the parent classification element:
            response = self._otcs_frontend.get_node_by_volume_and_path(
                volume_type=self._otcs_frontend.VOLUME_TYPE_CLASSIFICATION_VOLUME,
                path=classification_path,
                create_path=True,
            )
            parent_id = self._otcs_frontend.get_result_value(
                response=response,
                key="id",
            )
            if parent_id:
                # Try to find the node that has the given key attribute value:
                response = self._otcs_frontend.lookup_nodes(
                    parent_node_id=parent_id,
                    category=cat_name,
                    attribute=att_name,
                    attribute_set=set_name,
                    value=key,
                )
                classification_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
            else:
                # if we couldn't determine the parent ID this means there are
                # now classification instances for this classification type. Then we set
                # classification_id = None and let the code go into the else case below:
                classification_id = None

            if classification_id:
                # Case 1: key given + key found = name irrelevant, item exist
                classification_old_name = self._otcs_frontend.get_result_value(
                    response=response,
                    key="name",
                )
                self.logger.info(
                    "Found existing classification -> %s (%s) in classification with ID -> %s using key value -> '%s', category -> '%s', and attribute -> '%s'.",
                    classification_old_name,
                    classification_id,
                    parent_id,
                    key,
                    cat_name,
                    att_name,
                )
                if classification_old_name != classification_name:
                    self.logger.info(
                        "Existing classification has the old name -> '%s' which is different from the new name -> '%s.'",
                        classification_old_name,
                        classification_name,
                    )
                else:
                    classification_old_name = None
                # We get the modify date of the existing classification.
                classification_modify_date = self._otcs_frontend.get_result_value(
                    response=response,
                    key="modify_date",
                )
            else:
                # Case 2: key given + key not found = if name exist it is a name clash
                self.logger.info(
                    "Couldn't find existing classification with the key value -> '%s' in category -> '%s' and attribute -> '%s' in folder with ID -> %s.",
                    key,
                    cat_name,
                    att_name,
                    parent_id,
                )
                handle_name_clash = True
            # end if key_attribute
        # end if key
        else:
            # Get the parent classification element:
            response = self._otcs_frontend.get_node_by_volume_and_path(
                volume_type=self._otcs_frontend.VOLUME_TYPE_CLASSIFICATION_VOLUME,
                path=classification_path,
            )
            parent_id = self._otcs_frontend.get_result_value(
                response=response,
                key="id",
            )
            # If we haven't a key we try by parent + name
            response = self._otcs_frontend.get_node_by_parent_and_name(
                parent_id=parent_id,
                name=classification_name,
            )
            classification_id = self._otcs_frontend.get_result_value(
                response=response,
                key="id",
            )
            if classification_id:
                # Case 3: no key given + name found = item exist
                self.logger.info(
                    "Found existing classification -> '%s' (%s).",
                    classification_name,
                    classification_id,
                )
                # We get the modify date of the existing classification.
                classification_modify_date = self._otcs_frontend.get_result_value(
                    response=response,
                    key="modify_date",
                )
            else:
                # Case 4: no key given + name not found = item does not exist
                self.logger.info(
                    "No existing classification with name -> '%s' in path -> %s.",
                    classification_name,
                    classification_path,
                )

        # Check if we want to recreate an existing classification
        # then we handle the "delete" part of "recreate" first:
        if classification_id and "recreate" in row_operations:
            response = self._otcs_frontend.delete_node(
                node_id=classification_id,
                purge=True,
            )
            if not response:
                self.logger.error(
                    "Failed to recreate existing classification -> '%s' (%s)! Delete failed.",
                    (classification_name if classification_old_name is None else classification_old_name),
                    classification_id,
                )
                result["success"] = False
                result["failure_counter"] += 1
                continue
            result["delete_counter"] += 1
            self.logger.info(
                "Successfully deleted existing classification -> '%s' (%s) as part of the recreate operation.",
                (classification_name if classification_old_name is None else classification_old_name),
                classification_id,
            )
            classification_id = None

        # Check if classification does not exists - then we create a new classification
        # if this is requested ("create" or "recreate" value in operations list in payload)
        if not classification_id and ("create" in row_operations or "recreate" in row_operations):
            # for Case 2 (we looked up the classification by key but could have name collisions)
            # we need to see if we have a name collision:
            if handle_name_clash and parent_id:
                response = self._otcs_frontend.check_node_name(
                    parent_id=parent_id,
                    node_name=classification_name,
                )
                # result is a list of names that clash - if it is empty we have no clash
                if response and response["results"]:
                    # We add the suffix with the key which should be unique:
                    self.logger.warning(
                        "Classification -> '%s' does already exist in folder with ID -> %s and we need to handle the name clash by using name -> '%s'",
                        classification_name,
                        parent_id,
                        classification_name + " (" + key + ")",
                    )
                    classification_name = classification_name + " (" + key + ")"

            # If category data is in payload we substitute
            # the values with data from the current data row:
            if categories:
                # Make sure the threads are not changing data structures that
                # are shared between threads. categories is a list of dicts.
                # list and dicts are "mutable" data structures in Python!
                worker_categories = self.process_bulk_categories(
                    row=row,
                    index=index,
                    categories=categories,
                    replacements=replacements,
                )
                # classification_category_data = self.prepare_item_create_form(
                #     parent_id=parent_id,
                #     categories=worker_categories,
                #     subtype=self._otcs_frontend.ITEM_TYPE_CLASSIFICATION,
                # )
                classification_category_data = self.prepare_category_data(
                    categories_payload=worker_categories,
                    source_node_id=parent_id,
                )
                if not classification_category_data:
                    self.logger.error(
                        "Failed to prepare the category data for new classification -> '%s'!",
                        classification_name,
                    )
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue  # for index, row in partition.iterrows()
            else:
                classification_category_data = {}

            self.logger.info(
                "Bulk create classification -> '%s'...",
                classification_name,
            )

            # Create the classification with all provided information:
            response = self._otcs_frontend.create_item(
                parent_id=parent_id,
                item_type=self._otcs_frontend.ITEM_TYPE_CLASSIFICATION,
                item_name=classification_name,
                item_description=description,
                category_data=classification_category_data,
                external_create_date=external_create_date,
                external_modify_date=external_modify_date,
                show_error=False,
            )
            if response is None:
                # Potential race condition: see if the classification has been created by a concurrent thread.
                # So we better check if the classification is there even if the create_item() call delivered
                # a 'None' response:
                response = self._otcs_frontend.get_node_by_parent_and_name(
                    parent_id=parent_id,
                    name=classification_name,
                )
            classification_id = self._otcs_frontend.get_result_value(
                response=response,
                key="id",
            )
            if not classification_id:
                self.logger.error(
                    "Failed to bulk create classification -> '%s' under parent with ID -> %s!",
                    classification_name,
                    parent_id,
                )
                result["success"] = False
                result["failure_counter"] += 1
                continue

            # Set the categories separately:
            # if classification_category_data:
            #     response = self._otcs_frontend.update_item(
            #         node_id=classification_id,
            #         category_data=classification_category_data,
            #     )
            self.logger.info(
                "Created classification -> '%s' with ID -> %s!",
                classification_name,
                classification_id,
            )
            result["create_counter"] += 1

        # end if not classification_id and "create" or "recreate" in row_operations

        # If "updates" are an requested row operation we update the existing classification with
        # fresh metadata from the payload. Additionally we check the external
        # modify date to support incremental load for content that has really
        # changed.
        # In addition we check that "delete" is not requested as otherwise it will
        # never go in elif "delete" ... below (and it does not make sense to update a classification
        # that is deleted in the next step...)
        elif (
            classification_id
            and "update" in row_operations
            and "delete" not in row_operations  # note the NOT !
            and OTCS.date_is_newer(
                date_old=classification_modify_date,
                date_new=external_modify_date,
            )
        ):
            # get the specific update operations given in the payload
            # if not specified we do all 4 update operations (name, description, categories and nickname)
            update_operations = bulk_classification.get(
                "update_operations",
                ["name", "description", "categories", "nickname"],
            )

            # If category data is in payload we substitute
            # the values with data from the current data row.
            # This is only done if "categories" update is not
            # excluded from the update_operations:
            if categories and "categories" in update_operations:
                # Make sure the threads are not changing data structures that
                # are shared between threads. categories is a list of dicts.
                # list and dicts are "mutable" data structures in Python!
                worker_categories = self.process_bulk_categories(
                    row=row,
                    index=index,
                    categories=categories,
                    replacements=replacements,
                )
                # classification_category_data = self.prepare_item_create_form(
                #     parent_id=parent_id,
                #     categories=worker_categories,
                #     subtype=self._otcs_frontend.ITEM_TYPE_CLASSIFICATION,
                # )
                # Transform the payload structure into the format
                # the OTCS REST API requires:
                classification_category_data = self.prepare_category_data(
                    categories_payload=worker_categories,
                    source_node_id=classification_id,
                )
                if not classification_category_data:
                    self.logger.error(
                        "Failed to prepare the updated category data for classification -> '%s'!",
                        classification_name,
                    )
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue  # for index, row in partition.iterrows()
            # end if categories
            else:
                classification_category_data = {}

            self.logger.info(
                "Update existing classification -> '%s' (%s) with operations -> %s...",
                classification_old_name if classification_old_name else classification_name,
                str(classification_id),
                str(update_operations),
            )
            # Update the classification with all provided information:
            response = self._otcs_frontend.update_item(
                classification_id=classification_id,
                item_name=(classification_name if "name" in update_operations else None),
                item_description=(description if "description" in update_operations else None),
                category_data=(classification_category_data if "categories" in update_operations else None),
                external_create_date=external_create_date,
                external_modify_date=external_modify_date,
                show_error=True,
            )
            if not response:
                self.logger.error(
                    "Failed to update existing classification -> '%s' (%s)!",
                    (classification_old_name if classification_old_name else classification_name),
                    classification_id,
                )
                result["success"] = False
                result["failure_counter"] += 1
                continue
            self.logger.info(
                "Updated existing classification -> '%s' (%s).",
                classification_name
                if "name" in update_operations or not classification_old_name
                else classification_old_name,
                classification_id,
            )
            result["update_counter"] += 1

        # end elif "update" in row_operations...
        elif classification_id and "delete" in row_operations:
            # We delete with immediate purging to keep recycle bin clean
            # and to not run into issues with nicknames used in deleted items:
            response = self._otcs_frontend.delete_node(
                node_id=classification_id,
                purge=True,
            )
            if not response:
                self.logger.error(
                    "Failed to delete existing classification -> '%s' (%s)!",
                    (classification_old_name if classification_old_name else classification_name),
                    classification_id,
                )
                result["success"] = False
                result["failure_counter"] += 1
                continue
            self.logger.info(
                "Successfully deleted existing classification -> '%s' (%s).",
                classification_old_name if classification_old_name else classification_name,
                classification_id,
            )
            result["delete_counter"] += 1
            classification_id = None
        # end elif classification_id and "delete" in row_operations

        # this is the plain old "it does exist and we just skip it case":
        elif classification_id:
            result["skipped_counter"] += 1
            self.logger.info(
                "Skipped existing classification -> '%s' (%s).",
                classification_old_name if classification_old_name else classification_name,
                classification_id,
            )
        # this is the case where we just want to operate on existing classifications (update or delete)
        # but they do not exist:
        elif not classification_id and ("update" in row_operations or "delete" in row_operations):
            result["skipped_counter"] += 1
            self.logger.info(
                "Skipped update/delete of non-existing classification -> '%s'.",
                classification_old_name if classification_old_name else classification_name,
            )

        # The following code is executed for all operations
        # (create, recreate, update, delete):

        # Check if we want to set / update the nickname of the classification.
        # Precondition is we have determined a nickname, we have the classification ID
        # and the update of the nickname is either required (create, recreate)
        # or requested (update_operations include "nickname"):
        if (
            nickname
            and classification_id
            and (
                "create" in row_operations
                or "recreate" in row_operations
                or ("update" in row_operations and "nickname" in update_operations)
            )
        ):
            response = self._otcs_frontend.set_node_nickname(
                node_id=classification_id,
                nickname=nickname,
                show_error=True,
            )
            if not response:
                self.logger.error(
                    "Failed to assign nickname -> '%s' to classification -> '%s'!",
                    nickname,
                    classification_name,
                )

        # Depending on the bulk operations (create, update, delete)
        # and the related conditions it may well be that classification_id is None.
        # Only if classification ID is not none we want to count this row as success:
        if classification_id is not None:
            result["success_counter"] += 1
            # Record the classification name and ID to allow to read it from failure file
            # and speedup the process.
            result["classifications"][classification_name] = classification_id
    # end for index...

    self.logger.info("End working...")

    results.append(result)

process_bulk_datasource(data_source_name, force_reload=True)

Process a data source that is given by a payload element.

Parse its properties and deliver a 'Data' object which is a wrapper for a Pandas data frame.

Parameters:

Name	Type	Description	Default
data_source_name	str	The data source name.	required
force_reload	bool	Force a reload of the data source if True.	True

Returns:

Type	Description
Data | None	Data | None: Data source object of type Data. None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource")
def process_bulk_datasource(
    self,
    data_source_name: str,
    force_reload: bool = True,
) -> Data | None:
    """Process a data source that is given by a payload element.

    Parse its properties and deliver a 'Data' object which is a wrapper for
    a Pandas data frame.

    Args:
        data_source_name (str):
            The data source name.
        force_reload (bool):
            Force a reload of the data source if True.

    Returns:
        Data | None:
            Data source object of type Data. None in case of an error.

    """

    if not data_source_name:
        self.logger.error("Missing data source name!")
        return None

    self._log_header_callback(
        text="Process Bulk Data Source -> '{}'".format(data_source_name),
        char="-",
    )

    self.logger.info(
        "Found specified data source name -> '%s'. Lookup the data source payload...",
        data_source_name,
    )
    data_source = next(
        (item for item in self._bulk_datasources if item["name"] == data_source_name),
        None,
    )
    if not data_source:  # does this data source not exist?
        self.logger.error(
            "Cannot find specified data source -> '%s' in payload!",
            data_source_name,
        )
        return None

    # Check if data has already been loaded for the data source:
    if "data" in data_source and not force_reload:
        self.logger.info(
            "Data for data source -> '%s' is already loaded and reload is not enforced. Return existing data...",
            data_source_name,
        )
        return data_source["data"]
    elif force_reload:
        self.logger.info(
            "Reload of data from data source -> '%s' is enforced. Building data...",
            data_source_name,
        )
    else:
        self.logger.info(
            "Data for data source -> '%s' is not yet available. Building data...",
            data_source_name,
        )
        # Try to load the data source in the Admin Personal Workspace for investigations:
        if data_source.get("load_data_source", False):
            self.logger.info(
                "Payload allows to reload existing data source. Check if the data source -> '%s' can be reloaded...",
                data_source_name,
            )
            if self.read_data_source_file(data_source=data_source):
                return data_source["data"]
            else:
                self.logger.warning(
                    "Couldn't load the data source -> '%s' from file. Fall back to reloading it from the source...",
                    data_source_name,
                )

    data_source_type = data_source.get("type", None)
    if not data_source_type:
        self.logger.error(
            "Data source needs a type parameter. This is new - you may need to add it to your bulkDataSource payload definition file! Cannot load data.",
        )
        return None

    match data_source_type:
        case "excel":
            data = self.process_bulk_datasource_excel(data_source=data_source)
            if data is None:
                self.logger.error("Failure during load of Excel file!")
                return None
        case "servicenow":
            data = self.process_bulk_datasource_servicenow(data_source=data_source)
            if data is None:
                self.logger.error("Failure during load of ServiceNow articles!")
                return None
        case "otmm":
            data = self.process_bulk_datasource_otmm(data_source=data_source)
            if data is None:
                self.logger.error(
                    "Failure during load of OpenText Media Management assets!",
                )
                return None
        case "otcs":
            data = self.process_bulk_datasource_otcs(data_source=data_source)
            if data is None:
                self.logger.error(
                    "Failure during load of OpenText Content Server items!",
                )
                return None
        case "pht":
            data = self.process_bulk_datasource_pht(data_source=data_source)
            if data is None:
                self.logger.error(
                    "Failure during load of OpenText Product Hierarchy (PHT)!",
                )
                return None
        case "nhc":
            data = self.process_bulk_datasource_nhc(data_source=data_source)
            if data is None:
                self.logger.error(
                    "Failure during load of National Hurricane Center data (NHC)!",
                )
                return None
        case "json":
            data = self.process_bulk_datasource_json(data_source=data_source)
            if data is None:
                self.logger.error("Failure during load of JSON data source!")
                return None
        case "xml":
            data = self.process_bulk_datasource_xml(data_source=data_source)
            if data is None:
                self.logger.error("Failure during load of XML data source!")
                return None
        case "csv":
            data = self.process_bulk_datasource_csv(data_source=data_source)
            if data is None:
                self.logger.error("Failure during load of CSV data source!")
                return None
        case "web":
            data = self.process_bulk_datasource_web(data_source=data_source)
            if data is None:
                self.logger.error("Failure during load of Web data source!")
                return None
        case "filesystem":
            data = self.process_bulk_datasource_filesystem(data_source=data_source)
            if data is None:
                self.logger.error("Failure during load of Web data source!")
                return None
        case _:
            self.logger.error(
                "Illegal data source type. Types supported: 'excel', 'servicenow', 'otmm', 'otcs', 'pht', 'json', 'csv', 'xml', 'web', or 'filesystem'",
            )
            return None

    if data.get_data_frame() is None or data.get_data_frame().empty:
        self.logger.warning("Data source is empty - nothing loaded.")
        return None

    self.logger.info(
        "Data Frame for source -> '%s' has %s row(s) and %s column(s) after data loading.",
        data_source_name,
        data.get_data_frame().shape[0],
        data.get_data_frame().shape[1],
    )

    cleansings = data_source.get("cleansings", {})
    columns_to_drop = data_source.get("columns_to_drop", [])
    columns_to_keep = data_source.get("columns_to_keep", [])
    columns_to_add = data_source.get("columns_to_add", [])
    columns_to_add_list = data_source.get("columns_to_add_list", [])
    columns_to_add_concat = data_source.get("columns_to_add_concat", [])
    columns_to_add_table = data_source.get("columns_to_add_table", [])
    filters = data_source.get("filters", [])
    if not filters:
        # Backward compatibility. This used to be called "conditions" before
        # but we don't want to mix this with the conditions on row level
        # we have for bulkWorkspaces and bulkDocuments:
        filters = data_source.get("conditions", [])
    explosions = data_source.get("explosions", [])

    # Cleanup data if specified in data_source. We do this first!
    if cleansings:
        self.logger.info(
            "Start cleansing for data source -> '%s'...",
            data_source_name,
        )
        data.cleanse(cleansings=cleansings)
        self.logger.info(
            "Cleansing for data source -> '%s' completed...",
            data_source_name,
        )

    # Add columns if specified in data_source:
    for add_column in columns_to_add:
        if "source_column" not in add_column or "name" not in add_column:
            self.logger.error(
                "Add columns is missing name or source column. Column will not be added.",
            )
            continue
        data.add_column(
            source_column=add_column["source_column"],
            new_column=add_column["name"],
            reg_exp=add_column.get("regex", add_column.get("reg_exp", None)),
            prefix=add_column.get("prefix", ""),
            suffix=add_column.get("suffix", ""),
            length=add_column.get("length", None),
            group_chars=add_column.get("group_chars", None),
            group_separator=add_column.get("group_separator", "."),
        )

    # Add columns with list values from a list of other columns
    # if specified in data_source:
    for add_column in columns_to_add_list:
        if "source_columns" not in add_column or "name" not in add_column:
            self.logger.error(
                "Add list columns is missing name or source columns. Column will not be added.",
            )
            continue
        data.add_column_list(
            source_columns=add_column["source_columns"],
            new_column=add_column["name"],
        )

    # Add columns with list values from a list of other columns
    # if specified in data_source:
    for add_column in columns_to_add_concat:
        if "source_columns" not in add_column or "name" not in add_column:
            self.logger.error(
                "Add concatenation columns is missing name or source columns. Column will not be added.",
            )
            continue
        data.add_column_concat(
            source_columns=add_column["source_columns"],
            new_column=add_column["name"],
            concat_char=add_column.get("concat_chars", ""),
            lower=add_column.get("lower", False),
            upper=add_column.get("upper", False),
            capitalize=add_column.get("capitalize", False),
            title=add_column.get("title", False),
        )

    # Add columns with list values from a list of other columns
    # if specified in data_source:
    for add_column in columns_to_add_table:
        if "source_columns" not in add_column or "name" not in add_column:
            self.logger.error(
                "Add table columns is missing name or source columns. Column will not be added.",
            )
            continue
        data.add_column_table(
            source_columns=add_column["source_columns"],
            new_column=add_column["name"],
            delimiter=add_column.get("list_splitter", ","),
        )

    # Drop columns if specified in data_source:
    if columns_to_drop:
        data.drop_columns(columns_to_drop)

    # Keep only selected columns if specified in data_source:
    if columns_to_keep:
        data.keep_columns(columns_to_keep)

    # Check if fields with list substructures should be exploded:
    for explosion in explosions:
        # The explode field parameter can be a string or a list.
        # Exploding multiple fields at once avoids combinatorial explosions -
        # this is VERY different from exploding columns one after the other!
        if "explode_field" not in explosion and "explode_fields" not in explosion:
            self.logger.error("Missing explosion field(s)!")
            continue
        # We support both syntax (singular + plural):
        explode_fields = (
            explosion["explode_field"] if "explode_fields" not in explosion else explosion["explode_fields"]
        )

        flatten_fields = explosion.get("flatten_fields", [])
        split_string_to_list = explosion.get("split_string_to_list", False)
        list_splitter = explosion.get("list_splitter", None)
        self.logger.info(
            "Starting explosion of data source -> '%s' by field(s) -> '%s' (type -> '%s'). Size of data frame before explosion -> %s.",
            data_source_name,
            str(explode_fields),
            type(explode_fields),
            str(len(data)),
        )
        data.explode_and_flatten(
            explode_fields=explode_fields,
            flatten_fields=flatten_fields,
            make_unique=False,
            split_string_to_list=split_string_to_list,
            separator=list_splitter,
            reset_index=True,
        )
        self.logger.info("Size of data frame after explosion -> %s.", str(len(data)))

    # Keep only selected rows if filters are specified in data_source
    # We have this after "explosions" to allow access to subfields as well:
    if filters:
        data.filter(conditions=filters)

    # Keep the Data Frame for later lookups:
    data_source["data"] = data

    self._log_header_callback(
        text="Completed Bulk Data Source -> '{}'".format(data_source_name),
        char="-",
    )

    # Save the data source in the Admin Personal Workspace for investigations:
    if data_source.get("save_data_source", False):
        self.write_data_source_file(data_source=data_source)

    return data

process_bulk_datasource_csv(data_source)

Load data from CSV files (comma-separated values) into the data frame of the Data class.

See helper/data.py

Parameters:

Name	Type	Description	Default
data_source	dict	Payload dict element for the data source.	required

Returns:

Type	Description
Data | None	Data | None: Data class that includes a Pandas data frame. None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_csv")
def process_bulk_datasource_csv(self, data_source: dict) -> Data | None:
    """Load data from CSV files (comma-separated values) into the data frame of the Data class.

    See helper/data.py

    Args:
        data_source (dict):
            Payload dict element for the data source.

    Returns:
        Data | None:
            Data class that includes a Pandas data frame. None in case of an error.

    """

    # 1. Read and validate values from the data source payload:
    csv_files = data_source.get("csv_files", [])
    if not csv_files:
        self.logger.error(
            "CSV files not specified in payload of bulk data source. Cannot load data!",
        )
        return None
    csv_delimiter = data_source.get("csv_delimiter", ",")
    # The header index is an integer. The first row / line has the index 0.
    # But 0 is not the default! The default is the CSV does not have a header line
    # at all (None):
    csv_header_index = data_source.get("csv_header_index")
    csv_column_names = data_source.get("csv_column_names")
    csv_use_columns = data_source.get("csv_use_columns")

    # If no headers is specified it means the CSV does not have column
    # names in a row (typically row 0 = first row). If we also don't
    # have the names for the columns we will end with having coumn names
    # that a index values (1, 2, 3, ...). This may not be what the payload
    # author wants - so we issue a warning:
    if not csv_column_names and csv_header_index is None:  # "is None" is important here as the index can be 0
        self.logger.warning(
            "CSV loader parameters are missing both, header line index (no 'csv_header_index') and column names (no 'csv_column_names'). This could lead to numeric column names.",
        )

    # 2. Initialize Data object:
    data = Data(logger=self.logger)

    # 3. Iterate CSV files and load data into Data object:
    for csv_file in csv_files:
        if not data.load_csv_data(
            csv_path=csv_file,
            delimiter=csv_delimiter,
            names=csv_column_names,
            header=csv_header_index,
            usecols=csv_use_columns,
        ):
            self.logger.error("failed to load CSV file -> '%s'!", csv_file)
            return None

    return data

process_bulk_datasource_excel(data_source)

Load data from Excel files into the data frame of the Data class.

See helper/data.py

Parameters:

Name	Type	Description	Default
data_source	dict	Payload dict element for the data source.	required

Returns:

Type	Description
Data | None	Data | None: Data class that includes a Pandas data frame. None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_excel")
def process_bulk_datasource_excel(self, data_source: dict) -> Data | None:
    """Load data from Excel files into the data frame of the Data class.

    See helper/data.py

    Args:
        data_source (dict):
            Payload dict element for the data source.

    Returns:
        Data | None:
            Data class that includes a Pandas data frame. None in case of an error.

    """

    # 1. Read and validate values from the data source payload:
    xlsx_files = data_source.get("xlsx_files", [])
    if not xlsx_files:
        self.logger.error(
            "Excel files not specified in payload of bulk data source. Cannot load data!",
        )
        return None
    xlsx_sheets = data_source.get("xlsx_sheets", 0)  # use 0 not None!
    xlsx_columns = data_source.get("xlsx_columns")
    xlsx_skip_rows = data_source.get("xlsx_skip_rows", 0)
    xlsx_na_values = data_source.get("xlsx_na_values")

    # 2. Initialize Data object:
    data = Data(logger=self.logger)

    # 3. Iterate of Excel files and load them into the Data object:
    for xlsx_file in xlsx_files:
        if not data.load_excel_data(
            xlsx_path=xlsx_file,
            sheet_names=xlsx_sheets,
            usecols=xlsx_columns,
            skip_rows=xlsx_skip_rows,
            na_values=xlsx_na_values,
        ):
            self.logger.error("Failed to load Excel file -> '%s'!", xlsx_file)
            return None

    return data

process_bulk_datasource_filesystem(data_source)

Load data from filesystem into the data frame of the Data class.

See helper/data.py

Parameters:

Name	Type	Description	Default
data_source	dict	Payload dict element for the data source.	required

Returns:

Type	Description
Data | None	Data | None: Data class that includes a Pandas data frame. None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_filesystem")
def process_bulk_datasource_filesystem(self, data_source: dict) -> Data | None:
    """Load data from filesystem into the data frame of the Data class.

    See helper/data.py

    Args:
        data_source (dict):
            Payload dict element for the data source.

    Returns:
        Data | None:
            Data class that includes a Pandas data frame. None in case of an error.

    """

    # 1. Read and validate values from the data source payload:
    root_folders = data_source.get("root_folders", [])
    if not root_folders:
        self.logger.error(
            "Root folders not specified in payload of bulk data source. Cannot load data from filesystem!",
        )
        return None

    if not isinstance(root_folders, list):
        self.logger.warning(
            "The payload for root folders of the 'filesytem' data source should be a list! You should adjust the 'root_folders' setting.",
        )
        root_folders: list = [root_folders]

    # 2. Initialize Data object:
    data = Data(logger=self.logger)

    # 3. Iterate root folders and load data into Data object:
    for folder in root_folders:
        if not data.load_directory(path_to_root=folder):
            self.logger.error(
                "Failed to load data from root folder -> '%s'!",
                folder,
            )
            return None

    return data

process_bulk_datasource_json(data_source)

Load data from JSON files into the data frame of the Data class.

See helper/data.py

Parameters:

Name	Type	Description	Default
data_source	dict	Payload dict element for the data source.	required

Returns:

Type	Description
Data | None	Data | None: Data class that includes a Pandas data frame. None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_json")
def process_bulk_datasource_json(self, data_source: dict) -> Data | None:
    """Load data from JSON files into the data frame of the Data class.

    See helper/data.py

    Args:
        data_source (dict):
            Payload dict element for the data source.

    Returns:
        Data | None:
            Data class that includes a Pandas data frame. None in case of an error.

    """

    # 1. Read and validate values from the data source payload:
    json_files = data_source.get("json_files", [])
    if not json_files:
        self.logger.error(
            "JSON files not specified in payload of bulk data source. Cannot load data!",
        )
        return None

    # 2. Initialize Data object:
    data = Data(logger=self.logger)

    # 3. Iterate JSON files and load data into Data object:
    for json_file in json_files:
        json_file = self.prepare_datasource_file(
            data_source=data_source,
            filename=json_file,
        )

        if not data.load_json_data(json_path=json_file):
            self.logger.error(
                "Invalid JSON file -> '%s'. Cannot load it!",
                json_file,
            )
            return None

    return data

process_bulk_datasource_nhc(data_source)

Load data from National Hurricane Center data source into the data frame of the Data class.

See helper/data.py

Parameters:

Name	Type	Description	Default
data_source	dict	Payload dict element for the data source	required

Returns:

Type	Description
Data | None	Data | None: Data class that includes a Pandas data frame. None in cause of an error.

Side Effects

self._nhc is set to the NHC object created by this method.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_nhc")
def process_bulk_datasource_nhc(self, data_source: dict) -> Data | None:
    """Load data from National Hurricane Center data source into the data frame of the Data class.

    See helper/data.py

    Args:
        data_source (dict):
            Payload dict element for the data source

    Returns:
        Data | None:
            Data class that includes a Pandas data frame. None in cause of an error.

    Side Effects:
        self._nhc is set to the NHC object created by this method.

    """
    try:
        from pyxecm_internal.nhc import (
            NHC,
        )
    except ModuleNotFoundError:
        self.logger.info(
            "Python module 'pyxecm_internal' not installed. Cannot process NHC data source!",
        )
        return None

    # 1. Read and validate values from the data source payload:
    nhc_year_start = data_source.get("nhc_year_start", "2023")
    nhc_year_end = data_source.get("nhc_year_end", "2024")
    nhc_basin = data_source.get("nhc_basin", "north_atlantic")
    nhc_track_images = data_source.get("nhc_track_images", ["png"])
    nhc_track_data = data_source.get("nhc_track_data", [])
    nhc_rain_images = data_source.get("nhc_rain_images", ["png"])
    nhc_rain_data = data_source.get("nhc_rain_data", [])
    nhc_skip_existing_files = data_source.get("nhc_skip_existing_files", True)
    nhc_async = data_source.get("nhc_async", True)
    nhc_async_processes = data_source.get("nhc_async_processes", 5)
    nhc_async_processes = data_source.get("nhc_async_processes", 5)
    nhc_storm_plot_exclusions = data_source.get("nhc_storm_plot_exclusions", [])
    nhc_download_dir = data_source.get("nhc_download_dir", "/data/nhc")
    # We either get the download dir for images from the payload
    # directly or we construct it from the general download dir:
    nhc_download_dir_images = data_source.get(
        "nhc_download_dir_images",
        os.path.join(nhc_download_dir, "images"),
    )  # don't use "/images"
    # We either get the download dir for data files (JSON) from the
    # payload directly or we construct it from the general download dir:
    nhc_download_dir_data = data_source.get(
        "nhc_download_dir_images",
        os.path.join(nhc_download_dir, "json"),
    )  # don't use "/data"

    # 2. Creating the NHC object:
    self._nhc = NHC(
        basin=nhc_basin,
        storm_plot_exclusions=nhc_storm_plot_exclusions,
        download_dir_images=nhc_download_dir_images,
        download_dir_data=nhc_download_dir_data,
        logger=self.logger,
    )
    if not self._nhc:
        self.logger.error("Failed to initialize NHC object!")
        return None

    # 3. Load the NHC storms into the Data object (Pandas DataFrame):
    if not self._nhc.load_storms(
        year_start=nhc_year_start,
        year_end=nhc_year_end,
        save_track_images=nhc_track_images,
        save_track_data=nhc_track_data,
        save_rain_images=nhc_rain_images,
        save_rain_data=nhc_rain_data,
        skip_existing_files=nhc_skip_existing_files,
        load_async=nhc_async,
        async_processes=nhc_async_processes,
    ):
        self.logger.error("Failure during load of NHC storms!")
        return None

    data = self._nhc.get_data()
    if data is None:
        self.logger.error("Failure during load of NHC storm data!")
        return None

    return data

process_bulk_datasource_otcs(data_source)

Load data from Content Server (OTCS) data source into the data frame.

See helper/data.py

Parameters:

Name	Type	Description	Default
data_source	dict	Payload dict element for the data source.	required

Returns:

Name	Type	Description
Data	Data	Data class that includes a Pandas DataFrame

Side Effects

self._otcs_source is set to the OTCS object created by this method.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_otcs")
def process_bulk_datasource_otcs(self, data_source: dict) -> Data:
    """Load data from Content Server (OTCS) data source into the data frame.

    See helper/data.py

    Args:
        data_source (dict):
            Payload dict element for the data source.

    Returns:
        Data:
            Data class that includes a Pandas DataFrame

    Side Effects:
        self._otcs_source is set to the OTCS object created by this method.

    """

    # 1. Read and validate values from the data source payload:
    otcs_hostname = data_source.get("otcs_hostname", "")
    if not otcs_hostname:
        self.logger.error(
            "Content Server hostname (otcs_hostname) is not specified in payload of bulk data source. Cannot load data!",
        )
        return None
    otcs_protocol = data_source.get("otcs_protocol", "https")
    otcs_port = data_source.get("otcs_port", "443")
    otcs_basepath = data_source.get("otcs_basepath", "/cs/cs")
    otcs_username = data_source.get("otcs_username", "")
    otcs_password = data_source.get("otcs_password", "")
    if not otcs_username or not otcs_password:
        self.logger.error(
            "Content Server user name (otcs_username) or password (otcs_password) are missing in payload of bulk data source. Cannot load data!",
        )
        return None
    otcs_thread_number = data_source.get("otcs_thread_number", BULK_THREAD_NUMBER)
    otcs_download_dir = data_source.get("otcs_download_dir", "/data/contentserver")
    # Root nodes for the traversal of the data loader (can be a list or single ID)
    # we keep two spellings to remain backwards compatible with the payload syntax:
    otcs_root_node_ids = data_source.get(
        "otcs_root_node_ids",
        data_source.get("otcs_root_node_id"),
    )

    if not otcs_root_node_ids:
        self.logger.error(
            "Content Server root node ID(s) for traversal are missing in payload of bulk data source. Cannot load data!",
        )
        return None

    otcs_include_workspaces = data_source.get("otcs_include_workspaces", True)
    otcs_include_items = data_source.get("otcs_include_items", True)
    otcs_include_workspace_metadata = data_source.get("otcs_include_workspace_metadata", True)
    otcs_include_item_metadata = data_source.get("otcs_include_item_metadata", True)

    # Filter workspace by depth under the given root (only consider items as workspace if they have the right depth in the hierarchy):
    otcs_filter_workspace_depth = data_source.get("otcs_filter_workspace_depth", 0)
    # Filter workspace by subtype (only consider items as workspace if they have the right technical subtype):
    # This is NOT the workspace type but the technical subtype (like 848 for workspaces and 0 for folder)
    otcs_filter_workspace_subtypes = data_source.get(
        "otcs_filter_workspace_subtypes",
    )
    # Filter workspace by category name (only consider items as workspace if they have the category):
    otcs_filter_workspace_category = data_source.get(
        "otcs_filter_workspace_category",
    )
    # Filter workspace by attribute values (only consider items as workspace if they have the attributes with the defined values):
    otcs_filter_workspace_attributes = data_source.get(
        "otcs_filter_workspace_attributes",
    )

    # Filter item by depth under the given root:
    otcs_filter_item_depth = data_source.get("otcs_filter_item_depth")
    # Filter item by subtype (only consider items if they have the right technical subtype):
    # This is the technical subtype (like 0 for folder and 144 for documents)
    otcs_filter_item_subtypes = data_source.get(
        "otcs_filter_item_subtypes",
    )
    # Filter item by category name (only consider items as workspace if they have the category):
    otcs_filter_item_category = data_source.get("otcs_filter_item_category")
    # Filter item by attribute values (only consider items if they have the attributes with the defined values):
    otcs_filter_item_attributes = data_source.get("otcs_filter_item_attributes")
    # Filter item also if the are in workspaces (default is True):
    otcs_filter_item_in_workspace = data_source.get(
        "otcs_filter_item_in_workspace",
        True,
    )
    # List of node IDs to exclude in traversing the folders in the OTCS data source:
    otcs_exclude_node_ids = data_source.get("otcs_exclude_node_ids")

    # Document handling parameters:
    otcs_download_documents = data_source.get("otcs_download_documents", True)
    otcs_skip_existing_downloads = data_source.get("otcs_skip_existing_downloads", True)
    otcs_extract_zip = data_source.get("extract_zip", False)
    # The following parameter controls how column names are constructed. If it is true, then
    # attribute columns for workspaces and items will use the category ID in the column name.
    # Wokspace attributes always start with "workspace_cat_". Item attributes start with item_cat_".
    # If the value of 'otcs_use_numeric_category_identifier' is False then the category name
    # is converted to lower-case and spaces and non-alphanumeric characters are replaced with "_".
    # Example with otcs_use_numeric_category_identifier = True: workspace_cat_47110815_10
    # Example with otcs_use_numeric_category_identifier = False: workspace_cat_customer_use_case_10
    otcs_use_numeric_category_identifier = data_source.get("otcs_use_numeric_category_identifier", True)

    # Ensure Root_node_id is a list of integers
    if not isinstance(otcs_root_node_ids, list):
        otcs_root_node_ids = [otcs_root_node_ids]
    otcs_root_node_ids = [int(item) for item in otcs_root_node_ids]

    # 2. Creating the OTCS object:
    self._otcs_source = OTCS(
        protocol=otcs_protocol,
        hostname=otcs_hostname,
        port=otcs_port,
        base_path=otcs_basepath,
        username=otcs_username,
        password=otcs_password,
        thread_number=otcs_thread_number,
        download_dir=otcs_download_dir,
        use_numeric_category_identifier=otcs_use_numeric_category_identifier,
        logger=self.logger,
    )

    # 3. Authenticate at OTCS
    auth_data = self._otcs_source.authenticate()
    if not auth_data:
        self.logger.error(
            "Failed to authenticate at Content Server -> %s!",
            otcs_protocol + "://" + otcs_hostname + otcs_basepath,
        )
        return None
    else:
        self.logger.info(
            "Successfully authenticated at Content Server -> %s.",
            otcs_protocol + "://" + otcs_hostname + otcs_basepath,
        )

    # 4. Load the Content Server data into the Data object (Pandas DataFrame):

    self.logger.info(
        "Loading data (folder, workspaces, items) from Content Server -> %s using root IDs -> %s...",
        otcs_protocol + "://" + otcs_hostname + otcs_basepath,
        otcs_root_node_ids,
    )

    for otcs_root_node_id in otcs_root_node_ids:
        if not self._otcs_source.load_items(
            node_id=otcs_root_node_id,
            workspaces=otcs_include_workspaces,
            items=otcs_include_items,
            filter_workspace_depth=otcs_filter_workspace_depth,
            filter_workspace_subtypes=otcs_filter_workspace_subtypes,
            filter_workspace_category=otcs_filter_workspace_category,
            filter_workspace_attributes=otcs_filter_workspace_attributes,
            filter_item_depth=otcs_filter_item_depth,
            filter_item_subtypes=otcs_filter_item_subtypes,
            filter_item_category=otcs_filter_item_category,
            filter_item_attributes=otcs_filter_item_attributes,
            filter_item_in_workspace=otcs_filter_item_in_workspace,
            exclude_node_ids=otcs_exclude_node_ids,
            workspace_metadata=otcs_include_workspace_metadata,
            item_metadata=otcs_include_item_metadata,
            download_documents=otcs_download_documents,
            skip_existing_downloads=otcs_skip_existing_downloads,
            extract_zip=otcs_extract_zip,
            workers=otcs_thread_number,
        ):
            self.logger.error(
                "Failure during load of Content Server items from root node ID -> %s!",
                str(otcs_root_node_id),
            )
            return None

    data = self._otcs_source.get_data()
    if data is None:
        self.logger.error(
            "Failure during load of Content Server items! No data loaded!",
        )
        return None

    return data

process_bulk_datasource_otmm(data_source)

Load data from OTMM data source into the data frame of the Data class.

See helper/data.py

Parameters:

Name	Type	Description	Default
data_source	dict	Payload dict element for the data source.	required

Returns:

Type	Description
Data | None	Data | None: Data class that includes a Pandas data frame. None in case of an error.

Side Effects

self._otmm is set to the OTMM object created by this method

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_otmm")
def process_bulk_datasource_otmm(self, data_source: dict) -> Data | None:
    """Load data from OTMM data source into the data frame of the Data class.

    See helper/data.py

    Args:
        data_source (dict):
            Payload dict element for the data source.

    Returns:
        Data | None:
            Data class that includes a Pandas data frame. None in case of an error.

    Side Effects:
        self._otmm is set to the OTMM object created by this method

    """

    # 1. Read and validate values from the data source payload:
    otmm_base_url = data_source.get("otmm_base_url", "")
    if not otmm_base_url:
        self.logger.error(
            "OTMM base URL (otmm_base_url) is not specified in payload of bulk data source. Cannot load data!",
        )
        return None
    otmm_username = data_source.get("otmm_username", "")
    otmm_password = data_source.get("otmm_password", "")
    otmm_client_id = data_source.get("otmm_client_id")
    otmm_client_secret = data_source.get("otmm_client_secret")
    otmm_thread_number = data_source.get("otmm_thread_number", BULK_THREAD_NUMBER)
    otmm_download_dir = data_source.get("otmm_download_dir", "/data/mediaassets")
    otmm_business_unit_exclusions = data_source.get(
        "otmm_business_unit_exclusions",
        [],
    )
    otmm_business_unit_inclusions = data_source.get("otmm_business_unit_inclusions")
    otmm_product_exclusions = data_source.get("otmm_product_exclusions", [])
    otmm_product_inclusions = data_source.get("otmm_product_inclusions")
    otmm_asset_exclusions = data_source.get("otmm_asset_exclusions", [])
    otmm_asset_inclusions = data_source.get("otmm_asset_inclusions", [])

    self.logger.info(
        "Loading data from OpenText Media Management -> %s (Marketing Assets)...",
        otmm_base_url,
    )

    # 2. Creating the OTMM object:
    self._otmm = OTMM(
        base_url=otmm_base_url,
        client_id=otmm_client_id,
        client_secret=otmm_client_secret,
        username=otmm_username,
        password=otmm_password,
        thread_number=otmm_thread_number,
        download_dir=otmm_download_dir,
        business_unit_exclusions=otmm_business_unit_exclusions,
        business_unit_inclusions=otmm_business_unit_inclusions,
        product_exclusions=otmm_product_exclusions,
        product_inclusions=otmm_product_inclusions,
        asset_exclusions=otmm_asset_exclusions,
        asset_inclusions=otmm_asset_inclusions,
        logger=self.logger,
    )
    if not self._otmm:
        self.logger.error("Failed to initialize OTMM object!")
        return None

    # 3. Authenticate at OTMM
    token = self._otmm.authenticate()
    if not token:
        self.logger.error(
            "Failed to authenticate at OpenText Media Management -> %s!",
            otmm_base_url,
        )
        return None
    else:
        self.logger.info(
            "Successfully authenticated at OpenText Media Management -> %s.",
            otmm_base_url,
        )

    # 4. Load the OTMM assets into the Data object (Pandas DataFrame):
    if not self._otmm.load_assets():
        self.logger.error(
            "Failure during load of OpenText Media Management assets!",
        )
        return None

    data = self._otmm.get_data()
    if data is None:
        self.logger.error(
            "Failure during load of OpenText Media Management assets! No assets loaded!",
        )
        return None

    return data

process_bulk_datasource_pht(data_source)

Load data from OpenText PHT data source into the data frame of the Data class.

See helper/data.py

Parameters:

Name	Type	Description	Default
data_source	dict	Payload dict element for the data source.	required

Returns:

Type	Description
Data | None	Data | None: Data class that includes a Pandas data frame. None in case of an error.

Side Effects

self._pht is set to the PHT object created by this method.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_pht")
def process_bulk_datasource_pht(self, data_source: dict) -> Data | None:
    """Load data from OpenText PHT data source into the data frame of the Data class.

    See helper/data.py

    Args:
        data_source (dict):
            Payload dict element for the data source.

    Returns:
        Data | None:
            Data class that includes a Pandas data frame. None in case of an error.

    Side Effects:
        self._pht is set to the PHT object created by this method.

    """

    try:
        from pyxecm_internal.pht import PHT
    except ModuleNotFoundError:
        self.logger.info(
            "Python module 'pyxecm_internal' not installed. Cannot process PHT data source!",
        )
        return None

    # 1. Read and validate values from the data source payload:
    pht_base_url = data_source.get("pht_base_url", "")
    if not pht_base_url:
        self.logger.error(
            "PHT base URL (pht_base_url) is not specified in payload of bulk data source. Cannot load data!",
        )
        return None
    pht_username = data_source.get("pht_username", "")
    if not pht_username:
        self.logger.error(
            "PHT username (pht_username) is not specified in payload of bulk data source. Cannot load data!",
        )
        return None
    pht_password = data_source.get("pht_password", "")
    if not pht_password:
        self.logger.error(
            "PHT password (pht_password) is not specified in payload of bulk data source. Cannot load data!",
        )
        return None

    pht_business_unit_exclusions = data_source.get(
        "pht_business_unit_exclusions",
        [],
    )
    pht_business_unit_inclusions = data_source.get(
        "pht_business_unit_inclusions",
        [],
    )
    pht_product_exclusions = data_source.get("pht_product_exclusions", [])
    pht_product_inclusions = data_source.get("pht_product_inclusions", [])
    pht_product_category_exclusions = data_source.get(
        "pht_product_category_exclusions",
        [],
    )
    pht_product_category_inclusions = data_source.get(
        "pht_product_category_inclusions",
        [],
    )
    pht_product_status_exclusions = data_source.get(
        "pht_product_status_exclusions",
        [],
    )
    pht_product_status_inclusions = data_source.get(
        "pht_product_status_inclusions",
        [],
    )
    pht_product_attributes = data_source.get(
        "pht_product_attributes",
        [],
    )

    self.logger.info(
        "Loading data from OpenText PHT -> %s (Product Hierarchy)...",
        pht_base_url,
    )

    # 2. Creating the PHT object:
    self._pht = PHT(
        base_url=pht_base_url,
        username=pht_username,
        password=pht_password,
        business_unit_exclusions=pht_business_unit_exclusions,
        business_unit_inclusions=pht_business_unit_inclusions,
        product_exclusions=pht_product_exclusions,
        product_inclusions=pht_product_inclusions,
        product_category_exclusions=pht_product_category_exclusions,
        product_category_inclusions=pht_product_category_inclusions,
        product_status_exclusions=pht_product_status_exclusions,
        product_status_inclusions=pht_product_status_inclusions,
        logger=self.logger,
    )
    if not self._pht:
        self.logger.error("Failed to initialize PHT object!")
        return None

    # 3. Authenticate at PHT
    token = self._pht.authenticate()
    if not token:
        self.logger.error(
            "Failed to authenticate at OpenText PHT -> %s!",
            pht_base_url,
        )
        return None
    else:
        self.logger.info(
            "Successfully authenticated at OpenText PHT -> %s.",
            pht_base_url,
        )

    # 4. Load the PHT product information into the Data object (Pandas DataFrame):
    if not self._pht.load_products(attributes_to_extract=pht_product_attributes):
        self.logger.error("Failure during load of OpenText PHT products!")
        return None

    data = self._pht.get_data()
    if data is None:
        self.logger.error("Failure during load of OpenText PHT product data!")
        return None

    return data

process_bulk_datasource_servicenow(data_source)

Load data from ServiceNow data source into the data frame of the Data class.

See helper/data.py

Parameters:

Name	Type	Description	Default
data_source	dict	Payload dict element for the data source.	required

Returns:

Type	Description
Data | None	Data | None: Data class that includes a Pandas data frame. None in case of an error.

Side Effects

self._servicenow is set to the ServiceNow object created by this method

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_servicenow")
def process_bulk_datasource_servicenow(self, data_source: dict) -> Data | None:
    """Load data from ServiceNow data source into the data frame of the Data class.

    See helper/data.py

    Args:
        data_source (dict):
            Payload dict element for the data source.

    Returns:
        Data | None:
            Data class that includes a Pandas data frame. None in case of an error.

    Side Effects:
        self._servicenow is set to the ServiceNow object created by this method

    """

    # 1. Read and validate values from the data source payload:
    sn_base_url = data_source.get("sn_base_url", "")
    if not sn_base_url:
        self.logger.error(
            "ServiceNow base URL (sn_base_url) is not specified in payload of bulk data source. Cannot load data!",
        )
        return None
    sn_auth_type = data_source.get("sn_auth_type", "basic")
    sn_username = data_source.get("sn_username", "")
    sn_password = data_source.get("sn_password", "")
    sn_client_id = data_source.get("sn_client_id")
    sn_client_secret = data_source.get("sn_client_secret")
    sn_table_name = data_source.get(
        "sn_table_name",
        "u_kb_template_technical_article_public",
    )
    sn_queries = data_source.get("sn_queries", [])
    sn_query = data_source.get("sn_query")
    if sn_query is not None:
        sn_queries.append({"table": sn_table_name, "query": sn_query})

    sn_thread_number = data_source.get("sn_thread_number", BULK_THREAD_NUMBER)
    sn_download_dir = data_source.get("sn_download_dir", "/data/knowledgebase")
    sn_skip_existing_downloads = data_source.get("sn_skip_existing_downloads", True)
    sn_product_exclusions = data_source.get("sn_product_exclusions", [])

    if sn_product_exclusions:
        self.logger.info(
            "Excluding products -> %s from ServiceNow data loading.",
            sn_product_exclusions,
        )

    if sn_base_url and (sn_auth_type == "basic") and (not sn_username or not sn_password):
        self.logger.error(
            "ServiceNow basic authentication needs username and password in payload!",
        )
        return None
    if sn_base_url and (sn_auth_type == "oauth") and (not sn_client_id or not sn_client_secret):
        self.logger.error(
            "ServiceNow OAuth authentication needs client ID and client secret in payload!",
        )
        return None

    # 2. Creating the ServiceNow object:
    self._servicenow = ServiceNow(
        base_url=sn_base_url,
        auth_type=sn_auth_type,
        client_id=sn_client_id,
        client_secret=sn_client_secret,
        username=sn_username,
        password=sn_password,
        thread_number=sn_thread_number,
        download_dir=sn_download_dir,
        product_exclusions=sn_product_exclusions,
        logger=self.logger,
    )

    # 3. Authenticate at ServiceNow
    auth_data = self._servicenow.authenticate(auth_type=sn_auth_type)
    if not auth_data:
        self.logger.error("Failed to authenticate at ServiceNow -> %s!", sn_base_url)
        return None
    else:
        self.logger.info(
            "Successfully authenticated at ServiceNow -> %s.",
            sn_base_url,
        )

    # 4. Load the ServiceNow data into the Data object (Pandas DataFrame):
    for item in sn_queries:
        sn_table_name = item["sn_table_name"]
        sn_query = item["sn_query"]

        self.logger.info(
            "Loading data from ServiceNow table -> '%s' with query -> '%s'...",
            sn_table_name,
            sn_query,
        )

        if not self._servicenow.load_articles(
            table_name=sn_table_name,
            query=sn_query,
            skip_existing_downloads=sn_skip_existing_downloads,
        ):
            self.logger.error(
                "Failure during load of ServiceNow articles from table -> '%s' using query -> '%s' !",
                sn_table_name,
                sn_query,
            )
            continue

    data = self._servicenow.get_data()
    if data is None:
        self.logger.error(
            "Failure during load of ServiceNow articles! No articles loaded!",
        )
        return None

    return data

process_bulk_datasource_web(data_source)

Load data from Web page into the data frame of the Data class.

See helper/data.py

Parameters:

Name	Type	Description	Default
data_source	dict	Payload dict element for the data source	required

Returns:

Type	Description
Data | None	Data | None: Data class that includes a Pandas DataFrame. None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_web")
def process_bulk_datasource_web(self, data_source: dict) -> Data | None:
    """Load data from Web page into the data frame of the Data class.

    See helper/data.py

    Args:
        data_source (dict):
            Payload dict element for the data source

    Returns:
        Data | None:
            Data class that includes a Pandas DataFrame. None in case of an error.

    """

    # 1. Read and validate values from the data source payload:
    web_url_templates = data_source.get("web_url_templates")
    if not web_url_templates:
        self.logger.error(
            "Web URLs (templates) not specified in payload of bulk data source. Cannot load data!",
        )
        return None
    web_start_value = data_source.get("web_start_value")
    web_end_value = data_source.get("web_end_value")
    web_value_name = data_source.get(
        "web_value_name",
    )  # which column name to use for the value
    web_special_url_templates = data_source.get("web_special_url_templates", [])
    # Pattern to filter file links on the page
    web_file_url_pattern = data_source.get("web_file_url_pattern")

    web_values = list(range(web_start_value, web_end_value + 1)) if web_start_value and web_end_value else None
    web_special_values = data_source.get("web_special_values", [])

    # 2. Initialize Data object:
    data = Data(logger=self.logger)

    # 3. Iterate over Web pages and load data into Data object:
    if not data.load_web(
        values=web_values,
        value_name=web_value_name,
        url_templates=web_url_templates,
        special_values=web_special_values,
        special_url_templates=web_special_url_templates,
        pattern=web_file_url_pattern,
    ):
        self.logger.error(
            "Failed to load Web data from URLs -> %s!",
            str(web_url_templates),
        )
        return None

    return data

process_bulk_datasource_xml(data_source)

Load data from XML files or directories or zip files into the data frame of the Data class.

See helper/data.py

Parameters:

Name	Type	Description	Default
data_source	dict	Payload dict element for the data source.	required

Returns:

Type	Description
Data | None	Data | None: Data class that includes a Pandas data frame. None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_datasource_xml")
def process_bulk_datasource_xml(self, data_source: dict) -> Data | None:
    """Load data from XML files or directories or zip files into the data frame of the Data class.

    See helper/data.py

    Args:
        data_source (dict):
            Payload dict element for the data source.

    Returns:
        Data | None:
            Data class that includes a Pandas data frame. None in case of an error.

    """

    # 1. Read and validate values from the data source payload:
    xml_files = data_source.get("xml_files", [])
    xml_directories = data_source.get(
        "xml_directories",
        [],
    )  # can also be zip files
    xml_xpath = data_source.get("xml_xpath")

    # 2. Initialize Data object:
    data = Data(logger=self.logger)

    # 3. If no XML directories are specified we interpret the "xml_files"
    if not xml_directories:
        for xml_file in xml_files:
            self.logger.info("Loading XML file -> '%s'...", xml_file)
            if not data.load_xml_data(xml_path=xml_file, xpath=xml_xpath):
                self.logger.error("Failed to load XML file -> '%s'!", xml_file)
                return None

    # 4. If XML directories or zip files of XML files are given we traverse them instead:
    else:
        # we now produce a list of dictionaries:
        xml_data = XML.load_xml_files_from_directories(
            directories=xml_directories,
            filenames=xml_files,
            xpath=xml_xpath,
            logger=self.logger.getChild("xml"),
        )
        if xml_data is None:  # test on error
            self.logger.error(
                "Failed to load XML files from directories or ZIP files -> %s!",
                xml_directories,
            )
            return None
        if not xml_data:  # test on empty result
            self.logger.warning(
                "Found no XML files in directory or ZIP files -> %s!",
                xml_directories,
            )
        else:
            data.append(add_data=xml_data)

    return data

process_bulk_documents(section_name='bulkDocuments')

Process bulkDocuments in payload and bulk create them in Content Server (multi-threaded).

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'bulkDocuments'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_documents")
def process_bulk_documents(self, section_name: str = "bulkDocuments") -> bool:
    """Process bulkDocuments in payload and bulk create them in Content Server (multi-threaded).

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise.

    """

    if not self._bulk_documents:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    # For efficient idem-potent operation we may want to see which workspaces
    # have already been processed successfully before:
    if self.check_status_file(
        payload_section_name=section_name,
        payload_specific=True,
        prefix="failure_",
    ):
        self.logger.info(
            "Found failure file. Trying to reprocess just the failed ones...",
        )
        # Read payload from where we left it last time
        self._bulk_documents = self.get_status_file(
            payload_section_name=section_name,
            prefix="failure_",
        )
        if not self._bulk_documents:
            self.logger.error(
                "Cannot load existing bulkDocuments failure file. Bailing out!",
            )
            return False

    success: bool = True

    for bulk_document in self._bulk_documents:
        # Check if element has been disabled in payload (enabled = false).
        # In this case we skip the element:
        enabled = bulk_document.get("enabled", True)
        if not enabled:
            self.logger.info("Payload for Bulk Document is disabled. Skipping...")
            continue

        # The payload element must have a "data_source" key:
        data_source_name = bulk_document.get("data_source", None)
        if not data_source_name:
            self.logger.error("Missing data source name in Bulk Document!")
            success = False
            continue

        copy_data_source = bulk_document.get("copy_data_source", False)
        force_reload = bulk_document.get("force_reload", True)

        # Load and prepare the data source for the bulk processing:
        if copy_data_source:
            self.logger.info(
                "Take a copy of data source -> %s to avoid side-effects for repeative usage of the data source...",
                data_source_name,
            )
            data = Data(
                self.process_bulk_datasource(
                    data_source_name=data_source_name,
                    force_reload=force_reload,
                ),
                logger=self.logger,
            )
        else:
            self.logger.info(
                "Use original data source -> '%s' and not do a copy.",
                bulk_document["data_source"],
            )
            data = self.process_bulk_datasource(
                data_source_name=data_source_name,
                force_reload=force_reload,
            )
        if (
            data is None
            or data.get_data_frame() is None  # the 2nd check is important for the "copy_data_source" case!
        ):  # important to use "is None" here!
            self.logger.error(
                "Failed to load data source -> '%s' for bulk documents!",
                data_source_name,
            )
            success = False
            continue

        # Check if fields with list substructures should be exploded.
        # We may want to do this outside the bulkDatasource to only
        # explode for bulkDocuments and not for bulkWorkspaces or
        # bulkWorkspaceRelationships:
        explosions = bulk_document.get("explosions", [])
        for explosion in explosions:
            # explode field can be a string or a list
            # exploding multiple fields at once avoids
            # combinatorial explosions - this is VERY
            # different from exploding columns one after the other!
            if "explode_field" not in explosion and "explode_fields" not in explosion:
                self.logger.error("Missing explosion field(s)!")
                continue
            # we want to be backwards compatible...
            explode_fields = (
                explosion["explode_field"] if "explode_field" in explosion else explosion["explode_fields"]
            )
            flatten_fields = explosion.get("flatten_fields", [])
            split_string_to_list = explosion.get("split_string_to_list", False)
            list_splitter = explosion.get("list_splitter", None)
            self.logger.info(
                "Starting explosion of bulk documents by field(s) -> %s (type -> %s). Size of data frame before explosion -> %s.",
                explode_fields,
                str(type(explode_fields)),
                str(len(data)),
            )
            data.explode_and_flatten(
                explode_fields=explode_fields,
                flatten_fields=flatten_fields,
                make_unique=False,
                split_string_to_list=split_string_to_list,
                separator=list_splitter,
                reset_index=True,
            )
            self.logger.info(
                "Size of data frame after explosion -> %s.",
                str(len(data)),
            )
        # end for explosion in explosions

        # Keep only selected rows if filters are specified in bulkDocuments.
        # We have this _after_ "explosions" to allow access to subfields as well.
        # We have this _before_ "sorting" and "deduplication" as we may keep the wrong
        # rows otherwise (unique / deduplication always keeps the first matching row).
        # We may want to do this outside the bulkDatasource to only
        # filter for bulkDocuments and not for bulkWorkspaces or
        # bulkWorkspaceRelationships:
        filters = bulk_document.get("filters", [])
        if filters:
            data.filter(conditions=filters)

        # Sort the data frame if "sort" specified in payload. We may want to do this to have a
        # higher chance that rows with workspace names that may collapse into
        # one name are put into the same partition. This can avoid race conditions
        # between different Python threads.
        # We do this before deduplication (unique) below as sorting has an influence
        # on which duplicates are kept.
        sort_fields = bulk_document.get("sort", [])
        if sort_fields:
            self.logger.info(
                "Start sorting of bulk document data frame based on fields (columns) -> %s...",
                str(sort_fields),
            )
            data.sort(sort_fields=sort_fields, inplace=True)
            self.logger.info(
                "Sorting of bulk document data frame based on fields (columns) -> %s completed!",
                str(sort_fields),
            )

        # Check if duplicate rows for given fields should be removed. It is
        # important to do this after sorting as Pandas always keep the first occurance,
        # so ordering plays an important role in deduplication!
        unique_fields = bulk_document.get("unique", [])
        if unique_fields:
            self.logger.info(
                "Starting deduplication of data frame for bulk documents with unique fields -> %s. Size of data frame before deduplication -> %s.",
                str(unique_fields),
                str(len(data)),
            )
            data.deduplicate(unique_fields=unique_fields, inplace=True)
            self.logger.info(
                "Size of data frame after deduplication -> %s.",
                str(len(data)),
            )

        # Read name field from payload:
        name_field = bulk_document.get("name", None)
        if not name_field:
            self.logger.error(
                "Bulk document needs a name field! Skipping to next bulk document...",
            )
            success = False
            continue

        self._log_header_callback(
            text="Process bulk documents -> '{}' from data source -> '{}'".format(
                name_field,
                data_source_name,
            ),
            char="-",
        )

        # Read optional description field from payload:
        description_field = bulk_document.get("description", None)

        # Read the optional categories payload:
        categories = bulk_document.get("categories", None)
        if not categories:
            self.logger.info(
                "Bulk document payload has no category data! Will leave category attributes empty...",
            )

        # Should existing documents be updated? False (= no) is the default.
        operations = bulk_document.get("operations", ["create"])

        self.logger.info(
            "Bulk create documents (name field -> '%s', operations -> %s)...",
            name_field,
            str(operations),
        )

        # See if bulkWorkspace definition has a specific thread number
        # otherwise it is read from a global environment variable
        bulk_thread_number = int(
            bulk_document.get("thread_number", BULK_THREAD_NUMBER),
        )

        partitions = data.partitionate(bulk_thread_number)

        # Create a list to hold the threads
        threads = []
        results = []

        # Define source OTCS object and authenticate once and pass it to all workers if needed
        if bulk_document.get("source_type", "URL").lower() == "contentserver":
            if bulk_document.get("cs_hostname") is None:
                source_otcs = None
                self.logger.error(
                    "Required information for source type ContentServer is not configured -> cs_hostname",
                )
                success = False
                continue

            if bulk_document.get("cs_username") is None:
                source_otcs = None
                self.logger.error(
                    "Required information for source type ContentServer is not configured -> cs_username",
                )
                success = False
                continue

            if bulk_document.get("cs_password") is None:
                source_otcs = None
                self.logger.error(
                    "Required information for source type ContentServer is not configured -> cs_password",
                )
                success = False
                continue

            self.logger.info(
                "Generating reusable OTCS instance for bulk processing...",
            )
            source_otcs = OTCS(
                protocol=bulk_document.get("cs_protocol", "https"),
                hostname=bulk_document.get("cs_hostname"),
                port=bulk_document.get("cs_port", "443"),
                base_path=bulk_document.get("cs_basepath", "/cs/cs"),
                username=bulk_document.get("cs_username"),
                password=bulk_document.get("cs_password"),
                logger=self.logger,
            )
            source_otcs.authenticate()
        # end if bulk_document.get("source_type", "URL").lower() == "contentserver"
        else:
            source_otcs = None

        # Create and start a thread for each partition
        for index, partition in enumerate(partitions, start=1):
            # start a thread executing the process_bulk_documents_worker() method below:
            thread = threading.Thread(
                name=f"{section_name}_{index:02}",
                target=self.thread_wrapper,
                args=(
                    self.process_bulk_documents_worker,
                    bulk_document,
                    partition,
                    name_field,
                    description_field,
                    categories,
                    operations,
                    results,
                    source_otcs,
                ),
            )
            self.logger.info("Starting thread -> '%s'...", str(thread.name))
            thread.start()
            threads.append(thread)
            # Avoid that all threads start at the exact same time with
            # potentially expired cookies that cases race conditions:
            time.sleep(1)
        # end for index, partition in enumerate(partitions, start=1)

        # Wait for all threads to complete
        for thread in threads:
            self.logger.info(
                "Waiting for thread -> '%s' to complete...",
                str(thread.name),
            )
            thread.join()
            self.logger.info("Thread -> '%s' has completed.", str(thread.name))

        if "documents" not in bulk_document:
            bulk_document["documents"] = {}

        # Print statistics for each thread. In addition,
        # check if all threads have completed without error / failure.
        # If there's a single failure in on of the thread results we
        # set 'success' variable to False.
        results.sort(key=lambda x: x["thread_name"])
        for result in results:
            self.logger.info(
                "Thread -> '%s' completed with overall status '%s'.",
                str(result["thread_name"]),
                "successful" if result["success"] else "failed",
            )
            self.logger.info(
                "Thread -> '%s' processed %s data rows with %s successful, %s failed, and %s skipped.",
                str(result["thread_name"]),
                str(
                    result["success_counter"] + result["failure_counter"] + result["skipped_counter"],
                ),
                str(result["success_counter"]),
                str(result["failure_counter"]),
                str(result["skipped_counter"]),
            )
            self.logger.info(
                "Thread -> '%s' created %s documents, updated %s documents, and deleted %s documents.",
                str(result["thread_name"]),
                str(result["create_counter"]),
                str(result["update_counter"]),
                str(result["delete_counter"]),
            )

            if not result["success"]:
                success = False
            # Record all generated documents. This should allow us
            # to restart in case of failures and avoid trying to
            # uploading that have been successfully uploaded before.
            bulk_document["documents"].update(result["documents"])
        # end for result in results
        self._log_header_callback(
            text="Completed processing of bulk documents -> '{}' using data source -> '{}'".format(
                name_field,
                data_source_name,
            ),
            char="-",
        )

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._bulk_documents,
    )

    return success

process_bulk_documents_worker(bulk_document, partition, name_field, description_field, categories=None, operations=None, results=None, source_otcs=None)

Thread worker to download + create documents in bulk.

Each worker thread gets a partition of the rows that include the data required for the document creation.

Parameters:

Name	Type	Description	Default
bulk_document	dict	The bulkDocument payload element.	required
partition	DataFrame	Data partition with rows to process by this thread.	required
name_field	str	The payload field where the document name is stored.	required
description_field	str	The payload field where the document description is stored.	required
categories	list	A list of category dictionaries.	None
operations	list	A list of operations that should be applyed on workspaces. Possible values: "create", "update", "delete", "recreate".	None
results	list	A mutable list of thread results.	None
source_otcs	OTCS | None	Provide the OTCS object if a direct download from Content Server is done.	None

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_documents_worker")
def process_bulk_documents_worker(
    self,
    bulk_document: dict,
    partition: pd.DataFrame,
    name_field: str,
    description_field: str,
    categories: list | None = None,
    operations: list | None = None,
    results: list | None = None,
    source_otcs: OTCS | None = None,
) -> None:
    """Thread worker to download + create documents in bulk.

    Each worker thread gets a partition of the rows that include
    the data required for the document creation.

    Args:
        bulk_document (dict):
            The bulkDocument payload element.
        partition (pd.DataFrame):
            Data partition with rows to process by this thread.
        name_field (str):
            The payload field where the document name is stored.
        description_field (str):
            The payload field where the document description is stored.
        categories (list, optional):
            A list of category dictionaries.
        operations (list, optional):
            A list of operations that should be applyed on workspaces.
            Possible values: "create", "update", "delete", "recreate".
        results (list, optional):
            A mutable list of thread results.
        source_otcs (OTCS | None, optional):
            Provide the OTCS object if a direct download from Content Server is done.

    """

    thread_id = threading.get_ident()
    thread_name = threading.current_thread().name

    self.logger.info(
        "Start working on data frame partition of size -> %s to bulk create documents...",
        str(len(partition)),
    )

    # Avoid linter warnings - so make parameter default None while we
    # actually want ["create"] to be the default:
    if operations is None:
        operations = ["create"]

    # get the specific update operations given in the payload
    # if not specified we do the following update operations.
    # The 'purge' operation needs to be specified explicitly.
    update_operations = bulk_document.get(
        "update_operations",
        ["name", "description", "categories", "nickname", "version"],
    )

    result = {}
    result["thread_id"] = thread_id
    result["thread_name"] = thread_name
    result["success_counter"] = 0
    result["failure_counter"] = 0
    result["skipped_counter"] = 0
    result["create_counter"] = 0
    result["update_counter"] = 0
    result["delete_counter"] = 0
    result["documents"] = {}
    result["success"] = True

    # Check if documents have been processed before, e.i. testing
    # if a "documents" key exists and if it is pointing to a non-empty list:
    # Additionally we check that workspace updates are not enforced:
    if (
        bulk_document.get("documents")
        and "update" not in operations
        and "delete" not in operations
        and "recreate" not in operations
    ):
        existing_documents = bulk_document["documents"]
        self.logger.info(
            "Found %s already processed documents. Try to complete the job...",
            str(len(existing_documents)),
        )
    else:
        existing_documents = {}

    # See if external creation and modification dates are in the payload data:
    external_modify_date_field = bulk_document.get("external_modify_date")
    external_create_date_field = bulk_document.get("external_create_date")

    # See if we have a key field to uniquely identify an existing document:
    key_field = bulk_document.get("key")

    # Get dictionary of replacements for bulk document creations
    # this we will be used of all places data is read from the
    # data frame. Each dictionary item has the field name as the
    # dictionary key and a list of regular expressions as dictionary value
    replacements = bulk_document.get("replacements")

    # In case the name cannot be resolved we allow to
    # specify an alternative name field in the payload.
    name_field_alt = bulk_document.get("name_alt")

    # If download_name field is not in payload we use name_field instead.
    # It can still be that download_name is "" as name_field is only
    # used if the entry for "download_name" is not in payload at all.
    download_name_field = bulk_document.get("download_name", name_field)

    # Determine if we want to automatically extract ZIP files and upload
    # them as folders / documents:
    extract_zip = bulk_document.get("extract_zip", False)

    # Document names are limited in terms of allowed characters.
    # E.g. if you don't want any path elements and "/" dividers
    # in the document name you could use name_regex = r".*/" in the payload.
    document_name_additional_regex_list = [bulk_document.get("name_regex", r"")]

    # Fetch the nickname field from the payload (if it is specified):
    nickname_field = bulk_document.get("nickname")

    # Nicknames are very limited in terms of allowed characters.
    # For nicknames we need additional regexp as we need to
    # replace all non-alphanumeric, non-space characters with ""
    # We also preserve hyphens in the first step to replace
    # them below with underscores. This is important to avoid
    # that different spellings of names produce different nicknames.
    # We want spellings with spaces match spellings with hyphens.
    # For this the workspace names have a regexp "-| " in the payload.
    nickname_additional_regex_list = [r"[^\w\s-]"]

    total_rows = len(partition)

    # Process all rows in the partition that was given to this thread:
    for index, row in partition.iterrows():
        # Calculate percentage of completion:
        percent_complete = ((partition.index.get_loc(index) + 1) / total_rows) * 100

        self.logger.info(
            "Processing data row -> %s for bulk document creation (%.2f %% complete)...",
            str(index),
            percent_complete,
        )

        # Clear variables to ensure clean state for each row:
        parent_id = None
        document_id = None

        # Create a copy of the mutable operations list as we may
        # want to modify it per row:
        row_operations = list(operations)

        # Check if all data conditions to create the document are met
        conditions = bulk_document.get("conditions")
        if conditions:
            evaluated_condition = self.evaluate_conditions(
                conditions=conditions,
                row=row,
            )
            if not evaluated_condition:
                self.logger.info(
                    "Condition for bulk document row -> %s not met. Skipping row for document creation...",
                    str(index),
                )
                result["skipped_counter"] += 1
                continue

        # Check if all data conditions to create the document are met:
        if "create" in row_operations or "recreate" in row_operations:
            conditions_create = bulk_document.get("conditions_create")
            if conditions_create:
                evaluated_conditions_create = self.evaluate_conditions(
                    conditions=conditions_create,
                    row=row,
                    replacements=replacements,
                )
                if not evaluated_conditions_create:
                    self.logger.info(
                        "Create condition for bulk document row -> %s not met. Excluding create operation for current row...",
                        str(index),
                    )
                    if "create" in row_operations:
                        row_operations.remove("create")
                    if "recreate" in row_operations:
                        row_operations.remove("recreate")
            elif (
                "recreate" in row_operations
            ):  # we still create and recreate without conditions_create. But give a warning for 'recreate' without condition.
                self.logger.warning(
                    "No create condition provided but 'recreate' operation requested. This will recreate all existing documents!",
                )

        # Check if all data conditions to delete the document are met:
        if "delete" in row_operations:
            conditions_delete = bulk_document.get("conditions_delete")
            if conditions_delete:
                evaluated_conditions_delete = self.evaluate_conditions(
                    conditions=conditions_delete,
                    row=row,
                    replacements=replacements,
                )
                if not evaluated_conditions_delete:
                    self.logger.info(
                        "Delete condition for bulk document row -> %s not met. Excluding delete operation for current row...",
                        str(index),
                    )
                    row_operations.remove("delete")
            else:  # without delete_conditions we don't delete!!
                self.logger.warning(
                    "Delete operation requested for bulk documents but conditions for deletion are missing! (specify 'conditions_delete'!)",
                )
                row_operations.remove("delete")

        # Check if all data conditions to update the document are met:
        if "update" in row_operations:
            conditions_update = bulk_document.get("conditions_update")
            if conditions_update:
                evaluated_conditions_update = self.evaluate_conditions(
                    conditions=conditions_update,
                    row=row,
                    replacements=replacements,
                )
                if not evaluated_conditions_update:
                    self.logger.info(
                        "Update condition for bulk document row -> %s not met. Excluding update operation for current row...",
                        str(index),
                    )
                    row_operations.remove("update")

        # Determine the document name:
        document_name = self.replace_bulk_placeholders(
            input_string=name_field,
            row=row,
            replacements=replacements,
            additional_regex_list=document_name_additional_regex_list,
        )
        # If we cannot get the document_name from the
        # name_field we try the alternative name field
        # (if specified in payload via "name_alt"):
        if not document_name and name_field_alt:
            self.logger.debug(
                "Row -> %s does not have the data to resolve the placeholders in document name -> %s! Trying alternative name field -> %s...",
                str(index),
                name_field,
                name_field_alt,
            )
            # The additional regex list will make sure
            # that any path or URL element is stripped from
            # the value in the row element:
            document_name = self.replace_bulk_placeholders(
                input_string=name_field_alt,
                row=row,
                replacements=replacements,
                additional_regex_list=document_name_additional_regex_list,
            )
        if not document_name:
            self.logger.error(
                "Row -> %s does not have the data to resolve the placeholders in document name -> %s%s!",
                str(index),
                name_field,
                (" nor in alternative name field -> " + name_field_alt if name_field_alt else ""),
            )
            result["skipped_counter"] += 1
            continue

        # Cleanse the document name (allowed characters, maximum length):
        document_name = OTCS.cleanse_item_name(document_name)

        download_name = ""
        if download_name_field:
            # The additional regex list will make sure
            # that any path or URL element is stripped from
            # the value in the row element:
            download_name = self.replace_bulk_placeholders(
                input_string=download_name_field,
                row=row,
                replacements=replacements,
                additional_regex_list=document_name_additional_regex_list,
            )
        if not download_name:
            self.logger.warning(
                "Download name is empty or row -> %s does not have the data to resolve the placeholders in document download name -> '%s'. Using -> '%s' instead!",
                str(index),
                download_name_field,
                document_name,
            )
            # in this case we use the document_name also as the download_name:
            download_name = document_name

        # This is an optimization. We check if the document was created
        # in a former run. This helps if the customizer is re-run:
        if document_name and document_name in existing_documents:
            self.logger.info(
                "Document -> '%s' does already exist and has ID -> %s. Skipping...",
                document_name,
                existing_documents[document_name],
            )
            result["skipped_counter"] += 1
            continue

        # Determine the description field:
        description = (
            self.replace_bulk_placeholders(
                input_string=description_field,
                row=row,
            )
            if description_field
            else ""
        )

        # Determine the external creation date (if any):
        if external_create_date_field:
            external_create_date = self.replace_bulk_placeholders(
                input_string=external_create_date_field,
                row=row,
            )
        else:
            external_create_date = None

        # Determine the external modification date (if any):
        if external_modify_date_field:
            external_modify_date = self.replace_bulk_placeholders(
                input_string=external_modify_date_field,
                row=row,
            )
        else:
            external_modify_date = None

        # Determine the key field (if any):
        key = self.replace_bulk_placeholders(input_string=key_field, row=row) if key_field else None

        # check if workspace with this nickname does already exist.
        # we also store the nickname to assign it to the new workspace:
        if nickname_field:
            nickname = self.replace_bulk_placeholders(
                input_string=nickname_field,
                row=row,
                replacements=replacements,
                additional_regex_list=nickname_additional_regex_list,
            )
        else:
            nickname = None

        # Based on the evaluation of conditions_create, conditions_delete,
        # and conditions_update we could end up with an empty operations list.
        # In such cases we skip the further processing of this row:
        if not row_operations:
            self.logger.info(
                "Skip document -> '%s' as row operations is empty (no create, update, delete or recreate). This may be because conditions_create, conditions_delete, and conditions_update have been evaluated to false for this row.",
                document_name,
            )
            result["skipped_counter"] += 1
            continue

        self.logger.info(
            "Bulk process document -> '%s' using effective operations -> %s...",
            document_name,
            str(row_operations),
        )

        # We only need to download the files if we want more than just
        # the "delete" operation:
        if any(operation in ["create", "update", "recreate"] for operation in row_operations):
            file_name, mime_type = self.download_bulk_document(
                bulk_document=bulk_document,
                download_name=download_name,
                row=row,
                source_otcs=source_otcs,
                result=result,
            )

            if file_name is None:
                result["success"] = False
                result["failure_counter"] += 1
                continue

            self.logger.debug("Download name -> '%s'", download_name)
            self.logger.debug("Document name -> '%s'", document_name)
            self.logger.debug("File name -> '%s'", file_name)
            self.logger.debug("Mime type -> '%s'", mime_type)
        else:
            # if we only do a delete operation then there's no need for
            # processing local files for upload:
            file_name = None
            mime_type = None

        # Now we traverse a list of (multiple) workspaces
        # the document should be uploaded to:
        success = True
        workspaces = bulk_document.get("workspaces", [])
        for workspace in workspaces:
            # success will only be false if a config problem (failure)
            # and not just a data problem (skipped) has occured:
            parent_id, success = self.get_bulk_document_location(
                workspace=workspace,
                row=row,
                index=index,
                replacements=replacements,
            )

            if parent_id is None:
                continue

            #
            # Create the document in the target folder specified by parent_id:
            #

            # Check if the document does already exist.
            # First we try to look up the document by a unique
            # key that does not change over time.
            # For this we expect a "key" value to be defined for the
            # bulkDocument and one of the category / attribute item
            # to be marked with "is_key" = True. If we don't find the
            # document via key we try via parent and name.
            self.logger.info(
                "Check if document -> '%s' is already in target folder with ID -> %s%s...",
                document_name,
                parent_id,
                " (using key -> {})".format(key) if key is not None else "",
            )
            # Initialize variables - this is important!
            document_old_name = None
            document_id = None
            handle_name_clash = False
            document_modify_date = None

            # We have 4 cases to differentiate:

            # 1. key given + key found = name irrelevant, item exist
            # 2. key given + key not found = if name exist it is a name clash
            # 3. no key given + name found = item exist
            # 4. no key given + name not found = item does not exist

            # We are NOT trying to compensate for a failed key lookup with a name lookup!
            # Names are only relevant if no key is in payload!

            if key:
                key_attribute = next(
                    (cat_attr for cat_attr in categories if cat_attr.get("is_key", False) is True),
                    None,
                )
                if key_attribute:
                    cat_name = key_attribute.get("name", None)
                    att_name = key_attribute.get("attribute", None)
                    set_name = key_attribute.get("set", None)
                    # Try to find the node that has the given key attribute value:
                    response = self._otcs_frontend.lookup_nodes(
                        parent_node_id=parent_id,
                        category=cat_name,
                        attribute=att_name,
                        attribute_set=set_name,
                        value=key,
                    )
                    document_id = self._otcs_frontend.get_result_value(
                        response=response,
                        key="id",
                    )
                    if document_id:
                        # Case 1: key given + key found = name irrelevant, item exist
                        self.logger.info(
                            "Found existing document with the key value -> '%s' in category -> '%s' and attribute -> '%s' in folder with ID -> %s.",
                            key,
                            cat_name,
                            att_name,
                            parent_id,
                        )
                        document_old_name = self._otcs_frontend.get_result_value(
                            response=response,
                            key="name",
                        )
                        if (
                            document_old_name != document_name
                            and document_old_name != document_name + " (" + key + ")"
                        ):
                            self.logger.info(
                                "Existing document with ID -> %s has the old name -> '%s' which is different from the new name -> '%s.'",
                                document_id,
                                document_old_name,
                                document_name,
                            )
                        else:
                            # if there was a name clash before and got resolved
                            # then we need to stick to the resolved name also for updates:
                            if document_old_name == document_name + " (" + key + ")":
                                self.logger.info(
                                    "Document name conflict was resolved before. Changing document name to -> '%s' to match former resolution.",
                                    document_old_name,
                                )
                                document_name = document_old_name
                            document_old_name = None
                        # We get the modify date of the existing document.
                        document_modify_date = self._otcs_frontend.get_result_value(
                            response=response,
                            key="modify_date",
                        )
                    else:
                        # Case 2: key given + key not found = if name exist it is a name clash
                        self.logger.info(
                            "Couldn't find existing document with the key value -> '%s' in category -> '%s' and attribute -> '%s' in folder with ID -> %s.",
                            key,
                            cat_name,
                            att_name,
                            parent_id,
                        )
                        # Keep track that we need to handle a name clash as based on the key
                        # the document should not exist.
                        handle_name_clash = True
                else:
                    self.logger.error(
                        "Document has a key -> '%s' defined but none of the category attributes is marked as a key attribute ('is_key' is missing)!",
                        key,
                    )
                    success = False
                    continue
            # end if key
            else:
                # If we haven't a key we try by parent + name
                response = self._otcs_frontend.get_node_by_parent_and_name(
                    name=document_name,
                    parent_id=parent_id,
                )
                document_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
                if document_id:
                    # Case 3: no key given + name found = item exist
                    self.logger.info(
                        "Found existing document -> '%s' (%s) in parent with ID -> %s.",
                        document_name,
                        document_id,
                        parent_id,
                    )
                    # We get the modify date of the existing document.
                    document_modify_date = self._otcs_frontend.get_result_value(
                        response=response,
                        key="modify_date",
                    )
                elif not extract_zip or mime_type != "application/zip":
                    # Case 4: no key given + name not found = item does not exist
                    self.logger.info(
                        "Cannot find document -> '%s' in parent with ID -> %s.",
                        document_name,
                        parent_id,
                    )
                else:
                    # Edge case: no key given + name not found but it is a zip file that got extracted.
                    # So we don't really know if it exists yet.
                    self.logger.info(
                        "File -> '%s' is a zip file that potentially has been extracted into parent with ID -> %s before. Existence of extracted files will be determined later.",
                        document_name,
                        parent_id,
                    )

            # Check if we want to recreate an existing document
            # then we handle the "delete" part of "recreate" first:
            if document_id and "recreate" in row_operations:
                response = self._otcs_frontend.delete_node(
                    node_id=document_id,
                    purge=True,
                )
                if not response:
                    self.logger.error(
                        "Failed to recreate existing document -> '%s' (%s) under parent ID -> %s! Delete failed.",
                        (document_name if document_old_name is None else document_old_name),
                        document_id,
                        parent_id,
                    )
                    success = False
                    continue
                self.logger.info(
                    "Deleted existing document -> '%s' (%s) as part of the recreate operation.",
                    (document_name if document_old_name is None else document_old_name),
                    document_id,
                )
                result["delete_counter"] += 1
                document_id = None

            # Check if document does not exists - then we create a new document
            # if this is requested ("create" value in operations list in payload)
            if not document_id and ("create" in row_operations or "recreate" in row_operations):
                # The document does not exist in Content Server - so we
                # upload it now...

                # for Case 2 (we looked up the document by key but could have name collisions)
                # we need to see if we have name collisions:
                if handle_name_clash:
                    response = self._otcs_frontend.check_node_name(
                        parent_id=parent_id,
                        node_name=document_name,
                    )
                    # result is a list of names that clash - if it is empty we have no clash
                    if response.get("results"):
                        # We need to handle a race condition here: it could be that
                        # the document has been created by another thread. This is because
                        # the key could be in multiple rows of the data frame depending how the data loader works.
                        # We can also not assume that this can be resolved by making the key unique
                        # in the data source as it could be that a data set with the same key should
                        # go to multiple workspaces that then are folded into one by synonym resolution.
                        conflicting_node_id = response["results"][0].get("id")
                        conflicting_key = self._otcs_frontend.get_category_value_by_name(
                            node_id=conflicting_node_id,
                            category_name=cat_name,
                            attribute_name=att_name,
                            set_name=set_name,
                        )
                        if key == conflicting_key:
                            # We have a race condition as the two documents don't really clash but are identical.
                            # Just skip uploading the same document once more.
                            self.logger.warning(
                                "Detected a race condition in name clash handling! Skipping document upload!",
                            )
                            continue

                        # We add the suffix with the key which should be unique:
                        self.logger.warning(
                            "Document -> '%s' does already exist in workspace folder with ID -> %s and we need to handle the name clash and use name -> '%s'",
                            document_name,
                            parent_id,
                            document_name + " (" + key + ")",
                        )
                        document_name = document_name + " (" + key + ")"

                # If category data is in payload we substitute
                # the values with data from the current data row:
                if categories:
                    # Make sure the threads are not changing data structures that
                    # are shared between threads. categories is a list of dicts.
                    # list and dicts are "mutable" data structures in Python!
                    worker_categories = self.process_bulk_categories(
                        row=row,
                        index=index,
                        categories=categories,
                        replacements=replacements,
                    )
                    document_category_data = self.prepare_category_data(
                        categories_payload=worker_categories,
                        source_node_id=parent_id,
                    )
                # end if categories
                else:
                    document_category_data = {}

                response = self._otcs_frontend.upload_file_to_parent(
                    file_url=file_name,
                    file_name=document_name,
                    mime_type=mime_type,
                    parent_id=int(parent_id),
                    category_data=document_category_data,
                    description=description,
                    external_create_date=external_create_date,
                    external_modify_date=external_modify_date,
                    extract_zip=extract_zip,
                    replace_existing="update" in row_operations and "version" in update_operations,
                    show_error=False,
                )
                document_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
                if not document_id:
                    # We may have a race condition here. Double check the document does not exist
                    # before issuing an error:
                    response = self._otcs_frontend.get_node_by_parent_and_name(
                        parent_id=int(parent_id),
                        name=document_name,
                    )
                    document_id = self._otcs_frontend.get_result_value(
                        response=response,
                        key="id",
                    )
                if not document_id:
                    self.logger.error(
                        "Cannot create document -> '%s' (download name -> '%s') in folder/workspace with ID -> %s!",
                        document_name,
                        download_name,
                        parent_id,
                    )
                    success = False
                    continue
                self.logger.info(
                    "Created document -> '%s' (ID -> %s, file -> '%s', mime type -> '%s'%s%s) in parent with ID -> %s.",
                    document_name,
                    document_id,
                    file_name,
                    mime_type,
                    ", description -> {}".format(description) if description else "",
                    ", size -> {}".format(os.path.getsize(file_name))
                    if os.path.exists(file_name)
                    else "",  # this can happen for files in zip files that have been cleaned up already
                    parent_id,
                )
                result["create_counter"] += 1

                # Check if metadata or images embeddings need to be updated for the document:
                aviator_metadata = bulk_document.get("aviator_metadata", False)
                aviator_images = bulk_document.get("aviator_images", False)
                aviator_remove_existing = bulk_document.get("aviator_remove_existing", False)
                if aviator_metadata or aviator_images or aviator_remove_existing:
                    self.logger.info(
                        "%s Content Aviator %s%s%s embedding via FEME for document or image -> '%s' (%s)...",
                        "Trigger" if not aviator_remove_existing else "Remove",
                        "metadata" if aviator_metadata else "",
                        " and " if aviator_metadata and aviator_images else "",
                        "image" if aviator_images else "",
                        document_name,
                        document_id,
                    )

                    self._otcs_frontend.aviator_embed_metadata(
                        node_id=document_id,
                        workspace_metadata=False,
                        document_metadata=aviator_metadata,
                        images=aviator_images,
                        wait_for_completion=False,
                        remove_existing=aviator_remove_existing,
                    )

            # end if not workspace_id and "create" in row_operations

            # If updates are requested we update the existing document with
            # a new document version and with fresh metadata from the payload.
            # Additionally we check the external modify date to support
            # incremental load for content that has really changed.
            # In addition we check that "delete" is not requested as otherwise it will
            # never go in elif "delete" ... below (and it does not make sense to update a document
            # that is deleted in the next step...)

            elif (
                document_id
                and "update" in row_operations
                and "delete" not in row_operations  # note the NOT !
                and OTCS.date_is_newer(
                    date_old=document_modify_date,
                    date_new=external_modify_date,
                )
            ):
                # If category data is in payload we substitute
                # the values with data from the current data row.
                # This is only done if "categories" update is not
                # excluded from the update_operations:
                if categories and "categories" in update_operations:
                    # Make sure the threads are not changing data structures that
                    # are shared between threads. categories is a list of dicts.
                    # list and dicts are "mutable" data structures in Python!
                    worker_categories = self.process_bulk_categories(
                        row=row,
                        index=index,
                        categories=categories,
                        replacements=replacements,
                    )
                    # document_category_data = self.prepare_item_create_form(
                    #     parent_id=parent_id,
                    #     categories=worker_categories,
                    #     subtype=self._otcs_frontend.ITEM_TYPE_DOCUMENT,
                    # )
                    # Transform the payload structure into the format
                    # the OTCS REST API requires:
                    document_category_data = self.prepare_category_data(
                        categories_payload=worker_categories,
                        source_node_id=document_id,
                        source_is_document=True,
                    )
                    if not document_category_data:
                        self.logger.error(
                            "Failed to prepare the updated category data for document -> '%s' (%s)!",
                            document_name,
                            str(document_id),
                        )
                        success = False
                        continue  # for index, row in partition.iterrows()
                # end if categories
                else:
                    document_category_data = {}

                self.logger.info(
                    "Update existing document -> '%s' (%s) with operations -> %s...",
                    document_old_name if document_old_name else document_name,
                    document_id,
                    str(update_operations),
                )
                # check if adding a version is requested:
                if "version" in update_operations:
                    response = self._otcs_frontend.add_document_version(
                        node_id=document_id,
                        file_url=file_name,
                        file_name=document_name,
                        mime_type=mime_type,
                        description=description,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to add new version to existing document -> '%s' (%s)!",
                            (document_old_name if document_old_name else document_name),
                            document_id,
                        )
                        success = False
                        continue
                if "purge" in update_operations:
                    max_versions = bulk_document.get("max_versions", 1)
                    response = self._otcs_frontend.purge_document_versions(
                        node_id=document_id, versions_to_keep=max_versions
                    )
                    if not response:
                        self.logger.error(
                            "Failed to purge versions of document -> '%s' (%s) to %d version%s!",
                            (document_old_name if document_old_name else document_name),
                            document_id,
                            max_versions,
                            "s" if max_versions > 1 else "",
                        )
                        success = False
                        continue
                response = self._otcs_frontend.update_item(
                    node_id=document_id,
                    parent_id=None,  # None = do not move item
                    item_name=(document_name if "name" in update_operations else None),
                    item_description=(description if "description" in update_operations else None),
                    category_data=(document_category_data if "categories" in update_operations else None),
                    external_create_date=external_create_date,
                    external_modify_date=external_modify_date,
                )
                if not response:
                    self.logger.error(
                        "Failed to update metadata of existing document -> '%s' (%s) with metadata -> %s!",
                        (document_old_name if document_old_name else document_name),
                        document_id,
                        str(document_category_data),
                    )
                    success = False
                    continue
                self.logger.info(
                    "Updated existing document -> '%s' (ID -> %s, file -> '%s', mime type -> '%s', description -> '%s'%s).",
                    document_name,
                    document_id,
                    file_name,
                    mime_type,
                    description,
                    ", size -> {}".format(os.path.getsize(file_name))
                    if os.path.exists(file_name)
                    else "",  # this can happen for files in zip files that have been cleaned up already
                )
                result["update_counter"] += 1

                # Check if metadata or image embeddings need to be updated for the document:
                aviator_metadata = bulk_document.get("aviator_metadata", False)
                aviator_images = bulk_document.get("aviator_images", False)
                aviator_remove_existing = bulk_document.get("aviator_remove_existing", False)
                if aviator_metadata or aviator_images or aviator_remove_existing:
                    self.logger.info(
                        "%s Content Aviator %s%s%s embedding via FEME for document or image -> '%s' (%s)...",
                        "Update" if not aviator_remove_existing else "Remove",
                        "metadata" if aviator_metadata else "",
                        " and " if aviator_metadata and aviator_images else "",
                        "image" if aviator_images else "",
                        document_name,
                        document_id,
                    )

                    self._otcs_frontend.aviator_embed_metadata(
                        node_id=document_id,
                        workspace_metadata=False,
                        document_metadata=aviator_metadata,
                        images=aviator_images,
                        wait_for_completion=False,
                        remove_existing=aviator_remove_existing,
                    )
            # end if workspace_id and "update" in row_operations
            elif document_id and "delete" in row_operations:
                # We delete with immediate purging to keep recycle bin clean
                # and to not run into issues with nicknames used in deleted items:
                response = self._otcs_frontend.delete_node(
                    node_id=document_id,
                    purge=True,
                )
                if not response:
                    self.logger.error(
                        "Failed to bulk delete existing document -> '%s' (%s)!",
                        (document_old_name if document_old_name else document_name),
                        document_id,
                    )
                    success = False
                    continue
                self.logger.info(
                    "Successfully deleted existing document -> '%s' (%s).",
                    document_old_name if document_old_name else document_name,
                    document_id,
                )
                result["delete_counter"] += 1
                document_id = None
            # end elif document_id and "delete" in row_operations

            # this is the plain old "if it does exist we just skip it" case:
            elif document_id:
                self.logger.info(
                    "Skipped existing document -> '%s' (%s).",
                    document_old_name if document_old_name else document_name,
                    document_id,
                )
            # this is the case where we just want to operate on existing documents (update or delete)
            # but they do not exist:
            elif not document_id and ("update" in row_operations or "delete" in row_operations):
                result["skipped_counter"] += 1
                self.logger.info(
                    "Skipped update/delete of non-existing document -> '%s'.",
                    document_old_name if document_old_name else document_name,
                )

            # The following code is executed for all operations
            # (create, recreate, update, delete):

            # Check if we want to set / update the nickname of the document.
            # Precondition is we have determined a nickname, we have the document ID
            # and the update of the nickname is either required (create, recreate)
            # or requested (update_operations include "nickname"):
            if (
                nickname
                and document_id
                and (
                    "create" in row_operations
                    or "recreate" in row_operations
                    or ("update" in row_operations and "nickname" in update_operations)
                )
            ):
                response = self._otcs_frontend.set_node_nickname(
                    node_id=document_id,
                    nickname=nickname,
                    show_error=True,
                )
                if not response:
                    self.logger.error(
                        "Failed to assign nickname -> '%s' to document -> '%s'!",
                        nickname,
                        document_name,
                    )
            if document_id is not None:
                # Record the document name and ID to allow to read it from failure file
                # and speedup the process.
                result["documents"][document_name] = document_id

        # end for workspaces

        # We want the success, failure and skip counter
        # to consider only complete data frame rows. In
        # case of bulk documents we can potentially create
        # update or delete more than 1 document per row.
        # So we use the "success" variable to accumulate
        # this for all documents per row:
        if not success:
            result["success"] = False
            result["failure_counter"] += 1
        elif document_name not in result["documents"]:
            self.logger.info(
                "Bulk document -> '%s' was not uploaded to any workspace.",
                document_name,
            )
            result["skipped_counter"] += 1
        else:
            result["success_counter"] += 1

        # Make sure no temp documents are piling up except
        # we want it (e.g. if using cloud document storage):
        if file_name and os.path.exists(file_name) and bulk_document.get("delete_download", True):
            os.remove(file_name)
    # end for index, row in partition.iterrows()

    self.logger.info("End working...")

    results.append(result)

process_bulk_items(section_name='bulkItems')

Process bulkItems in payload and bulk create them in Content Server (multi-threaded).

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'bulkItems'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_items")
def process_bulk_items(self, section_name: str = "bulkItems") -> bool:
    """Process bulkItems in payload and bulk create them in Content Server (multi-threaded).

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise.

    """

    if not self._bulk_items:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    # For efficient idem-potent operation we may want to see which workspaces
    # have already been processed before:
    if self.check_status_file(
        payload_section_name=section_name,
        payload_specific=True,
        prefix="failure_",
    ):
        self.logger.info(
            "Found failure file. Trying to reprocess just the failed ones...",
        )
        # Read payload from where we left it last time
        self._bulk_items = self.get_status_file(
            payload_section_name=section_name,
            prefix="failure_",
        )
        if not self._bulk_items:
            self.logger.error(
                "Cannot load existing bulk items failure file. Bailing out!",
            )
            return False

    success: bool = True

    for bulk_item in self._bulk_items:
        # Check if element has been disabled in payload (enabled = false).
        # In this case we skip the element:
        enabled = bulk_item.get("enabled", True)
        if not enabled:
            self.logger.info("Payload for bulk items is disabled. Skipping...")
            continue

        # The payload element must have a "data_source" key:
        data_source_name = bulk_item.get("data_source", None)
        if not data_source_name:
            self.logger.error("Missing data source name in bulk items!")
            success = False
            continue

        copy_data_source = bulk_item.get("copy_data_source", False)
        force_reload = bulk_item.get("force_reload", True)

        # Load and prepare the data source for the bulk processing:
        if copy_data_source:
            self.logger.info(
                "Take a copy of data source -> %s to avoid side-effects for repeative usage of the data source...",
                data_source_name,
            )
            data = Data(
                self.process_bulk_datasource(
                    data_source_name=data_source_name,
                    force_reload=force_reload,
                ),
                logger=self.logger,
            )
        else:
            self.logger.info(
                "Use original data source -> '%s' and not do a copy.",
                bulk_item["data_source"],
            )
            data = self.process_bulk_datasource(
                data_source_name=data_source_name,
                force_reload=force_reload,
            )
        if (
            data is None
            or data.get_data_frame() is None  # the 2nd check is important for the "copy_data_source" case!
        ):  # important to use "is None" here!
            self.logger.error(
                "Failed to load data source -> '%s' for bulk items!",
                data_source_name,
            )
            success = False
            continue

        # Check if fields with list substructures should be exploded.
        # We may want to do this outside the bulkDatasource to only
        # explode for bulkItems and not for bulkWorkspaces or
        # bulkWorkspaceRelationships:
        explosions = bulk_item.get("explosions", [])
        for explosion in explosions:
            # explode field can be a string or a list
            # exploding multiple fields at once avoids
            # combinatorial explosions - this is VERY
            # different from exploding columns one after the other!
            if "explode_field" not in explosion and "explode_fields" not in explosion:
                self.logger.error("Missing explosion field(s)!")
                continue
            # we want to be backwards compatible...
            explode_fields = (
                explosion["explode_field"] if "explode_field" in explosion else explosion["explode_fields"]
            )
            flatten_fields = explosion.get("flatten_fields", [])
            split_string_to_list = explosion.get("split_string_to_list", False)
            list_splitter = explosion.get("list_splitter", None)
            self.logger.info(
                "Starting explosion of bulk items by field(s) -> %s (type -> %s). Size of data frame before explosion -> %s.",
                explode_fields,
                str(type(explode_fields)),
                str(len(data)),
            )
            data.explode_and_flatten(
                explode_fields=explode_fields,
                flatten_fields=flatten_fields,
                make_unique=False,
                split_string_to_list=split_string_to_list,
                separator=list_splitter,
                reset_index=True,
            )
            self.logger.info(
                "Size of data frame after explosion -> %s.",
                str(len(data)),
            )
        # end for explosion in explosions

        # Keep only selected rows if filters are specified in bulkDocuments.
        # We have this _after_ "explosions" to allow access to subfields as well.
        # We have this _before_ "sorting" and "deduplication" as we may keep the wrong
        # rows otherwise (unique / deduplication always keeps the first matching row).
        # We may want to do this outside the bulkDatasource to only
        # filter for bulkDocuments and not for bulkWorkspaces or
        # bulkWorkspaceRelationships:
        filters = bulk_item.get("filters", [])
        if filters:
            data.filter(conditions=filters)

        # Sort the data frame if "sort" specified in payload. We may want to do this to have a
        # higher chance that rows with workspace names that may collapse into
        # one name are put into the same partition. This can avoid race conditions
        # between different Python threads.
        # We do this before deduplication (unique) below as sorting has an influence
        # on which duplicates are kept.
        sort_fields = bulk_item.get("sort", [])
        if sort_fields:
            self.logger.info(
                "Start sorting of bulk items data frame based on fields (columns) -> %s...",
                str(sort_fields),
            )
            data.sort(sort_fields=sort_fields, inplace=True)
            self.logger.info(
                "Sorting of bulk items data frame based on fields (columns) -> %s completed.",
                str(sort_fields),
            )

        # Check if duplicate rows for given fields should be removed. It is
        # important to do this after sorting as Pandas always keep the first occurance,
        # so ordering plays an important role in deduplication!
        unique_fields = bulk_item.get("unique", [])
        if unique_fields:
            self.logger.info(
                "Starting deduplication of data frame for bulk items with unique fields -> %s. Size of data frame before deduplication -> %s.",
                str(unique_fields),
                str(len(data)),
            )
            data.deduplicate(unique_fields=unique_fields, inplace=True)
            self.logger.info(
                "Size of data frame after deduplication -> %s.",
                str(len(data)),
            )

        # Read name field from payload:
        name_field = bulk_item.get("name", None)
        if not name_field:
            self.logger.error(
                "Bulk item needs a name field! Skipping to next bulk item...",
            )
            success = False
            continue

        self._log_header_callback(
            text="Process bulk items -> '{}' from data source -> '{}'".format(
                name_field,
                data_source_name,
            ),
            char="-",
        )

        # Read optional description field from payload:
        description_field = bulk_item.get("description", None)

        # Read the optional categories payload:
        categories = bulk_item.get("categories", None)
        if not categories:
            self.logger.info(
                "Bulk item payload has no category data! Will leave category attributes empty...",
            )

        # Should existing items be updated? False (= no) is the default.
        operations = bulk_item.get("operations", ["create"])

        self.logger.info(
            "Bulk create items (name field -> %s. Operations -> %s.)",
            name_field,
            str(operations),
        )

        # See if bulkWorkspace definition has a specific thread number
        # otherwise it is read from a global environment variable
        bulk_thread_number = int(
            bulk_item.get("thread_number", BULK_THREAD_NUMBER),
        )

        partitions = data.partitionate(bulk_thread_number)

        # Create a list to hold the threads
        threads = []
        results = []

        # Create and start a thread for each partition
        for index, partition in enumerate(partitions, start=1):
            # start a thread executing the process_bulk_items_worker() method below:
            thread = threading.Thread(
                name=f"{section_name}_{index:02}",
                target=self.thread_wrapper,
                args=(
                    self.process_bulk_items_worker,
                    bulk_item,
                    partition,
                    name_field,
                    description_field,
                    categories,
                    operations,
                    results,
                ),
            )
            self.logger.info("Starting thread -> '%s'...", str(thread.name))
            thread.start()
            threads.append(thread)
            # Avoid that all threads start at the exact same time with
            # potentially expired cookies that cases race conditions:
            time.sleep(1)
        # end for index, partition in enumerate(partitions, start=1)

        # Wait for all threads to complete
        for thread in threads:
            self.logger.info(
                "Waiting for thread -> '%s' to complete...",
                str(thread.name),
            )
            thread.join()
            self.logger.info("Thread -> '%s' has completed.", str(thread.name))

        if "items" not in bulk_item:
            bulk_item["items"] = {}

        # Print statistics for each thread. In addition,
        # check if all threads have completed without error / failure.
        # If there's a single failure in on of the thread results we
        # set 'success' variable to False.
        results.sort(key=lambda x: x["thread_name"])
        for result in results:
            self.logger.info(
                "Thread -> '%s' completed with overall status '%s'.",
                str(result["thread_name"]),
                "successful" if result["success"] else "failed",
            )
            self.logger.info(
                "Thread -> '%s' processed %s data rows with %s successful, %s failed, and %s skipped.",
                str(result["thread_name"]),
                str(
                    result["success_counter"] + result["failure_counter"] + result["skipped_counter"],
                ),
                str(result["success_counter"]),
                str(result["failure_counter"]),
                str(result["skipped_counter"]),
            )
            self.logger.info(
                "Thread -> '%s' created %s items, updated %s items, and deleted %s items.",
                str(result["thread_name"]),
                str(result["create_counter"]),
                str(result["update_counter"]),
                str(result["delete_counter"]),
            )

            if not result["success"]:
                success = False
            # Record all generated items. This should allow us
            # to restart in case of failures and avoid trying to
            # create items that have been successfully created before.
            bulk_item["items"].update(result["items"])
        # end for result in results
        self._log_header_callback(
            text="Completed processing of bulk items -> '{}' using data source -> '{}'".format(
                name_field,
                data_source_name,
            ),
            char="-",
        )

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._bulk_items,
    )

    return success

process_bulk_items_worker(bulk_item, partition, name_field, description_field, categories=None, operations=None, results=None)

Thread worker to create items in bulk.

Each worker thread gets a partition of the rows that include the data required for the item creation.

Parameters:

Name	Type	Description	Default
bulk_item	dict	The bulkItem payload element.	required
partition	DataFrame	Data partition with rows to process by this thread.	required
name_field	str	The payload field where the item name is stored.	required
description_field	str	The payload field where the item description is stored.	required
categories	list	A list of category dictionaries.	None
operations	list	A list of operations that should be applyed on workspaces. Possible values: "create", "update", "delete", "recreate".	None
results	list	A mutable list of thread results.	None

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_items_worker")
def process_bulk_items_worker(
    self,
    bulk_item: dict,
    partition: pd.DataFrame,
    name_field: str,
    description_field: str,
    categories: list | None = None,
    operations: list | None = None,
    results: list | None = None,
) -> None:
    """Thread worker to create items in bulk.

    Each worker thread gets a partition of the rows that include
    the data required for the item creation.

    Args:
        bulk_item (dict):
            The bulkItem payload element.
        partition (pd.DataFrame):
            Data partition with rows to process by this thread.
        name_field (str):
            The payload field where the item name is stored.
        description_field (str):
            The payload field where the item description is stored.
        categories (list, optional):
            A list of category dictionaries.
        operations (list, optional):
            A list of operations that should be applyed on workspaces.
            Possible values: "create", "update", "delete", "recreate".
        results (list, optional):
            A mutable list of thread results.

    """

    thread_id = threading.get_ident()
    thread_name = threading.current_thread().name

    self.logger.info(
        "Start working on data frame partition of size -> %s to bulk create items...",
        str(len(partition)),
    )

    # Avoid linter warnings - so make parameter default None while we
    # actually want ["create"] to be the default:
    if operations is None:
        operations = ["create"]

    result = {}
    result["thread_id"] = thread_id
    result["thread_name"] = thread_name
    result["success_counter"] = 0
    result["failure_counter"] = 0
    result["skipped_counter"] = 0
    result["create_counter"] = 0
    result["update_counter"] = 0
    result["delete_counter"] = 0
    result["items"] = {}
    result["success"] = True

    # Check if items have been processed before, e.i. testing
    # if a "items" key exists and if it is pointing to a non-empty list:
    # Additionally we check that workspace updates are not enforced:
    if (
        bulk_item.get("items")
        and "update" not in operations
        and "delete" not in operations
        and "recreate" not in operations
    ):
        existing_items = bulk_item["items"]
        self.logger.info(
            "Found %s already processed items. Try to complete the job...",
            str(len(existing_items)),
        )
    else:
        existing_items = {}

    # See if external creation and modification dates are in the payload data:
    external_modify_date_field = bulk_item.get("external_modify_date")
    external_create_date_field = bulk_item.get("external_create_date")

    # See if we have a key field to uniquely identify an existing item:
    key_field = bulk_item.get("key")

    # Get dictionary of replacements for bulk item creations
    # this we will be used of all places data is read from the
    # data frame. Each dictionary item has the field name as the
    # dictionary key and a list of regular expressions as dictionary value
    replacements = bulk_item.get("replacements")

    # In case the name cannot be resolved we allow to
    # specify an alternative name field in the payload.
    name_field_alt = bulk_item.get("name_alt")

    # Determine the type of the item (OTCS subtype ID):
    type_field = bulk_item.get("type", self._otcs_frontend.ITEM_TYPE_URL)

    # In case the type is URL, determine the URL field:
    url_field = bulk_item.get("url")

    # In case the type is Shortcut, determine the origin field:
    original_nickname_field = bulk_item.get("original_nickname")
    original_path = bulk_item.get("original_path")

    # Document names are limited in terms of allowed characters.
    # E.g. if you don't want any path elements and "/" dividers
    # in the item name you could use name_regex = r".*/" in the payload.
    item_name_additional_regex_list = [bulk_item.get("name_regex", r"")]

    # Fetch the nickname field from the payload (if it is specified):
    nickname_field = bulk_item.get("nickname")

    # Nicknames are very limited in terms of allowed characters.
    # For nicknames we need additional regexp as we need to
    # replace all non-alphanumeric, non-space characters with ""
    # We also preserve hyphens in the first step to replace
    # them below with underscores. This is important to avoid
    # that different spellings of names produce different nicknames.
    # We want spellings with spaces match spellings with hyphens.
    # For this the workspace names have a regexp "-| " in the payload.
    nickname_additional_regex_list = [r"[^\w\s-]"]

    total_rows = len(partition)

    # Process all rows in the partition that was given to this thread:
    for index, row in partition.iterrows():
        # Calculate percentage of completion:
        percent_complete = ((partition.index.get_loc(index) + 1) / total_rows) * 100

        self.logger.info(
            "Processing data row -> %s for bulk item creation (%.2f %% complete)...",
            str(index),
            percent_complete,
        )

        # Clear variables to ensure clean state for each row:
        parent_id = None
        item_id = None

        # Create a copy of the mutable operations list as we may
        # want to modify it per row:
        row_operations = list(operations)

        # Check if all data conditions to create the item are met
        conditions = bulk_item.get("conditions")
        if conditions:
            evaluated_condition = self.evaluate_conditions(
                conditions=conditions,
                row=row,
            )
            if not evaluated_condition:
                self.logger.info(
                    "Condition for bulk item row -> %s not met. Skipping row for item creation...",
                    str(index),
                )
                result["skipped_counter"] += 1
                continue

        # Check if all data conditions to create the item are met:
        if "create" in row_operations or "recreate" in row_operations:
            conditions_create = bulk_item.get("conditions_create")
            if conditions_create:
                evaluated_conditions_create = self.evaluate_conditions(
                    conditions=conditions_create,
                    row=row,
                    replacements=replacements,
                )
                if not evaluated_conditions_create:
                    self.logger.info(
                        "Create condition for bulk item row -> %s not met. Excluding create operation for current row...",
                        str(index),
                    )
                    if "create" in row_operations:
                        row_operations.remove("create")
                    if "recreate" in row_operations:
                        row_operations.remove("recreate")
            elif (
                "recreate" in row_operations
            ):  # we still create and recreate without conditions_create. But give a warning for 'recreate' without condition.
                self.logger.warning(
                    "No create condition provided but 'recreate' operation requested. This will recreate all existing items!",
                )

        # Check if all data conditions to delete the item are met:
        if "delete" in row_operations:
            conditions_delete = bulk_item.get("conditions_delete")
            if conditions_delete:
                evaluated_conditions_delete = self.evaluate_conditions(
                    conditions=conditions_delete,
                    row=row,
                    replacements=replacements,
                )
                if not evaluated_conditions_delete:
                    self.logger.info(
                        "Delete condition for bulk item row -> %s not met. Excluding delete operation for current row...",
                        str(index),
                    )
                    row_operations.remove("delete")
            else:  # without delete_conditions we don't delete!!
                self.logger.warning(
                    "Delete operation requested for bulk items but conditions for deletion are missing! (specify 'conditions_delete'!)",
                )
                row_operations.remove("delete")

        # Check if all data conditions to update the item are met:
        if "update" in row_operations:
            conditions_update = bulk_item.get("conditions_update")
            if conditions_update:
                evaluated_conditions_update = self.evaluate_conditions(
                    conditions=conditions_update,
                    row=row,
                    replacements=replacements,
                )
                if not evaluated_conditions_update:
                    self.logger.info(
                        "Update condition for bulk item row -> %s not met. Excluding update operation for current row...",
                        str(index),
                    )
                    row_operations.remove("update")

        # Determine the item name:
        item_name = self.replace_bulk_placeholders(
            input_string=name_field,
            row=row,
            replacements=replacements,
            additional_regex_list=item_name_additional_regex_list,
        )
        # If we cannot get the item_name from the
        # name_field we try the alternative name field
        # (if specified in payload via "name_alt"):
        if not item_name and name_field_alt:
            self.logger.debug(
                "Row -> %s does not have the data to resolve the placeholders in item name -> %s! Trying alternative name field -> %s...",
                str(index),
                name_field,
                name_field_alt,
            )
            # The additional regex list will make sure
            # that any path or URL element is stripped from
            # the value in the row element:
            item_name = self.replace_bulk_placeholders(
                input_string=name_field_alt,
                row=row,
                replacements=replacements,
                additional_regex_list=item_name_additional_regex_list,
            )
        if not item_name:
            self.logger.error(
                "Row -> %s does not have the data to resolve the placeholders in item name -> %s%s!",
                str(index),
                name_field,
                (" nor in alternative name field -> " + name_field_alt if name_field_alt else ""),
            )
            result["skipped_counter"] += 1
            continue

        # Cleanse the item name (allowed characters, maximum length):
        item_name = OTCS.cleanse_item_name(item_name)

        # This is an optimization. We check if the item was created
        # in a former run. This helps if the customizer is re-run:
        if item_name and item_name in existing_items:
            self.logger.info(
                "Item -> '%s' does already exist and has ID -> %s. Skipping...",
                item_name,
                existing_items[item_name],
            )
            result["skipped_counter"] += 1
            continue

        # Determine the description field:
        description = (
            self.replace_bulk_placeholders(
                input_string=description_field,
                row=row,
            )
            if description_field
            else ""
        )

        # Determine the item type:
        item_type = self.replace_bulk_placeholders(
            input_string=type_field,
            row=row,
        )

        # Determine the item type:
        item_url = self.replace_bulk_placeholders(
            input_string=url_field,
            row=row,
        )

        # Determine the item origin:
        original_nickname = self.replace_bulk_placeholders(
            input_string=original_nickname_field,
            row=row,
        )

        # Determine the item origin:
        self.replace_bulk_placeholders_list(
            input_list=original_path,
            row=row,
        )

        item_original_node = None
        # First try to get the original item for the shortcut via the nickname:
        if original_nickname:
            item_original_node = self._otcs_frontend.get_node_from_nickname(nickname=original_nickname)
        # FIf that didn't resolve the original node we try a path if provided in the payload:
        if not item_original_node and original_path:
            item_original_node = self._otcs_frontend.get_node_by_volume_and_path(
                volume_type=self._otcs_frontend.VOLUME_TYPE_ENTERPRISE_WORKSPACE,
                path=original_path,
            )
        item_original_node_id = self._otcs_frontend.get_result_value(response=item_original_node, key="id")

        # Determine the external creation date (if any):
        if external_create_date_field:
            external_create_date = self.replace_bulk_placeholders(
                input_string=external_create_date_field,
                row=row,
            )
        else:
            external_create_date = None

        # Determine the external modification date (if any):
        if external_modify_date_field:
            external_modify_date = self.replace_bulk_placeholders(
                input_string=external_modify_date_field,
                row=row,
            )
        else:
            external_modify_date = None

        # Determine the key field (if any):
        key = self.replace_bulk_placeholders(input_string=key_field, row=row) if key_field else None

        # check if workspace with this nickname does already exist.
        # we also store the nickname to assign it to the new workspace:
        if nickname_field:
            nickname = self.replace_bulk_placeholders(
                input_string=nickname_field,
                row=row,
                replacements=replacements,
                additional_regex_list=nickname_additional_regex_list,
            )
        else:
            nickname = None

        # Based on the evaluation of conditions_create, conditions_delete,
        # and conditions_update we could end up with an empty operations list.
        # In such cases we skip the further processing of this row:
        if not row_operations:
            self.logger.info(
                "Skip item -> '%s' as row operations is empty (no create, update, delete or recreate). This may be because conditions_create, conditions_delete, and conditions_update have been evaluated to false for this row.",
                item_name,
            )
            result["skipped_counter"] += 1
            continue

        self.logger.info(
            "Bulk process item -> '%s' using effective operations -> %s...",
            item_name,
            str(row_operations),
        )

        # Now we traverse a list of (multiple) workspaces
        # the item should be added to:
        success = True
        workspaces = bulk_item.get("workspaces", [])
        for workspace in workspaces:
            # success will only be false if a config problem (failure)
            # and not just a data problem (skipped) has occured:
            parent_id, success = self.get_bulk_document_location(
                workspace=workspace,
                row=row,
                index=index,
                replacements=replacements,
            )

            if parent_id is None:
                continue

            #
            # Create the item in the target folder specified by parent_id:
            #

            # Check if the item does already exist.
            # First we try to look up the item by a unique
            # key that does not change over time.
            # For this we expect a "key" value to be defined for the
            # bulkDocument and one of the category / attribute item
            # to be marked with "is_key" = True. If we don't find the
            # item via key we try via parent and name.
            self.logger.info(
                "Check if item -> '%s' is already in target folder with ID -> %s%s...",
                item_name,
                parent_id,
                " (using key -> {})".format(key) if key is not None else "",
            )
            # Initialize variables - this is important!
            item_old_name = None
            item_id = None
            handle_name_clash = False
            item_modify_date = None

            # We have 4 cases to differentiate:

            # 1. key given + key found = name irrelevant, item exist
            # 2. key given + key not found = if name exist it is a name clash
            # 3. no key given + name found = item exist
            # 4. no key given + name not found = item does not exist

            # We are NOT trying to compensate for a failed key lookup with a name lookup!
            # Names are only relevant if no key is in payload!

            if key:
                key_attribute = next(
                    (cat_attr for cat_attr in categories if cat_attr.get("is_key", False) is True),
                    None,
                )
                if key_attribute:
                    cat_name = key_attribute.get("name", None)
                    att_name = key_attribute.get("attribute", None)
                    set_name = key_attribute.get("set", None)
                    # Try to find the node that has the given key attribute value:
                    response = self._otcs_frontend.lookup_nodes(
                        parent_node_id=parent_id,
                        category=cat_name,
                        attribute=att_name,
                        attribute_set=set_name,
                        value=key,
                    )
                    item_id = self._otcs_frontend.get_result_value(
                        response=response,
                        key="id",
                    )
                    if item_id:
                        # Case 1: key given + key found = name irrelevant, item exist
                        self.logger.info(
                            "Found existing item with the key value -> '%s' in category -> '%s' and attribute -> '%s' in folder with ID -> %s.",
                            key,
                            cat_name,
                            att_name,
                            parent_id,
                        )
                        item_old_name = self._otcs_frontend.get_result_value(
                            response=response,
                            key="name",
                        )
                        if item_old_name != item_name and item_old_name != item_name + " (" + key + ")":
                            self.logger.info(
                                "Existing item with ID -> %s has the old name -> '%s' which is different from the new name -> '%s.'",
                                item_id,
                                item_old_name,
                                item_name,
                            )
                        else:
                            # if there was a name clash before and got resolved
                            # then we need to stick to the resolved name also for updates:
                            if item_old_name == item_name + " (" + key + ")":
                                self.logger.info(
                                    "Document name conflict was resolved before. Changing item name to -> '%s' to match former resolution.",
                                    item_old_name,
                                )
                                item_name = item_old_name
                            item_old_name = None
                        # We get the modify date of the existing item.
                        item_modify_date = self._otcs_frontend.get_result_value(
                            response=response,
                            key="modify_date",
                        )
                    else:
                        # Case 2: key given + key not found = if name exist it is a name clash
                        self.logger.info(
                            "Couldn't find existing item with the key value -> '%s' in category -> '%s' and attribute -> '%s' in folder with ID -> %s.",
                            key,
                            cat_name,
                            att_name,
                            parent_id,
                        )
                        handle_name_clash = True
                else:
                    self.logger.error(
                        "Item has a key -> '%s' defined but none of the category attributes is marked as a key attribute ('is_key' is missing)!",
                        key,
                    )
                    success = False
                    continue
                # end if key_attribute:
            # end if key:
            else:
                # If we haven't a key we try by parent + name
                response = self._otcs_frontend.get_node_by_parent_and_name(
                    name=item_name,
                    parent_id=parent_id,
                )
                item_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
                if item_id:
                    # Case 3: no key given + name found = item exist
                    self.logger.info(
                        "Found existing item -> '%s' (%s) in parent with ID -> %s.",
                        item_name,
                        item_id,
                        parent_id,
                    )
                    # We get the modify date of the existing item.
                    item_modify_date = self._otcs_frontend.get_result_value(
                        response=response,
                        key="modify_date",
                    )
                else:
                    # Case 4: no key given + name not found = item does not exist
                    self.logger.info(
                        "No existing item -> '%s' in parent with ID -> %s.",
                        item_name,
                        parent_id,
                    )

            # Check if we want to recreate an existing item
            # then we handle the "delete" part of "recreate" first:
            if item_id and "recreate" in row_operations:
                response = self._otcs_frontend.delete_node(
                    node_id=item_id,
                    purge=True,
                )
                if not response:
                    self.logger.error(
                        "Failed to recreate existing item -> '%s' (%s) under parent ID -> %s! Delete failed.",
                        (item_name if item_old_name is None else item_old_name),
                        item_id,
                        parent_id,
                    )
                    success = False
                    continue
                self.logger.info(
                    "Successfully deleted existing item -> '%s' (%s) as part of the recreate operation.",
                    (item_name if item_old_name is None else item_old_name),
                    item_id,
                )
                result["delete_counter"] += 1
                item_id = None

            # Check if item does not exists - then we create a new item
            # if this is requested ("create" value in operations list in payload)
            if not item_id and ("create" in row_operations or "recreate" in row_operations):
                # The item does not exist in Content Server - so we
                # add it now...

                # for Case 2 (we looked up the item by key but could have name collisions)
                # we need to see if we have name collisions:
                if handle_name_clash:
                    response = self._otcs_frontend.check_node_name(
                        parent_id=parent_id,
                        node_name=item_name,
                    )
                    # result is a list of names that clash - if it is empty we have no clash
                    if response.get("results"):
                        # We need to handle a race condition here: it could be that
                        # the right item has been created by another thread. This is because
                        # the key could be in multiple rows of the data frame depending how the data loader works.
                        # We can also not assume that this can be resolved by making the key unique
                        # in the data source as it could be that a data set with the same key should
                        # go to multiple workspaces that then are folded into one by synonym resolution.
                        conflicting_node_id = response["results"][0].get("id")
                        conflicting_key = self._otcs_frontend.get_category_value_by_name(
                            node_id=conflicting_node_id,
                            category_name=cat_name,
                            attribute_name=att_name,
                            set_name=set_name,
                        )
                        if key == conflicting_key:
                            # We have a race condition as the two items don't really clash but are identical.
                            # Just skip creating the same item once more.
                            self.logger.warning(
                                "Detected a race condition in name clash handling! Skipping item creation!",
                            )
                            continue

                        # We add the suffix with the key which should be unique:
                        self.logger.warning(
                            "Item -> '%s' does already exist in workspace folder with ID -> %s and we need to handle the name clash and use name -> '%s'",
                            item_name,
                            parent_id,
                            item_name + " (" + key + ")",
                        )
                        item_name = item_name + " (" + key + ")"

                # If category data is in payload we substitute
                # the values with data from the current data row:
                if categories:
                    # Make sure the threads are not changing data structures that
                    # are shared between threads. categories is a list of dicts.
                    # list and dicts are "mutable" data structures in Python!
                    worker_categories = self.process_bulk_categories(
                        row=row,
                        index=index,
                        categories=categories,
                        replacements=replacements,
                    )
                    item_category_data = self.prepare_category_data(
                        categories_payload=worker_categories,
                        source_node_id=parent_id,
                    )
                # end if categories
                else:
                    item_category_data = {}

                response = self._otcs_frontend.create_item(
                    parent_id=int(parent_id),
                    item_type=int(item_type),
                    item_name=item_name,
                    item_description=description,
                    url=item_url,
                    original_id=int(item_original_node_id),
                    category_data=item_category_data,
                    external_create_date=external_create_date,
                    external_modify_date=external_modify_date,
                    show_error=False,
                )
                item_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
                if not item_id:
                    # We may have a race condition here. Double check the item does not yet exist:
                    response = self._otcs_frontend.get_node_by_parent_and_name(
                        parent_id=int(parent_id),
                        name=item_name,
                    )
                    item_id = self._otcs_frontend.get_result_value(
                        response=response,
                        key="id",
                    )
                if not item_id:
                    self.logger.error(
                        "Cannot create item -> '%s' in folder/workspace with ID -> %s!",
                        item_name,
                        parent_id,
                    )
                    success = False
                    continue
                self.logger.info(
                    "Created item -> '%s' (%s), description -> '%s' in parent with ID -> %s.",
                    item_name,
                    item_id,
                    description,
                    parent_id,
                )
                result["create_counter"] += 1

            # end if not workspace_id and "create" in row_operations

            # If updates are requested we update the existing item with
            # a new item version and with fresh metadata from the payload.
            # Additionally we check the external modify date to support
            # incremental load for content that has really changed.
            # In addition we check that "delete" is not requested as otherwise it will
            # never go in elif "delete" ... below (and it does not make sense to update a item
            # that is deleted in the next step...)
            elif (
                item_id
                and "update" in row_operations
                and "delete" not in row_operations  # note the NOT !
                and OTCS.date_is_newer(
                    date_old=item_modify_date,
                    date_new=external_modify_date,
                )
            ):
                # get the specific update operations given in the payload
                # if not specified we do all 4 update operations (name, description, categories and version)
                update_operations = bulk_item.get(
                    "update_operations",
                    ["name", "description", "categories", "nickname", "url"],
                )

                # If category data is in payload we substitute
                # the values with data from the current data row.
                # This is only done if "categories" update is not
                # excluded from the update_operations:
                if categories and "categories" in update_operations:
                    # Make sure the threads are not changing data structures that
                    # are shared between threads. categories is a list of dicts.
                    # list and dicts are "mutable" data structures in Python!
                    worker_categories = self.process_bulk_categories(
                        row=row,
                        index=index,
                        categories=categories,
                        replacements=replacements,
                    )
                    # Transform the payload structure into the format
                    # the OTCS REST API requires:
                    item_category_data = self.prepare_category_data(
                        categories_payload=worker_categories,
                        source_node_id=item_id,
                        source_is_document=True,
                    )
                    if not item_category_data:
                        self.logger.error(
                            "Failed to prepare the updated category data for item -> '%s' (%s)!",
                            item_name,
                            str(item_id),
                        )
                        success = False
                        continue  # for index, row in partition.iterrows()
                # end if categories
                else:
                    item_category_data = {}

                self.logger.info(
                    "Update existing item -> '%s' (%s) with operations -> %s...",
                    item_old_name if item_old_name else item_name,
                    item_id,
                    str(update_operations),
                )
                response = self._otcs_frontend.update_item(
                    node_id=item_id,
                    parent_id=None,  # None = do not move item
                    item_name=(item_name if "name" in update_operations else None),
                    item_description=(description if "description" in update_operations else None),
                    category_data=(item_category_data if "categories" in update_operations else None),
                    url=(item_url if "url" in update_operations else None),
                    external_create_date=external_create_date,
                    external_modify_date=external_modify_date,
                )
                if not response:
                    self.logger.error(
                        "Failed to update metadata of existing item -> '%s' (%s) with metadata -> %s!",
                        (item_old_name if item_old_name else item_name),
                        item_id,
                        str(item_category_data),
                    )
                    success = False
                    continue
                self.logger.info(
                    "Updated existing item -> '%s' (%s, description -> '%s')",
                    item_name,
                    item_id,
                    description,
                )
                result["update_counter"] += 1
            # end if item_id and "update" in row_operations
            elif item_id and "delete" in row_operations:
                # We delete with immediate purging to keep recycle bin clean
                # and to not run into issues with nicknames used in deleted items:
                response = self._otcs_frontend.delete_node(
                    node_id=item_id,
                    purge=True,
                )
                if not response:
                    self.logger.error(
                        "Failed to bulk delete existing item -> '%s' (%s)!",
                        (item_old_name if item_old_name else item_name),
                        item_id,
                    )
                    success = False
                    continue
                self.logger.info(
                    "Successfully deleted existing item -> '%s' (%s).",
                    item_old_name if item_old_name else item_name,
                    item_id,
                )
                result["delete_counter"] += 1
                item_id = None
            # end elif item_id and "delete" in row_operations

            # this is the plain old "if it does exist we just skip it" case:
            elif item_id:
                self.logger.info(
                    "Skipped existing item -> '%s' (%s).",
                    item_old_name if item_old_name else item_name,
                    item_id,
                )
            # this is the case where we just want to operate on existing items (update or delete)
            # but they do not exist:
            elif not item_id and ("update" in row_operations or "delete" in row_operations):
                result["skipped_counter"] += 1
                self.logger.info(
                    "Skipped update/delete of non-existing item -> '%s'.",
                    item_old_name if item_old_name else item_name,
                )

            # The following code is executed for all operations
            # (create, recreate, update, delete):

            # Check if we want to set / update the nickname of the item.
            # Precondition is we have determined a nickname, we have the item ID
            # and the update of the nickname is either required (create, recreate)
            # or requested (update_operations include "nickname"):
            if (
                nickname
                and item_id
                and (
                    "create" in row_operations
                    or "recreate" in row_operations
                    or ("update" in row_operations and "nickname" in update_operations)
                )
            ):
                response = self._otcs_frontend.set_node_nickname(
                    node_id=item_id,
                    nickname=nickname,
                    show_error=True,
                )
                if not response:
                    self.logger.error(
                        "Failed to assign nickname -> '%s' to item -> '%s'!",
                        nickname,
                        item_name,
                    )
            if item_id is not None:
                # Record the item name and ID to allow to read it from failure file
                # and speedup the process.
                result["items"][item_name] = item_id

        # end for workspaces

        # We want the success, failure and skip counter
        # to consider only complete data frame rows. In
        # case of bulk items we can potentially create
        # update or delete more than 1 item per row.
        # So we use the "success" variable to accumulate
        # this for all items per row:
        if not success:
            result["success"] = False
            result["failure_counter"] += 1
        elif item_name not in result["items"]:
            self.logger.info(
                "Bulk item -> '%s' was not added to any workspace.",
                item_name,
            )
            result["skipped_counter"] += 1
        else:
            result["success_counter"] += 1

    # end for index, row in partition.iterrows()

    self.logger.info("End working...")

    results.append(result)

process_bulk_workspace_relationships(section_name='bulkWorkspaceRelationships')

Process workspaces in payload and bulk create them in Content Server (multi-threaded).

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'bulkWorkspaceRelationships'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_workspace_relationships")
def process_bulk_workspace_relationships(
    self,
    section_name: str = "bulkWorkspaceRelationships",
) -> bool:
    """Process workspaces in payload and bulk create them in Content Server (multi-threaded).

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise.

    """

    if not self._bulk_workspace_relationships:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    # For efficient idem-potent operation we may want to see which workspaces
    # have already been processed before:
    if self.check_status_file(
        payload_section_name=section_name,
        payload_specific=True,
        prefix="failure_",
    ):
        # Read payload from where we left it last time
        self._bulk_workspace_relationships = self.get_status_file(
            payload_section_name=section_name,
            prefix="failure_",
        )
        if not self._bulk_workspace_relationships:
            self.logger.error(
                "Cannot load existing bulkWorkspaceRelationships failure file. Bailing out!",
            )
            return False

    success: bool = True

    for bulk_workspace_relationship in self._bulk_workspace_relationships:
        # Check if element has been disabled in payload (enabled = false).
        # In this case we skip the element:
        enabled = bulk_workspace_relationship.get("enabled", True)
        if not enabled:
            self.logger.info(
                "Payload for bulk workspace relationship is disabled. Skipping...",
            )
            continue

        # Read Pattern for "From" Workspace from payload:
        from_workspace = bulk_workspace_relationship.get("from_workspace", None)
        if not from_workspace:
            self.logger.error(
                "Bulk workspace relationship creation needs a workspace nickname in from_workspace! Skipping to next bulk workspace relationship...",
            )
            success = False
            continue
        from_sub_workspace = bulk_workspace_relationship.get(
            "from_sub_workspace_name",
            None,
        )

        # Read Pattern for "To" Workspace from payload:
        to_workspace = bulk_workspace_relationship.get("to_workspace", None)
        if not to_workspace:
            self.logger.error(
                "Bulk workspace relationship creation needs a workspace nickname in to_workspace! Skipping to next bulk workspace relationship...",
            )
            success = False
            continue
        to_sub_workspace = bulk_workspace_relationship.get(
            "to_sub_workspace_name",
            None,
        )

        # The payload element must have a "data_source" key:
        data_source_name = bulk_workspace_relationship.get("data_source", None)
        if not data_source_name:
            self.logger.error(
                "Missing data source name in bulk workspace relationship!",
            )
            success = False
            continue

        self._log_header_callback(
            text="Process bulk workspace relationships from -> '{}' to -> '{}'".format(
                from_sub_workspace if from_sub_workspace else from_workspace,
                to_sub_workspace if to_sub_workspace else to_workspace,
            ),
            char="-",
        )

        # Determine which operations should be done for this bulk workspace relationship:
        operations = bulk_workspace_relationship.get("operations", ["create"])
        if "delete" in operations and "create" in operations:
            self.logger.error(
                "Bulk workspace relationships can either have 'create' or 'delete' operation - not both. Use separate payloads for this purpose!",
            )
            success = False
            continue

        copy_data_source = bulk_workspace_relationship.get(
            "copy_data_source",
            False,
        )
        force_reload = bulk_workspace_relationship.get("force_reload", True)

        # Load and prepare the data source for the bulk processing:
        if copy_data_source:
            self.logger.info(
                "Take a copy of data source -> '%s' to avoid side-effects for repeative usage of the data source...",
                bulk_workspace_relationship["data_source"],
            )
            data = Data(
                self.process_bulk_datasource(
                    data_source_name=data_source_name,
                    force_reload=force_reload,
                ),
                logger=self.logger,
            )
        else:
            self.logger.info(
                "Use original data source -> '%s' and not do a copy.",
                bulk_workspace_relationship["data_source"],
            )
            # Load and prepare the data source for the bulk processing:
            data = self.process_bulk_datasource(
                data_source_name=data_source_name,
                force_reload=force_reload,
            )
        if data is None:  # important to use "is None" here!
            self.logger.error(
                "Failed to load data source -> '%s' for bulk workspace relationships from -> '%s' to -> '%s'",
                data_source_name,
                from_sub_workspace if from_sub_workspace else from_workspace,
                to_sub_workspace if to_sub_workspace else to_workspace,
            )
            success = False
            continue
        if data.get_data_frame() is None:  # important to use "is None" here!
            self.logger.warning(
                "Data source for bulk workspace relationships from -> '%s' to -> '%s' is empty!",
                from_sub_workspace if from_sub_workspace else from_workspace,
                to_sub_workspace if to_sub_workspace else to_workspace,
            )
            continue

        # Check if fields with list substructures should be exploded.
        # We may want to do this outside the bulkDatasource to only
        # explode for bulkDocuments and not for bulkWorkspaces or
        # bulkWorkspaceRelationships:
        explosions = bulk_workspace_relationship.get("explosions", [])
        for explosion in explosions:
            # The explode field parameter can be a string or a list.
            # Exploding multiple fields at once avoids combinatorial explosions -
            # this is VERY different from exploding columns one after the other!
            if "explode_field" not in explosion and "explode_fields" not in explosion:
                self.logger.error("Missing explosion field(s)!")
                continue
            # we want to be backwards compatible...
            explode_fields = (
                explosion["explode_field"] if "explode_field" in explosion else explosion["explode_fields"]
            )
            flatten_fields = explosion.get("flatten_fields", [])
            split_string_to_list = explosion.get("split_string_to_list", False)
            list_splitter = explosion.get(
                "list_splitter",
                ",",
            )  # don't have None as default!
            self.logger.info(
                "Starting explosion of bulk relationships by field(s) -> %s (type -> %s). Size of data frame before explosion -> %s.",
                str(explode_fields),
                type(explode_fields),
                str(len(data)),
            )
            data.explode_and_flatten(
                explode_fields=explode_fields,
                flatten_fields=flatten_fields,
                make_unique=False,
                split_string_to_list=split_string_to_list,
                separator=list_splitter,
                reset_index=True,
            )
            self.logger.info(
                "Size of data frame after explosion -> %s.",
                str(len(data)),
            )
        # end for explosion in explosions

        # Keep only selected rows if filters are specified in bulkWorkspaceRelationship.
        # We have this _after_ "explosions" to allow access to subfields as well.
        # We have this _before_ "sorting" and "deduplication" as we may keep the wrong
        # rows otherwise (unique / deduplication always keeps the first matching row).
        # We may want to do this filtering outside the bulkDatasource to only
        # filter the data frame for bulkDocuments and not for bulkWorkspaces or
        # bulkWorkspaceRelationships:
        filters = bulk_workspace_relationship.get("filters", [])
        if filters:
            data.filter(conditions=filters)

        # Sort the data frame if "sort" specified in payload. We may want to do this to have a
        # higher chance that rows with common values that may collapse into
        # one name are put into the same partition. This can avoid race conditions
        # between different Python threads.
        sort_fields = bulk_workspace_relationship.get("sort", [])
        if sort_fields:
            self.logger.info(
                "Start sorting of bulk workspace relationships data frame based on fields (columns) -> %s...",
                str(sort_fields),
            )
            data.sort(sort_fields=sort_fields, inplace=True)
            self.logger.info(
                "Sorting of bulk workspace relationships data frame based on fields (columns) -> %s completed.",
                str(sort_fields),
            )

        # Check if duplicate rows for given fields should be removed. It is
        # important to do this after sorting as Pandas always keep the first occurance,
        # so ordering plays an important role in deduplication!
        unique_fields = bulk_workspace_relationship.get("unique", [])
        if unique_fields:
            self.logger.info(
                "Starting deduplication of data frame for bulk workspace relationships with unique fields -> %s. Size of data frame before deduplication -> %s",
                str(unique_fields),
                str(len(data)),
            )
            data.deduplicate(unique_fields=unique_fields, inplace=True)
            self.logger.info(
                "Size of data frame after deduplication -> %s.",
                str(len(data)),
            )

        self.logger.info(
            "Bulk create workspace relationships (from workspace -> '%s' to workspace -> '%s'). Operations -> %s.",
            from_sub_workspace if from_sub_workspace else from_workspace,
            to_sub_workspace if to_sub_workspace else to_workspace,
            str(operations),
        )

        # See if bulkWorkspace definition has a specific thread number
        # otherwise it is read from a global environment variable
        bulk_thread_number = int(
            bulk_workspace_relationship.get("thread_number", BULK_THREAD_NUMBER),
        )

        partitions = data.partitionate(bulk_thread_number)

        # Create a list to hold the threads
        threads = []
        results = []

        # Create and start a thread for each partition
        for index, partition in enumerate(partitions, start=1):
            # start a thread executing the process_bulk_workspace_relationships_worker() method below:
            thread = threading.Thread(
                name=f"{section_name}_{index:02}",
                target=self.thread_wrapper,
                args=(
                    self.process_bulk_workspace_relationships_worker,
                    bulk_workspace_relationship,
                    partition,
                    operations,
                    results,
                ),
            )
            self.logger.info("Starting thread -> '%s'...", str(thread.name))
            thread.start()
            threads.append(thread)
            # Avoid that all threads start at the exact same time with
            # potentially expired cookies that cases race conditions:
            time.sleep(1)
        # end for index, partition in enumerate(partitions, start=1)

        # Wait for all threads to complete
        for thread in threads:
            self.logger.info(
                "Waiting for thread -> '%s' to complete...",
                str(thread.name),
            )
            thread.join()
            self.logger.info("Thread -> '%s' has completed.", str(thread.name))

        if "relationships" not in bulk_workspace_relationship:
            bulk_workspace_relationship["relationships"] = {}

        # Print statistics for each thread. In addition,
        # check if all threads have completed without error / failure.
        # If there's a single failure in on of the thread results we
        # set 'success' variable to False.
        results.sort(key=lambda x: x["thread_name"])
        for result in results:
            if not result["success"]:
                self.logger.info(
                    "Thread -> '%s' completed with %s failed, %s skipped, and %s created workspace relationships.",
                    str(result["thread_name"]),
                    str(result["failure_counter"]),
                    str(result["skipped_counter"]),
                    str(result["success_counter"]),
                )
                success = False
            else:
                self.logger.info(
                    "Thread -> '%s' completed successful with %s skipped, and %s created workspace relationships.",
                    str(result["thread_name"]),
                    str(result["skipped_counter"]),
                    str(result["success_counter"]),
                )
            # Record all generated workspaces relationships. This should
            # allow us to restart in case of failures and avoid trying to
            # create workspaces that have been created before.
            bulk_workspace_relationship["relationships"].update(
                result["relationships"],
            )
        self._log_header_callback(
            text="Completed processing of bulk workspace relationships from -> '{}' to -> '{}'".format(
                from_sub_workspace if from_sub_workspace else from_workspace,
                to_sub_workspace if to_sub_workspace else to_workspace,
            ),
            char="-",
        )

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._bulk_workspace_relationships,
    )

    return success

process_bulk_workspace_relationships_worker(bulk_workspace_relationship, partition, operations=None, results=None)

Thread worker to create workspaces relationships in bulk.

Each worker thread gets a partition of the rows that include the data required for the workspace relationship creation.

Parameters:

Name	Type	Description	Default
bulk_workspace_relationship	dict	The payload of the bulkWorkspaceRelationship.	required
partition	DataFrame	The data partition with rows to process.	required
from_workspace	str	The string pattern for nickname of workspace (from).	required
to_workspace	str	The string pattern for nickname of workspace (to).	required
operations	list	Defines which operations should be applyed on workspace relationships. Possible values are "create", "delete", "recreate".	None
results	list	A mutable list of thread results.	None

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(
    attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_workspace_relationships_worker"
)
def process_bulk_workspace_relationships_worker(
    self,
    bulk_workspace_relationship: dict,
    partition: pd.DataFrame,
    operations: list | None = None,
    results: list | None = None,
) -> None:
    """Thread worker to create workspaces relationships in bulk.

    Each worker thread gets a partition of the rows that include
    the data required for the workspace relationship creation.

    Args:
        bulk_workspace_relationship (dict):
            The payload of the bulkWorkspaceRelationship.
        partition (pd.DataFrame):
            The data partition with rows to process.
        from_workspace (str):
            The string pattern for nickname of workspace (from).
        to_workspace (str):
            The string pattern for nickname of workspace (to).
        operations (list, optional):
            Defines which operations should be applyed on workspace relationships.
            Possible values are "create", "delete", "recreate".
        results (list, optional):
            A mutable list of thread results.

    """

    thread_id = threading.get_ident()
    thread_name = threading.current_thread().name

    self.logger.info(
        "Start working on data frame partition of size -> %s...",
        str(len(partition)),
    )

    # Avoid linter warnings - so make parameter default None while we
    # actually want ["create"] to be the default:
    if operations is None:
        operations = ["create"]

    result = {}
    result["thread_id"] = thread_id
    result["thread_name"] = thread_name
    result["success_counter"] = 0
    result["failure_counter"] = 0
    result["skipped_counter"] = 0
    result["delete_counter"] = 0
    result["relationships"] = {}
    result["success"] = True

    # Check if workspace relationships have been processed before, e.i. testing
    # if a "relationships" key exists and if it is pointing to a non-empty list:
    if bulk_workspace_relationship.get("relationships") and "delete" not in operations:
        existing_workspace_relationships = bulk_workspace_relationship["relationships"]
        self.logger.info(
            "Found %s already processed workspace relationships. Try to complete the job...",
            str(len(existing_workspace_relationships)),
        )
    else:
        existing_workspace_relationships = {}

    # Get dictionary of replacements for bulk workspace relationships
    # this we will be used of all places data is read from the
    # current data frame row. Each dictionary item has the field name as the
    # dictionary key and a list of regular expressions as dictionary value
    replacements = bulk_workspace_relationship.get("replacements")

    # Check if all data conditions to create the workspace relationship are met
    conditions = bulk_workspace_relationship.get("conditions")

    # Type of the relationship - can either be child or parent.
    relationship_type = bulk_workspace_relationship.get("type", "child")

    # Whether or not we want to show an error if the lookup of related
    # workspaces fails.
    show_lookup_error_from = bulk_workspace_relationship.get(
        "from_workspace_lookup_error",
        True,
    )
    show_lookup_error_to = bulk_workspace_relationship.get(
        "to_workspace_lookup_error",
        True,
    )

    # Nicknames are very limited in terms of allowed characters.
    # For nicknames we need additional regexp as we need to
    # replace all non-alphanumeric, non-space characters with ""
    # We also preserve hyphens in the first step to replace
    # them below with underscores. This is important to avoid
    # that different spellings of names produce different nicknames.
    # We want spellings with spaces match spellings with hyphens.
    # For this the workspace names have a regexp "-| " in the payload.
    nickname_additional_regex_list = [r"[^\w\s-]"]

    total_rows = len(partition)

    # Process all datasets in the partion that was given to the thread:
    for index, row in partition.iterrows():
        # Calculate percentage of completion
        percent_complete = ((partition.index.get_loc(index) + 1) / total_rows) * 100

        self.logger.info(
            "Processing data row -> %s for bulk workspace relationship creation (%.2f %% complete)...",
            str(index),
            percent_complete,
        )

        # Create a copy of the mutable operations list as we may
        # want to modify it per row:
        row_operations = list(operations)

        # check if we have any exlusions that apply here:
        if conditions:
            evaluated_condition = self.evaluate_conditions(
                conditions=conditions,
                row=row,
                replacements=replacements,
            )
            if not evaluated_condition:
                self.logger.info(
                    "Condition for row -> %s not met. Skipping row for workspace relationship...",
                    str(index),
                )
                result["skipped_counter"] += 1
                continue

        (from_workspace_id, from_workspace_name) = self.get_bulk_workspace_relationship_endpoint(
            bulk_workspace_relationship=bulk_workspace_relationship,
            row=row,
            index=index,
            endpoint="from",
            replacements=replacements,
            nickname_additional_regex_list=nickname_additional_regex_list,
            show_error=show_lookup_error_from,
        )

        # Check we have "from" endpoint:
        if not from_workspace_id:
            self.logger.warning(
                "%s%s",
                (
                    "Failed to retrieve 'from' endpoint for bulk workspace relationship! "
                    if not from_workspace_id and not from_workspace_name
                    else ""
                ),
                (
                    "Failed to retrieve 'from' endpoint (workspace name -> {}) for bulk workspace relationship! ".format(
                        from_workspace_name,
                    )
                    if not from_workspace_id and from_workspace_name
                    else ""
                ),
            )
            result["skipped_counter"] += 1
            continue

        # If relationships should be deleted we do it only if the
        # relationships for this workspace ID have not been deleted before.
        # This is checked by a dictionary result["relationships"].
        # If a "delete" operation is requested there cannot be a "create"
        # operation in the same bulk workspace relationships payload!
        # (this would create too many interferences between threads)
        # Because of this we can continue with the next row then:
        if "delete" in row_operations and from_workspace_id not in result["relationships"]:
            # Get the target (to) workspace type if specified:
            to_workspace_type = bulk_workspace_relationship.get("to_workspace_type")
            if not to_workspace_type:
                self.logger.error(
                    "Cannot delete workspace relationships! Need the target (to) workspace type!",
                )
                result["success"] = False
                result["failure_counter"] += 1
                continue
            result = self._otcs_frontend.delete_workspace_relationships(
                workspace_id=from_workspace_id,
                related_workspace_type_id=to_workspace_type,
            )
            if not result:
                result["success"] = False
                result["failure_counter"] += 1
            else:
                result["delete_counter"] += 1
                # Mark that we have deleted the relationships originating from
                # the current workspace ID (from) pointing to workspace of
                # the given type.
                result["relationships"][from_workspace_id] = [to_workspace_type]
            # We are continuing here as we cannot do delete and create
            # operations in one thread. This needs separate bulkRelationships
            # payloads!
            continue

        (to_workspace_id, to_workspace_name) = self.get_bulk_workspace_relationship_endpoint(
            bulk_workspace_relationship=bulk_workspace_relationship,
            row=row,
            index=index,
            endpoint="to",
            replacements=replacements,
            nickname_additional_regex_list=nickname_additional_regex_list,
            show_error=show_lookup_error_to,
        )

        # Check we have "to" endpoint:
        if not from_workspace_id or not to_workspace_id:
            self.logger.warning(
                "%s%s",
                (
                    "Failed to retrieve 'to' endpoint for bulk workspace relationship!"
                    if not to_workspace_id and not to_workspace_name
                    else ""
                ),
                (
                    "Failed to retrieve 'to' endpoint (workspace name -> {}) for bulk workspace relationship!".format(
                        to_workspace_name,
                    )
                    if not to_workspace_id and to_workspace_name
                    else ""
                ),
            )
            result["skipped_counter"] += 1
            continue

        # Check if workspace relationship has been created before (either in this run
        # or in a former run of the customizer):
        if (  # processed in former run?
            from_workspace_id in existing_workspace_relationships
            and to_workspace_id in existing_workspace_relationships[from_workspace_id]
        ) or (  # processed in current run?
            from_workspace_id in result["relationships"]
            and to_workspace_id in result["relationships"][from_workspace_id]
        ):
            self.logger.info(
                "Workspace relationship between workspace -> '%s' (%s) and related workspace -> '%s' (%s) has successfully been processed before. Skipping...",
                from_workspace_name,
                str(from_workspace_id),
                to_workspace_name,
                str(to_workspace_id),
            )
            result["skipped_counter"] += 1
            continue

        # Check if workspace relationship does already exist in Extended ECM
        # (this is an additional safety measure to avoid errors):
        response = self._otcs_frontend.get_workspace_relationships(
            workspace_id=from_workspace_id,
            relationship_type=relationship_type,
            related_workspace_name=to_workspace_name,
        )
        current_workspace_relationships = self._otcs_frontend.exist_result_item(
            response=response,
            key="id",
            value=to_workspace_id,
        )
        if current_workspace_relationships:
            self.logger.info(
                "Workspace relationship between workspace -> '%s' (%s) and related workspace -> '%s' (%s) does already exist. Skipping...",
                from_workspace_name,
                str(from_workspace_id),
                to_workspace_name,
                str(to_workspace_id),
            )
            result["skipped_counter"] += 1
            continue

        self.logger.info(
            "Bulk create workspace relationship '%s' (%s) -> '%s' (%s)...",
            from_workspace_name,
            str(from_workspace_id),
            to_workspace_name,
            str(to_workspace_id),
        )

        response = self._otcs_frontend.create_workspace_relationship(
            workspace_id=from_workspace_id,
            related_workspace_id=to_workspace_id,
            relationship_type=relationship_type,
            show_error=False,
        )

        if response is None:
            # Potential race condition: see if the workspace-2-workspace relationship has been created by a concurrent thread.
            # So we better check if the relationship is there even if the create_workspace_relationship() call delivered
            # a 'None' response:
            response = self._otcs_frontend.get_workspace_relationships(
                workspace_id=from_workspace_id,
                relationship_type=relationship_type,
                related_workspace_name=to_workspace_name,
            )
            current_workspace_relationships = self._otcs_frontend.exist_result_item(
                response=response,
                key="id",
                value=to_workspace_id,
            )
            if current_workspace_relationships:
                self.logger.info(
                    "Workspace relationship between workspace -> '%s' (%s) and related workspace -> '%s' (%s) has been created concurrently. Skipping...",
                    from_workspace_name,
                    str(from_workspace_id),
                    to_workspace_name,
                    str(to_workspace_id),
                )
                result["skipped_counter"] += 1
                continue
            self.logger.error(
                "Failed to bulk create workspace relationship (%s) from -> '%s' (%s) to -> '%s' (%s)!",
                relationship_type,
                from_workspace_name,
                str(from_workspace_id),
                to_workspace_name,
                str(to_workspace_id),
            )
            result["success"] = False
            result["failure_counter"] += 1
        else:
            self.logger.info(
                "Successfully created bulk workspace relationship (%s) from -> '%s' (%s) to -> '%s' (%s).",
                relationship_type,
                from_workspace_name,
                str(from_workspace_id),
                to_workspace_name,
                str(to_workspace_id),
            )
            result["success_counter"] += 1
            # Record the workspace name and ID to allow to read it from failure file
            # and speedup the process.
            if from_workspace_id not in result["relationships"]:
                # Initialize the "to" list:
                result["relationships"][from_workspace_id] = [to_workspace_id]
            else:
                result["relationships"][from_workspace_id].append(to_workspace_id)

    self.logger.info("End working...")

    results.append(result)

process_bulk_workspaces(section_name='bulkWorkspaces')

Process workspaces in payload and bulk create them in Content Server (multi-threaded).

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'bulkWorkspaces'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_workspaces")
def process_bulk_workspaces(self, section_name: str = "bulkWorkspaces") -> bool:
    """Process workspaces in payload and bulk create them in Content Server (multi-threaded).

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise.

    """

    if not self._bulk_workspaces:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    # For efficient idem-potent operation we may want to see which workspaces
    # have already been processed before:
    if self.check_status_file(
        payload_section_name=section_name,
        payload_specific=True,
        prefix="failure_",
    ):
        # Read payload from where we left it last time
        self._bulk_workspaces = self.get_status_file(
            payload_section_name=section_name,
            prefix="failure_",
        )
        if not self._bulk_workspaces:
            self.logger.error(
                "Cannot load existing bulkWorkspaces failure file. Bailing out!",
            )
            return False

    success: bool = True

    for bulk_workspace in self._bulk_workspaces:
        # Check if element has been disabled in payload (enabled = false).
        # In this case we skip the element:
        enabled = bulk_workspace.get("enabled", True)
        if not enabled:
            self.logger.info("Payload for bulk workspace is disabled. Skipping...")
            continue

        # Read workspace type name from payload:
        type_name = bulk_workspace.get("type_name", None)
        if not type_name:
            self.logger.error(
                "Bulk workspace needs a workspace type name! Skipping to next workspace...",
            )
            success = False
            continue

        # The payload element must have a "data_source" key:
        data_source_name = bulk_workspace.get("data_source", None)
        if not data_source_name:
            self.logger.error("Missing data source name in bulk workspace!")
            success = False
            continue

        self._log_header_callback(
            text="Process bulk workspaces for -> '{}' using data source -> '{}'".format(
                type_name,
                data_source_name,
            ),
            char="-",
        )

        copy_data_source = bulk_workspace.get("copy_data_source", False)
        force_reload = bulk_workspace.get("force_reload", True)

        # Load and prepare the data source for the bulk processing:
        if copy_data_source:
            self.logger.info(
                "Take a copy of data source -> '%s' to avoid side-effects for repeative usage of the data source...",
                data_source_name,
            )
            data = Data(
                self.process_bulk_datasource(
                    data_source_name=data_source_name,
                    force_reload=force_reload,
                ),
                logger=self.logger,
            )
        else:
            self.logger.info(
                "Use original data source -> '%s' and not do a copy.",
                bulk_workspace["data_source"],
            )
            data = self.process_bulk_datasource(
                data_source_name=data_source_name,
                force_reload=force_reload,
            )
        if data is None:  # important to use "is None" here!
            self.logger.error(
                "Failed to load data source -> '%s' for bulk workspace type -> '%s'!",
                data_source_name,
                type_name,
            )
            success = False
            continue
        if data.get_data_frame() is None:  # important to use "is None" here!
            self.logger.error(
                "Data source for bulk workspace type -> '%s' is empty!",
                type_name,
            )
            continue

        # Check if fields with list substructures should be exploded.
        # We may want to do this outside the bulkDatasource to only
        # explode for bulkDocuments and not for bulkWorkspaces or
        # bulkWorkspaceRelationships:
        explosions = bulk_workspace.get("explosions", [])
        for explosion in explosions:
            # explode field can be a string or a list
            # exploding multiple fields at once avoids
            # combinatorial explosions - this is VERY
            # different from exploding columns one after the other!
            if "explode_field" not in explosion and "explode_fields" not in explosion:
                self.logger.error("Missing explosion field(s)!")
                continue
            # we want to be backwards compatible...
            explode_fields = (
                explosion["explode_field"] if "explode_field" in explosion else explosion["explode_fields"]
            )
            flatten_fields = explosion.get("flatten_fields", [])
            split_string_to_list = explosion.get("split_string_to_list", False)
            list_splitter = explosion.get(
                "list_splitter",
                ",",
            )  # don't have None as default!
            self.logger.info(
                "Starting explosion of bulk workspaces by field(s) -> %s (type -> %s). Size of data frame before explosion -> %s.",
                explode_fields,
                type(explode_fields),
                str(len(data)),
            )
            data.explode_and_flatten(
                explode_fields=explode_fields,
                flatten_fields=flatten_fields,
                make_unique=False,
                split_string_to_list=split_string_to_list,
                separator=list_splitter,
                reset_index=True,
            )
            self.logger.info(
                "Size of data frame after explosion -> %s.",
                str(len(data)),
            )

        # Keep only selected rows if filters are specified in bulkWorkspaces.
        # We have this _after_ "explosions" to allow access to subfields as well.
        # We have this _before_ "sorting" and "deduplication" as may keep the wrong
        # rows otherwise (unique / deduplication always keeps the first matching row).
        # We may want to do this outside the bulkDatasource to only
        # filter for bulkDocuments and not for bulkWorkspaces or
        # bulkWorkspaceRelationships:
        filters = bulk_workspace.get("filters", [])
        if filters:
            data.filter(conditions=filters)

        # Sort the data frame if "sort" specified in payload. We may want to do this to have a
        # higher chance that rows with workspace names that may collapse into
        # one name are put into the same partition. This can avoid race conditions
        # between different Python threads.
        sort_fields = bulk_workspace.get("sort", [])
        if sort_fields:
            self.logger.info(
                "Start sorting of data frame for workspace type -> '%s' based on fields (columns) -> %s...",
                type_name,
                str(sort_fields),
            )
            data.sort(sort_fields=sort_fields, inplace=True)
            self.logger.info(
                "Sorting of data frame for workspace type -> '%s' based on fields (columns) -> %s completed.",
                type_name,
                str(sort_fields),
            )

        # Check if duplicate rows for given fields should be removed. It is
        # important to do this after sorting as Pandas always keep the first occurance,
        # so ordering plays an important role in deduplication!
        unique_fields = bulk_workspace.get("unique", [])
        if unique_fields:
            self.logger.info(
                "Starting deduplication of data frame for workspace type -> '%s' with unique fields -> %s. Size of data frame before deduplication -> %s.",
                type_name,
                str(unique_fields),
                str(len(data)),
            )
            data.deduplicate(unique_fields=unique_fields, inplace=True)
            self.logger.info(
                "Size of data frame after deduplication -> %s.",
                str(len(data)),
            )

        # Read name field from payload:
        workspace_name_field = bulk_workspace.get("name", None)
        if not workspace_name_field:
            self.logger.error(
                "Bulk workspace needs a name field! Skipping to next workspace...",
            )
            success = False
            continue

        # Read optional description field from payload:
        workspace_description_field = bulk_workspace.get("description", None)

        # Find the workspace type with the name given in the payload:
        workspace_type = next(
            (item for item in self._workspace_types if item["name"] == type_name),
            None,
        )
        if workspace_type is None:
            self.logger.error(
                "Workspace type -> '%s' not found. Skipping to next bulk workspace...",
                type_name,
            )
            success = False
            continue
        if workspace_type["templates"] == []:
            self.logger.error(
                "Workspace type -> '%s' does not have templates. Skipping to next bulk workspace...",
                type_name,
            )
            success = False
            continue

        # check if the template to be used is specified in the payload:
        if "template_name" in bulk_workspace:
            template_name_field = bulk_workspace["template_name"]
            workspace_template = next(
                (item for item in workspace_type["templates"] if item["name"] == template_name_field),
                None,
            )
            if workspace_template:  # does this template exist?
                self.logger.info(
                    "Workspace template -> '%s' has been specified in payload and it does exist.",
                    template_name_field,
                )
            elif "{" in template_name_field and "}" in template_name_field:
                self.logger.info(
                    "Workspace template -> '%s' has been specified in payload and contains placeholders, validation cannot be performed.",
                    template_name_field,
                )
            else:
                self.logger.error(
                    "Workspace template -> '%s' has been specified in payload but it doesn't exist!",
                    template_name_field,
                )
                self.logger.error(
                    "Workspace type -> '%s' has only these templates -> %s.",
                    type_name,
                    workspace_type["templates"],
                )
                success = False
                continue
        # template to be used is NOT specified in the payload - then we just take the first one:
        else:
            workspace_template = workspace_type["templates"][0]
            template_name_field = None
            self.logger.info(
                "Workspace template has not been specified in payload - taking the first one (%s)...",
                workspace_template,
            )

        # Read the optional categories payload:
        categories = bulk_workspace.get("categories", None)
        if not categories:
            self.logger.info(
                "Bulk workspace payload has no category data! Will leave category attributes empty...",
            )

        # Should existing workspaces be updated? No is the default.
        operations = bulk_workspace.get("operations", ["create"])

        self.logger.info(
            "Bulk create workspaces (name field -> %s, type -> '%s') from workspace template -> '%s'. Operations -> %s.",
            workspace_name_field,
            type_name,
            template_name_field,
            str(operations),
        )

        # See if bulkWorkspace definition has a specific thread number
        # otherwise it is read from a global environment variable
        bulk_thread_number = int(
            bulk_workspace.get("thread_number", BULK_THREAD_NUMBER),
        )

        partitions = data.partitionate(bulk_thread_number)

        # Create a list to hold the threads
        threads = []
        results = []

        # Create and start a thread for each partition
        for index, partition in enumerate(partitions, start=1):
            # start a thread executing the process_bulk_workspaces_worker() method below:
            thread = threading.Thread(
                name=f"{section_name}_{index:02}",
                target=self.thread_wrapper,
                args=(
                    self.process_bulk_workspaces_worker,
                    bulk_workspace,
                    partition,
                    workspace_type,
                    template_name_field,
                    workspace_name_field,
                    workspace_description_field,
                    categories,
                    operations,
                    results,
                ),
            )
            self.logger.info("Starting thread -> '%s'...", str(thread.name))
            thread.start()
            threads.append(thread)
            # Avoid that all threads start at the exact same time with
            # potentially expired cookies that cases race conditions:
            time.sleep(1)
        # end for index, partition in enumerate(partitions, start=1)

        # Wait for all threads to complete
        for thread in threads:
            self.logger.info(
                "Waiting for thread -> '%s' to complete...",
                str(thread.name),
            )
            thread.join()
            self.logger.info("Thread -> '%s' has completed.", str(thread.name))

        if "workspaces" not in bulk_workspace:
            bulk_workspace["workspaces"] = {}

        # Print statistics for each thread. In addition,
        # check if all threads have completed without error / failure.
        # If there's a single failure in on of the thread results we
        # set 'success' variable to False.
        results.sort(key=lambda x: x["thread_name"])
        for result in results:
            self.logger.info(
                "Thread -> '%s' completed with overall status '%s'.",
                str(result["thread_name"]),
                "successful" if result["success"] else "failed",
            )
            self.logger.info(
                "Thread -> '%s' processed %s data rows with %s successful, %s failed, and %s skipped.",
                str(result["thread_name"]),
                str(
                    result["success_counter"] + result["failure_counter"] + result["skipped_counter"],
                ),
                str(result["success_counter"]),
                str(result["failure_counter"]),
                str(result["skipped_counter"]),
            )
            self.logger.info(
                "Thread -> '%s' created %s workspaces, updated %s workspaces, and deleted %s workspaces.",
                str(result["thread_name"]),
                str(result["create_counter"]),
                str(result["update_counter"]),
                str(result["delete_counter"]),
            )

            if not result["success"]:
                success = False
            # Record all generated workspaces. This should allow us
            # to restart in case of failures and avoid trying to
            # create workspaces that have been created before.
            bulk_workspace["workspaces"].update(result["workspaces"])
        self._log_header_callback(
            text="Completed processing of bulk workspaces for -> '{}' using data source -> '{}'".format(
                type_name,
                data_source_name,
            ),
            char="-",
        )

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._bulk_workspaces,
    )

    return success

process_bulk_workspaces_lookup(workspace_nickname=None, workspace_name=None, workspace_type=None, parent_id=None, data_source_name=None, show_error=True)

Look the workspace name and ID.

Use a combination of workspace name, workspace type, and workspace data source (using synonyms) to do so.

Parameters:

Name	Type	Description	Default
workspace_nickname	str	The nickname of the workspace.	None
workspace_name	str	The name as input for lookup. This must be either the name of an existing workspace instance or one of the synonyms of the workspace name.	None
workspace_type	str	The Name of the workspace type.	None
parent_id	int	ID of parent workspace (if it is a sub-workspace) or parent folder.	None
data_source_name	str	The workspace data source name.	None
show_error	bool	Whether or not to log an error if it occurs. If False, just log a warning.	True

Returns:

Type	Description
tuple[int | None, str | None]	tuple[int | None, str | None]: The workspace ID and the looked up workspace name.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_workspaces_lookup")
def process_bulk_workspaces_lookup(
    self,
    workspace_nickname: str | None = None,
    workspace_name: str | None = None,
    workspace_type: str | None = None,
    parent_id: int | None = None,
    data_source_name: str | None = None,
    show_error: bool = True,
) -> tuple[int | None, str | None]:
    """Look the workspace name and ID.

    Use a combination of workspace name, workspace type, and workspace
    data source (using synonyms) to do so.

    Args:
        workspace_nickname (str, optional):
            The nickname of the workspace.
        workspace_name (str, optional):
            The name as input for lookup. This must be either the name of
            an existing workspace instance or one of the synonyms of the workspace name.
        workspace_type (str, optional):
            The Name of the workspace type.
        parent_id (int, optional):
            ID of parent workspace (if it is a sub-workspace) or parent folder.
        data_source_name (str, optional):
            The workspace data source name.
        show_error (bool, optional):
            Whether or not to log an error if it occurs. If False, just log a warning.

    Returns:
        tuple[int | None, str | None]:
            The workspace ID and the looked up workspace name.

    """

    # First we try to find the workspace by a nickname (if provided)
    if workspace_nickname:
        # Nicknames for sure are not allowed to include spaces and dashes:
        workspace_nickname = workspace_nickname.replace(" ", "_")
        workspace_nickname = workspace_nickname.replace("-", "_")
        workspace_nickname = workspace_nickname.replace("___", "_")
        workspace_nickname = workspace_nickname.lower()

        response = self._otcs_frontend.get_node_from_nickname(
            nickname=workspace_nickname,
            show_error=False,
        )
        workspace_id = self._otcs_frontend.get_result_value(
            response=response,
            key="id",
        )
        if workspace_id:
            # If we don't have the name yet, we can get it from the response above:
            if not workspace_name:
                workspace_name = self._otcs_frontend.get_result_value(
                    response=response,
                    key="name",
                )
            return (workspace_id, workspace_name)
        # DON'T RETURN FAILURE AT THIS POINT!

    # Our 2nd try is to find the workspace by a workspace name and workspace type combination:
    if workspace_name:
        workspace_name = workspace_name.strip()
    else:
        self.logger.error(
            "No workspace name specified. Cannot find the workspace by nickname%s, nor by type and name, nor by parent ID and name, nor by synonym.",
            " -> {}".format(workspace_nickname) if workspace_nickname else "",
        )
        return (None, None)

    # If we have workspace name and workspace parent ID then we try this:
    if workspace_name and parent_id is not None:
        response = self._otcs_frontend.get_node_by_parent_and_name(
            parent_id=parent_id,
            name=workspace_name,
        )
        workspace_id = self._otcs_frontend.get_result_value(
            response=response,
            key="id",
        )
        if workspace_id:
            return (workspace_id, workspace_name)

    # If we have workspace name and workspace type then we try to get
    # the workspace by type + name:
    if workspace_name and workspace_type:
        response = self._otcs_frontend.get_workspace_by_type_and_name(
            type_name=workspace_type,
            name=workspace_name,
        )
        workspace_id = self._otcs_frontend.get_result_value(
            response=response,
            key="id",
        )
        if workspace_id:
            return (workspace_id, workspace_name)

    # if the code gets to here we dont have a nickname and the workspace with given name
    # type, or parent ID was not found either. Now we see if we can find the workspace name
    # as a synonym in the workspace data source to find the real/correct workspace name:
    if data_source_name:
        self.logger.info(
            "Try to find the workspace with the synonym -> '%s' using data source -> '%s'...",
            workspace_name,
            data_source_name,
        )

        (workspace_id, workspace_synonym_name) = self.process_bulk_workspaces_synonym_lookup(
            data_source_name=data_source_name,
            workspace_name_synonym=workspace_name,  # see if workspace_name is a synonym
            workspace_type=workspace_type,
        )
        if workspace_id is not None:
            return (workspace_id, workspace_synonym_name)

    # As this message may be hunderds of times in the log
    # we invest some effort to make it look nice:
    message = "Couldn't find a workspace "
    concat_string = ""
    if workspace_nickname:
        message += "by nickname -> '{}'".format(workspace_nickname)
        concat_string = ", nor "
    if workspace_name:
        message += "{}by name -> '{}'".format(concat_string, workspace_name)
        concat_string = ", nor "
    if parent_id:
        message += "{}by parent ID -> {}".format(concat_string, parent_id)
        concat_string = ", nor "
    if data_source_name:
        message += "{}as synonym in data source -> '{}'".format(
            concat_string,
            data_source_name,
        )
    if show_error:
        self.logger.error(message)
    else:
        self.logger.warning(message)

    return (
        None,
        workspace_name,
    )  # it is important to return the name - used by process_bulk_categories()

process_bulk_workspaces_synonym_lookup(data_source_name, workspace_name_synonym='', workspace_type='')

Use a data source to lookup the workspace name (or all fields) and ID using a given synonym.

Parameters:

Name	Type	Description	Default
data_source_name	str	The data source name.	required
workspace_name_synonym	str	The synonym of the workspace name as input for lookup.	''
workspace_type	str	The name of the workspace type. Default is "".	''

Returns:

Type	Description
tuple[int | None, str | None] | None	tuple[int | None, int | None]: The workspace ID and the looked up workspace name or None in case the loomkup has failed.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_workspaces_synonym_lookup")
def process_bulk_workspaces_synonym_lookup(
    self,
    data_source_name: str,
    workspace_name_synonym: str = "",
    workspace_type: str = "",
) -> tuple[int | None, str | None] | None:
    """Use a data source to lookup the workspace name (or all fields) and ID using a given synonym.

    Args:
        data_source_name (str):
            The data source name.
        workspace_name_synonym (str, optional):
            The synonym of the workspace name as input for lookup.
        workspace_type (str, optional):
            The name of the workspace type. Default is "".

    Returns:
        tuple[int | None, int | None]:
            The workspace ID and the looked up workspace name or None in case the loomkup has failed.

    """

    # Determine the data source to use for synonym lookup:
    if not data_source_name:
        self.logger.error(
            "No workspace data source name specified. Cannot lookup the workspace by synonym -> '%s'!",
            workspace_name_synonym,
        )
        return (None, None)
    workspace_data_source = next(
        (item for item in self._bulk_datasources if item["name"] == data_source_name),
        None,
    )
    if not workspace_data_source:
        self.logger.error(
            "Workspace data source -> '%s' not found in payload. Cannot lookup the workspace by synonym -> '%s'!",
            data_source_name,
            workspace_name_synonym,
        )
        return (None, None)

    # Read the synonym column and the name column from the data source payload item:
    workspace_data_source_name_column = workspace_data_source.get(
        "name_column",
        None,  # e.g. "Name"
    )
    workspace_data_source_synonyms_column = workspace_data_source.get(
        "synonyms_column",
        None,  # e.g. "Synonyms"
    )

    if not workspace_data_source_name_column or not workspace_data_source_synonyms_column:
        self.logger.warning(
            "Workspace data source -> '%s' has no synonym lookup columns. Cannot find the workspace by synonym -> '%s'!",
            data_source_name,
            workspace_name_synonym,
        )
        return (None, None)

    # Get the row that has the synonym in the synonym column:
    lookup_row = self.lookup_data_source_value(
        data_source=workspace_data_source,
        lookup_column=workspace_data_source_synonyms_column,
        lookup_value=workspace_name_synonym,
    )

    if lookup_row is None:
        # Handle an edge case where the actual workspace name
        # is already correct (using the name column):
        lookup_row = self.lookup_data_source_value(
            data_source=workspace_data_source,
            lookup_column=workspace_data_source_name_column,
            lookup_value=workspace_name_synonym,
        )

    if lookup_row is not None:
        # Now we determine the real workspace name be taking it from
        # the name column in the result row:
        workspace_name = lookup_row[workspace_data_source_name_column]
        self.logger.info(
            "Found workspace name -> '%s' using synonym -> '%s'.",
            workspace_name,
            workspace_name_synonym,
        )

        # We now have the real name. If the workspace type name is
        # provided as well we should be able to lookup the workspace ID now:
        if workspace_type:
            response = self._otcs_frontend.get_workspace_by_type_and_name(
                type_name=workspace_type,
                name=workspace_name,
            )
            workspace_id = self._otcs_frontend.get_result_value(
                response=response,
                key="id",
            )
        else:
            # This method may be called with workspace_type=None.
            # In this case we can return the synonym but cannot
            # lookup the workspace ID:
            workspace_id = None

        # Return the tuple with workspace_id and the real workspace name
        return (workspace_id, workspace_name)

    return (None, None)

process_bulk_workspaces_worker(bulk_workspace, partition, workspace_type, template_name_field, workspace_name_field, workspace_description_field, categories=None, operations=None, results=None)

Thread worker to create workspaces in bulk.

Each worker thread gets a partition of the rows that include the data required for the workspace creation.

Parameters:

Name	Type	Description	Default
bulk_workspace	dict	The payload of the bulkWorkspace.	required
partition	DataFrame	Data partition with rows to process.	required
template_id	int	ID of the workspace template to use.	required
workspace_type	dict	Workspace type data.	required
template_name_field	str | None	Field where the template name is stored.	required
workspace_name_field	str	Field where the workspace name is stored.	required
workspace_description_field	str	Field where the workspace description is stored.	required
categories	list	List of category dictionaries.	None
operations	list	Defines which operations should be applyed on workspaces. Possible values are "create", "update", "delete", "recreate".	None
results	list	A mutable list of thread results.	None

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_bulk_workspaces_worker")
def process_bulk_workspaces_worker(
    self,
    bulk_workspace: dict,
    partition: pd.DataFrame,
    workspace_type: dict,
    template_name_field: str | None,
    workspace_name_field: str,
    workspace_description_field: str,
    categories: list | None = None,
    operations: list | None = None,
    results: list | None = None,
) -> None:
    """Thread worker to create workspaces in bulk.

    Each worker thread gets a partition of the rows that include
    the data required for the workspace creation.

    Args:
        bulk_workspace (dict):
            The payload of the bulkWorkspace.
        partition (pd.DataFrame):
            Data partition with rows to process.
        template_id (int):
            ID of the workspace template to use.
        workspace_type (dict):
            Workspace type data.
        template_name_field (str | None):
            Field where the template name is stored.
        workspace_name_field (str):
            Field where the workspace name is stored.
        workspace_description_field (str):
            Field where the workspace description is stored.
        categories (list, optional):
            List of category dictionaries.
        operations (list, optional):
            Defines which operations should be applyed on workspaces.
            Possible values are "create", "update", "delete", "recreate".
        results (list, optional):
            A mutable list of thread results.

    """

    thread_id = threading.get_ident()
    thread_name = threading.current_thread().name

    t = trace.get_current_span()
    t.set_attributes(
        {
            "thread.id": thread_id,
            "thread.name": thread_name,
        }
    )

    self.logger.info(
        "Start working on data frame partition of size -> %s to bulk create workspaces...",
        str(len(partition)),
    )

    # Avoid linter warnings - so make parameter default None while we
    # actually want ["create"] to be the default:
    if operations is None:
        operations = ["create"]

    result = {}
    result["thread_id"] = thread_id
    result["thread_name"] = thread_name
    result["success_counter"] = 0
    result["failure_counter"] = 0
    result["skipped_counter"] = 0
    result["create_counter"] = 0
    result["update_counter"] = 0
    result["delete_counter"] = 0
    result["workspaces"] = {}
    result["success"] = True

    # Check if workspaces have been processed before, e.i. testing
    # if a "workspaces" key exists and if it is pointing to a non-empty list.
    # Additionally we check that workspace updates are not enforced:
    if (
        bulk_workspace.get("workspaces")
        and "update" not in operations
        and "delete" not in operations
        and "recreate" not in operations
    ):
        existing_workspaces = bulk_workspace["workspaces"]
        self.logger.info(
            "Found %s already processed workspaces. Try to complete the job...",
            str(len(existing_workspaces)),
        )
    else:
        existing_workspaces = {}

    # See if external creation and modification dates are in the payload data:
    external_modify_date_field = bulk_workspace.get("external_modify_date")
    external_create_date_field = bulk_workspace.get("external_create_date")

    # See if we have a key field to uniquely identify an existing workspace:
    key_field = bulk_workspace.get("key")

    # Get dictionary of replacements for bulk workspace creations
    # this we will be used of all places data is read from the
    # data frame. Each dictionary item has the field name as the
    # dictionary key and a list of regular expressions as dictionary value
    replacements = bulk_workspace.get("replacements")

    # In case the name cannot be resolved we allow to
    # specify an alternative name field in the payload.
    name_field_alt = bulk_workspace.get("name_alt")

    # In case the description cannot be resolved we allow to
    # specify an alternative description field in the payload.
    description_field_alt = bulk_workspace.get("description_alt")

    # Fetch the nickname field from the payload (if it is specified):
    nickname_field = bulk_workspace.get("nickname")

    # In case the nickname cannot be resolved we allow to
    # specify an alternative nickname field in the payload.
    nickname_field_alt = bulk_workspace.get("nickname_alt")

    # Nicknames are very limited in terms of allowed characters.
    # For nicknames we need additional regexp as we need to
    # replace all non-alphanumeric, non-space characters with ""
    # We also preserve hyphens in the first step to replace
    # them below with underscores. This is important to avoid
    # that different spellings of names produce different nicknames.
    # We want spellings with spaces match spellings with hyphens.
    # For this the workspace names have a regexp "-| " in the payload.
    nickname_additional_regex_list = [r"[^\w\s-]"]

    # Classification can either be provided by classification pathes
    # or by nicknames:
    classification_pathes = bulk_workspace.get("classification_pathes", [])
    classification_nicknames = bulk_workspace.get("classification_nicknames", [])

    total_rows = len(partition)

    # Process all rows in the partition that was given to this thread:
    for index, row in partition.iterrows():
        with tracer.start_as_current_span("process_bulk_workspaces_worker-workspace") as t:
            # Calculate percentage of completion:
            percent_complete = ((partition.index.get_loc(index) + 1) / total_rows) * 100

            self.logger.info(
                "Processing data row -> %s for bulk workspace creation (%.2f %% complete)...",
                str(index),
                percent_complete,
            )

            # Clear variables to esure clean state for each row:
            workspace_id = None

            workspace_template = None
            if template_name_field is None:
                workspace_template = workspace_type["templates"][0]

            else:
                workspace_template_name = self.replace_bulk_placeholders(
                    input_string=template_name_field,
                    row=row,
                    replacements=replacements,
                )

                workspace_template = next(
                    (item for item in workspace_type["templates"] if item["name"] == workspace_template_name),
                    None,
                )

            if workspace_template is None:
                self.logger.error(
                    "Workspace template -> '%s' has been specified in payload but it doesn't exist!",
                    workspace_template_name,
                )
                self.logger.error(
                    "Workspace type -> '%s' has only these templates -> %s",
                    workspace_type["name"],
                    workspace_type["templates"],
                )
                result["success"] = False
                result["failure_counter"] += 1
                continue

            template_id = workspace_template["id"]
            template_name = workspace_template["name"]
            workspace_type_id = workspace_type["id"]

            # Determine the workspace name:
            workspace_name = self.replace_bulk_placeholders(
                input_string=workspace_name_field,
                row=row,
                replacements=replacements,
            )
            # If we cannot get the workspace name from the
            # workspace_name_field we try the alternative name field
            # (if specified in payload via "name_alt"):
            if not workspace_name and name_field_alt:
                self.logger.debug(
                    "Row -> %s does not have the data to resolve the placeholders in workspace name -> %s! Trying alternative name field -> %s...",
                    str(index),
                    workspace_name_field,
                    name_field_alt,
                )
                workspace_name = self.replace_bulk_placeholders(
                    input_string=name_field_alt,
                    row=row,
                    replacements=replacements,
                )

            if not workspace_name:
                self.logger.warning(
                    "Row -> %s does not have the required data to resolve -> %s for the workspace name!%s",
                    str(index),
                    workspace_name_field,
                    " There's no alternative field name given in the payload (via 'name_alt')."
                    if not name_field_alt
                    else " The alternative field -> '{}' could not be resolved either!".format(
                        name_field_alt,
                    ),
                )
                result["skipped_counter"] += 1
                continue

            # Cleanse the workspace name (allowed characters, maximum length):
            workspace_name = OTCS.cleanse_item_name(workspace_name)

            # Check if workspace has been created before (either in this run
            # or in a former run of the customizer):
            if (
                workspace_name in existing_workspaces  # processed in former run?
                or workspace_name in result["workspaces"]  # processed in current run?
            ):
                self.logger.info(
                    "Workspace -> '%s' does already exist. Skipping...",
                    workspace_name,
                )
                result["skipped_counter"] += 1
                continue

            # Determine the description field:
            if workspace_description_field:
                description = self.replace_bulk_placeholders(
                    input_string=workspace_description_field,
                    row=row,
                )
                # If we cannot get the description from the
                # workspace_description_field we try the alternative name field
                # (if specified in payload via "description_alt"):
                if not description and description_field_alt:
                    self.logger.debug(
                        "Row -> %s does not have the data to resolve the placeholders in workspace description -> %s! Trying alternative description field -> %s...",
                        str(index),
                        workspace_name_field,
                        description_field_alt,
                    )
                    description = self.replace_bulk_placeholders(
                        input_string=description_field_alt,
                        row=row,
                    )
            else:
                description = ""

            # Create a copy of the mutable operations list as we may
            # want to modify it per row:
            row_operations = list(operations)

            # Check if all data conditions to create the workspace are met
            conditions = bulk_workspace.get("conditions")
            if conditions:
                evaluated_condition = self.evaluate_conditions(
                    conditions=conditions,
                    row=row,
                    replacements=replacements,
                )
                if not evaluated_condition:
                    self.logger.info(
                        "Condition for bulk workspace row -> %s not met. Skipping row for workspace creation...",
                        str(index),
                    )
                    result["skipped_counter"] += 1
                    continue

            # Check if all data conditions to create or recreate the workspace are met:
            if "create" in row_operations or "recreate" in row_operations:
                conditions_create = bulk_workspace.get("conditions_create")
                if conditions_create:
                    evaluated_conditions_create = self.evaluate_conditions(
                        conditions=conditions_create,
                        row=row,
                        replacements=replacements,
                    )
                    if not evaluated_conditions_create:
                        self.logger.info(
                            "Create condition for bulk workspace row -> %s not met. Excluding create operation for current row...",
                            str(index),
                        )
                        if "create" in row_operations:
                            row_operations.remove("create")
                        if "recreate" in row_operations:
                            row_operations.remove("recreate")
                elif (
                    "recreate" in row_operations
                ):  # we still create and recreate without conditions_create. But give a warning for 'recreate' without condition.
                    self.logger.warning(
                        "No create condition provided but 'recreate' operation requested. This will recreate all existing workspaces!",
                    )

            # Check if all data conditions to delete the workspace are met:
            if "delete" in row_operations:
                conditions_delete = bulk_workspace.get("conditions_delete")
                if conditions_delete:
                    evaluated_conditions_delete = self.evaluate_conditions(
                        conditions=conditions_delete,
                        row=row,
                        replacements=replacements,
                    )
                    if not evaluated_conditions_delete:
                        self.logger.info(
                            "Delete condition for bulk workspace row -> %s not met. Excluding delete operation for current row...",
                            str(index),
                        )
                        row_operations.remove("delete")
                else:  # without delete_conditions we don't delete!!
                    self.logger.warning(
                        "Delete operation requested for bulk workspaces but conditions for deletion are missing! (specify 'conditions_delete')!",
                    )
                    row_operations.remove("delete")

            # Check if all data conditions to delete the workspace are met:
            if "update" in row_operations:
                conditions_update = bulk_workspace.get("conditions_update")
                if conditions_update:
                    evaluated_conditions_update = self.evaluate_conditions(
                        conditions=conditions_update,
                        row=row,
                        replacements=replacements,
                    )
                    if not evaluated_conditions_update:
                        self.logger.info(
                            "Update condition for bulk workspace row -> %s not met. Excluding update operation for current row...",
                            str(index),
                        )
                        row_operations.remove("update")

            # Determine the external modification date (if any):
            if external_modify_date_field:
                external_modify_date = self.replace_bulk_placeholders(
                    input_string=external_modify_date_field,
                    row=row,
                )
            else:
                external_modify_date = None

            # Determine the external creation date (if any):
            if external_create_date_field:
                external_create_date = self.replace_bulk_placeholders(
                    input_string=external_create_date_field,
                    row=row,
                )
            else:
                external_create_date = None

            # Determine the key field (if any):
            key = self.replace_bulk_placeholders(input_string=key_field, row=row) if key_field else None

            # check if workspace with this nickname does already exist.
            # we also store the nickname to assign it to the new workspace:
            found_workspace_name = None
            found_workspace_id = None
            if nickname_field:
                nickname = self.replace_bulk_placeholders(
                    input_string=nickname_field,
                    row=row,
                    replacements=replacements,
                    additional_regex_list=nickname_additional_regex_list,
                )
                # If we cannot get the nickname from the
                # workspace_nickname_field we try the alternative nickname field
                # (if specified in payload via "nickname_alt"):
                if not nickname and nickname_field_alt:
                    nickname = self.replace_bulk_placeholders(
                        input_string=nickname_field_alt,
                        row=row,
                        replacements=replacements,
                        additional_regex_list=nickname_additional_regex_list,
                    )

                # Nicknames for sure should not have leading or trailing spaces:
                nickname = nickname.strip()
                # Nicknames for sure are not allowed to include spaces:
                nickname = nickname.replace(" ", "_")
                # We also want to replace hyphens with underscores
                # to make sure that workspace name spellings with
                # spaces and with hyphens are mapped to the same
                # workspace nicknames (aligned with the workspace names
                # that have a regexp rule for this in the payload):
                nickname = nickname.replace("-", "_")
                nickname = nickname.replace("___", "_")
                nickname = nickname.lower()
                response = self._otcs_frontend.get_node_from_nickname(nickname=nickname)
                if response:
                    found_workspace_name = self._otcs_frontend.get_result_value(
                        response=response,
                        key="name",
                    )
                    found_workspace_id = self._otcs_frontend.get_result_value(
                        response=response,
                        key="id",
                    )
                    if found_workspace_name != workspace_name:
                        self.logger.warning(
                            "Clash of nicknames for -> '%s' and -> '%s' (%s)! We will not create the workspace but allow for updates and deletes",
                            workspace_name,
                            found_workspace_name,
                            found_workspace_id,
                        )
                        # Remove 'create' from row operations as it would produce errors about uniqueness of nicknames.
                        if "create" in row_operations:
                            row_operations.remove("create")
                    # Only skip, if workspace 'update' or 'delete' is NOT requested:
                    elif "update" not in row_operations and "delete" not in row_operations:
                        self.logger.info(
                            "Workspace -> '%s' with nickname -> '%s' does already exist (found -> '%s'). No update or delete operations requested or allowed. Skipping...",
                            workspace_name,
                            nickname,
                            found_workspace_name,
                        )
                        result["skipped_counter"] += 1
                        continue
            # end if nickname_field
            else:
                nickname = None

            t.add_event(name="Preparations done.", timestamp=time.time_ns())

            # Based on the evaluation of conditions_create, conditions_delete,
            # and conditions_update we could end up with an empty operations list.
            # In such cases we skip the further processing of this row:
            if not row_operations:
                self.logger.info(
                    "Skip workspace -> '%s' as row operations is empty (no create, update, delete or recreate). This may be because conditions_create, conditions_delete, and conditions_update have been evaluated to false for this row.",
                    workspace_name,
                )
                result["skipped_counter"] += 1
                continue

            self.logger.info(
                "Bulk process workspace -> '%s' using effective operations -> %s...",
                workspace_name,
                str(row_operations),
            )

            # Check if the workspace does already exist.
            # First we try to look up the workspace by a unique
            # key that does not change over time.
            # For this we expect a "key" value to be defined for the
            # bulkWorkspace and one of the category / attribute item
            # to be marked with "is_key" = True. If we don't find the
            # workspace via key we try via name and type.
            self.logger.info(
                "Check if workspace -> '%s' does already exist...",
                workspace_name,
            )
            # Initialize variables - this is important!
            workspace_old_name = None
            workspace_id = None
            handle_name_clash = False
            workspace_modify_date = None
            parent_id = None

            # We have 4 cases to differentiate:

            # 1. key given + key found = name irrelevant, item exist
            # 2. key given + key not found = if name exist it is a name clash
            # 3. no key given + name found = item exist
            # 4. no key given + name not found = item does not exist

            # We are NOT trying to compensate for a failed key lookup with a name lookup!
            # Names are only relevant if no key is in payload!

            if key:
                # if we have a key we need to also know which category attribute is
                # storing the value for the key:
                key_attribute = next(
                    (cat_attr for cat_attr in categories if cat_attr.get("is_key", False) is True),
                    None,
                )
                if not key_attribute:
                    self.logger.error(
                        "Bulk Workspace has key -> '%s' defined but none of the category attributes is marked as key ('is_key' is missing)!",
                        key,
                    )
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue
                cat_name = key_attribute.get("name", None)
                att_name = key_attribute.get("attribute", None)
                set_name = key_attribute.get("set", None)

                # determine where workspaces of this type typically reside (IMPORTANT: this
                # may return None if no instances of this workspace type exist!):
                parent_id = self._otcs_frontend.get_workspace_type_location(
                    type_id=workspace_type_id,
                )
                if parent_id:
                    # Try to find the node that has the given key attribute value:
                    response = self._otcs_frontend.lookup_nodes(
                        parent_node_id=parent_id,
                        category=cat_name,
                        attribute=att_name,
                        attribute_set=set_name,
                        value=key,
                    )
                    workspace_id = self._otcs_frontend.get_result_value(
                        response=response,
                        key="id",
                    )
                else:
                    # if we couldn't determine the parent ID this means there are
                    # now workspace instances for this workspace type. Then we set
                    # workspace_id = None and let the code go into the else case below:
                    workspace_id = None

                if workspace_id:
                    # Case 1: key given + key found = name irrelevant, item exist
                    workspace_old_name = self._otcs_frontend.get_result_value(
                        response=response,
                        key="name",
                    )
                    self.logger.info(
                        "Found existing workspace -> %s (%s) in folder with ID -> %s using key value -> '%s', category -> '%s', and attribute -> '%s'.",
                        workspace_old_name,
                        workspace_id,
                        parent_id,
                        key,
                        cat_name,
                        att_name,
                    )
                    if workspace_old_name != workspace_name:
                        self.logger.info(
                            "Existing workspace has the old name -> '%s' which is different from the new name -> '%s'.",
                            workspace_old_name,
                            workspace_name,
                        )
                    else:
                        workspace_old_name = None
                    # We get the modify date of the existing workspace.
                    workspace_modify_date = self._otcs_frontend.get_result_value(
                        response=response,
                        key="modify_date",
                    )
                else:
                    # Case 2: key given + key not found = if name exist it is a name clash
                    self.logger.info(
                        "Couldn't find existing workspace with the key value -> '%s' in category -> '%s' and attribute -> '%s' in folder with ID -> %s.",
                        key,
                        cat_name,
                        att_name,
                        parent_id,
                    )
                    handle_name_clash = True
                # end if key_attribute
            # end if key
            else:
                # If we haven't a key we try by type + name
                response = self._otcs_frontend.get_workspace_by_type_and_name(
                    type_id=workspace_type_id,
                    name=workspace_name,
                )
                workspace_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
                if workspace_id:
                    # Case 3: no key given + name found = item exist
                    self.logger.info(
                        "Found existing workspace -> '%s' (%s) with type ID -> %s.",
                        workspace_name,
                        workspace_id,
                        workspace_type_id,
                    )
                    # We get the modify date of the existing workspace.
                    workspace_modify_date = self._otcs_frontend.get_result_value(
                        response=response,
                        key="modify_date",
                    )
                else:
                    # Case 4: no key given + name not found = item does not exist
                    self.logger.info(
                        "No existing workspace with name -> '%s' and type ID -> %s.",
                        workspace_name,
                        workspace_type_id,
                    )
                    # Check if we found an existing workspace by the same nickname:
                    if found_workspace_id and found_workspace_name:
                        self.logger.info(
                            "But there's a workspace -> '%s' (%s) that has a matching nickname -> '%s'. Using this workspace instead...",
                            found_workspace_name,
                            found_workspace_id,
                            nickname,
                        )
                        workspace_id = found_workspace_id

            # Check if we want to recreate an existing workspace
            # then we handle the "delete" part of "recreate" first:
            if workspace_id and "recreate" in row_operations:
                response = self._otcs_frontend.delete_node(
                    node_id=workspace_id,
                    purge=True,
                )
                if not response:
                    self.logger.error(
                        "Failed to recreate existing workspace -> '%s' (%s) with type ID -> %s! Delete failed.",
                        (workspace_name if workspace_old_name is None else workspace_old_name),
                        workspace_id,
                        workspace_type_id,
                    )
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue
                result["delete_counter"] += 1
                self.logger.info(
                    "Deleted existing workspace -> '%s' (%s) as part of the recreate operation.",
                    (workspace_name if workspace_old_name is None else workspace_old_name),
                    workspace_id,
                )
                workspace_id = None

            # Check if workspace does not exists - then we create a new workspace
            # if this is requested ("create" or "recreate" value in operations list in payload)
            if not workspace_id and ("create" in row_operations or "recreate" in row_operations):
                # for Case 2 (we looked up the workspace by key but could have name collisions)
                # we need to see if we have a name collision:
                if handle_name_clash and parent_id:
                    response = self._otcs_frontend.check_node_name(
                        parent_id=parent_id,
                        node_name=workspace_name,
                    )
                    # result is a list of names that clash - if it is empty we have no clash
                    if response["results"]:
                        # We add the suffix with the key which should be unique:
                        self.logger.warning(
                            "Workspace -> '%s' does already exist in folder with ID -> %s and we need to handle the name clash by using name -> '%s'",
                            workspace_name,
                            parent_id,
                            workspace_name + " (" + key + ")",
                        )
                        workspace_name = workspace_name + " (" + key + ")"

                # If category data is in payload we substitute
                # the values with data from the current data row:
                if categories:
                    # Make sure the threads are not changing data structures that
                    # are shared between threads. categories is a list of dicts.
                    # list and dicts are "mutable" data structures in Python!
                    worker_categories = self.process_bulk_categories(
                        row=row,
                        index=index,
                        categories=categories,
                        replacements=replacements,
                    )
                    workspace_category_data = self.prepare_workspace_create_form(
                        categories=worker_categories,
                        template_id=template_id,
                    )
                    if not workspace_category_data:
                        self.logger.error(
                            "Failed to prepare the category data for new workspace -> '%s'!",
                            workspace_name,
                        )
                        result["success"] = False
                        result["failure_counter"] += 1
                        continue  # for index, row in partition.iterrows()
                else:
                    workspace_category_data = {}

                self.logger.info(
                    "Bulk create workspace -> '%s'...",
                    workspace_name,
                )

                if classification_nicknames or classification_pathes:
                    classification_ids = self.process_bulk_classification_assignments(
                        row=row,
                        index=index,
                        classifications=classification_pathes + classification_nicknames,
                        replacements=replacements,
                    )
                else:
                    classification_ids = None

                # Create the workspace with all provided information:
                response = self._otcs_frontend.create_workspace(
                    workspace_template_id=template_id,
                    workspace_name=workspace_name,
                    workspace_description=description,
                    workspace_type=workspace_type_id,
                    category_data=workspace_category_data,
                    classifications=classification_ids,
                    external_create_date=external_create_date,
                    external_modify_date=external_modify_date,
                    show_error=False,
                )
                if response is None:
                    # Potential race condition: see if the workspace has been created by a concurrent thread.
                    # So we better check if the workspace is there even if the create_workspace() call delivered
                    # a 'None' response:
                    response = self._otcs_frontend.get_workspace_by_type_and_name(
                        type_id=workspace_type_id,
                        name=workspace_name,
                    )
                workspace_id = self._otcs_frontend.get_result_value(
                    response=response,
                    key="id",
                )
                if not workspace_id:
                    self.logger.error(
                        "Failed to bulk create workspace -> '%s' with type ID -> %s from template -> '%s' (%s)!",
                        workspace_name,
                        workspace_type_id,
                        template_name,
                        template_id,
                    )
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue
                self.logger.info(
                    "Successfully created workspace -> '%s' with ID -> %s from template -> '%s' (%s).",
                    workspace_name,
                    workspace_id,
                    template_name,
                    template_id,
                )
                result["create_counter"] += 1

                # Is Content Aviator enabled system-wide and is it enabled for this
                # bulkWorkspaces?
                if self._aviator_enabled and bulk_workspace.get("enable_aviator", False):
                    response = self._otcs_frontend.update_workspace_aviator(
                        workspace_id=workspace_id,
                        status=True,
                    )
                    if not response:
                        self.logger.error(
                            "Failed to enable Content Aviator for workspace -> '%s' (%s)!",
                            workspace_name,
                            workspace_id,
                        )

                # Check if metadata embeddings need to be updated
                aviator_metadata = bulk_workspace.get("aviator_metadata", False)
                if aviator_metadata:
                    self.logger.info(
                        "Trigger Content Aviator metadata embedding for workspace -> '%s' (%s)...",
                        workspace_name,
                        workspace_id,
                    )

                    self._otcs_frontend.aviator_embed_metadata(
                        node_id=workspace_id,
                        wait_for_completion=False,
                    )

            # end if not workspace_id and "create" or "recreate" in row_operations

            # If "updates" are an requested row operation we update the existing workspace with
            # fresh metadata from the payload. Additionally we check the external
            # modify date to support incremental load for content that has really
            # changed.
            # In addition we check that "delete" is not requested as otherwise it will
            # never go in elif "delete" ... below (and it does not make sense to update a workspace
            # that is deleted in the next step...)
            elif (
                workspace_id
                and "update" in row_operations
                and "delete" not in row_operations  # note the NOT !
                and OTCS.date_is_newer(
                    date_old=workspace_modify_date,
                    date_new=external_modify_date,
                )
            ):
                # get the specific update operations given in the payload
                # if not specified we do all 4 update operations (name, description, categories and nickname)
                update_operations = bulk_workspace.get(
                    "update_operations",
                    ["name", "description", "categories", "nickname", "classifications", "members"],
                )

                # If category data is in payload we substitute
                # the values with data from the current data row.
                # This is only done if "categories" update is not
                # excluded from the update_operations:
                if categories and "categories" in update_operations:
                    # Make sure the threads are not changing data structures that
                    # are shared between threads. categories is a list of dicts.
                    # list and dicts are "mutable" data structures in Python!
                    worker_categories = self.process_bulk_categories(
                        row=row,
                        index=index,
                        categories=categories,
                        replacements=replacements,
                    )
                    # response = self._otcs_frontend.get_node(node_id=workspace_id)
                    # parent_id = self._otcs_frontend.get_result_value(response=response, key="parent_id")
                    # workspace_category_data = self.prepare_item_create_form(
                    #     parent_id=parent_id,
                    #     categories=worker_categories,
                    #     subtype=self._otcs_frontend.ITEM_TYPE_BUSINESS_WORKSPACE,
                    # )
                    # Transform the payload structure into the format
                    # the OTCS REST API requires:
                    workspace_category_data = self.prepare_category_data(
                        categories_payload=worker_categories,
                        source_node_id=workspace_id,
                    )
                    if not workspace_category_data:
                        self.logger.error(
                            "Failed to prepare the updated category data for workspace -> '%s'!",
                            workspace_name,
                        )
                        result["success"] = False
                        result["failure_counter"] += 1
                        continue  # for index, row in partition.iterrows()
                # end if categories
                else:
                    workspace_category_data = {}

                self.logger.info(
                    "Update existing workspace -> '%s' (%s) with operations -> %s...",
                    workspace_old_name if workspace_old_name else workspace_name,
                    str(workspace_id),
                    str(update_operations),
                )
                # Update the workspace with all provided information:
                response = self._otcs_frontend.update_workspace(
                    workspace_id=workspace_id,
                    workspace_name=(workspace_name if "name" in update_operations else None),
                    workspace_description=(description if "description" in update_operations else None),
                    category_data=(workspace_category_data if "categories" in update_operations else None),
                    external_create_date=external_create_date,
                    external_modify_date=external_modify_date,
                    show_error=True,
                )
                if not response:
                    self.logger.error(
                        "Failed to update existing workspace -> '%s' (%s) with type ID -> %s!",
                        (workspace_old_name if workspace_old_name else workspace_name),
                        workspace_id,
                        workspace_type_id,
                    )
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue
                self.logger.info(
                    "Updated existing workspace -> '%s' (%s) with type ID -> %s.",
                    workspace_name if "name" in update_operations or not workspace_old_name else workspace_old_name,
                    workspace_id,
                    workspace_type_id,
                )
                result["update_counter"] += 1

                if "classifications" in update_operations and (classification_nicknames or classification_pathes):
                    classification_ids = self.process_bulk_classification_assignments(
                        row=row,
                        index=index,
                        classifications=classification_pathes + classification_nicknames,
                    )
                    response = self._otcs.assign_classifications(
                        node_id=workspace_id,
                        classifications=classification_ids,
                        apply_to_sub_items=False,
                    )
                    if response is None:
                        self.logger.error(
                            "Failed to assign classifications -> %s to workspace -> '%s' (%s)!",
                            classification_ids,
                            workspace_name
                            if "name" in update_operations or not workspace_old_name
                            else workspace_old_name,
                            workspace_id,
                        )
                    else:
                        self.logger.info(
                            "Successfully assigned classifications -> %s to workspace -> '%s' (%s).",
                            classification_ids,
                            workspace_name
                            if "name" in update_operations or not workspace_old_name
                            else workspace_old_name,
                            workspace_id,
                        )

                # Check if metadata embeddings need to be updated
                aviator_metadata = bulk_workspace.get("aviator_metadata", False)
                if aviator_metadata:
                    self.logger.info(
                        "Update Content Aviator metadata embedding for workspace -> %s (%s)...",
                        workspace_name,
                        workspace_id,
                    )

                    self._otcs_frontend.aviator_embed_metadata(
                        node_id=workspace_id,
                        wait_for_completion=False,
                    )
            # end elif "update" in row_operations...
            elif workspace_id and "delete" in row_operations:
                # We delete with immediate purging to keep recycle bin clean
                # and to not run into issues with nicknames used in deleted items:
                response = self._otcs_frontend.delete_node(
                    node_id=workspace_id,
                    purge=True,
                )
                if not response:
                    self.logger.error(
                        "Failed to delete existing workspace -> '%s' (%s) with type ID -> %s!",
                        (workspace_old_name if workspace_old_name else workspace_name),
                        workspace_id,
                        workspace_type_id,
                    )
                    result["success"] = False
                    result["failure_counter"] += 1
                    continue
                self.logger.info(
                    "Successfully deleted existing workspace -> '%s' (%s).",
                    workspace_old_name if workspace_old_name else workspace_name,
                    workspace_id,
                )
                result["delete_counter"] += 1
                workspace_id = None
            # end elif workspace_id and "delete" in row_operations

            # this is the plain old "it does exist and we just skip it" case:
            elif workspace_id:
                result["skipped_counter"] += 1
                self.logger.info(
                    "Skipped existing workspace -> '%s' (%s)",
                    workspace_old_name if workspace_old_name else workspace_name,
                    workspace_id,
                )
            # this is the case where we just want to operate on existing workspaces (update or delete)
            # but they do not exist:
            elif not workspace_id and ("update" in row_operations or "delete" in row_operations):
                result["skipped_counter"] += 1
                self.logger.info(
                    "Skipped update/delete of non-existing workspace -> '%s'.",
                    workspace_old_name if workspace_old_name else workspace_name,
                )

            # The following code is executed for all operations
            # (create, recreate, update, delete):

            # Check if we want to set / update the nickname of the workspace.
            # Precondition is we have determined a nickname, we have the workspace ID
            # and the update of the nickname is either required (create, recreate)
            # or requested (update_operations include "nickname"):
            if (
                nickname
                and workspace_id
                and (
                    "create" in row_operations
                    or "recreate" in row_operations
                    or ("update" in row_operations and "nickname" in update_operations)
                )
            ):
                response = self._otcs_frontend.set_node_nickname(
                    node_id=workspace_id,
                    nickname=nickname,
                    show_error=True,
                )
                if not response:
                    self.logger.error(
                        "Failed to assign nickname -> '%s' to workspace -> '%s'!",
                        nickname,
                        workspace_name,
                    )

            # Check if we want to set / update the members for the workspace roles.
            # Precondition is we have members configured in the payload and setting
            # of roles is requested (workspaces is created or update_operations
            # include "members" operation):
            members = bulk_workspace.get("members")
            if (
                members
                and workspace_id
                and (
                    "create" in row_operations
                    or "recreate" in row_operations
                    or ("update" in row_operations and "members" in update_operations)
                )
            ):
                self.looger.info(
                    "Update workspace role members for workspace -> '%s' (%s)...",
                    workspace_name,
                    workspace_id,
                )
                workspace_roles = self._otcs_frontend.get_workspace_roles(
                    workspace_id=workspace_id,
                )

                # Traverse the members payload:
                for member in members:
                    # Get the member role from payload first. It is mandatory:
                    member_role = member.get("role", None)
                    if not member_role:
                        self.logger.warning(
                            "Members of workspace -> '%s' is missing the role name.",
                            workspace_name,
                        )
                        continue
                    member_role = self.replace_bulk_placeholders(input_string=member_role, row=row)
                    if not member_role:
                        continue
                    role_id = self._otcs.lookup_result_value(
                        response=workspace_roles,
                        key="name",
                        value=member_role,
                        return_key="id",
                    )
                    if not role_id:
                        self.logger.warning(
                            "Cannot find workspace role -> '%s' for workspace -> '%s' (%s). Skipping...",
                            member_role,
                            workspace_name,
                            workspace_id,
                        )
                        continue

                    # read user list and role name from payload:
                    member_users = member.get("users", [])
                    member_groups = member.get("groups", [])

                    if member_users:
                        self.replace_bulk_placeholders_list(input_list=member_users, row=row)
                    if member_groups:
                        self.replace_bulk_placeholders_list(input_list=member_groups, row=row)

                    if not member_users and not member_groups:
                        self.logger.debug(
                            "Role -> '%s' of workspace -> '%s' does not have any members (no users nor groups).",
                            member_role,
                            workspace_name,
                        )
                        continue

                    # check if we want to clear (remove) existing members of this role:
                    member_clear = member.get("clear", False)
                    if member_clear:
                        self.logger.info(
                            "Clear existing members of role -> '%s' (%s) for workspace -> '%s' (%s)...",
                            member_role,
                            role_id,
                            workspace_name,
                            workspace_id,
                        )
                        self._otcs.remove_workspace_members(
                            workspace_id=workspace_id,
                            role_id=role_id,
                        )
                # TODO: complete the implementation...
            # end if members...

            # Depending on the bulk operations (create, update, delete)
            # and the related conditions it may well be that workspace_id is None.
            # Only if workspace ID is not none we want to count this row as success:
            if workspace_id is not None:
                result["success_counter"] += 1
                # Record the workspace name and ID to allow to read it from failure file
                # and speedup the process.
                result["workspaces"][workspace_name] = workspace_id
    # end for index...

    self.logger.info("End working...")

    results.append(result)

process_business_object_types(section_name='businessObjectTypes')

Create a data structure for all business object types in the Extended ECM system.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'businessObjectTypes'

Returns:

Name	Type	Description
list	list	A list of business object types. Each list element is a dict with these values: - id (str) - name (str) - type (str) - ext_system_id (str) - business_properties (list) - business_property_groups (list)

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_business_object_types")
def process_business_object_types(
    self,
    section_name: str = "businessObjectTypes",
) -> list:
    """Create a data structure for all business object types in the Extended ECM system.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        list:
            A list of business object types. Each list element is a dict with these values:
            - id (str)
            - name (str)
            - type (str)
            - ext_system_id (str)
            - business_properties (list)
            - business_property_groups (list)

    """

    # If this payload section has been processed successfully before we
    # still need to read the data structure from the status file and
    # initialize self._business_object_types:
    if self.check_status_file(payload_section_name=section_name):
        # read the list from the json file in admin Home
        # this is important for restart of customizer pod
        # as this data structure is used later on for workspace processing
        self.logger.info(
            "Re-Initialize business object types list from status file -> '%s' for later use...",
            self.get_status_file_name(payload_section_name=section_name),
        )
        self._business_object_types = self.get_status_file(
            payload_section_name=section_name,
        )
        if self._business_object_types:
            self.logger.info(
                "Found -> %s business object types.",
                str(len(self._business_object_types)),
            )
            self.logger.debug(
                "Business object types -> %s",
                str(self._business_object_types),
            )
            return self._business_object_types
        else:
            self.logger.warning(
                "Couldn't read business object types from status file -> '%s'. Regenerate list...",
                self.get_status_file_name(payload_section_name=section_name),
            )

    success: bool = True

    # Read payload_section "businessObjectTypes" if available.
    # For some (external) payload we may want to skip this time-
    # consuming processing as there are not BO specific payload
    # elements:
    for section in self._payload_sections:
        # Check if the section does exist and it is explicitly disabled
        # (enabled = False). Then we can skip further processing:
        if section.get("name", "") == "businessObjectTypes" and not section.get(
            "enabled",
            True,
        ):
            self.logger.info(
                "Skip business object type processing as its section is disabled in payload.",
            )
            return self._business_object_types

    # get all business object types (these have been created by the transports and are not in the payload!)
    # we need to do this each time as it needs to work across multiple payload files...
    response = self._otcs.get_business_object_types()
    if response is None:
        self.logger.info("No business object types found!")
        self._business_object_types = []
    else:
        # Store result list in self._business_object_types for later use:
        self._business_object_types = response["results"]
        self.logger.info(
            "Found -> %s business object types.",
            str(len(self._business_object_types)),
        )
        self.logger.debug(
            "Business object types -> %s",
            str(self._business_object_types),
        )

    otcs_version = float(self._otcs.get_server_version())
    if otcs_version >= 25.3:
        self.logger.info(
            "Using new REST API of OTCS %s to retrieve business object type information...",
            str(otcs_version),
        )
    else:
        self.logger.info(
            "Have to use transport extraction with OTCS %s to retrieve business object type information...",
            str(otcs_version),
        )

    # now we enrich the business_object_type list elments (which are dicts)
    # with additional dict elements for further processing:
    for business_object_type in self._business_object_types:
        # Flatten the response structure for more easy retrieval:
        # Get BO Type (e.g. KNA1):
        bo_type = business_object_type["data"]["properties"]["bo_type"]
        self.logger.debug("Business object type -> %s", bo_type)
        business_object_type["type"] = bo_type
        # Get BO Type ID:
        bo_type_id = business_object_type["data"]["properties"]["bo_type_id"]
        self.logger.debug("Business object type ID -> %s", bo_type_id)
        business_object_type["id"] = bo_type_id
        # Get BO Type Name:
        bo_type_name = business_object_type["data"]["properties"]["bo_type_name"]
        self.logger.debug("Business object type name -> %s", bo_type_name)
        business_object_type["name"] = bo_type_name
        # Get External System ID:
        ext_system_id = business_object_type["data"]["properties"]["ext_system_id"]
        self.logger.debug("External system ID -> %s", ext_system_id)
        business_object_type["ext_system_id"] = ext_system_id
        # Get External System ID:
        workspace_type_id = business_object_type["data"]["properties"]["workspace_type_id"]
        self.logger.debug("Workspace type ID -> %s", workspace_type_id)
        business_object_type["workspace_type_id"] = workspace_type_id

        if not bo_type_name or not bo_type_id:
            self.logger.info(
                "Business object type %sis void (dummy data for workspace types without a business object connection)! Skipping...",
                "for workspace type ID -> {} ".format(workspace_type_id) if workspace_type_id else "",
            )
            continue
        self.logger.info(
            "Processing business object type -> '%s' (%s) with ID -> %s for workspace type with ID -> %d...",
            bo_type_name,
            bo_type,
            bo_type_id,
            workspace_type_id,
        )

        # Get additional information per BO Type (before 25.3 REST API is severly
        # limited) - it does not return Property names from External System
        # and is also missing Business Property Groups. Thus, for older versions
        # we extract that information from the Transport XML files (see else case):
        if otcs_version >= 25.3:
            response = self._otcs.get_business_object_type(type_id=bo_type_id)
            if response is None or not response["results"]:
                self.logger.warning(
                    "Cannot retrieve additional information for business object type -> %s (%s). Skipping...",
                    bo_type,
                    bo_type_id,
                )
                success = False
                continue

            business_properties = response["results"]["GeneralTab"]["data"]["properties"]["fWkspCreationConfig"][
                "propertyMappings"
            ]
            business_object_type["business_properties"] = business_properties

            business_property_groups = response["results"]["GeneralTab"]["data"]["properties"][
                "fWkspCreationConfig"
            ]["propertyGroups"]
            business_object_type["business_properties_groups"] = business_property_groups
        elif not self.extract_properties_from_transport_packages(business_object_type, bo_type_id, bo_type_name):
            success = False
    # end for business_object_type in self._business_object_types

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._business_object_types,
    )

    return self._business_object_types

process_cs_applications(otcs_object, section_name='csApplications')

Process CS applications in payload and install them in Content Server.

The CS Applications need to be installed in all frontend and backends.

Parameters:

Name	Type	Description	Default
otcs_object	object	This can either be the OTCS frontend or OTCS backend. If None then the otcs_backend is used.	required
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'csApplications'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_cs_applications")
def process_cs_applications(
    self,
    otcs_object: OTCS,
    section_name: str = "csApplications",
) -> bool:
    """Process CS applications in payload and install them in Content Server.

    The CS Applications need to be installed in all frontend and backends.

    Args:
        otcs_object (object):
            This can either be the OTCS frontend or OTCS backend. If None
            then the otcs_backend is used.
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._cs_applications:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # OTCS backend is the default:
    if not otcs_object:
        otcs_object = self._otcs_backend

    for cs_application in self._cs_applications:
        application_name = cs_application.get("name")
        if not application_name:
            self.logger.error("Missing CS application name in payload! Skipping...")
            continue

        # Check if CS Application has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not cs_application.get("enabled", True):
            self.logger.info(
                "Payload for CS Application -> '%s' is disabled. Skipping...",
                application_name,
            )
            continue

        application_description = cs_application.get("description", "")

        self.logger.info(
            "Install CS Application -> '%s'%s...",
            application_name,
            " ({})".format(application_description) if application_description else "",
        )
        response = otcs_object.install_cs_application(
            application_name=application_name,
        )
        if response is None:
            self.logger.error(
                "Failed to install CS Application -> '%s'!",
                application_name,
            )
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._cs_applications,
    )

    return success

process_docgen_settings(section_name='docgenSettings')

Process OTPD settings in payload and configure them in OTPD.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'docgenSettings'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_docgen_settings")
def process_docgen_settings(self, section_name: str = "docgenSettings") -> bool:
    """Process OTPD settings in payload and configure them in OTPD.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._docgen_settings:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for docgen_setting in self._docgen_settings:
        # Check if the setting has a proper name in the payload:
        setting_name = docgen_setting.get("name")
        if not setting_name:
            self.logger.error(
                "OTPD document generation setting needs a name in payload! Skipping...",
            )
            success = False
            continue

        # Check if the document geneneration has been explicitly disabled in payload
        # (enabled = false). In this case we skip the payload element:
        if not docgen_setting.get("enabled", True):
            self.logger.info(
                "Payload for OTPD setting -> '%s' is disabled. Skipping...",
                setting_name,
            )
            continue

        setting_value = docgen_setting.get("value")
        if not setting_value:
            self.logger.error(
                "OTPD setting -> '%s' needs a value in payload! Skipping...",
                setting_name,
            )
            continue

        tenant_name = docgen_setting.get("tenant", None)
        description = docgen_setting.get("description", "")

        # Apply the setting to PowerDocs (OTPD):
        response = self._otpd.apply_setting(
            setting_name=setting_name,
            setting_value=setting_value,
            tenant_name=tenant_name,
        )
        if response:
            self.logger.info(
                "Added OTPD setting -> '%s' with value -> '%s'%s%s.",
                setting_name,
                setting_value,
                " for tenant -> '{}'".format(tenant_name) if tenant_name else "",
                " ({})".format(description) if description else "",
            )
        else:
            self.logger.error(
                "Failed to configure OTPD setting -> '%s' with value -> '%s'%s%s!",
                setting_name,
                setting_value,
                " for tenant -> '{}'".format(tenant_name) if tenant_name else "",
                " ({})".format(description) if description else "",
            )
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._docgen_settings,
    )

    return success

process_document_generators(section_name='documentGenerators')

Generate documents for a defined workspace type based on template.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'documentGenerators'

Returns:

Name	Type	Description
bool	bool	, True if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_document_generators")
def process_document_generators(
    self,
    section_name: str = "documentGenerators",
) -> bool:
    """Generate documents for a defined workspace type based on template.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:,
            True if payload has been processed without errors, False otherwise.

    """

    if not self._doc_generators:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # save admin credentials for later switch back to admin user:
    #        admin_credentials = self._otcs.credentials()
    authenticated_user = "admin"

    for doc_generator in self._doc_generators:
        if "workspace_type" not in doc_generator:
            self.logger.error(
                "To generate documents for workspaces the workspace type needs to be specified in the payload! Skipping to next document generator...",
            )
            success = False
            continue
        workspace_type = doc_generator["workspace_type"]

        self._log_header_callback(
            text="Process Document Generator for workspace type -> '{}'".format(workspace_type),
            char="-",
        )

        # Check if doc generator has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not doc_generator.get("enabled", True):
            self.logger.info(
                "Payload for document generator of workspace type -> '%s' is disabled. Skipping...",
                workspace_type,
            )
            continue

        template_path = doc_generator.get("template_path")
        if not template_path:
            self.logger.error(
                "To generate documents for workspaces of type -> '%s' the path to the document template needs to be specified in the payload! Skipping to next document generator...",
                workspace_type,
            )
            success = False
            continue

        template = self._otcs.get_node_by_volume_and_path(
            volume_type=self._otcs.VOLUME_TYPE_CONTENT_SERVER_DOCUMENT_TEMPLATES,
            path=template_path,
        )
        if not template:
            self.logger.error(
                "Cannot find document template in path -> %s. Skipping to next document generator...",
                template_path,
            )
            success = False
            continue
        template_id = self._otcs.get_result_value(response=template, key="id")
        template_name = self._otcs.get_result_value(response=template, key="name")

        if "classification_path" not in doc_generator:
            self.logger.error(
                "To generate documents for workspaces of type -> '%s' the path to the document classification needs to be specified in the payload! Skipping to next document generator...",
                workspace_type,
            )
            success = False
            continue
        classification_path = doc_generator["classification_path"]
        classification = self._otcs.get_node_by_volume_and_path(
            volume_type=self._otcs.VOLUME_TYPE_CLASSIFICATION_VOLUME,
            path=classification_path,
        )
        if not classification:
            self.logger.error(
                "Cannot find document classification in path -> %s. Skipping to next document generator...",
                classification_path,
            )
            success = False
            continue
        classification_id = self._otcs.get_result_value(
            response=classification,
            key="id",
        )

        # "category_name" is optional. But if it is specified
        # then also "attributes" needs to be specified:
        if "category_name" not in doc_generator:
            self.logger.info(
                "No metadata (category name) specified in the payload for this document generator.",
            )
            category_name = ""
            attributes = {}
            category_data = {}
        else:
            category_name = doc_generator["category_name"]
            if "attributes" not in doc_generator:
                self.logger.error(
                    "To generate documents for workspaces of type -> '%s' with metadata, the attributes needs to be specified in the payload! Skipping to next document generator...",
                    workspace_type,
                )
                success = False
                continue
            attributes = doc_generator["attributes"]

            # The following method returns two values: the category ID and
            # a dict of the attributes. If the category is not found
            # on the document template it returns -1 for the category ID
            # and an empty dict for the attribute definitions:
            (
                category_id,
                attribute_definitions,
            ) = self._otcs.get_node_category_definition(
                node_id=template_id,
                category_name=category_name,
            )
            if category_id == -1:
                self.logger.error(
                    "The document template -> '%s' does not have the specified category -> %s. Skipping to next document generator...",
                    template_name,
                    category_name,
                )
                success = False
                continue

            category_data = {str(category_id): {}}

            # now we fill the prepared (but empty) category_data
            # with the actual attribute values from the payload:
            for attribute in attributes:
                attribute_name = attribute["name"]
                attribute_value = attribute["value"]
                attribute_type = attribute_definitions[attribute_name]["type"]
                attribute_id = attribute_definitions[attribute_name]["id"]

                # Special treatment for type user: determine the ID for the login name.
                # the ID is the actual value we have to put in the attribute:
                if attribute_type == "user":
                    user = self._otcs.get_user(
                        name=attribute_value,
                        show_error=True,
                    )
                    user_id = self._otcs.get_result_value(response=user, key="id")
                    if not user_id:
                        self.logger.error(
                            "Cannot find user with login name -> '%s'. Skipping...",
                            attribute_value,
                        )
                        success = False
                        continue
                    attribute_value = user_id
                category_data[str(category_id)][attribute_id] = attribute_value

        if "workspace_folder_path" not in doc_generator:
            self.logger.info(
                "No workspace folder path defined for workspaces of type -> '%s'. Documents will be stored in workspace root.",
                workspace_type,
            )
            workspace_folder_path = []
        else:
            workspace_folder_path = doc_generator["workspace_folder_path"]

        if "exec_as_user" in doc_generator:
            exec_as_user = doc_generator["exec_as_user"]

            # Find the user in the users payload:
            exec_user = next(
                (item for item in self._users if item["name"] == exec_as_user),
                None,
            )
            # Have we found the user in the payload?
            if exec_user is not None:
                self.logger.info(
                    "Executing document generator with user -> '%s'...",
                    exec_as_user,
                )
                # Impersonate as the user specified in the payload:
                self.logger.info("Impersonate user -> '%s'...", exec_as_user)
                result = self.start_impersonation(username=exec_as_user)
                if not result:
                    self.logger.error(
                        "Couldn't impersonate user -> '%s'!",
                        exec_as_user,
                    )
                    continue
                admin_context = False
                authenticated_user = exec_as_user
            else:
                self.logger.error(
                    "Cannot find user with login name -> '%s' for executing document generator. Executing as admin...",
                    exec_as_user,
                )
                admin_context = True
                success = False
        else:
            admin_context = True
            exec_as_user = "admin"

        if admin_context and authenticated_user != "admin":
            # Impersonate as the admin user:
            self.logger.info(
                "Impersonate as admin user -> '%s'...",
                self._otcs.credentials()["username"],
            )
            # Stop the impersonation as a user:
            result = self.stop_impersonation()
            authenticated_user = "admin"

        if category_data:
            self.logger.info(
                "Generate documents for workspace type -> '%s' based on template -> '%s' with metadata -> %s...",
                workspace_type,
                template_name,
                category_data,
            )
        else:
            self.logger.info(
                "Generate documents for workspace type -> '%s' based on template -> '%s' without metadata...",
                workspace_type,
                template_name,
            )

        # Find the workspace type with the name given in the _workspace_types
        # datastructure that has been generated by process_workspace_types() method before:
        workspace_type_id = next(
            (item["id"] for item in self._workspace_types if item["name"] == workspace_type),
            None,
        )

        # The workspace type name is used as a "starts with" search on the
        # workspace type name. This can cause issues, so we prefer to use the type ID
        # if we have it. Otherwise the REST API prefers the name over the type:
        workspace_instances = self._otcs.get_workspace_instances_iterator(
            type_name=workspace_type if not workspace_type_id else None,
            type_id=workspace_type_id,
        )
        for workspace_instance in workspace_instances:
            workspace = workspace_instance.get("data").get("properties")
            workspace_id = workspace["id"]
            workspace_name = workspace["name"]
            if workspace_folder_path:
                workspace_folder = self._otcs.get_node_by_workspace_and_path(
                    workspace_id=workspace_id,
                    path=workspace_folder_path,
                )
                if workspace_folder:
                    workspace_folder_id = self._otcs.get_result_value(
                        response=workspace_folder,
                        key="id",
                    )
                else:
                    # If the workspace template is not matching
                    # the path we may have an error here. Then
                    # we fall back to workspace root level.
                    self.logger.warning(
                        "Folder path does not exist in workspace -> '%s'. Using workspace root level instead...",
                        workspace_name,
                    )
                    workspace_folder_id = workspace_id
            else:
                workspace_folder_id = workspace_id

            document_name = workspace_name + " - " + template_name
            self.logger.info("Generate document -> '%s'...", document_name)

            response = self._otcs.check_node_name(
                parent_id=int(workspace_folder_id),
                node_name=document_name,
            )
            if response["results"]:
                self.logger.warning(
                    "Node -> '%s' does already exist in workspace folder with ID -> %s",
                    document_name,
                    workspace_folder_id,
                )
                continue
            response = self._otcs.create_document_from_template(
                template_id=int(template_id),
                parent_id=int(workspace_folder_id),
                classification_id=int(classification_id),
                category_data=category_data,
                doc_name=document_name,
                doc_description="This document has been auto-generated by Terrarium",
            )
            if not response:
                self.logger.error(
                    "Failed to generate document -> '%s' in workspace -> '%s' (%s) as user -> '%s'!",
                    document_name,
                    workspace_name,
                    workspace_id,
                    exec_as_user,
                )
                success = False
            else:
                self.logger.info(
                    "Successfully generated document -> '%s' in workspace -> '%s' (%s) as user -> '%s'.",
                    document_name,
                    workspace_name,
                    workspace_id,
                    exec_as_user,
                )

    if authenticated_user != "admin":
        # Impersonate as the admin user:
        self.logger.info(
            "Impersonate as admin user -> '%s'...",
            self._otcs.credentials()["username"],
        )
        result = self.stop_impersonation()
        if not result:
            self.logger.error(
                "Couldn't impersonate as admin user -> '%s'!",
                self._otcs.credentials()["username"],
            )
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._doc_generators,
    )

    return success

process_embeddings(section_name='embeddings')

Process additional Aviator embeddings.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'embeddings'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_embeddings")
def process_embeddings(self, section_name: str = "embeddings") -> bool:
    """Process additional Aviator embeddings.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._embeddings:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for embedding in self._embeddings:
        # Check if item has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not embedding.get("enabled", True):
            self.logger.info(
                "Payload for embedding -> '%s' is disabled for FEME. Skipping...",
                str(embedding),
            )
            continue

        # Backwards-compatibility for old syntax "documents":
        if "document_metadata" not in embedding:
            document_metadata = bool(embedding.get("documents", False))
        else:
            document_metadata = bool(embedding.get("document_metadata", False))

        # Backwards-compatibility for old syntax "workspaces":
        if "workspace_metadata" not in embedding:
            workspace_metadata = bool(embedding.get("workspaces", False))
        else:
            workspace_metadata = bool(embedding.get("workspace_metadata", False))

        wait_for_completion = bool(embedding.get("wait_for_completion", True))
        crawl = bool(embedding.get("crawl", False))
        images = bool(embedding.get("images", False))

        # We support 3 ways to determine the node ID(s):
        # 1. Node ID is specified directly in the payload using 'id = "..."'
        # 2. Node is is specified via a nickname of the node using 'nickname = "..."'
        # 3. Nodes are specified via the name of a workspace type. In this
        #    case all nodes of workspace instances are considered.

        node_id = None

        if embedding.get("id", None) is None and "nickname" in embedding:
            response = self._otcs.get_node_from_nickname(
                nickname=embedding.get("nickname"),
            )
            node_id = self._otcs.get_result_value(response=response, key="id")

        else:
            node_id = embedding.get("id", None)
            response = None

        if node_id:
            result = self._otcs.aviator_embed_metadata(
                node_id=node_id,
                node=response,
                wait_for_completion=wait_for_completion,
                crawl=crawl,
                document_metadata=document_metadata,
                workspace_metadata=workspace_metadata,
                images=images,
            )
            if not result:
                success = False
        elif "workspace_types" in embedding:
            workspace_types = embedding["workspace_types"]
            # Handle the case of a single workspace type name:
            if isinstance(workspace_types, str):
                workspace_types = [workspace_types]
            for workspace_type_name in workspace_types:
                self.logger.info(
                    "Embedding metadata of workspace instances with type -> '%s'...", workspace_type_name
                )
                workspace_instances = self._otcs.get_workspace_instances_iterator(type_name=workspace_type_name)
                for workspace in workspace_instances:
                    properties = workspace.get("data").get("properties")
                    self.logger.info(
                        "Embedding metadata of workspace instance -> '%s' (%s)...",
                        properties["name"],
                        properties["id"],
                    )
                    result = self._otcs.aviator_embed_metadata(
                        node_id=None,
                        node=workspace,
                        wait_for_completion=wait_for_completion,
                        crawl=crawl,
                        document_metadata=document_metadata,
                        workspace_metadata=workspace_metadata,
                        images=images,
                    )
                    if not result:
                        success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._embeddings,
    )

    return success

process_exec_commands(section_name='execCommands')

Process Payload items to execute a command.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'execCommands'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_exec_commands")
def process_exec_commands(self, section_name: str = "execCommands") -> bool:
    """Process Payload items to execute a command.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise

    """

    if not self._exec_commands:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for exec_command in self._exec_commands:
        if "command" not in exec_command or not exec_command.get("command"):
            self.logger.error(
                "Command is not specified! It needs to be a non-empty list! Skipping to next command...",
            )
            success = False

            continue
        command = exec_command["command"]

        # Check if exec command has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not exec_command.get("enabled", True):
            self.logger.info(
                "Payload for exec command is disabled. Skipping...",
            )
            continue

        description = exec_command.get("description")
        if not description:
            self.logger.info(
                "Executing command -> %s...",
                command,
            )
        else:
            self.logger.info(
                "Executing command -> %s (%s)...",
                command,
                description,
            )

        try:
            output = subprocess.run(command, stdout=subprocess.PIPE, check=False)
            result = output.stdout.decode("utf-8")
        except Exception:
            self.logger.error(
                "Execution of command -> %s failed!",
                command,
            )
            success = False

        # we need to differentiate 3 cases here:
        # 1. result = None is returned - this is an error (exception)
        # 2. result is empty string - this is OK
        # 3. result is a non-empty string - this is OK - print it to log
        if result is None:
            self.logger.error(
                "Execution of command -> %s failed",
                command,
            )
            success = False
        elif result != "":
            self.logger.info(
                "Execution of command -> %s returned result -> %s.",
                command,
                result,
            )
        else:
            # It is not an error if no result is returned. It depends on the nature of the command
            # if a result is written to stdout or stderr.
            self.logger.info(
                "Execution of command -> %s did not return a result.",
                command,
            )

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._exec_pod_commands,
    )

    return success

process_exec_database_commands(section_name='execDatabaseCommands')

Process commands that should be executed in the PostgreSQL database.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'execDatabaseCommands'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_exec_database_commands")
def process_exec_database_commands(self, section_name: str = "execDatabaseCommands") -> bool:
    """Process commands that should be executed in the PostgreSQL database.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise.

    """

    if not self._exec_database_commands:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    if not psycopg_installed:
        self.logger.warning("Python module 'psycopg' not installed. Cannot execute database commands! Skipping...")
        return False

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for database_command_set in self._exec_database_commands:
        # Check if exec pod command has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not database_command_set.get("enabled", True):
            self.logger.info(
                "Payload for database command set is disabled. Skipping...",
            )
            continue

        db_connection = database_command_set.get("db_connection")
        if not db_connection:
            self.logger.error(
                "To execute a command in a database the connection information needs to be specified in the payload! Skipping to next database command set...",
            )
            success = False
            continue

        db_name = db_connection.get("db_name", None)
        if not db_name:
            self.logger.error(
                "Database connection information is missing the database name! Skipping to next database command set...",
            )
            success = False
            continue
        db_hostname = db_connection.get("db_hostname", None)
        if not db_hostname:
            self.logger.error(
                "Database connection information is missing the database hostname! Skipping to next database command set...",
            )
            success = False
            continue
        db_port = int(db_connection.get("db_port", 5432))
        db_username = db_connection.get("db_username", None)
        if not db_username:
            self.logger.error(
                "Database connection information is missing the database username! Skipping to next database command set...",
            )
            success = False
            continue
        db_password = db_connection.get("db_password", None)
        if not db_password:
            self.logger.error(
                "Database connection information is missing the database password! Skipping to next database command set...",
            )
            success = False
            continue

        connect_string = "dbname={} user={} password={} host={} port={}".format(
            db_name, db_username, db_password, db_hostname, db_port
        )

        db_commands = database_command_set.get("db_commands", [])

        # TODO: Add support for sql file

        db_connection = None  # Predefine for safe access in except
        allowed_verbs = {"SELECT", "INSERT", "UPDATE", "CREATE"}

        try:
            # Using a context managers (with ...) for automatic resource management:
            with psycopg.connect(connect_string) as db_connection:
                self.logger.info(
                    "Connected to database -> '%s' (%s) with user -> '%s'...", db_name, db_hostname, db_username
                )
                with db_connection.cursor() as cursor:
                    for db_command in db_commands:
                        cmd = db_command.get("command", None)
                        if not cmd:
                            self.logger.warning(
                                "Cannot execute database command without SQL statement. Skipping..."
                            )
                            continue
                        params = db_command.get("params", None)
                        if params is not None and isinstance(params, (list, tuple)):
                            self.logger.error(
                                "Database parameters -> %s must be given as a list or tuple!", str(params)
                            )
                            continue
                        # Get the command verb (like "SELECT", "CREATE")
                        verb = cmd.strip().split()[0].upper()
                        if verb not in allowed_verbs:
                            self.logger.error("Database command -> '%s' is not allowed!", verb)
                            continue
                        if params:
                            self.logger.info(
                                "Execute database command -> '%s' with parameters -> %s...", cmd, str(params)
                            )
                        else:
                            self.logger.info("Execute database command -> '%s' without parameters...", cmd)
                        cursor.execute(cmd, params)
                        if verb == "SELECT":
                            response = cursor.fetchall()
                            self.logger.debug("Database response -> '%s'", response)
                db_connection.commit()
        except psycopg.Error as e:
            success = False
            self.logger.error("Database error -> %s", e)
            if db_connection is not None:
                db_connection.rollback()

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._exec_database_commands,
    )

    return success

process_exec_pod_commands(section_name='execPodCommands')

Process commands that should be executed in the Kubernetes pods.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'execPodCommands'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_exec_pod_commands")
def process_exec_pod_commands(self, section_name: str = "execPodCommands") -> bool:
    """Process commands that should be executed in the Kubernetes pods.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise

    """

    self.logger.warning("execPodCommand is deprecated - use kubernetes section with action 'execPodCommands'!")

    if not isinstance(self._k8s, K8s):
        self.logger.error(
            "K8s not setup properly -> Skipping payload section -> '%s'...",
            section_name,
        )
        return False

    if not self._exec_pod_commands:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for exec_pod_command in self._exec_pod_commands:
        pod_name = exec_pod_command.get("pod_name")
        if not pod_name:
            self.logger.error(
                "To execute a command in a pod the pod name needs to be specified in the payload! Skipping to next pod command...",
            )
            success = False
            continue

        command = exec_pod_command.get("command", [])
        if not command:
            self.logger.error(
                "Command is not specified for pod -> %s! It needs to be a non-empty list! Skipping to next pod command...",
                pod_name,
            )
            success = False
            continue

        container = exec_pod_command.get("container", None)
        timeout = int(exec_pod_command.get("timeout", 60))

        # Check if exec pod command has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not exec_pod_command.get("enabled", True):
            self.logger.info(
                "Payload for exec pod command in pod -> '%s' is disabled. Skipping...",
                pod_name,
            )
            continue

        if "description" not in exec_pod_command:
            self.logger.info(
                "Executing command -> %s in pod -> '%s'...",
                command,
                pod_name,
            )
        else:
            description = exec_pod_command["description"]
            self.logger.info(
                "Executing command -> %s in pod -> '%s' (%s)...",
                command,
                pod_name,
                description,
            )

        if "interactive" not in exec_pod_command or exec_pod_command["interactive"] is False:
            result = self._k8s.exec_pod_command(
                pod_name=pod_name, command=command, container=container, timeout=timeout
            )
        else:
            result = self._k8s.exec_pod_command_interactive(
                pod_name=pod_name,
                commands=command,
                timeout=timeout,
            )

        # we need to differentiate 3 cases here:
        # 1. result = None is returned - this is an error (exception)
        # 2. result is empty string - this is OK
        # 3. result is a non-empty string - this is OK - print it to log
        if result is None:
            self.logger.error(
                "Execution of command -> %s in pod -> '%s' failed",
                command,
                pod_name,
            )
            success = False
        elif result != "":
            self.logger.info(
                "Execution of command -> %s in pod -> '%s' returned result -> %s.",
                command,
                pod_name,
                result,
            )
        else:
            # It is not an error if no result is returned. It depends on the nature of the command
            # if a result is written to stdout or stderr.
            self.logger.info(
                "Execution of command -> %s in pod -> '%s' did not return a result.",
                command,
                pod_name,
            )

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._exec_pod_commands,
    )

    return success

process_external_systems(section_name='externalSystems')

Process external systems in payload and create them in Content Server.

The payload section is a list of dicts (each representing one external
system) with these items:
{
    enabled: True or False to enable or disable the payload item
    external_system_name: Name of the external systen.
    external_system_type: Type of the external system.
                          Possible values are
                          * SAP
                          * SuccessFactors
                          * Salesforce
                          * Guidewire
                          * AppWorks Platform
                          * Business Scenario Sample
    base_url: Base URL of the external system
    as_url: Application Server URL of the external system
    username: (Technical) User Name for the connection
    password: Passord of the (technical) user
    oauth_client_id: OAuth client ID
    oauth_client_secret: OAuth client secret
    archive_logical_name: Logical name of Archive for SAP
    archive_certificate_file: Path and filename to certificate file.
                              This file is inside the customizer
                              pof file system.
    skip_connection_test: Should we skip the connection test for this
                          external system?
}
If OAuth Client ID and Client Secret are provided then username
and password are no longer used.

In the payload for SAP external systems there are additional
items "client", "destination" that are processed by init_sap()

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'externalSystems'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Side Effects

• based on system_type different other settings in the dict are set
• reachability is tested and a flag is set in the payload dict

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_external_systems")
def process_external_systems(self, section_name: str = "externalSystems") -> bool:
    """Process external systems in payload and create them in Content Server.

        The payload section is a list of dicts (each representing one external
        system) with these items:
        {
            enabled: True or False to enable or disable the payload item
            external_system_name: Name of the external systen.
            external_system_type: Type of the external system.
                                  Possible values are
                                  * SAP
                                  * SuccessFactors
                                  * Salesforce
                                  * Guidewire
                                  * AppWorks Platform
                                  * Business Scenario Sample
            base_url: Base URL of the external system
            as_url: Application Server URL of the external system
            username: (Technical) User Name for the connection
            password: Passord of the (technical) user
            oauth_client_id: OAuth client ID
            oauth_client_secret: OAuth client secret
            archive_logical_name: Logical name of Archive for SAP
            archive_certificate_file: Path and filename to certificate file.
                                      This file is inside the customizer
                                      pof file system.
            skip_connection_test: Should we skip the connection test for this
                                  external system?
        }
        If OAuth Client ID and Client Secret are provided then username
        and password are no longer used.

        In the payload for SAP external systems there are additional
        items "client", "destination" that are processed by init_sap()

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    Side Effects:
        - based on system_type different other settings in the dict are set
        - reachability is tested and a flag is set in the payload dict

    """

    if not self._external_systems:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # Even if this payload section has been processed successfully before we
    # cannot skip processing it once more to re-initialize self._sap and self._salesforce!

    success: bool = True

    for external_system in self._external_systems:
        #
        # 1: Do sanity checks for the payload:
        #
        if "external_system_name" not in external_system:
            self.logger.error(
                "External System connection needs a logical system name! Skipping to next external system...",
            )
            success = False
            continue
        system_name = external_system["external_system_name"]

        if "external_system_type" not in external_system:
            self.logger.error(
                "External System connection -> '%s' needs a type (SAP, Salesfoce, SuccessFactors, AppWorks Platform)! Skipping...",
                system_name,
            )
            success = False
            continue
        system_type = external_system["external_system_type"]

        self._log_header_callback(
            text="Process External System -> '{}' ({})".format(system_name, system_type),
            char="-",
        )

        # Check if external system has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not external_system.get("enabled", True):
            self.logger.info(
                "Payload for External System -> '%s' (%s) is disabled. Skipping...",
                system_name,
                system_type,
            )
            continue

        """
        Possible Connection Types for external systems:
        "Business Scenario Sample" (Business Scenarios Sample Adapter)
        "ot.sap.c4c.SpiAdapter" (SAP C4C SPI Adapter)
        "ot.sap.c4c.SpiAdapterV2" (C4C SPI Adapter V2)
        "HTTP" (Default WebService Adapter)
        "ot.sap.S4HANAAdapter" (S/4HANA SPI Adapter)
        "SF" (SalesForce Adapter)
        "SFInstance" (SFWebService)
        "GWInstance" (Guidewire Adapter)
        """

        # Set the default settings for the different system types:
        match system_type:
            # Check if we have a SuccessFactors system:
            case "SuccessFactors":
                connection_type = "SFInstance"
                auth_method = "OAUTH"
                username = None
                password = None
            case "SAP":
                connection_type = "HTTP"
                auth_method = "BASIC"
                oauth_client_id = None
                oauth_client_secret = None
            case "Salesforce":
                connection_type = "SF"
                auth_method = "OAUTH"
                username = None
                password = None
            case "Guidewire":
                connection_type = "GWInstance"
                auth_method = "BASIC"
                oauth_client_id = None
                oauth_client_secret = None
            case "AppWorks Platform":
                connection_type = "HTTP"
                auth_method = "BASIC"
                oauth_client_id = None
                oauth_client_secret = None
            case "Business Scenario Sample":
                connection_type = "Business Scenario Sample"
                auth_method = "BASIC"
                oauth_client_id = None
                oauth_client_secret = None
            case _:
                self.logger.error(
                    "Unsupported system_type defined -> '%s'",
                    system_type,
                )
                return False

        base_url = external_system.get("base_url", "")

        if "as_url" not in external_system:
            self.logger.warning(
                "External system connection -> '%s' needs an application server URL! Skipping to next external system...",
                system_name,
            )
            success = False
            continue
        as_url = external_system["as_url"]

        self.logger.info(
            "Processing external system -> '%s' (type -> '%s', connection type -> '%s', endpoint -> '%s')...",
            system_name,
            system_type,
            connection_type,
            as_url,
        )

        skip_connection_test = external_system.get("skip_connection_test", False)
        # If skip_connection_test is not enabled in payload and the external system is
        # not of type 'Business Scenario Sample', we run the external system check:
        if not skip_connection_test and system_type != "Business Scenario Sample":
            # Check if external system is reachable and
            # update the payload dict with a "reachable" key/value pair:
            if not self.check_external_system(external_system=external_system):
                self.logger.warning(
                    "External System connection -> '%s' (%s) is not reachable! Skipping to next external system...",
                    system_name,
                    system_type,
                )
                success = False
                continue
        else:
            self.logger.info(
                "Skipped reachability check for external system -> '%s' (%s)...",
                system_name,
                system_type,
            )

        # Read either username/password (BASIC) or client ID / secret (OAuth)
        match auth_method:
            case "BASIC":
                if "username" not in external_system:
                    self.logger.warning(
                        "External System connection -> '%s' needs a user name for BASIC authentication! Skipping to next external system...",
                        system_name,
                    )
                    continue
                if "password" not in external_system:
                    self.logger.warning(
                        "External System connection -> '%s' needs a password for BASIC authentication! Skipping to next external system...",
                        system_name,
                    )
                    continue
                username = external_system["username"]
                password = external_system["password"]
                oauth_client_id = ""
                oauth_client_secret = ""

            case "OAUTH":
                if "oauth_client_id" not in external_system:
                    self.logger.error(
                        "External System connection -> '%s' is missing OAuth client ID! Skipping to next external system...",
                        system_name,
                    )
                    success = False
                    continue
                if "oauth_client_secret" not in external_system:
                    self.logger.error(
                        "External System connection -> '%s' is missing OAuth client secret! Skipping to next external system...",
                        system_name,
                    )
                    success = False
                    continue
                oauth_client_id = external_system["oauth_client_id"]
                oauth_client_secret = external_system["oauth_client_secret"]
                # For backward compatibility we also read username/password
                # with OAuth settings:
                username = external_system["username"] if external_system.get("username") else None
                password = external_system["password"] if external_system.get("password") else None
            case _:
                self.logger.error(
                    "Unsupported authorization method specified (%s) , Skipping ... ",
                    auth_method,
                )
                return False
        # end match

        # We do this existance test late in this function to make sure the payload
        # datastructure is properly updated for debugging purposes.
        self.logger.info(
            "Test if external system -> '%s' does already exist...",
            system_name,
        )
        if self._otcs.get_external_system_connection(connection_name=system_name):
            self.logger.info(
                "External system connection -> '%s' already exists.",
                system_name,
            )
            # This is for handling re-runs of customizer pod where the transports
            # are skipped and thus self._sap or self._salesforce may not be initialized:
            if system_type == "SAP" and not self._sap:
                self.logger.info(
                    "Re-Initialize SAP connection for external system -> '%s'...",
                    system_name,
                )
                # Initialize SAP object responsible for communication to SAP:
                self._sap = self.init_sap(external_system)
            if system_type == "Salesforce" and not self._salesforce:
                self.logger.info(
                    "Re-Initialize Salesforce connection for external system -> '%s'...",
                    system_name,
                )
                # Initialize Salesforce object responsible for communication to Salesforce:
                self._salesforce = self.init_salesforce(
                    salesforce_external_system=external_system,
                )
            if system_type == "SuccessFactors" and not self._successfactors:
                self.logger.info(
                    "Re-Initialize SuccessFactors connection for external system -> '%s'...",
                    system_name,
                )
                # Initialize SuccessFactors object responsible for communication to SuccessFactors:
                self._successfactors = self.init_successfactors(
                    sucessfactors_external_system=external_system,
                )
            if system_type == "Guidewire" and "policy" in system_name.lower() and not self._guidewire_policy_center:
                self.logger.info(
                    "Re-Initialize Guidewire connection for external system -> '%s'...",
                    system_name,
                )
                # Initialize Guidewire object responsible for communication to Guidewire Policy Center:
                self._guidewire_policy_center = self.init_guidewire(
                    guidewire_external_system=external_system,
                )
            if system_type == "Guidewire" and "claim" in system_name.lower() and not self._guidewire_claims_center:
                self.logger.info(
                    "Re-Initialize Guidewire connection for external system -> '%s'...",
                    system_name,
                )
                # Initialize Guidewire object responsible for communication to Guidewire Claims Center:
                self._guidewire_claims_center = self.init_guidewire(
                    guidewire_external_system=external_system,
                )

            continue

        #
        # Create External System:
        #

        response = self._otcs.add_external_system_connection(
            connection_name=system_name,
            connection_type=connection_type,
            as_url=as_url,
            base_url=base_url,
            username=str(username),
            password=str(password),
            authentication_method=auth_method,
            client_id=oauth_client_id,
            client_secret=oauth_client_secret,
        )
        if response is None:
            self.logger.error(
                "Failed to create external system -> '%s'; type -> '%s'!",
                system_name,
                connection_type,
            )
            success = False
        else:
            self.logger.info(
                "Successfully created external system -> '%s'.",
                system_name,
            )

        #
        # In case of an SAP external system we also initialize the SAP object
        #
        if system_type == "SAP":
            # Initialize SAP object responsible for communication to SAP:
            self._sap = self.init_sap(sap_external_system=external_system)

        #
        # In case of an SuccessFactors external system we also initialize the SuccessFactors object
        #
        if system_type == "SuccessFactors":
            # Initialize SuccessFactors object responsible for communication to SuccessFactors:
            self._successfactors = self.init_successfactors(
                sucessfactors_external_system=external_system,
            )

        #
        # In case of an Salesforce external system we also initialize the Salesforce object
        #
        if system_type == "Salesforce":
            # Initialize Salesforce object responsible for communication to Salesforce:
            self._salesforce = self.init_salesforce(
                salesforce_external_system=external_system,
            )

        #
        # In case of an Guidewire external system we also initialize the Guidewire objects:
        #
        if system_type == "Guidewire":
            if "claim" in system_name.lower():
                # Initialize Guidewire Claim Center object:
                self._guidewire_claims_center = self.init_guidewire(
                    guidewire_external_system=external_system,
                )
            elif "policy" in system_name.lower():
                # Initialize Guidewire Policy Center object:
                self._guidewire_policy_center = self.init_guidewire(
                    guidewire_external_system=external_system,
                )

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._external_systems,
    )

    return success

process_group_placeholders()

Replace a group placeholder (sourrounded by %%...%%) with the actual ID of the Content Server group.

For this we prepare a lookup dict. The dict self._placeholder_values already includes lookups for the OTCS and OTAWP OTDS resource IDs (see main.py)

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_group_placeholders")
def process_group_placeholders(self) -> None:
    """Replace a group placeholder (sourrounded by %%...%%) with the actual ID of the Content Server group.

    For this we prepare a lookup dict. The dict self._placeholder_values already includes
    lookups for the OTCS and OTAWP OTDS resource IDs (see main.py)
    """

    for group in self._groups:
        group_name = group.get("name")
        if not group_name:
            self.logger.error(
                "Group needs a name for placeholder definition in payload! Skipping...",
            )
            continue

        # Check if group has been explicitly disabled in payload
        # (enabled = false). In this case we skip the payload element:
        if not group.get("enabled", True):
            self.logger.info(
                "Payload for group -> '%s' is disabled. Skipping...",
                group_name,
            )
            continue

        # Now we determine the ID. Either it is in the payload section from
        # the current customizer run or we try to look it up in the system.
        # The latter case may happen if the customizer pod got restarted.
        group_id = self.determine_group_id(group=group)
        if not group_id:
            self.logger.warning(
                "Group needs an ID for placeholder definition. Skipping...",
            )
            continue

        # Add Group with its ID to the dict self._placeholder_values:
        self._placeholder_values["OTCS_GROUP_ID_" + group_name.upper().replace(" & ", "_").replace(" ", "_")] = str(
            group_id,
        )

    self.logger.debug(
        "Placeholder values after group processing = %s",
        self._placeholder_values,
    )

process_groups(section_name='groups')

Process groups in payload and create them in Content Server.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'groups'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Side Effects

The group items are modified by adding an "id" dict element that includes the technical ID of the group in Content Server

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_groups")
def process_groups(self, section_name: str = "groups") -> bool:
    """Process groups in payload and create them in Content Server.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    Side Effects:
        The group items are modified by adding an "id" dict element that
        includes the technical ID of the group in Content Server

    """

    if not self._groups:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # First run through groups: create all groups in payload
    # and store the IDs of the created groups:
    for group in self._groups:
        group_name = group.get("name")
        if not group_name:
            self.logger.error("Group needs a name inb payload! Skipping...")
            success = False
            continue

        # Check if group has been explicitly disabled in payload
        # (enabled = false). In this case we skip the payload element:
        if not group.get("enabled", True):
            self.logger.info(
                "Payload for group -> '%s' is disabled. Skipping...",
                group_name,
            )
            continue

        # Check if the group does already exist (e.g. if job is restarted)
        group_id = self.determine_group_id(group=group)
        if group_id:
            self.logger.info(
                "Found existing group -> '%s' (%s). Skipping to next group...",
                group_name,
                group_id,
            )
            continue

        # Now we know it is a new group...
        new_group = self._otcs.add_group(name=group_name)
        if new_group:
            new_group_id = self._otcs.get_result_value(response=new_group, key="id")
            self.logger.info(
                "Successfully created group -> '%s' (%s).",
                group_name,
                new_group_id,
            )
            # Remember the OTCS group ID in the payload structure:
            group["id"] = new_group_id
        else:
            self.logger.error("Failed to create group -> '%s'!", group_name)
            success = False
            continue

        # Assign usage privileges to the new group:
        usage_privileges = group.get("usage_privileges", [])
        for usage_privilege in usage_privileges:
            response = self._otcs.assign_usage_privilege(
                usage_privilege=usage_privilege,
                member_id=new_group_id,
            )
            if response:
                self.logger.info(
                    "Successfully assigned usage privilege -> '%s' to group -> '%s' (%s).",
                    usage_privilege,
                    group_name,
                    new_group_id,
                )
            else:
                self.logger.info(
                    "Failed to assign usage privilege -> '%s' to group -> '%s' (%s)!",
                    usage_privilege,
                    group_name,
                    new_group_id,
                )

        # Assign usage privileges to the new group:
        object_privileges = group.get("object_privileges", [])
        for object_type in object_privileges:
            response = self._otcs.assign_object_privilege(
                object_type=object_type,
                member_id=new_group_id,
            )
            if response:
                self.logger.info(
                    "Successfully assigned object privilege -> '%s' to group -> '%s' (%s).",
                    object_type,
                    group_name,
                    new_group_id,
                )
            else:
                self.logger.info(
                    "Failed to assign object privilege -> '%s' to group -> '%s' (%s)!",
                    object_type,
                    group_name,
                    new_group_id,
                )

    # end for group in self._groups: (first run)

    # Second run through groups: create all group memberships
    # (nested groups) based on the IDs created in first run:
    for group in self._groups:
        group_name = group.get("name")
        group_id = group.get("id")  # this should have been set in the first loop
        if not group_id:
            self.logger.error(
                "Group -> '%s' does not have an ID! Creation may have failed before. Skipping...", group_name
            )
            success = False
            continue
        parent_group_names = group.get("parent_groups", [])
        for parent_group_name in parent_group_names:
            # First, try to find parent group in payload by parent group name:
            parent_group = next(
                (item for item in self._groups if item["name"] == parent_group_name),
                None,
            )
            if parent_group is None:
                # If this didn't work, try to get the parent group from OTCS. This covers
                # cases where the parent group is system-generated or part
                # of a former payload processing run (or part of another payload):
                parent_group = self._otcs.get_group(name=parent_group_name)
                parent_group_id = self._otcs.get_result_value(
                    response=parent_group,
                    key="id",
                )
                if not parent_group_id:
                    self.logger.error(
                        "Parent group -> '%s' not found! Skipping...",
                        parent_group_name,
                    )
                    success = False
                    continue
            elif "id" not in parent_group:
                self.logger.error(
                    "Parent group -> '%s' does not have an ID! Cannot establish group nesting. Skipping...",
                    parent_group["name"],
                )
                success = False
                continue
            else:  # we can read the ID from the
                parent_group_id = parent_group["id"]

            # retrieve all members of the parent group (1 = get only groups)
            # to check if the current group is already a member in the parent group:
            members = self._otcs.get_group_members(
                group=parent_group_id,
                member_type=1,
            )
            if self._otcs.exist_result_item(
                response=members,
                key="id",
                value=group["id"],
            ):
                self.logger.info(
                    "Group -> '%s' (%s) is already a member of parent group -> '%s' (%s). Skipping to next parent group...",
                    group_name,
                    group_id,
                    parent_group_name,
                    parent_group_id,
                )
            else:
                self.logger.info(
                    "Add group -> '%s' (%s) to parent group -> '%s' (%s)...",
                    group_name,
                    group_id,
                    parent_group_name,
                    parent_group_id,
                )
                self._otcs.add_group_member(
                    member_id=group["id"],
                    group_id=parent_group_id,
                )
        # end for parent_group_name in parent_group_names:

        # Assign application roles to the new group:
        application_roles = group.get("application_roles", [])
        if application_roles:
            self.logger.info(
                "Group -> '%s' has application roles -> %s. Assigning...", group_name, str(application_roles)
            )
        for role in application_roles:
            group_partition = self._otcs.config()["partition"]
            if not group_partition:
                self.logger.error("Group partition not found! Skipping application role -> %s...", str(role))
                success = False
                continue

            # Split role at the @ sign to identify Partitions
            role_parts = role.split("@", 1)
            role_name = role_parts[0]
            role_partition = role_parts[1] if len(role_parts) > 1 else "OAuthClients"

            # This is on OTDS method (not OTCS) thuis the group ID is the group name!
            response = self._otds.assign_group_to_application_role(
                group_id=group_name,
                group_partition=group_partition,
                role_name=role_name,
                role_partition=role_partition,
            )

            if response:
                self.logger.info(
                    "Successfully assigned application role -> '%s' in partition -> '%s' to group -> '%s' (%s) in partition -> '%s'.",
                    role_name,
                    role_partition,
                    group_name,
                    group_id,
                    group_partition,
                )
            else:
                self.logger.info(
                    "Failed to assign application role -> '%s' in partition -> %s to group -> '%s' (%s) in partition -> '%s'!",
                    role_name,
                    role_partition,
                    group_name,
                    group_id,
                    group_partition,
                )
        # end for role in application_roles:
    # end for group in self._groups: (second run)

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._groups,
    )

    return success

process_groups_core_share(section_name='groupsCoreShare')

Process groups in payload and create them in Core Share.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'groupsCoreShare'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_groups_core_share")
def process_groups_core_share(self, section_name: str = "groupsCoreShare") -> bool:
    """Process groups in payload and create them in Core Share.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not isinstance(self._core_share, CoreShare):
        self.logger.error(
            "Core Share connection not setup properly. Skipping payload section '%s'...",
            section_name,
        )
        return False

    if not self._groups:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # Create all groups specified in payload
    # and store the IDs of the created Core Share groups:
    for group in self._groups:
        group_name = group.get("name")
        if not group_name:
            self.logger.error("Group needs a name. Skipping...")
            success = False
            continue

        # Check if group has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not group.get("enabled", True):
            self.logger.info(
                "Payload for group -> '%s' is disabled. Skipping...",
                group_name,
            )
            continue

        # Core Share is disabled per default. There needs to be "enable_core_share" in payload
        # and it needs to be True:
        if not group.get("enable_core_share", False):
            self.logger.info(
                "Group -> '%s' is not enabled for Core Share. Skipping...",
                group_name,
            )
            continue

        # Check if the group does already exist (e.g. if job is restarted)
        core_share_group = self._core_share.get_group_by_name(name=group_name)
        core_share_group_id = self._core_share.get_result_value(
            response=core_share_group,
            key="id",
        )
        if core_share_group_id:
            self.logger.info(
                "Found existing Core Share group -> '%s' (%s). Just do cleanup of potential left-overs...",
                group_name,
                core_share_group_id,
            )
            # Write Core Share group ID back into the payload (for the success file)
            group["core_share_id"] = core_share_group_id

            # For existing users we want to cleanup possible left-overs form old deployments
            self.logger.info(
                "Cleanup existing file shares of Core Share group -> '%s' (%s)...",
                group_name,
                core_share_group_id,
            )
            response = self._core_share.cleanup_group_shares(
                group_id=core_share_group_id,
            )
            if not response:
                self.logger.error("Failed to cleanup group shares!")

            continue

        self.logger.info(
            "Creating a new Core Share group -> '%s'...",
            group_name,
        )

        # Now we know it is a new group...
        new_group = self._core_share.add_group(group_name=group_name)
        new_group_id = self._core_share.get_result_value(
            response=new_group,
            key="id",
        )
        if new_group_id:
            # Store the Microsoft 365 group ID in payload:
            group["core_share_id"] = new_group_id
            self.logger.info(
                "Successfully created Core Share group -> '%s' with ID -> %s.",
                group_name,
                new_group_id,
            )
        else:
            self.logger.error(
                "Failed to create Core Share group -> %s!",
                group_name,
            )
            success = False

    # Core Share groups cannot be nested. So we are fone here.

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._groups,
    )

    return success

process_groups_m365(section_name='groupsM365')

Process groups in payload and create them in Microsoft 365.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'groupsM365'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_groups_m365")
def process_groups_m365(self, section_name: str = "groupsM365") -> bool:
    """Process groups in payload and create them in Microsoft 365.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not isinstance(self._m365, M365):
        self.logger.error(
            "Microsoft 365 connection not setup properly. Skipping payload section '%s'...",
            section_name,
        )
        return False

    if not self._groups:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # First run through groups: create all groups in payload
    # and store the IDs of the created groups:
    for group in self._groups:
        group_name = group.get("name")
        if not group_name:
            self.logger.error("Group needs a name. Skipping...")
            success = False
            continue

        # Check if group has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not group.get("enabled", True):
            self.logger.info(
                "Payload for group -> '%s' is disabled. Skipping...",
                group_name,
            )
            continue
        # M365 is disabled per default. There needs to be "enable_o365" in payload
        # and it needs to be True:
        if not group.get("enable_o365", False):
            self.logger.info(
                "Microsoft 365 is not enabled in payload for group -> '%s'. Skipping...",
                group_name,
            )
            continue

        # Check if the group does already exist (e.g. if job is restarted)
        # as this is a pattern search it could return multiple groups:
        existing_groups = self._m365.get_group(group_name=group_name)

        if existing_groups and existing_groups["value"]:
            self.logger.debug(
                "Found existing Microsoft 365 groups -> %s",
                existing_groups["value"],
            )
            # Get list of all matching groups:
            existing_groups_list = existing_groups["value"]
            # Find the group with the exact match of the name:
            existing_group = next(
                (item for item in existing_groups_list if item["displayName"] == group_name),
                None,
            )
            # Have we found an exact match?
            if existing_group is not None:
                self.logger.info(
                    "Found existing Microsoft 365 group -> '%s' (%s) - skip creation of group...",
                    existing_group["displayName"],
                    existing_group["id"],
                )
                # Write M365 group ID back into the payload (for the success file)
                group["m365_id"] = existing_group["id"]
                continue

        self.logger.info(
            "Creating a new Microsoft 365 group -> '%s'...",
            group_name,
        )

        # Now we know it is a new group...
        new_group = self._m365.add_group(name=group_name)
        if new_group is not None:
            # Store the Microsoft 365 group ID in payload:
            group["m365_id"] = new_group["id"]
            self.logger.info(
                "New Microsoft 365 group -> '%s' with ID -> %s has been created",
                group_name,
                group["m365_id"],
            )
        else:
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._groups,
    )

    return success

process_groups_salesforce(section_name='groupsSalesforce')

Process groups in payload and create them in Salesforce.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'groupsSalesforce'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_groups_salesforce")
def process_groups_salesforce(self, section_name: str = "groupsSalesforce") -> bool:
    """Process groups in payload and create them in Salesforce.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not isinstance(self._salesforce, Salesforce):
        self.logger.error(
            "Salesforce connection not setup properly. Skipping payload section '%s'...",
            section_name,
        )
        return False

    if not self._groups:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # First run through groups: create all groups in payload
    # and store the IDs of the created groups:
    for group in self._groups:
        group_name = group.get("name")
        if not group_name:
            self.logger.error("Group needs a name. Skipping...")
            success = False
            continue

        # Check if group has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not group.get("enabled", True):
            self.logger.info(
                "Payload for group -> '%s' is disabled. Skipping...",
                group_name,
            )
            continue

        # Salesforce is disabled per default. There needs to be "enable_salesforce" in payload
        # and it needs to be True:
        if not group.get("enable_salesforce", False):
            self.logger.info(
                "Salesforce is not enabled in payload for group -> '%s'. Skipping...",
                group_name,
            )
            continue

        # Check if the group does already exist (e.g. if job is restarted)
        existing_group_id = self._salesforce.get_group_id(group_name=group_name)
        if existing_group_id:
            self.logger.info(
                "Found existing Salesforce group -> '%s' (%s). Skipping...",
                group_name,
                existing_group_id,
            )
            # Write M365 group ID back into the payload (for the success file)
            group["salesforce_id"] = existing_group_id
            continue

        self.logger.info(
            "Creating a new Salesforce group -> '%s'...",
            group_name,
        )

        # Now we know it is a new group...
        new_group = self._salesforce.add_group(group_name=group_name)
        new_group_id = self._salesforce.get_result_value(
            response=new_group,
            key="id",
        )
        if new_group_id:
            # Store the Microsoft 365 group ID in payload:
            group["salesforce_id"] = new_group_id
            self.logger.info(
                "Successfully created Salesforce group -> '%s' with ID -> %s.",
                group_name,
                new_group_id,
            )
        else:
            self.logger.error(
                "Failed to create Salesforce group -> '%s'!",
                group_name,
            )
            success = False

    # Second run through groups: create all group memberships
    # (nested groups) based on the IDs created in first run:
    for group in self._groups:
        if "salesforce_id" not in group:
            self.logger.info(
                "Group -> '%s' does not have an Salesforce ID. Skipping...",
                group["name"],
            )
            # Not all groups may be enabled for Salesforce. This is not an error.
            continue
        group_id = group["salesforce_id"]
        parent_group_names = group["parent_groups"]
        for parent_group_name in parent_group_names:
            # First, try to find parent group in payload by parent group name:
            parent_group = next(
                (item for item in self._groups if item["name"] == parent_group_name),
                None,
            )
            if not parent_group:
                self.logger.error(
                    "Parent group -> '%s' not found. Cannot establish group nesting. Skipping...",
                    parent_group["name"],
                )
                success = False
                continue
            if "salesforce_id" not in parent_group:
                self.logger.info(
                    "Parent group -> '%s' does not have a Salesforce ID. Cannot establish group nesting. Parent group may not be enabled for Salesforce. Skipping...",
                    parent_group["name"],
                )
                # We don't treat this as an error - there may be payload groups which are not enabled for Salesforce!
                continue

            parent_group_id = parent_group["salesforce_id"]

            # retrieve all members of the parent group
            # to check if the current group is already a member in the parent group:
            members = self._salesforce.get_group_members(group_id=parent_group_id)
            if self._salesforce.exist_result_item(
                members,
                "UserOrGroupId",
                group_id,
            ):
                self.logger.info(
                    "Salesforce group -> '%s' (%s) is already a member of parent Salesforce group -> '%s' (%s). Skipping to next parent group...",
                    group["name"],
                    group["id"],
                    parent_group_name,
                    parent_group_id,
                )
                continue
            self.logger.info(
                "Add Salesforce group -> '%s' (%s) to parent Salesforce group -> '%s' (%s)...",
                group["name"],
                group_id,
                parent_group_name,
                parent_group_id,
            )
            self._salesforce.add_group_member(
                group_id=parent_group_id,
                member_id=group_id,
            )

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._groups,
    )

    return success

process_holds(section_name='holds')

Process Records Management Holds.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'holds'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_holds")
def process_holds(self, section_name: str = "holds") -> bool:
    """Process Records Management Holds.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._holds:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for hold in self._holds:
        hold_name = hold.get("name")
        if not hold_name:
            self.logger.error("Cannot create a hold without a name! Skipping...")
            continue

        hold_type = hold.get("type")
        if not hold_type:
            self.logger.error(
                "Cannot create hold -> '%s' without a type! Skipping...",
                hold_name,
            )
            success = False
            continue

        # Check if hold has been explicitly disabled in payload
        # (enabled = false). In this case we skip this payload element:
        if not hold.get("enabled", True):
            self.logger.info(
                "Payload for hold -> '%s' (%s) is disabled. Skipping...",
                hold_name,
                hold_type,
            )
            continue

        hold_group = hold.get("group")
        hold_comment = hold.get("comment", "")
        hold_alternate_id = hold.get("alternate_id")
        hold_date_applied = hold.get("date_applied")
        hold_date_to_remove = hold.get("date_to_remove")

        response = self._otcs.get_node_by_volume_and_path(
            volume_type=self._otcs.VOLUME_TYPE_RECORDS_MANAGEMENT,
            path=["Hold Maintenance"],
        )
        if not response:
            self.logger.error("Cannot find 'Records Management' volume!")
            continue
        holds_maintenance_id = self._otcs.get_result_value(
            response=response,
            key="id",
        )
        if not holds_maintenance_id:
            self.logger.error(
                "Cannot find 'Holds Maintenance' folder in 'Records Management' volume!",
            )
            continue

        if hold_group:
            # Check if the Hold Group (folder) does already exist.
            response = self._otcs.get_node_by_parent_and_name(
                parent_id=holds_maintenance_id,
                name=hold_group,
            )
            parent_id = self._otcs.get_result_value(response=response, key="id")
            if not parent_id:
                response = self._otcs.create_item(
                    parent_id=holds_maintenance_id,
                    item_type=self._otcs.ITEM_TYPE_HOLD,
                    item_name=hold_group,
                )
                parent_id = self._otcs.get_result_value(response=response, key="id")
                if not parent_id:
                    self.logger.error(
                        "Failed to create hold group -> '%s'!",
                        hold_group,
                    )
                    continue
        else:
            parent_id = holds_maintenance_id

        # Holds are special - they ahve folders that cannot be traversed
        # in the normal way - we need to get the whole list of holds and use
        # specialparameters for the exist_result_items() method as the REST
        # API calls delivers a results->data->holds structure (not properties)
        response = self._otcs.get_records_management_holds()
        if self._otcs.exist_result_item(
            response=response,
            key="HoldName",
            value=hold_name,
            property_name="holds",
        ):
            self.logger.info(
                "Hold -> '%s' does already exist. Skipping...",
                hold_name,
            )
            continue

        hold = self._otcs.create_records_management_hold(
            hold_type=hold_type,
            name=hold_name,
            comment=hold_comment,
            alternate_id=hold_alternate_id,
            parent_id=int(parent_id),
            date_applied=hold_date_applied,
            date_to_remove=hold_date_to_remove,
        )

        if hold and hold["holdID"]:
            self.logger.info(
                "Successfully created hold -> '%s' with ID -> %s.",
                hold_name,
                hold["holdID"],
            )
        else:
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._holds,
    )

    return success

process_items(items, section_name='items')

Process items specified in payload and create them in Content Server.

Parameters:

Name	Type	Description	Default
items	list	List of items to create (need this as parameter as we have multiple lists). Each list item in the payload is a dict with this structure: { enabled = "..." name = "..." description = "..." nickname = "..." parent_nickname = "..." parent_path = [...] parent_volume = ... original_nickname = "..." original_path = [] type = ... url = "..." details = { "scheduledbotdetails" : ... } # additional parameters }	required
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'items'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_items")
def process_items(self, items: list, section_name: str = "items") -> bool:
    """Process items specified in payload and create them in Content Server.

    Args:
        items (list):
            List of items to create (need this as parameter as we
            have multiple lists).
            Each list item in the payload is a dict with this structure:
            {
                enabled = "..."
                name = "..."
                description = "..."
                nickname = "..."
                parent_nickname = "..."
                parent_path = [...]
                parent_volume = ...
                original_nickname = "..."
                original_path = []
                type = ...
                url = "..."
                details = {
                    "scheduledbotdetails" : ...
                } # additional parameters
            }
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not items:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )

        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for item in items:
        item_name = item.get("name")
        if not item_name:
            self.logger.error("Item needs a name. Skipping...")
            continue

        # Check if element has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not item.get("enabled", True):
            self.logger.info(
                "Payload for item -> '%s' is disabled. Skipping...",
                item_name,
            )
            continue

        item_description = item.get("description", "")
        item_nickname = item.get("nickname", None)
        parent_nickname = item.get("parent_nickname", None)
        parent_path = item.get("parent_path", None)

        if parent_nickname:  # parent nickname has preference over parent path
            parent_node = self._otcs.get_node_from_nickname(
                nickname=parent_nickname,
            )
            parent_id = self._otcs.get_result_value(response=parent_node, key="id")
            if not parent_id:
                self.logger.error(
                    "Item -> '%s' has a parent nickname -> '%s' that does not exist. Skipping...",
                    item_name,
                    parent_nickname,
                )
                success = False
                continue
        elif parent_path is not None:  # parent_path can be [] which is valid for top-level items!
            parent_volume = item.get("parent_volume", self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE)
            parent_node = self._otcs.get_node_by_volume_and_path(
                volume_type=parent_volume,
                path=parent_path,
                create_path=True,
            )
            parent_id = self._otcs.get_result_value(response=parent_node, key="id")
            if not parent_id:
                # if not parent_node:
                self.logger.error(
                    "Item -> '%s' has a parent path -> %s that does not exist and couldn't be created in volume -> %d. Skipping...",
                    item_name,
                    parent_path,
                    self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE,
                )
                success = False
                continue
        else:
            self.logger.error("The parent for the item -> '%s' is not specified by nickname nor path!", item_name)
            success = False
            continue

        # Handling for shortcut items that have an orginal node:
        original_nickname = item.get("original_nickname")
        original_path = item.get("original_path")

        if original_nickname:
            original_node = self._otcs.get_node_from_nickname(
                nickname=original_nickname,
            )
            original_id = self._otcs.get_result_value(
                response=original_node,
                key="id",
            )
            if not original_id:
                # if not original_node:
                self.logger.error(
                    "Item -> '%s' has a original nickname -> '%s' that does not exist. Skipping...",
                    item_name,
                    original_nickname,
                )
                success = False
                continue
        elif original_path is not None:  # original_path can be [] which is valid for top-level items!
            original_volume = item.get("original_volume", self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE)
            original_node = self._otcs.get_node_by_volume_and_path(
                volume_type=original_volume,
                path=original_path,
            )
            original_id = self._otcs.get_result_value(
                response=original_node,
                key="id",
            )
            if not original_id:
                # if not original_node:
                self.logger.error(
                    "Item -> '%s' has a original path that does not exist. Skipping...",
                    item_name,
                )
                success = False
                continue
        else:
            original_id = 0

        if "type" not in item:
            self.logger.error("Item -> '%s' needs a type. Skipping...", item_name)
            success = False
            continue

        item_type = int(item.get("type", self._otcs.ITEM_TYPE_FOLDER))
        item_url = item.get("url", "")
        item_details = item.get("details", {})
        create_item_details = item.get("create_details", {})
        # check that we have the required information
        # for the given item type:
        match item_type:
            case self._otcs.ITEM_TYPE_URL:  # URL
                if item_url == "":
                    self.logger.error(
                        "Item -> '%s' has type URL but the URL is not in the payload. Skipping...",
                        item_name,
                    )
                    success = False
                    continue
            case self._otcs.ITEM_TYPE_SHORTCUT:  # Shortcut
                if not original_id:
                    self.logger.error(
                        "Item -> '%s' has type Shortcut but the original item is not in the payload. Skipping...",
                        item_name,
                    )
                    success = False
                    continue
            case self._otcs.ITEM_TYPE_COLLECTION:  # Collection
                item_ids = item.get("ids", None)
                if item_ids is None:
                    self.logger.error(
                        "Item -> '%s' has type Collection but the list of collected items is not provided in payload. Skipping...",
                        item_name,
                    )
                    success = False
                    continue
            case self._otcs.ITEM_TYPE_SCHEDULED_BOT:  # Scheduled Bots
                if any(
                    key not in item_details for key in ["xecmpfJobType", "scheduledbotdetails"]
                ):  #  not in item_details:
                    self.logger.error(
                        "Item -> '%s' has type Scheduled Bot but the mandatory details are not provided in payload (xecmpfJobType, scheduledbotdetails). Skipping...",
                        item_name,
                    )
                    success = False
                    continue

                create_item_details["xecmpfJobType"] = item_details["xecmpfJobType"]

        # Check if an item with the same name does already exist.
        # This can also be the case if the python container runs a 2nd time.
        # For this reason we are also not issuing an error but just an info (False):
        response = self._otcs.get_node_by_parent_and_name(
            parent_id=int(parent_id),
            name=item_name,
            show_error=False,
        )
        if self._otcs.get_result_value(response=response, key="name") == item_name:
            self.logger.info(
                "Item with name -> '%s' does already exist in parent folder with ID -> %s.",
                item_name,
                parent_id,
            )
            continue
        response = self._otcs.create_item(
            parent_id=int(parent_id),
            item_type=item_type,
            item_name=item_name,
            item_description=item_description,
            url=item_url,
            original_id=int(original_id),
            **create_item_details,
        )
        node_id = self._otcs.get_result_value(response=response, key="id")
        if not node_id:
            self.logger.error(
                "Failed to create item -> '%s' under parent%s!",
                item_name,
                " with nickname -> '{}'".format(parent_nickname)
                if parent_nickname
                else " path -> {} in volume -> {}".format(parent_path, parent_volume),
            )
            success = False
            continue

        self.logger.info(
            "Successfully created item -> '%s' with ID -> %s under parent%s.",
            item_name,
            node_id,
            " with nickname -> '{}'".format(parent_nickname)
            if parent_nickname
            else " path -> {} in volume -> {}".format(parent_path, parent_volume),
        )

        # Special handling for scheduled bot items:
        if item_type == self._otcs.ITEM_TYPE_SCHEDULED_BOT:
            scheduled_bot_details = item_details.get("scheduledbotdetails", {})
            if not scheduled_bot_details:
                self.logger.error("Failed to get details of scheduled bot item -> '%s'!", item_name)
                success = False
                continue
            start_mode = scheduled_bot_details.get("startmodus")
            if not start_mode:
                self.logger.error("Failed to get start mode of scheduled bot item -> '%s'!", item_name)
                success = False
                continue
            start_mode = start_mode.get("startMode")
            if not start_mode:
                self.logger.error("Failed to get start mode of scheduled bot item -> '%s'!", item_name)
                success = False
                continue
            old_schedule_data = scheduled_bot_details.get("oldscheduleData")
            # Check if this bot should start after another bot:
            if start_mode == "AfterJob":
                after_job_nickname = scheduled_bot_details["startmodus"].get("afterJob")
                if not after_job_nickname:
                    after_job_nickname = item_details.get("xecmpfAfterJobDataId")
                self.logger.info(
                    "Scheduled bot item -> '%s' starts after another scheduled bot with nickname -> '%s'. Resolving nickname...",
                    item_name,
                    after_job_nickname,
                )
                # Get the Scheduled Bot node that this bot depends on:
                response = self._otcs.get_node_from_nickname(nickname=after_job_nickname)
                after_job_id = self._otcs.get_result_value(
                    response=response,
                    key="id",
                )
                # Update the bot details - changing the name coming from the
                # payload to the actual node ID (both in details and in 'old' details):
                scheduled_bot_details["startmodus"]["afterJob"] = after_job_id
                old_schedule_data["afterJob"] = after_job_id

                item_details["xecmpfAfterJobDataId"] = after_job_id
            # Check if this bot should start based on a schedule.
            # Then we need to configure the agent to run it:
            elif str(start_mode) == "7558":  # 7558 is NOT a object ID but an internal agent type ID!
                self.logger.info(
                    "Scheduled bot item -> '%s' starts based on a schedule. Setting the agent ID to '7558'...",
                    item_name,
                )
                # Make sure the agent ID is configured:
                item_details["xecmpfAgentId"] = 7558
            self.logger.debug("Scheduled bot details -> %s", str(scheduled_bot_details))

            # The REST API requires to have the scheduled bot details wrapped into a JSON structure:
            item_details["scheduledbotdetails"] = json.dumps(item_details.get("scheduledbotdetails", {}))

            response = self._otcs.update_item(node_id=node_id, body=False, **item_details)
            if not response:
                self.logger.error("Failed to update scheduled bot item -> '%s'!", item_name)
                success = False
                continue

            # Check if we want to execute an action immediately after creation, like "Runnow":
            actions = item_details.get("actions", [])
            for action in actions:
                self.logger.info("Execute action -> '%s' for scheduled bot -> '%s'...", action, item_name)
                response = self._otcs.update_item(node_id=node_id, body=False, actionName=action)
                if not response:
                    self.logger.error(
                        "Failed to execute action -> '%s' for scheduled bot item -> '%s'!", action, item_name
                    )
                    success = False
                    continue
            if not actions:
                self.logger.info("No immediate actions specified for scheduled bot -> '%s'.", item_name)

        # end if item_type == self._otcs.ITEM_TYPE_SCHEDULED_BOT:

        # Special handling for collection items:
        elif item_type == self._otcs.ITEM_TYPE_COLLECTION:
            item_node_ids = []
            for collection_item in item_ids or []:
                # First see if the collection item is a workspace we can
                # lookup in the payload by its logical payload ID
                member_item = next(
                    (item for item in self._workspaces if item["id"] == collection_item),
                    None,
                )
                if member_item:
                    if member_item.get("enabled", True):
                        self.logger.info(
                            "Found collection item (workspace) -> '%s' in payload and it is enabled.",
                            member_item["name"],
                        )
                    else:
                        self.logger.info(
                            "Found collection item (workspace) -> '%s' in payload but it is not enabled. Skipping...",
                            member_item["name"],
                        )
                        continue
                    member_id = self.determine_workspace_id(workspace=member_item)
                    if not member_id:
                        self.logger.warning(
                            "Workspace of type -> '%s' and name -> '%s' does not exist. Cannot create collection item. Skipping...",
                            member_item["type_name"],
                            member_item["name"],
                        )
                        continue
                else:
                    # alternatively try to find the item as a nickname:
                    member_item = self._otcs.get_node_from_nickname(
                        nickname=collection_item,
                    )
                    member_id = self._otcs.get_result_value(
                        response=member_item,
                        key="id",
                    )
                    if not member_id:
                        self.logger.warning(
                            "Item -> '%s' does not exist. Cannot create collection item. Skipping...",
                            member_item["name"],
                        )
                        continue
                item_node_ids.append(member_id)
            # end for collection_item in item_ids
            if item_node_ids:
                response = self._otcs.add_node_to_collection(
                    collection_id=node_id,
                    node_ids=item_node_ids,
                )
        # end if item_type == self._otcs.ITEM_TYPE_COLLECTION

        # Do we have a nickname for the item in the payload? Then assign it:
        if item_nickname:
            self.logger.info("Assign nickname -> '%s' to item -> '%s' (%s)...", item_nickname, item_name, node_id)
            self._otcs.set_node_nickname(node_id=node_id, nickname=item_nickname)
    # end for item in items:

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=items,
    )

    return success

process_kubernetes(section_name='kubernetes')

Process actions that should be executed in the Kubernetess.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'kubernetes'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_kubernetes")
def process_kubernetes(self, section_name: str = "kubernetes") -> bool:
    """Process actions that should be executed in the Kubernetess.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise.

    """

    if not isinstance(self._k8s, K8s):
        self.logger.error(
            "K8s not setup properly -> Skipping payload section -> '%s'...",
            section_name,
        )
        return False

    if not self._kubernetes:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for item in self._kubernetes:
        # Check if element has been disabled in payload (enabled = false).
        # In this case we skip the element:
        if isinstance(item, dict) and not item.get("enabled", True):
            self.logger.info("Skipping disabled item -> %s", item)
            continue

        match item.get("action"):
            case "execPodCommand":
                pod_name = item.get("pod_name")

                if not pod_name:
                    self.logger.error(
                        "To execute a command in a pod the pod name needs to be specified in the payload! Skipping to next kubernetes item...",
                    )
                    success = False
                    continue

                command = item.get("command", [])
                if not command:
                    self.logger.error(
                        "Command is not specified for pod -> %s! It needs to be a non-empty list! Skipping to next kubernetes item...",
                        pod_name,
                    )
                    success = False
                    continue

                container = item.get("container", None)
                timeout = int(item.get("timeout", 60))

                # Check if exec pod command has been explicitly disabled in payload
                # (enabled = false). In this case we skip the element:
                if not item.get("enabled", True):
                    self.logger.info(
                        "Payload for exec pod command in pod -> '%s' is disabled. Skipping...",
                        pod_name,
                    )
                    continue

                if "description" not in item:
                    self.logger.info(
                        "Executing command -> %s in pod -> '%s'",
                        command,
                        pod_name,
                    )

                else:
                    description = item["description"]
                    self.logger.info(
                        "Executing command -> %s in pod -> '%s' (%s)...",
                        command,
                        pod_name,
                        description,
                    )

                if "interactive" not in item or item["interactive"] is False:
                    result = self._k8s.exec_pod_command(
                        pod_name=pod_name, command=command, container=container, timeout=timeout
                    )
                else:
                    result = self._k8s.exec_pod_command_interactive(
                        pod_name=pod_name,
                        commands=command,
                        timeout=timeout,
                    )

                # we need to differentiate 3 cases here:
                # 1. result = None is returned - this is an error (exception)
                # 2. result is empty string - this is OK
                # 3. result is a non-empty string - this is OK - print it to log
                if result is None:
                    self.logger.error(
                        "Execution of command -> %s in pod -> '%s' failed!",
                        command,
                        pod_name,
                    )
                    success = False
                elif result != "":
                    self.logger.info(
                        "Execution of command -> %s in pod -> '%s' returned result -> %s.",
                        command,
                        pod_name,
                        result,
                    )
                else:
                    # It is not an error if no result is returned. It depends on the nature of the command
                    # if a result is written to stdout or stderr.
                    self.logger.info(
                        "Execution of command -> %s in pod -> '%s' did not return a result.",
                        command,
                        pod_name,
                    )

            case "restart":
                k8s_type = item.get("type")
                name = item.get("name")
                message = item.get("message")

                if not name:
                    self.logger.error("Name not specified for kubernetes item -> %s", item)
                if not k8s_type:
                    self.logger.error("Type not specified for kubernetes item -> %s", item)

                if message:
                    self.logger.info("%s", message)

                if k8s_type.lower() == "statefulset":
                    self.logger.info("Restarting statefulset -> %s...", name)
                    restart_result = self._k8s.restart_stateful_set(sts_name=name)

                elif k8s_type.lower() == "deployment":
                    self.logger.info("Restarting deployment -> %s...", name)
                    restart_result = self._k8s.restart_deployment(deployment_name=name)

                elif k8s_type.lower() == "pod":
                    self.logger.info("Deleting pod -> %s...", name)
                    restart_result = self._k8s.delete_pod(pod_name=name)

                if not restart_result:
                    success = False

            case _:
                self.logger.error("Action not defined for work item -> %s", item)
                continue

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._kubernetes,
    )

    return success

process_licenses(section_name='licenses')

Process OTDS licenses in payload and create them in OTDS.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'licenses'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_licenses")
def process_licenses(self, section_name: str = "licenses") -> bool:
    """Process OTDS licenses in payload and create them in OTDS.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise

    """

    if not self._licenses:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for lic in self._licenses:
        self.logger.debug("Start processing License -> '%s'", lic)
        if not lic.get("enabled", True):
            self.logger.info(
                "Payload for License -> '%s' is disabled. Skipping...",
                lic,
            )
            continue

        path = lic.get("path")
        if not path:
            self.logger.error("Required attribute path not specified (%s)! Skipping ...", lic)
            success = False
            continue

        product_name = lic.get("product_name")
        if not product_name:
            self.logger.error("Required attribute 'product_name' not specified (%s)! Skipping ...", lic)
            success = False
            continue

        if "resource" in lic:
            try:
                resource_id = self._otds.get_resource(name=lic["resource"])["resourceID"]
            except Exception:
                self.logger.error("Error getting resource ID from resource -> '%s'!", lic["resource"])
                success = False
                continue
        else:
            self.logger.error("Required attribute 'resource' not specified (%s)! Skipping ...", lic)
            success = False
            continue

        self.logger.info(
            "Adding license -> '%s' for product -> '%s' to resource -> '%s'...", path, product_name, resource_id
        )

        add_license = self._otds.add_license_to_resource(
            path_to_license_file=path,
            product_name=product_name,
            product_description=lic.get("description", ""),
            resource_id=resource_id,
            update=lic.get("update", True),
        )

        if not add_license:
            success = False
            continue

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._licenses,
    )

    return success

process_nifi_flows(section_name='nifi')

Process Knowledge Discovery Nifi flows in payload and create them in Nifi.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'nifi'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_nifi_flows")
def process_nifi_flows(self, section_name: str = "nifi") -> bool:
    """Process Knowledge Discovery Nifi flows in payload and create them in Nifi.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise

    """

    if not self._nifi_flows:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for nifi_flow in self._nifi_flows:
        if "name" not in nifi_flow:
            self.logger.error(
                "Knowledge Discovery Nifi flow needs a name! Skipping to next Nifi flow...",
            )
            success = False
            continue
        name = nifi_flow["name"]

        # Check if element has been disabled in payload (enabled = false).
        # In this case we skip the element:
        if not nifi_flow.get("enabled", True):
            self.logger.info(
                "Payload for Knowledge Discovery Nifi flow -> '%s' is disabled. Skipping...",
                name,
            )
            continue

        if "file" not in nifi_flow:
            self.logger.error(
                "Knowledge Discovery Nifi flow -> '%s' needs a file! Skipping to next Nifi flow...", name
            )
            success = False
            continue
        filename = nifi_flow["file"]

        parameters = nifi_flow.get("parameters", [])

        if not self._otkd:
            self.logger.error("Knowledge Discovery is not initialized. Stop processing Nifi flows.")
            success = False
            break

        # Optional layout positions of the flow:
        position_x = nifi_flow.get("position_x", 0.0)
        position_y = nifi_flow.get("position_y", 0.0)
        start = nifi_flow.get("start", False)

        self.logger.info("Processing Knowledge Discovery Nifi flow -> '%s'...", name)

        existing = self._otkd.get_process_group_by_name(name=name)
        if existing:
            self.logger.warning("Nifi flow -> '%s' does already exist. Updating parameters only...", name)
            # We better don't start existing flows!? Otherwise this may produce errors.
            start = False
        else:
            response = self._otkd.upload_process_group(
                file_path=filename, name=name, position_x=position_x, position_y=position_y
            )
            if not response:
                self.logger.error("Failed to upload new Nifi flow -> '%s' for Knowledge Discovery!", name)
                success = False
                continue
            self.logger.info("Sucessfully uploaded new Nifi flow -> '%s' for Knowledge Discovery.", name)

        for parameter in parameters:
            component = parameter.get("component", None)
            if not component:
                self.logger.error("Missing component in parameter of Nifi flow -> '%s'!", name)
                success = False
                continue
            parameter_name = parameter.get("name", None)
            if not parameter_name:
                self.logger.error(
                    "Missing name in parameter of Nifi flow -> '%s', component -> '%s'!", name, component
                )
                success = False
                continue
            parameter_description = parameter.get("description", "")
            parameter_value = parameter.get("value", None)
            if not parameter_value:
                self.logger.error(
                    "Missing value in parameter of Nifi flow -> '%s', component -> '%s'!", name, component
                )
                success = False
                continue
            parameter_sensitive = parameter.get("sensitive", False)

            response = self._otkd.update_parameter(
                component=component,
                parameter=parameter_name,
                value=parameter_value,
                sensitive=parameter_sensitive,
                description=parameter_description,
            )
            if not response:
                self.logger.error("Failed to update parameter -> '%s' of Nifi flow -> '%s'!", parameter_name, name)
                success = False
                continue
            self.logger.info(
                "Successfully updated parameter -> '%s' of component -> '%s' in Nifi flow -> '%s' to value -> '%s'.",
                parameter_name,
                component,
                name,
                parameter_value if not parameter_sensitive else "<sensitive>",
            )
        # end for parameter in parameters:
        if start:
            response = self._otkd.start_all_processors(name=name)
            if response:
                self.logger.info("Successfully started Nifi flow -> '%s'.", name)
            else:
                self.logger.error("Failed to start Nifi flow -> '%s'!", name)
                success = False

            response = self._otkd.set_controller_services_state(name=name, state="ENABLED")
            if response:
                self.logger.info("Successfully enabled Nifi Controller Services for Nifi flow -> '%s'.", name)
            else:
                self.logger.error("Failed to enable Nifi Controller Services for Nifi flow -> '%s'!", name)
                success = False

        else:
            self.logger.info("Don't (re)start Nifi flow -> '%s'.", name)
    # end for nifi_flow in self._nifi_flows:

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._nifi_flows,
    )

    return success

process_oauth_clients(section_name='oauthClients')

Process OTDS OAuth clients in payload and create them in OTDS.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'oauthClients'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_oauth_clients")
def process_oauth_clients(self, section_name: str = "oauthClients") -> bool:
    """Process OTDS OAuth clients in payload and create them in OTDS.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._oauth_clients:
        self.logger.info("Payload section -> %s is empty. Skipping...", section_name)
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for oauth_client in self._oauth_clients:
        client_name = oauth_client.get("name")
        if not client_name:
            self.logger.error("OAuth client does not have a name in payload. Skipping...")
            success = False
            continue

        # Check if oauth client has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not oauth_client.get("enabled", True):
            self.logger.info(
                "Payload for OAuth client -> '%s' is disabled in payload. Skipping...",
                client_name,
            )
            continue

        client_description = oauth_client.get("description", "")
        client_confidential = oauth_client.get("confidential", True)
        client_partition = oauth_client.get("partition", "Global")
        if client_partition == "Global":
            client_partition = []
        client_redirect_urls = oauth_client.get("redirect_urls", [])
        client_permission_scopes = oauth_client.get("permission_scopes")
        client_default_scopes = oauth_client.get("default_scopes")
        client_allow_impersonation = oauth_client.get("allow_impersonation", True)
        client_secret = oauth_client.get("secret", "")

        # Check if OAuth client does already exist
        # (in an attempt to make the code idem-potent)
        self.logger.info(
            "Check if OTDS OAuth client -> '%s' does already exist...",
            client_name,
        )
        response = self._otds.get_oauth_client(
            client_id=client_name,
            show_error=False,
        )
        if response:
            self.logger.info(
                "OAuth client -> '%s' does already exist. Skipping...",
                client_name,
            )
            continue
        self.logger.info(
            "OAuth client -> '%s' does not exist. Creating...",
            client_name,
        )

        response = self._otds.add_oauth_client(
            client_id=client_name,
            description=client_description,
            redirect_urls=client_redirect_urls,
            allow_impersonation=client_allow_impersonation,
            confidential=client_confidential,
            auth_scopes=client_partition,
            allowed_scopes=client_permission_scopes,
            default_scopes=client_default_scopes,
            secret=client_secret,
        )
        if response:
            self.logger.info("Added OTDS OAuth client -> '%s'.", client_name)
        else:
            self.logger.error(
                "Failed to add OTDS OAuth client -> '%s'!",
                client_name,
            )
            success = False
            continue

        # in case the secret has not been provided in the payload we retrieve
        # the automatically created secret:
        client_secret = response.get("secret")
        if not client_secret:
            self.logger.error(
                "OAuth client -> '%s' does not have a secret!",
                client_name,
            )
            continue

        client_description += " Client Secret: " + str(client_secret)
        response = self._otds.update_oauth_client(
            client_id=client_name,
            updates={"description": client_description},
        )
        # Write the secret back into the payload
        oauth_client["secret"] = client_secret

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._oauth_clients,
    )

    return success

process_partitions(section_name='partitions')

Process OTDS partitions in payload and create them in OTDS.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'partitions'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_partitions")
def process_partitions(self, section_name: str = "partitions") -> bool:
    """Process OTDS partitions in payload and create them in OTDS.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise

    """

    if not self._partitions:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for partition in self._partitions:
        partition_name = partition.get("name")
        if not partition_name:
            self.logger.error("Partition does not have a name in payload! Skipping...")
            success = False
            continue

        # Check if partition has explicitly been disabled in payload
        # (enabled = false). In this case we skip the element:
        if not partition.get("enabled", True):
            self.logger.info(
                "Payload for partition -> '%s' is disabled. Skipping...",
                partition_name,
            )
            continue

        partition_description = partition.get("description", "")

        # Check if Partition does already exist (to make the code idem-potent):
        self.logger.info(
            "Check if OTDS partition -> '%s' does already exist...",
            partition_name,
        )
        response = self._otds.get_partition(name=partition_name, show_error=False)
        if response:
            self.logger.info(
                "Partition -> '%s' does already exist.",
                partition_name,
            )
        else:
            # Only continue if Partition does not exist already
            self.logger.info(
                "Partition -> '%s' does not yet exist. Creating...",
                partition_name,
            )

            response = self._otds.add_partition(
                name=partition_name,
                description=partition_description,
            )
            if response:
                self.logger.info(
                    "Added OTDS partition -> '%s'%s.",
                    partition_name,
                    " ({})".format(partition_description) if partition_description else "",
                )
            else:
                self.logger.error(
                    "Failed to add OTDS partition -> '%s'!",
                    partition_name,
                )
                success = False
                continue

        access_role = partition.get("access_role")
        if access_role:
            response = self._otds.add_partition_to_access_role(
                access_role=access_role,
                partition=partition_name,
            )
            if response:
                self.logger.info(
                    "Added OTDS partition -> '%s' to access role -> '%s'.",
                    partition_name,
                    access_role,
                )
            else:
                self.logger.error(
                    "Failed to add OTDS partition -> '%s' to access role -> '%s'!",
                    partition_name,
                    access_role,
                )
                success = False
        # end if access_role

        # Partions may have an optional list of licenses in
        # the payload. Assign the partition to all these licenses:
        partition_specific_licenses = partition.get("licenses")
        if partition_specific_licenses:
            # We assume these licenses are Extended ECM licenses!
            otcs_resource_name = self._otcs.config()["resource"]
            otcs_resource = self._otds.get_resource(name=otcs_resource_name)
            if not otcs_resource:
                self.logger.error(
                    "Cannot find OTCS resource -> '%s'!",
                    otcs_resource_name,
                )
                success = False
                continue
            otcs_resource_id = otcs_resource["resourceID"]
            otcs_license_name = "EXTENDED_ECM"
            for license_item in partition_specific_licenses:
                if isinstance(license_item, dict):
                    license_feature = license_item.get("feature")
                    license_name = license_item.get("name", "EXTENDED_ECM")
                    if "enabled" in license_item and not license_item["enabled"]:
                        self.logger.info(
                            "Payload for License '%s' -> '%s' is disabled. Skipping...",
                            license_name,
                            license_feature,
                        )
                        continue

                    if "resource" in license_item:
                        try:
                            resource_id = self._otds.get_resource(name=license_item["resource"])["resourceID"]
                        except Exception:
                            self.logger.error(
                                "Error getting resource ID from resource -> '%s'!", license_item["resource"]
                            )
                    else:
                        resource_id = otcs_resource_id

                elif isinstance(license_item, str):
                    license_feature = license_item
                    resource_id = otcs_resource_id
                    license_name = otcs_license_name
                else:
                    self.logger.error("Invalid License feature specified -> %s!", license_item)
                    success = False
                    continue

                if self._otds.is_partition_licensed(
                    partition_name=partition_name,
                    resource_id=resource_id,
                    license_feature=license_feature,
                    license_name=license_name,
                ):
                    self.logger.info(
                        "Partition -> '%s' is already licensed for -> '%s' (%s).",
                        partition_name,
                        license_name,
                        license_feature,
                    )
                    continue

                assigned_license = self._otds.assign_partition_to_license(
                    partition_name=partition_name,
                    resource_id=resource_id,
                    license_feature=license_feature,
                    license_name=license_name,
                )

                if not assigned_license:
                    self.logger.error(
                        "Failed to assign partition -> '%s' to license feature -> '%s' of license -> '%s'!",
                        partition_name,
                        license_feature,
                        license_name,
                    )
                    success = False
                else:
                    self.logger.info(
                        "Successfully assigned partition -> '%s' to license feature -> '%s' of license -> '%s'.",
                        partition_name,
                        license_feature,
                        license_name,
                    )
        # end if partition_specific_licenses:
    # end for partition in self._partitions:

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._partitions,
    )

    return success

process_payload()

Process a payload file.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_payload")
def process_payload(self) -> None:
    """Process a payload file."""

    if not self._payload_sections:
        self.logger.warning("No payload sections defined!")
        return

    for payload_section in self._payload_sections:
        with tracer.start_as_current_span("process_payload-section") as t:
            t.set_attributes({"payload_section": payload_section["name"]})
            self.logger.debug("processing payload_section -> %s", payload_section)
            if not payload_section.get("enabled", True):
                self.logger.info(
                    "Payload section -> '%s' is disabled. Skipping...",
                    payload_section.get("name", "Name not defined"),
                )
                continue

            match payload_section["name"]:
                case "embeddings" | "feme":
                    self._log_header_callback(
                        text="Process additional Embeddings for Content Aviator",
                    )
                    self.process_embeddings()
                case "avtsRepositories":
                    self._log_header_callback(
                        text="Process Aviator Search repositories",
                    )
                    self.process_avts_repositories()
                case "avtsQuestions":
                    self._log_header_callback(
                        text="Process Aviator Search Sample Questions",
                    )
                    self.process_avts_questions()
                case "appworks":
                    self._log_header_callback(text="Process AppWorks Configurations")
                    self.process_appworks_configurations()
                case "webHooks":
                    self._log_header_callback(text="Process Web Hooks")
                    self.process_web_hooks(webhooks=self._webhooks)
                case "webHooksPost":
                    self._log_header_callback(text="Process Web Hooks (post)")
                    self.process_web_hooks(
                        webhooks=self._webhooks_post,
                        section_name="webHooksPost",
                    )
                case "resources":
                    self._log_header_callback(text="Process OTDS Resources")
                    self.process_resources()
                case "partitions":
                    self._log_header_callback(text="Process OTDS Partitions")
                    self.process_partitions()
                case "licenses":
                    self._log_header_callback(text="Process OTDS Licenses")
                    self.process_licenses()
                case "synchronizedPartitions":
                    self._log_header_callback(
                        text="Process OTDS Synchronized Partitions",
                    )
                    self.process_synchronized_partitions()
                case "oauthClients":
                    self._log_header_callback(text="Process OTDS OAuth Clients")
                    self.process_oauth_clients()
                case "applicationRoles":
                    self._log_header_callback(text="Process OTDS Application Roles")
                    self.process_application_roles()
                case "authHandlers":
                    self._log_header_callback(text="Process OTDS Auth Handlers")
                    self.process_auth_handlers()
                case "trustedSites":
                    self._log_header_callback(text="Process OTDS Trusted Sites")
                    self.process_trusted_sites()
                case "systemAttributes":
                    self._log_header_callback(text="Process OTDS System Attributes")
                    self.process_system_attributes()
                case "docgenSettings":
                    if self._otpd and isinstance(self._otpd, OTPD):
                        self._log_header_callback(text="Process OTPD Settings")
                        self.process_docgen_settings()
                case "groups":
                    self._log_header_callback(text="Process Groups")
                    self.process_groups()
                    # Add all groups with ID the a lookup dict for placeholder replacements
                    # in adminSetting. This also updates the payload with group IDs from OTCS
                    # if the group already exists in Content Server. This is important especially
                    # if the customizer pod is restarted / run multiple times:
                    self.process_group_placeholders()
                    if self._core_share and isinstance(self._core_share, CoreShare):
                        self._log_header_callback(text="Process Core Share Groups", char="-")
                        self.process_groups_core_share()
                    if self._m365 and isinstance(self._m365, M365):
                        self._log_header_callback(text="Cleanup existing M365 Teams", char="-")
                        self.cleanup_all_teams_m365()
                        self._log_header_callback(text="Process M365 Groups", char="-")
                        self.process_groups_m365()
                case "users":
                    self._log_header_callback(text="Process Users")
                    self._user_customization = bool(
                        payload_section.get("user_customization", "True"),
                    )
                    self.process_users()
                    # Add all users with ID the a lookup dict for placeholder replacements
                    # in adminSetting. This also updates the payload with user IDs from OTCS
                    # if the user already exists in Content Server. This is important especially
                    # if the cutomizer pod is restarted / run multiple times:
                    self.process_user_placeholders()
                    self._log_header_callback(
                        text="Assign OTCS licenses to users",
                        char="-",
                    )
                    self.process_user_licenses(
                        resource_name=self._otcs.config()["resource"],
                        license_feature=self._otcs.config()["license"],
                        user_specific_payload_field="licenses",
                    )
                    self._log_header_callback(
                        text="Process OTDS user settings",
                        char="-",
                    )
                    self.process_user_settings()
                    if self._core_share and isinstance(self._core_share, CoreShare):
                        self._log_header_callback(text="Process Core Share users", char="-")
                        self.process_users_core_share()
                    if self._m365 and isinstance(self._m365, M365):
                        self._log_header_callback(text="Process M365 users", char="-")
                        self.process_users_m365()
                        # We need to do the MS Teams creation after the creation of
                        # the M365 users as we require Group Owners to create teams.
                        # Note: this is just for the teams of the top-level OTCS groups
                        # (departments), not the MS Teams for the Workspaces. These
                        # are created via the scheduled bots!
                        self._log_header_callback(text="Process M365 Teams for departmental groups", char="-")
                        self.process_teams_m365()
                case "adminSettings":
                    self._log_header_callback(
                        text="Process OTCS administration settings",
                    )
                    restart_required = self.process_admin_settings(
                        admin_settings=self._admin_settings,
                    )
                    if restart_required:
                        self.logger.info(
                            "Admin Settings require a restart of OTCS services...",
                        )
                        # Restart OTCS frontend and backend pods:
                        self._otcs_restart_callback(
                            backend=self._otcs_backend,
                            frontend=self._otcs_frontend,
                        )
                case "adminSettingsPost":
                    self._log_header_callback(
                        text="Process OTCS administration settings (post)",
                    )
                    restart_required = self.process_admin_settings(
                        self._admin_settings_post,
                        "adminSettingsPost",
                    )
                    if restart_required:
                        self.logger.info(
                            "Admin settings (Post) require a restart of OTCS services...",
                        )
                        # Restart OTCS frontend and backend pods:
                        self._otcs_restart_callback(
                            backend=self._otcs_backend,
                            frontend=self._otcs_frontend,
                        )
                case "execPodCommands":
                    self._log_header_callback(text="Process Pod Commands")
                    self.process_exec_pod_commands()
                case "kubernetes":
                    self._log_header_callback(text="Process Kubernetes Commands")
                    self.process_kubernetes()
                case "execCommands":
                    self._log_header_callback(text="Process Commands in Customizer Pod")
                    self.process_exec_commands()
                case "execDatabaseCommands":
                    self._log_header_callback(text="Process Database Commands")
                    self.process_exec_database_commands()
                case "csApplications":
                    self._log_header_callback(text="Process CS Apps (backend)")
                    self.process_cs_applications(
                        otcs_object=self._otcs_backend,
                        section_name="csApplicationsBackend",
                    )
                    self._log_header_callback("Process CS Apps (frontend)")
                    self.process_cs_applications(
                        otcs_object=self._otcs_frontend,
                        section_name="csApplicationsFrontend",
                    )
                case "externalSystems":
                    self._log_header_callback(
                        text="Process External System Connections",
                    )
                    self.process_external_systems()
                    # Now the SAP, SuccessFactors and Salesforce objects
                    # should be initialized and we can process users and groups
                    # in these external systems:
                    if self._sap and isinstance(self._sap, SAP):
                        self._log_header_callback(text="Process SAP Users")
                        self.process_users_sap()
                    if self._successfactors and isinstance(
                        self._successfactors,
                        SuccessFactors,
                    ):
                        self._log_header_callback(text="Process SuccessFactors Users")
                        self.process_users_successfactors()
                    if self._salesforce and isinstance(self._salesforce, Salesforce):
                        self._log_header_callback(text="Process Salesforce Groups")
                        self.process_groups_salesforce()
                        self._log_header_callback(text="Process Salesforce Users")
                        self.process_users_salesforce()
                case "transportPackages":
                    self._log_header_callback(text="Process Transport Packages")
                    self.process_transport_packages(self._transport_packages)
                    # Right after the transport that create the business object types
                    # and the workspace types we extract them and put them into
                    # generated payload lists:
                    self._log_header_callback(text="Process Business Object Types")
                    self.process_business_object_types()
                    self._log_header_callback(text="Process Workspace Types")
                    self.process_workspace_types()
                    if self._m365 and isinstance(self._m365, M365):
                        # Right after the transport that creates the top level folders
                        # we can add the M365 Teams apps for Extended ECM as its own tab:
                        self._log_header_callback(text="Process M365 Teams Apps")
                        self.process_teams_m365_apps()
                        # The SharePoint sites require the top-level departmental folder.
                        # So we can do this only after the transport packages have been
                        # processed:
                        self._log_header_callback(text="Process M365 SharePoint sites for departmental groups")
                        self.process_sites_m365()
                case "contentTransportPackages":
                    self._log_header_callback("Process Content Transport Packages")
                    self.process_transport_packages(
                        transport_packages=self._content_transport_packages,
                        section_name="contentTransportPackages",
                    )
                    # Process workspace permissions after content has been added:
                    # (this is a workaround for a flaw in transport warehouse that don't
                    # set workspace role permissions for content transported into workspaces)
                    self._log_header_callback(
                        text="Process Workspace Member Permissions",
                        char="-",
                    )
                    self.process_workspace_member_permissions()
                case "transportPackagesPost":
                    self._log_header_callback("Process Transport Packages (post)")
                    self.process_transport_packages(
                        transport_packages=self._transport_packages_post,
                        section_name="transportPackagesPost",
                    )
                case "workspaceTemplates":
                    # If a payload file (e.g. additional ones) does not have
                    # transportPackages then it can happen that the
                    # self._business_object_types and self._workspace_types are
                    # not yet initialized. As we need these structures for
                    # workspaceTemplates we initialize them here:
                    if not self._business_object_types:
                        self._log_header_callback("Process Business Object Types")
                        self.process_business_object_types()
                    if not self._workspace_types:
                        self._log_header_callback("Process Workspace Types")
                        self.process_workspace_types()

                    self._log_header_callback(
                        "Process Workspace Templates (Template Role Assignments)",
                    )
                    self.process_workspace_templates()
                case "workspaces":
                    # If a payload file (e.g. additional ones) does not have
                    # transportPackages then it can happen that the self._business_object_types and
                    # self._workspace_types are not yet initialized. As we need
                    # these structures for workspaces we initialize it here:
                    if not self._business_object_types:
                        self._log_header_callback("Process Business Object Types")
                        self.process_business_object_types()
                    if not self._workspace_types:
                        self._log_header_callback("Process Workspace Types")
                        self.process_workspace_types()

                    self._log_header_callback("Process Workspaces")
                    self.process_workspaces()
                    self._log_header_callback("Process Workspace Relationships")
                    self.process_workspace_relationships()
                    self._log_header_callback("Process Workspace Memberships")
                    self.process_workspace_members()

                    # This has to run after the processing of webReports that are
                    # used to enable Content Aviator in KINI database table:
                    if self._aviator_enabled:
                        self._log_header_callback("Process Workspace Aviators")
                        self.process_workspace_aviators()
                case "bulkDatasources":
                    # this is here just to avoid an error in catch all below
                    # the bulkDatasources dictionary will be processed in
                    # the other bulk* sections
                    pass
                case "bulkWorkspaces":
                    if not self._workspace_types:
                        self._log_header_callback("Process Workspace Types")
                        self.process_workspace_types()
                    self._log_header_callback("Process Bulk Workspaces")
                    self.process_bulk_workspaces()
                case "bulkWorkspaceRelationships":
                    if not self._workspace_types:
                        self._log_header_callback("Process Workspace Types")
                        self.process_workspace_types()
                    self._log_header_callback("Process Bulk Workspace Relationships")
                    self.process_bulk_workspace_relationships()
                case "bulkDocuments":
                    if not self._workspace_types:
                        self._log_header_callback("Process Workspace Types")
                        self.process_workspace_types()
                    self._log_header_callback("Process Bulk Documents")
                    self.process_bulk_documents()
                case "bulkItems":
                    if not self._workspace_types:
                        self._log_header_callback("Process Workspace Types")
                        self.process_workspace_types()
                    self._log_header_callback("Process Bulk Items")
                    self.process_bulk_items()
                case "bulkClassifications":
                    self._log_header_callback("Process Bulk Classifications")
                    self.process_bulk_classifications()
                case "sapRFCs":
                    if self._sap and isinstance(self._sap, SAP):
                        self._log_header_callback("Process SAP RFCs")
                        self.process_sap_rfcs(sap_rfcs=self._sap_rfcs)
                    else:
                        self.logger.warning(
                            "SAP RFC in payload but SAP external system is not configured or not enabled! RFCs will not be processed.",
                        )
                case "sapRFCsPost":
                    if self._sap and isinstance(self._sap, SAP):
                        self._log_header_callback("Process SAP RFCs (post)")
                        self.process_sap_rfcs(
                            sap_rfcs=self._sap_rfcs_post,
                            section_name="sapRFCsPost",
                        )
                    else:
                        self.logger.warning(
                            "SAP RFC in payload but SAP external system is not configured or not enabled! RFCs will not be processed.",
                        )
                case "webReports":
                    self._log_header_callback(text="Process Web Reports")
                    restart_required = self.process_web_reports(
                        web_reports=self._web_reports,
                    )
                    if restart_required:
                        self.logger.info(
                            "Web Reports require a restart of OTCS services...",
                        )
                        # Restart OTCS frontend and backend pods:
                        self._otcs_restart_callback(
                            backend=self._otcs_backend,
                            frontend=self._otcs_frontend,
                        )
                case "webReportsPost":
                    self._log_header_callback(text="Process Web Reports (post)")
                    restart_required = self.process_web_reports(
                        web_reports=self._web_reports_post,
                        section_name="webReportsPost",
                    )
                    if restart_required:
                        self.logger.info(
                            "WebReports (Post) require a restart of OTCS services...",
                        )
                        # Restart OTCS frontend and backend pods:
                        self._otcs_restart_callback(
                            backend=self._otcs_backend,
                            frontend=self._otcs_frontend,
                        )
                case "additionalGroupMemberships":
                    self._log_header_callback(
                        text="Process additional group members for OTDS",
                    )
                    self.process_additional_group_members()
                case "additionalAccessRoleMemberships":
                    self._log_header_callback(
                        text="Process additional access role members for OTDS",
                    )
                    self.process_additional_access_role_members()
                case "additionalApplicationRoleAssignments":
                    self._log_header_callback(
                        text="Process additional application role assignments for OTDS",
                    )
                    self.process_additional_application_role_assignments()
                case "renamings":
                    self._log_header_callback(text="Process Node Renamings")
                    self.process_renamings()
                case "items":
                    self._log_header_callback(text="Process Items")
                    self.process_items(items=self._items)
                case "itemsPost":
                    self._log_header_callback(text="Process Items (post)")
                    self.process_items(items=self._items_post, section_name="itemsPost")
                case "permissions":
                    self._log_header_callback(text="Process Permissions")
                    self.process_permissions(permissions=self._permissions)
                case "permissionsPost":
                    self._log_header_callback(text="Process Permissions (post)")
                    self.process_permissions(
                        permissions=self._permissions_post,
                        section_name="permissionsPost",
                    )
                case "workspacePermissions":
                    self._log_header_callback(text="Process Workspace Permissions")
                    self.process_workspace_permissions()
                case "assignments":
                    self._log_header_callback(text="Process Assignments")
                    self.process_assignments()
                case "securityClearances":
                    self._log_header_callback(text="Process Security Clearances")
                    self.process_security_clearances()
                case "supplementalMarkings":
                    self._log_header_callback(text="Process Supplemental Markings")
                    self.process_supplemental_markings()
                case "recordsManagementSettings":
                    self._log_header_callback(
                        text="Process Records Management Settings",
                    )
                    self.process_records_management_settings()
                case "holds":
                    self._log_header_callback(text="Process Records Management Holds")
                    self.process_holds()
                case "documentGenerators":
                    # If a payload file (e.g. additional ones) does not have
                    # transportPackages then it can happen that the
                    # self._workspace_types is not yet initialized. As we need
                    # this structure for documentGenerators we initialize it here:
                    if not self._workspace_types:
                        self._log_header_callback(text="Process Workspace Types")
                        self.process_workspace_types()

                    self._log_header_callback(text="Process Document Generators")
                    self.process_document_generators()
                case "workflowInitiations":
                    # If a payload file (e.g. additional ones) does not have
                    # transportPackages then it can happen that the
                    # self._workspace_types is not yet initialized. As we need
                    # this structure for workflowInitiations we initialize it here:
                    if not self._workspace_types:
                        self._log_header_callback(text="Process Workspace Types")
                        self.process_workspace_types()

                    self._log_header_callback(text="Process Workflow Initiations")
                    self.process_workflows()
                case "browserAutomations":
                    self._log_header_callback(text="Process Browser Automations")
                    self.process_browser_automations(
                        browser_automations=self._browser_automations,
                    )
                case "browserAutomationsPost":
                    self._log_header_callback(text="Process Browser Automations (post)")
                    self.process_browser_automations(
                        browser_automations=self._browser_automations_post,
                        section_name="browserAutomationsPost",
                    )
                case "testAutomations":
                    self._log_header_callback(text="Process Test Automations")
                    self.process_browser_automations(
                        browser_automations=self._test_automations, section_name="testAutomations"
                    )
                case "workspaceTypes":
                    if not self._workspace_types:
                        self._log_header_callback(text="Process Workspace Types")
                        self.process_workspace_types()
                case "businessObjectTypes":
                    if not self._business_object_types:
                        self._log_header_callback(text="Process Business Object Types")
                        self.process_business_object_types()
                case "nifi":
                    self._log_header_callback("Process Knowledge Discovery Nifi Flows")
                    self.process_nifi_flows()
                case _:
                    self.logger.error(
                        "Illegal payload section name -> '%s' in payloadSections!",
                        payload_section["name"],
                    )
            payload_section_restart = payload_section.get("restart", False)
            if payload_section_restart:
                self.logger.info(
                    "Payload section -> '%s' requests a restart of OTCS services...",
                    payload_section["name"],
                )
                # Restart OTCS frontend and backend pods:
                self._otcs_restart_callback(
                    backend=self._otcs_backend,
                    frontend=self._otcs_frontend,
                )
            # Avoid out of cycle message for bulkDatasources if it is
            # passed in the payload:
            elif payload_section["name"] != "bulkDatasources":
                self.logger.info(
                    "Payload section -> '%s' does not require a restart of OTCS services.",
                    payload_section["name"],
                )

    with tracer.start_as_current_span("process_payload_users"):
        if self._users and self._user_customization:
            self._log_header_callback("Process User Profile Photos")
            self.process_user_photos()
            if self._m365 and isinstance(self._m365, M365):
                self._log_header_callback("Process M365 User Profile Photos")
                self.process_user_photos_m365()
            if self._salesforce and isinstance(self._salesforce, Salesforce):
                self._log_header_callback("Process Salesforce User Profile Photos")
                self.process_user_photos_salesforce()
            if self._core_share and isinstance(self._core_share, CoreShare):
                self._log_header_callback("Process Core Share User Profile Photos")
                self.process_user_photos_core_share()
            self._log_header_callback("Process User Favorites and Profiles")
            self.process_user_favorites_and_profiles()
            self._log_header_callback("Process User Security")
            self.process_user_security()

process_permission(node_id, node_name, permission, apply_to, workspace_id=None)

Process a single permission payload item for a given node.

Parameters:

Name	Type	Description	Default
node_id	int	The ID of the node.	required
node_name	str	The name of the node.	required
permission	dict	The permission payload.	required
apply_to	int	0 = Apply to this item only 1 = Apply to sub-items only 2 = Apply to this item and its sub-items (default) 3 = Apply to this item and its immediate sub-items	required
workspace_id	int | None	If role permissions should be set we also need the workspace_id. Use None if node is not part of a workspace or no role permissions should be set.	None

Returns:

Name	Type	Description
bool	bool	True = success, False = Error

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_permission")
def process_permission(
    self,
    node_id: int,
    node_name: str,
    permission: dict,
    apply_to: int,
    workspace_id: int | None = None,
) -> bool:
    """Process a single permission payload item for a given node.

    Args:
        node_id (int):
            The ID of the node.
        node_name (str):
            The name of the node.
        permission (dict):
            The permission payload.
        apply_to (int):
            - 0 = Apply to this item only
            - 1 = Apply to sub-items only
            - 2 = Apply to this item and its sub-items (default)
            - 3 = Apply to this item and its immediate sub-items
        workspace_id (int | None):
            If role permissions should be set we also need the workspace_id.
            Use None if node is not part of a workspace or no role permissions
            should be set.

    Returns:
        bool:
            True = success, False = Error

    """

    # 1. Process Owner Permissions
    if "owner_permissions" in permission:
        owner_permissions = permission["owner_permissions"]
        self.logger.info(
            "Update owner permissions for item -> '%s' (%d) to -> %s.",
            node_name,
            node_id,
            str(owner_permissions),
        )
        response = self._otcs.assign_permission(
            node_id=int(node_id),
            assignee_type="owner",
            assignee=0,
            permissions=owner_permissions,
            apply_to=apply_to,
        )
        if not response:
            self.logger.error(
                "Failed to update owner permissions for item -> '%s' (%d)!",
                node_name,
                node_id,
            )
            return False

    # 2. Process Owner Group Permissions
    if "owner_group_permissions" in permission:
        owner_group_permissions = permission["owner_group_permissions"]
        self.logger.info(
            "Update owner group permissions for item -> '%s' (%d) to -> %s.",
            node_name,
            node_id,
            str(owner_group_permissions),
        )
        response = self._otcs.assign_permission(
            node_id=int(node_id),
            assignee_type="group",
            assignee=0,
            permissions=owner_group_permissions,
            apply_to=apply_to,
        )
        if not response:
            self.logger.error(
                "Failed to update group permissions for item -> '%s' (%d)!",
                node_name,
                node_id,
            )
            return False

    # 3. Process Public Permissions
    if "public_permissions" in permission:
        public_permissions = permission["public_permissions"]
        self.logger.info(
            "Update public permissions for item -> '%s' (%d) to -> %s.",
            node_name,
            node_id,
            str(public_permissions),
        )
        response = self._otcs.assign_permission(
            node_id=int(node_id),
            assignee_type="public",
            assignee=0,
            permissions=public_permissions,
            apply_to=apply_to,
        )
        if not response:
            self.logger.error(
                "Failed to update public permissions for item -> '%s' (%d)!",
                node_name,
                node_id,
            )
            return False
    # end if "public_permissions" in permission

    # 4. Process Assigned User Permissions (if specified and not empty)
    users = permission.get("users", [])
    for user in users:
        if "name" not in user or "permissions" not in user:
            self.logger.error(
                "Missing user name in user permission specificiation. Cannot set user permissions. Skipping...",
            )
            return False
        user_name = user["name"]
        if "permissions" not in user:
            self.logger.error(
                "Missing permissions in user -> '%s' permission specificiation. Cannot set user permissions. Skipping...",
                user_name,
            )
            return False
        user_permissions = user["permissions"]
        response = self._otcs.get_user(name=user_name, show_error=True)
        user_id = self._otcs.get_result_value(response=response, key="id")
        if not user_id:
            self.logger.error(
                "Cannot find user -> '%s'; cannot set user permissions. Skipping user...",
                user_name,
            )
            return False
        user["id"] = user_id  # write ID back into payload

        self.logger.info(
            "Update permission of user -> '%s' for item -> '%s' (%d) to -> %s.",
            user_name,
            node_name,
            node_id,
            str(user_permissions),
        )
        response = self._otcs.assign_permission(
            node_id=int(node_id),
            assignee_type="custom",
            assignee=user_id,
            permissions=user_permissions,
            apply_to=apply_to,
        )
        if not response:
            self.logger.error(
                "Failed to update assigned user permissions for item -> %s!",
                node_id,
            )
            return False
    # end for user in users

    # 5. Process Assigned Group Permissions (if specified and not empty)
    groups = permission.get("groups", [])
    for group in groups:
        if "name" not in group:
            self.logger.error(
                "Missing group name in group permission specificiation. Cannot set group permissions. Skipping...",
            )
            return False
        group_name = group["name"]
        if "permissions" not in group:
            self.logger.error(
                "Missing permissions in group -> '%s' permission specificiation. Cannot set group permissions. Skipping...",
                group_name,
            )
            return False
        group_permissions = group["permissions"]
        self.logger.info(
            "Update permissions of group -> '%s' for item -> '%s' (%d) to -> %s.",
            group_name,
            node_name,
            node_id,
            str(group_permissions),
        )
        otcs_group = self._otcs.get_group(name=group_name, show_error=True)
        group_id = self._otcs.get_result_value(response=otcs_group, key="id")
        if not group_id:
            self.logger.error(
                "Cannot find group -> '%s'; cannot set group permissions. Skipping group...",
                group_name,
            )
            return False
        group["id"] = group_id  # write ID back into payload
        response = self._otcs.assign_permission(
            node_id=int(node_id),
            assignee_type="custom",
            assignee=group_id,
            permissions=group_permissions,
            apply_to=apply_to,
        )
        if not response:
            self.logger.error(
                "Failed to update assigned group permissions for item -> '%s' (%d)!",
                node_name,
                node_id,
            )
            return False
    # end for group in groups

    # 6. Process Workspace Role Permissions (if specified and not empty)
    roles = permission.get("roles", [])
    if roles and not workspace_id:
        self.logger.error(
            "Cannot set workspace roles if no workspace ID is provided!",
        )
        return False
    for role in roles:
        if "name" not in role:
            self.logger.error(
                "Missing role name in role permission specificiation. Cannot set role permissions. Skipping...",
            )
            return False
        role_name = role["name"]
        if "permissions" not in role:
            self.logger.error(
                "Missing permissions in role -> '%s' permission specificiation. Cannot set role permissions. Skipping...",
                group_name,
            )
            return False
        role_permissions = role["permissions"]
        self.logger.info(
            "Update permissions of role -> '%s' for workspace item -> '%s' (%d) in workspace with ID -> %d to -> %s.",
            role_name,
            node_name,
            node_id,
            workspace_id,
            str(role_permissions),
        )
        response = self._otcs.get_workspace_roles(workspace_id=workspace_id)
        role_id = self._otcs.lookup_result_value(
            response=response,
            key="name",
            value=role_name,
            return_key="id",
        )
        if not role_id:
            self.logger.error(
                "Cannot find role -> '%s'; cannot set role permissions.",
                role_name,
            )
            return False
        response = self._otcs.assign_permission(
            node_id=int(node_id),
            assignee_type="custom",
            assignee=role_id,
            permissions=role_permissions,
            apply_to=apply_to,
        )
        if not response:
            self.logger.error(
                "Failed to update role permissions for item -> '%s' (%d)!",
                node_name,
                node_id,
            )
            return False
    # end for role in roles

    return True

process_permissions(permissions, section_name='permissions')

Process items specified in payload and upadate permissions.

Parameters:

Name	Type	Description	Default
permissions	list	A list of items to apply permissions to. Each list item in the payload is a dict with this structure: { nodeid = "..." volume = "..." nickname = "..." public_permissions = ["see", "see_content", ...] owner_permissions = [] owner_group_permissions = [] groups = [ { name = "..." permissions = [] } ] users = [ { name = "..." permissions = [] } ] apply_to = 2 }	required
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'permissions'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_permissions")
def process_permissions(
    self,
    permissions: list,
    section_name: str = "permissions",
) -> bool:
    """Process items specified in payload and upadate permissions.

    Args:
        permissions (list):
            A list of items to apply permissions to.
            Each list item in the payload is a dict with this structure:
            {
                nodeid = "..."
                volume = "..."
                nickname = "..."
                public_permissions = ["see", "see_content", ...]
                owner_permissions = []
                owner_group_permissions = []
                groups = [
                    {
                        name = "..."
                        permissions = []
                    }
                ]
                users = [
                    {
                        name = "..."
                        permissions = []
                    }
                ]
                apply_to = 2
            }
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not permissions:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for permission in permissions:
        if "path" not in permission and "volume" not in permission and "nickname" not in permission:
            self.logger.error(
                "Item to change permission is not specified (needs path, volume, or nickname). Skipping...",
            )
            success = False
            continue

        # Check if permission has been explicitly disabled in payload
        # (enabled = false). In this case we skip this payload element:
        if not permission.get("enabled", True):
            self.logger.info("Payload for Permission is disabled. Skipping...")
            continue

        node_id = 0

        # Check if "volume" is in payload and not empty string
        # we try to get the node ID from the volume type:
        if permission.get("volume"):
            volume_type = permission["volume"]
            self.logger.info(
                "Found volume type -> '%s' in permission payload. Determine volume ID...",
                volume_type,
            )
            node = self._otcs.get_volume(volume_type=volume_type)
            node_id = self._otcs.get_result_value(response=node, key="id")
            if not node_id:
                self.logger.error(
                    "Illegal volume -> '%s' in permission payload. Skipping...",
                    volume_type,
                )
                success = False
                continue
        else:
            # the following path block requires a value for the volume - if it is
            # not specified we take the Enterprise Workspace:
            volume_type = self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE

        # Check if "path" is in payload and not empty list
        # (path can be combined with volume so we need to take volume into account):
        if permission.get("path"):
            path = permission["path"]
            self.logger.info(
                "Found path -> '%s' in permission payload. Determine node ID...",
                path,
            )
            node = self._otcs.get_node_by_volume_and_path(
                volume_type=volume_type,
                path=path,
            )
            node_id = self._otcs.get_result_value(response=node, key="id")
            if not node_id:
                self.logger.error("Path -> '%s' does not exist. Skipping...", path)
                success = False
                continue

        # Check if "nickname" is in payload and not empty string:
        if permission.get("nickname"):
            nickname = permission["nickname"]
            self.logger.info(
                "Found nickname -> '%s' in permission payload. Determine node ID...",
                nickname,
            )
            node = self._otcs.get_node_from_nickname(nickname=nickname)
            node_id = self._otcs.get_result_value(response=node, key="id")
            if not node_id:
                self.logger.error(
                    "Nickname -> '%s' does not exist. Skipping...",
                    nickname,
                )
                success = False
                continue

        # Now we should have a value for node_id:
        if not node_id:
            self.logger.error("No node ID found! Skipping permission...")
            success = False
            continue

        node_name = self._otcs.get_result_value(response=node, key="name")
        self.logger.info(
            "Found node -> '%s' with ID -> %s to apply permission to.",
            node_name,
            node_id,
        )
        # write node information back into payload
        # for better debugging
        permission["node_name"] = node_name
        permission["node_id"] = node_id

        # Make item + sub-items (2) the default:
        apply_to = permission.get("apply_to", 2)

        # Prcess a single permission payload item:
        if not self.process_permission(
            node_id=node_id,
            node_name=node_name,
            permission=permission,
            apply_to=apply_to,
        ):
            success = False
            continue
    # end for permission in permissions

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=permissions,
    )

    return success

process_records_management_settings(section_name='recordsManagementSettings')

Process Records Management Settings for Content Server.

The setting files need to be placed in the OTCS file system file via a transport into the Support Asset Volume.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'recordsManagementSettings'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_records_management_settings")
def process_records_management_settings(
    self,
    section_name: str = "recordsManagementSettings",
) -> bool:
    """Process Records Management Settings for Content Server.

    The setting files need to be placed in the OTCS file system file via
    a transport into the Support Asset Volume.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._records_management_settings:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    if (
        "records_management_system_settings" in self._records_management_settings
        and self._records_management_settings["records_management_system_settings"] != ""
    ):
        filename = (
            self._custom_settings_dir + self._records_management_settings["records_management_system_settings"]
        )
        response = self._otcs.import_records_management_settings(file_path=filename)
        if not response:
            success = False

    if (
        "records_management_codes" in self._records_management_settings
        and self._records_management_settings["records_management_codes"] != ""
    ):
        filename = self._custom_settings_dir + self._records_management_settings["records_management_codes"]
        response = self._otcs.import_records_management_codes(file_path=filename)
        if not response:
            success = False

    if (
        "records_management_rsis" in self._records_management_settings
        and self._records_management_settings["records_management_rsis"] != ""
    ):
        filename = self._custom_settings_dir + self._records_management_settings["records_management_rsis"]
        response = self._otcs.import_records_management_rsis(file_path=filename)
        if not response:
            success = False

    if (
        "physical_objects_system_settings" in self._records_management_settings
        and self._records_management_settings["physical_objects_system_settings"] != ""
    ):
        filename = self._custom_settings_dir + self._records_management_settings["physical_objects_system_settings"]
        response = self._otcs.import_physical_objects_settings(file_path=filename)
        if not response:
            success = False

    if (
        "physical_objects_codes" in self._records_management_settings
        and self._records_management_settings["physical_objects_codes"] != ""
    ):
        filename = self._custom_settings_dir + self._records_management_settings["physical_objects_codes"]
        response = self._otcs.import_physical_objects_codes(file_path=filename)
        if not response:
            success = False

    if (
        "physical_objects_locators" in self._records_management_settings
        and self._records_management_settings["physical_objects_locators"] != ""
    ):
        filename = self._custom_settings_dir + self._records_management_settings["physical_objects_locators"]
        response = self._otcs.import_physical_objects_locators(file_path=filename)
        if not response:
            success = False

    if (
        "security_clearance_codes" in self._records_management_settings
        and self._records_management_settings["security_clearance_codes"] != ""
    ):
        filename = self._custom_settings_dir + self._records_management_settings["security_clearance_codes"]
        response = self._otcs.import_security_clearance_codes(file_path=filename)
        if not response:
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._records_management_settings,
    )

    return success

process_renamings(section_name='renamings')

Process renamings specified in payload and rename existing Content Server items.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'renamings'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_renamings")
def process_renamings(self, section_name: str = "renamings") -> bool:
    """Process renamings specified in payload and rename existing Content Server items.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._renamings:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for renaming in self._renamings:
        if "name" not in renaming:
            self.logger.error("Renamings require the new name!")
            continue
        if "nodeid" in renaming:
            node_id = renaming["nodeid"]
        elif "volume" in renaming:
            path = renaming.get("path")
            volume = renaming.get("volume")
            if path:
                self.logger.info(
                    "Found path -> '%s' in renaming payload. Determine node ID by volume and path...",
                    path,
                )
                node = self._otcs.get_node_by_volume_and_path(
                    volume_type=volume,
                    path=path,
                )
            else:
                # Determine object ID of volume:
                node = self._otcs.get_volume(volume_type=volume)
            node_id = self._otcs.get_result_value(response=node, key="id")
        elif "nickname" in renaming:
            nickname = renaming["nickname"]
            self.logger.info(
                "Found nickname -> '%s' in renaming payload. Determine node ID by nickname...",
                nickname,
            )
            node = self._otcs.get_node_from_nickname(nickname=nickname)
            node_id = self._otcs.get_result_value(response=node, key="id")
        else:
            self.logger.error(
                "Renamings require either a node ID or a volume (with an optional path) or a nickname! Skipping to next renaming...",
            )
            continue

        # Check if renaming has been explicitly disabled in payload
        # (enabled = false). In this case we skip this payload element:
        if not renaming.get("enabled", True):
            self.logger.info("Payload for renaming is disabled. Skipping...")
            continue

        response = self._otcs.rename_node(
            node_id=int(node_id),
            name=renaming["name"],
            description=renaming.get("description", ""),
        )
        if not response:
            self.logger.error(
                "Failed to rename node ID -> '%s' to new name -> '%s'!",
                node_id,
                renaming["name"],
            )
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._renamings,
    )

    return success

process_resources(section_name='resources')

Process OTDS resources in payload and create them in OTDS.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'resources'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_resources")
def process_resources(self, section_name: str = "resources") -> bool:
    """Process OTDS resources in payload and create them in OTDS.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._resources:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for resource in self._resources:
        resource_name = resource.get("name")
        if not resource_name:
            self.logger.error("OTDS resource payload does not have a name! Skipping...")
            success = False
            continue

        # Check if element has been disabled in payload (enabled = false).
        # In this case we skip the element (default is enabled):
        if not resource.get("enabled", True):
            self.logger.info(
                "Payload for OTDS Resource -> '%s' is disabled. Skipping...",
                resource_name,
            )
            continue

        resource_description = resource.get("description", "")
        display_name = resource.get("display_name", "")
        additional_payload = resource.get("additional_payload", {})
        activate_resource = resource.get("activate", True)
        resource_id = resource.get("resource_id", None)
        allow_impersonation = resource.get("allow_impersonation", True)
        secret = resource.get("secret", None)

        # Check if Partition does already exist
        # (in an attempt to make the code idem-potent)
        self.logger.info(
            "Check if OTDS resource -> '%s' does already exist...",
            resource_name,
        )
        response = self._otds.get_resource(name=resource_name, show_error=False)
        if response:
            self.logger.info(
                "OTDS Resource -> '%s' does already exist. Skipping...",
                resource_name,
            )
            continue

        # Only continue if Partition does not exist already
        self.logger.info(
            "Resource -> '%s' does not exist. Creating...",
            resource_name,
        )

        response = self._otds.add_resource(
            name=resource_name,
            description=resource_description,
            display_name=display_name,
            allow_impersonation=allow_impersonation,
            resource_id=resource_id,
            secret=secret,
            additional_payload=additional_payload,
        )
        if response:
            self.logger.info("Added OTDS resource -> '%s'.", resource_name)
        else:
            self.logger.error("Failed to add OTDS resource -> '%s'!", resource_name)
            success = False
            continue

        # If resource_id and secret are provided then the resource will
        # automatically be activated.
        if activate_resource and not secret:
            resource_id = response["resourceID"]
            self.logger.info(
                "Activate OTDS resource -> '%s' with ID -> %s...",
                resource_name,
                resource_id,
            )
            response = self._otds.activate_resource(resource_id=resource_id)

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._resources,
    )

    return success

process_sap_rfcs(sap_rfcs, section_name='sapRFCs')

Process SAP RFCs in payload and run them in SAP S/4HANA.

Parameters:

Name	Type	Description	Default
sap_rfcs	list	The payload list of SAP RFCs. As we have two different list (pre and post) we need to pass the actual list as parameter.	required
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'sapRFCs'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_sap_rfcs")
def process_sap_rfcs(self, sap_rfcs: list, section_name: str = "sapRFCs") -> bool:
    """Process SAP RFCs in payload and run them in SAP S/4HANA.

    Args:
        sap_rfcs (list):
            The payload list of SAP RFCs. As we have two different list (pre and post)
            we need to pass the actual list as parameter.
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise.

    """

    if not self._sap:
        self.logger.info("SAP object is undefined. Cannot call RFCs. Bailing out.")
        return False

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for sap_rfc in sap_rfcs:
        rfc_name = sap_rfc.get("name")
        if not rfc_name:
            self.logger.error("SAP RFC needs a name! Skipping...")
            success = False
            continue

        # Check if SAP RFC has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not sap_rfc.get("enabled", True):
            self.logger.info(
                "Payload for SAP RFC -> '%s' is disabled. Skipping...",
                rfc_name,
            )
            continue

        rfc_description = sap_rfc.get("description", "")

        # Check if we have SAP RFC call parameters in the payload. They are optional:
        rfc_params = sap_rfc.get("parameters", {})
        # Check if we have SAP RFC call options in the payload. They are optional:
        rfc_call_options = sap_rfc.get("call_options", {})

        if rfc_params:
            self.logger.info(
                "Calling SAP RFC -> '%s' (%s) with parameters -> %s%s...",
                rfc_name,
                rfc_description,
                str(rfc_params),
                " and options -> {}".format(rfc_call_options) if rfc_call_options else "",
            )
        else:
            self.logger.info(
                "Calling SAP RFC -> '%s' (%s)%s...",
                rfc_name,
                rfc_description,
                " with options -> {}".format(rfc_call_options) if rfc_call_options else "",
            )

        if rfc_call_options:
            self.logger.debug("Using call options -> '%s' ...", rfc_call_options)

        result = self._sap.call(rfc_name, rfc_call_options, rfc_params)
        if result is None:
            self.logger.error("Failed to call SAP RFC -> '%s'!", rfc_name)
            success = False
        elif result.get("RESULT") == "WARNING":
            self.logger.warning(
                "Result of SAP RFC -> '%s' has a warning -> '%s'.",
                rfc_name,
                str(result.get("COMMENT")).strip() if result.get("COMMENT") else str(result),
            )
        elif result.get("RESULT") != "OK":
            self.logger.error(
                "Result of SAP RFC -> '%s' is not OK, %sresult -> %s",
                rfc_name,
                "it returned -> '{}' failed items in".format(result.get("FAILED")) if result.get("FAILED") else "",
                str(result),
            )
            success = False
        else:
            self.logger.info(
                "Successfully called RFC -> '%s'. Result -> %s.",
                rfc_name,
                str(result),
            )
            # Save result for status file content
            sap_rfc["result"] = result

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=sap_rfcs,
    )

    return success

process_security_clearances(section_name='securityClearances')

Process Security Clearances for Content Server.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'securityClearances'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_security_clearances")
def process_security_clearances(
    self,
    section_name: str = "securityClearances",
) -> bool:
    """Process Security Clearances for Content Server.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._security_clearances:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for security_clearance in self._security_clearances:
        clearance_level = security_clearance.get("level")
        if not clearance_level:
            self.logger.error(
                "Security clearance requires a level in the payload. Skipping.",
            )
            continue
        clearance_name = security_clearance.get("name")
        if not clearance_name:
            self.logger.error(
                "Security clearance requires a name in the payload. Skipping.",
            )
            continue

        # Check if security clearance has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not security_clearance.get("enabled", True):
            self.logger.info(
                "Payload for security clearance -> '%s' is disabled. Skipping...",
                clearance_name,
            )
            continue

        clearance_description = security_clearance.get("description", "")
        if clearance_level and clearance_name:
            self.logger.info(
                "Creating security clearance -> '%s' : %s%s...",
                clearance_level,
                clearance_name,
                " ({})".format(clearance_description) if clearance_description else "",
            )
            self._otcs.run_web_report(
                nickname="web_report_security_clearance",
                web_report_parameters=security_clearance,
            )
        else:
            self.logger.error(
                "Cannot create security clearance - either level or name is missing!",
            )
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._security_clearances,
    )

    return success

process_sites_m365(section_name='sitesM365')

Process M365 groups in payload and configure SharePoint sites in Microsoft 365.

These are NOT the SharePoint sites for Business Workspaces which are created by Scheduled Bots (Jobs) from OTCS via the creation of MS teams (each MS Team has a SharePoint site behind it)!

These are the SharePoint sites for the departmental groups such as "Sales", "Procurement", "Enterprise Asset Management", ... Only departmental group that have a top-level folder with the exact same name as the Department are configured.

For each departmental group: 1. Determine a departmental folder in the Enterprise Workspace 2. Determine the M365 Group 3. Determine the SharePoint Site (based on the M365 group ID) 4. Determine the Page in the SharePoint site 5. Determine or create the SharePoint webpart for the OTCS browser 6. Create URL object pointing to SharePoint site inside top level department folder

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'sitesM365'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_sites_m365")
def process_sites_m365(self, section_name: str = "sitesM365") -> bool:
    """Process M365 groups in payload and configure SharePoint sites in Microsoft 365.

    These are NOT the SharePoint sites for Business Workspaces which are created
    by Scheduled Bots (Jobs) from OTCS via the creation of MS teams
    (each MS Team has a SharePoint site behind it)!

    These are the SharePoint sites for the departmental groups such as "Sales",
    "Procurement", "Enterprise Asset Management", ...
    Only departmental group that have a top-level folder with the exact same
    name as the Department are configured.

    For each departmental group:
    1. Determine a departmental folder in the Enterprise Workspace
    2. Determine the M365 Group
    3. Determine the SharePoint Site (based on the M365 group ID)
    4. Determine the Page in the SharePoint site
    5. Determine or create the SharePoint webpart for the OTCS browser
    6. Create URL object pointing to SharePoint site inside top level department folder

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not isinstance(self._m365, M365):
        self.logger.error(
            "Microsoft 365 connection not setup properly. Skipping payload section -> '%s'...",
            section_name,
        )
        return False

    if not self._groups:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for group in self._groups:
        if "name" not in group:
            self.logger.error("Group needs a name. Cannot configure SharePoint site for it. Skipping...")
            success = False
            continue
        group_name = group["name"]

        # Check if group has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not group.get("enabled", True):
            self.logger.info(
                "Payload for group -> '%s' is disabled. Cannot configure SharePoint site. Skipping...",
                group_name,
            )
            continue

        if "enable_o365" not in group or not group["enable_o365"]:
            self.logger.info(
                "Microsoft 365 is not enabled in payload for group -> '%s'. Skipping...",
                group_name,
            )
            continue

        #
        # 1. Determine a departmental folder in the Enterprise Workspace:
        #

        if group_name == "Innovate":
            folder_id = 2000
        else:
            folder = self._otcs.get_node_by_volume_and_path(
                volume_type=self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE,
                path=[group_name],
                show_error=False,
            )
            folder_id = self._otcs.get_result_value(response=folder, key="id")
        if not folder_id:
            self.logger.warning(
                "Group -> '%s' has no folder associated. Cannot configure SharePoint site. Skipping...",
                group_name,
            )
            continue

        #
        # 2. Determine the M365 Group
        #

        # Check if the M365 group does not exist (this should actually never happen at this point)
        m365_group_id = self.determine_group_id_m365(group=group)
        if not m365_group_id:
            # The "m365_id" value is set by the method process_groups_m365()
            self.logger.error(
                "No M365 group exist for group -> '%s' (M365 group creation may have failed). Skipping...",
                group_name,
            )
            success = False
            continue

        #
        # 3. Determine the SharePoint Site:
        #

        site = self._m365.get_sharepoint_site_for_group(group_id=m365_group_id)
        site_id = self._m365.get_result_value(response=site, key="id")
        if not site_id:
            self.logger.info(
                "M365 group -> '%s' has no M365 SharePoint site connected. Skipping...",
                group_name,
            )
            success = False
            continue
        site_name = self._m365.get_result_value(response=site, key="name")

        # Store the SharePoint site ID in the payload.
        group["m365_site_id"] = site_id
        group["m365_site_name"] = site_name
        group["m365_folder_id"] = folder_id

        self.logger.info(
            "Configure M365 SharePoint site -> '%s' (%s) for M365 group -> '%s'...",
            site_name,
            site_id,
            group_name,
        )

        #
        # 4. Determine the Page in the SharePoint site:
        #

        site_page_name = group.get("o365_site_page_name", "OpenText Content Management")

        site_page_id = None
        site_pages = self._m365.get_sharepoint_pages(site_id=site_id)
        if site_pages:
            site_page_id = self._m365.lookup_result_value(
                response=site_pages,
                key="title",
                value=site_page_name,
                return_key="id",
            )
            if site_page_id:
                self.logger.info(
                    "Found existing page -> '%s' (%s) for SharePoint site -> '%s' (%s).",
                    site_page_name,
                    site_page_id,
                    site_name,
                    site_id,
                )
        if not site_page_id:
            site_page = self._m365.add_sharepoint_page(site_id=site_id, page_name=site_page_name)
            if site_page:
                site_page_id = self._m365.get_result_value(response=site_page, key="id")
                self.logger.info(
                    "Successfully created page -> '%s' (%s) for SharePoint site -> '%s' (%s).",
                    site_page_name,
                    site_page_id,
                    site_name,
                    site_id,
                )
            else:
                self.logger.error(
                    "Failed to create page -> '%s' for SharePoint site -> '%s' (%s)!",
                    site_page_name,
                    site_name,
                    site_id,
                )
                success = False
                continue

        #
        # 5. Determine or create the SharePoint webpart for the OTCS browser:
        #

        # Get all webparts on the page:
        site_webparts = self._m365.get_sharepoint_webparts(site_id=site_id, page_id=site_page_id)
        # Check if there's already a webpart for the OTCS browser:
        site_webpart_id = self._m365.lookup_result_value(
            response=site_webparts,
            key="webPartType",
            value="cecfdba4-2e82-4538-9436-dbd1c4c01a80",  # OTCS Browser Type
            return_key="id",
        )
        if site_webpart_id:
            self.logger.info(
                "Found existing OTCS Browser webpart -> '%s'. Updating with folder ID -> %s...",
                site_webpart_id,
                str(folder_id),
            )
            # Update the webpart with the new ID (which has changed after redeployment):
            update_data = {
                "properties": {
                    "ContentServerFolderParentWP": "2000",
                    "ContentServerFolderSelectedWP": str(folder_id),
                },
            }
            response = self._m365.update_sharepoint_webpart(
                site_id=site_id,
                page_id=site_page_id,
                webpart_id=site_webpart_id,
                update_data=update_data,
            )
            if response:
                self.logger.info(
                    "Successfully updated OTCS Browser webpart -> '%s' with new folder ID -> %s.",
                    site_webpart_id,
                    str(folder_id),
                )
            else:
                self.logger.error(
                    "Failed to update OTCS Browser webpart -> '%s' with new folder ID -> %s!",
                    site_webpart_id,
                    str(folder_id),
                )
                success = False
                continue
        else:
            # Check if the section we want for the webpart does already
            # exist. Otherwise create it. Per default we use horizontal
            # section 2. This allows to have a banner as first section
            # that is not affected by the customizer and can be configured
            # manually:
            site_page_section_id = group.get("o365_site_page_section_id", 1)
            site_page_section_type = group.get("o365_site_page_section_type", "horizontalSections")
            self.logger.info(
                "Check if %s section -> %s on page -> '%s' (%s) of SharePoint site -> '%s' (%s) does already exist...",
                "horizontal" if site_page_section_type == "horizontalSections" else "vertical",
                site_page_section_id,
                site_page_name,
                site_page_id,
                site_name,
                site_id,
            )
            site_page_section = self._m365.get_sharepoint_section(
                site_id=site_id,
                page_id=site_page_id,
                section_type=site_page_section_type,
                section_id=site_page_section_id,
                show_error=False,
            )
            if not site_page_section:
                site_page_section = self._m365.add_sharepoint_section(
                    site_id=site_id,
                    page_id=site_page_id,
                    section_type=site_page_section_type,
                    section_id=site_page_section_id,
                )
                if not site_page_section:
                    success = False
                    continue
                self.logger.info(
                    "Created %s section -> %s on page -> '%s' (%s) of SharePoint site -> '%s' (%s)",
                    "horizontal" if site_page_section_type == "horizontalSections" else "vertical",
                    site_page_section_id,
                    site_page_name,
                    site_page_id,
                    site_name,
                    site_id,
                )
            else:
                self.logger.info(
                    "Using existing %s section -> %s on page -> '%s' (%s) of SharePoint site -> '%s' (%s)",
                    "horizontal" if site_page_section_type == "horizontalSections" else "vertical",
                    site_page_section_id,
                    site_page_name,
                    site_page_id,
                    site_name,
                    site_id,
                )

            # If the sharepointAppRootSite is not configured
            # we try to extract the site URL from the site ID
            # which typically has a format like this:
            # ideateqa.sharepoint.com,2c59000d-f3e7-44d1-9a8e-e5df82b8ab01,34b48533-af41-4743-8b41-185a21f0b80f
            site_url = (
                self._m365.config()["sharepointAppRootSite"]
                if self._m365.config()["sharepointAppRootSite"]
                else "https://" + site_id.split(",", 1)[0]
            )
            # Build the web part create data:
            create_data = {
                "@odata.type": "microsoft.graph.webPartData",
                "audiences": [],
                "dataVersion": "1.0",
                "description": "Browse, access and work with documents from OpenText Content Server.",
                "title": "Content Server Browser",
                "properties": {
                    "AzureAppId": self._m365.config()["clientId"],
                    "ContentServerURLWP": "",
                    "URLPrefixWP": "",
                    "SSOEnabledWP": "",
                    "ShowPersonalWorkspaceWP": "",
                    "ShowNavigationBreadcrumbWP": "",
                    "PageSizeWP": "",
                    "ContentServerFolderParentWP": "2000",
                    "ContentServerFolderSelectedWP": str(folder_id),
                    "ContentServerFolderDisplayWP": group_name,
                    "SettingStorageURLSite": site_url,
                    "ContentServerURLSite": self._otcs.config()["csPublicUrl"],
                    "URLPrefixSite": "/cssupport",
                    "ShowPersonalWorkspaceSite": "",
                    "ShowNavigationBreadcrumbSite": "",
                    "PageSizeSite": "",
                    "ContentServerFolderParentSite": "",
                    "ContentServerFolderSelectedSite": "",
                    "ContentServerFolderDisplaySite": "",
                    "SSOEnabledSite": "",
                    "SettingStorageURLGL": "{}/sites/appcatalog".format(
                        site_url,
                    ),
                    "ContentServerURLGL": self._otcs.config()["csPublicUrl"],
                    "URLPrefixGL": "/cssupport",
                    "ShowPersonalWorkspaceGL": "No",
                    "ShowNavigationBreadcrumbGL": "No",
                    "PageSizeGL": "",
                    "ContentServerFolderParentGL": "",
                    "ContentServerFolderSelectedGL": "",
                    "ContentServerFolderDisplayGL": "",
                    "SSOEnabledGL": "No",
                    "TargetPlatform": "SPOnline",
                    "ConfigurationSiteUrl": "",
                    "WebPartVersion": "23.4.0.0",
                    "ContentServerURLAltGL": "",
                    "SavedQueryIdGL": "2344",
                    "SavedQueryParentIdGL": "2329",
                    "SavedQueryNameGL": "Full Text Business Workspaces Search",
                    "isCurrentUserSiteAdmin": True,
                    "isCurrentUserGlobalAdmin": True,
                    "isTeamsContext": False,
                    "ContentServerUrlLocal": self._otcs.config()["csPublicUrl"],
                },
            }

            response = self._m365.add_sharepoint_webpart(
                site_id=site_id,
                page_id=site_page_id,
                webpart_type_id="cecfdba4-2e82-4538-9436-dbd1c4c01a80",
                section_id=site_page_section_id,
                create_data=create_data,
            )
            if response:
                self.logger.info(
                    "Successfully added OTCS browser webpart -> '%s' to page -> '%s' (%s) on site ->'%s' (%s).",
                    self._m365.get_result_value(response=response, key="id"),
                    site_page_name,
                    site_page_id,
                    site_name,
                    site_id,
                )
            else:
                self.logger.error(
                    "Failed to add OTCS browser webpart to page -> '%s' (%s) on site ->'%s' (%s)!",
                    site_page_name,
                    site_page_id,
                    site_name,
                    site_id,
                )
                success = False
            # end else
        # end else

        #
        # 6. Create URL object pointing to SharePoint site inside top level department folder
        #

        item_name = (
            "SharePoint site for {} department".format(group_name)
            if folder_id != 2000
            else "SharePoint site for Innovate"
        )
        response = self._otcs.get_node_by_parent_and_name(parent_id=folder_id, name=item_name)
        item_id = self._otcs.get_result_value(response=response, key="id")
        if not item_id:
            response = self._otcs.create_item(
                parent_id=folder_id,
                item_type=self._otcs.ITEM_TYPE_URL,
                item_name=item_name,
                url=site["webUrl"],
            )
            self.logger.info(
                "Created URL item -> '%s' (%s).",
                item_name,
                site["webUrl"],
            )
        else:
            self.logger.info(
                "URL item -> '%s' (%s) does already exist.",
                item_name,
                site["webUrl"],
            )
    # end for group in self._groups:

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._groups,
    )

    return success

process_supplemental_markings(section_name='supplementalMarkings')

Process Supplemental Markings for Content Server.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'supplementalMarkings'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_supplemental_markings")
def process_supplemental_markings(
    self,
    section_name: str = "supplementalMarkings",
) -> bool:
    """Process Supplemental Markings for Content Server.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._supplemental_markings:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for supplemental_marking in self._supplemental_markings:
        code = supplemental_marking.get("code")

        # Check if supplemental marking has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not supplemental_marking.get("enabled", True):
            self.logger.info(
                "Payload for supplemental marking -> '%s' is disabled. Skipping...",
                code,
            )
            continue

        description = supplemental_marking.get("description", "")

        if code:
            self.logger.info(
                "Creating supplemental marking -> '%s'%s...",
                code,
                " ({})".format(description) if description else "",
            )
            self._otcs.run_web_report(
                nickname="web_report_supplemental_marking",
                web_report_parameters=supplemental_marking,
            )
        else:
            self.logger.error(
                "Cannot create supplemental marking - either code or description is missing!",
            )
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._supplemental_markings,
    )

    return success

process_synchronized_partitions(section_name='synchronizedPartitions')

Process OTDS synchronized partitions in payload and create them in OTDS.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'synchronizedPartitions'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_synchronized_partitions")
def process_synchronized_partitions(
    self,
    section_name: str = "synchronizedPartitions",
) -> bool:
    """Process OTDS synchronized partitions in payload and create them in OTDS.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise.

    """

    # check if section present, if not return True
    if not self._synchronized_partitions:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True
    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success = True

    for partition in self._synchronized_partitions:
        partition_name = partition["spec"].get("profileName") if "spec" in partition else None
        if not partition_name:
            self.logger.error(
                "Synchronized partition does not have a profile name! Skipping...",
            )
            success = False
            continue

        # Check if partition has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not partition.get("enabled", True):
            self.logger.info(
                "Payload for synchronized partition -> '%s' is disabled. Skipping...",
                partition_name,
            )
            continue

        partition_description = partition["spec"].get("description", "")

        # Check if Partition does already exist
        # (in an attempt to make the code idem-potent)
        self.logger.info(
            "Check if OTDS synchronized partition -> '%s' does already exist...",
            partition_name,
        )
        response = self._otds.get_partition(name=partition_name, show_error=False)
        if response:
            self.logger.info(
                "Synchronized partition -> '%s' does already exist.",
                partition_name,
            )
        else:
            # Only continue if synchronized Partition does not exist already
            self.logger.info(
                "Synchronized partition -> '%s' does not yet exist. Creating...",
                partition_name,
            )

            response = self._otds.add_synchronized_partition(
                name=partition_name,
                description=partition_description,
                data=partition["spec"],
            )
            if response:
                self.logger.info(
                    "Added synchronized partition -> '%s' to OTDS.",
                    partition_name,
                )
            else:
                self.logger.error(
                    "Failed to add synchronized partition -> '%s' to OTDS!",
                    partition_name,
                )
                success = False
                continue

        result = self._otds.import_synchronized_partition_members(
            name=partition_name,
        )
        if result:
            self.logger.info(
                "Successfully imported members to OTDS synchronized partition -> '%s'.",
                partition_name,
            )
        else:
            self.logger.error(
                "Failed to import members to OTDS synchronized partition -> '%s'!",
                partition_name,
            )
            success = False

        access_role = partition.get("access_role")
        if access_role:
            response = self._otds.add_partition_to_access_role(
                access_role=access_role,
                partition=partition_name,
            )
            if response:
                self.logger.info(
                    "Added OTDS synchronized partition -> '%s' to access role -> '%s'.",
                    partition_name,
                    access_role,
                )
            else:
                self.logger.error(
                    "Failed to add OTDS synchronized partition -> '%s' to access role -> '%s'!",
                    partition_name,
                    access_role,
                )
                success = False
        # end if access_role

        # Partions may have an optional list of licenses in
        # the payload. Assign the partition to all these licenses:
        partition_specific_licenses = partition.get("licenses")
        if partition_specific_licenses:
            # We assume these licenses are Extended ECM licenses!
            otcs_resource_name = self._otcs.config()["resource"]
            otcs_resource = self._otds.get_resource(name=otcs_resource_name)
            if not otcs_resource:
                self.logger.error(
                    "Cannot find OTCS resource -> '%s'! Skipping...",
                    otcs_resource_name,
                )
                success = False
                continue
            otcs_resource_id = otcs_resource["resourceID"]
            otcs_license_name = "EXTENDED_ECM"
            for license_item in partition_specific_licenses:
                if isinstance(license_item, dict):
                    license_feature = license_item.get("feature")
                    license_name = license_item.get("name", "EXTENDED_ECM")
                    if "enabled" in license_item and not license_item["enabled"]:
                        self.logger.info(
                            "Payload for License '%s' -> '%s' is disabled. Skipping...",
                            license_name,
                            license_feature,
                        )
                        continue
                    if "resource" in license_item:
                        try:
                            resource_id = self._otds.get_resource(name=license_item["resource"])["resourceID"]
                        except Exception:
                            self.logger.error(
                                "Error getting resource ID from resource -> %s", license_item["resource"]
                            )
                    else:
                        resource_id = otcs_resource_id

                elif isinstance(license_item, str):
                    license_feature = license_item
                    resource_id = otcs_resource_id
                    license_name = otcs_license_name
                else:
                    self.logger.error("Invalid License feature specified -> %s!", license_item)
                    success = False
                    continue

                if self._otds.is_partition_licensed(
                    partition_name=partition_name,
                    resource_id=resource_id,
                    license_feature=license_feature,
                    license_name=license_name,
                ):
                    self.logger.info(
                        "Partition -> '%s' is already licensed for -> '%s' (%s).",
                        partition_name,
                        license_name,
                        license_feature,
                    )
                    continue

                assigned_license = self._otds.assign_partition_to_license(
                    partition_name=partition_name,
                    resource_id=resource_id,
                    license_feature=license_feature,
                    license_name=license_name,
                )

                if not assigned_license:
                    self.logger.error(
                        "Failed to assign synchronized partition -> '%s' to license feature -> '%s' of license -> '%s'!",
                        partition_name,
                        license_feature,
                        license_name,
                    )
                    success = False
                else:
                    self.logger.info(
                        "Successfully assigned synchronized partition -> '%s' to license feature -> '%s' of license -> '%s'.",
                        partition_name,
                        license_feature,
                        license_name,
                    )
        # end if partition_specific_licenses

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._synchronized_partitions,
    )

    return success

process_system_attributes(section_name='systemAttributes')

Process OTDS system attributes in payload and create them in OTDS.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'systemAttributes'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_system_attributes")
def process_system_attributes(self, section_name: str = "systemAttributes") -> bool:
    """Process OTDS system attributes in payload and create them in OTDS.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._system_attributes:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for system_attribute in self._system_attributes:
        attribute_name = system_attribute.get("name")
        if not attribute_name:
            self.logger.error("OTDS system attribute needs a name in payload! Skipping...")
            success = False
            continue

        # Check if system attribute has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not system_attribute.get("enabled", True):
            self.logger.info(
                "Payload for OTDS system attribute -> '%s' is disabled. Skipping...",
                attribute_name,
            )
            continue

        attribute_value = system_attribute.get("value")
        if not attribute_value:
            self.logger.error(
                "OTDS system attribute -> '%s' needs a value in payload! Skipping...",
                attribute_name,
            )
            continue

        attribute_description = system_attribute.get("description", "")

        # Add the system attribute to OTDS:
        response = self._otds.add_system_attribute(
            name=attribute_name,
            value=attribute_value,
            description=attribute_description,
        )
        if response:
            self.logger.info(
                "Added OTDS system attribute -> '%s' with value -> '%s'.",
                attribute_name,
                attribute_value,
            )
        else:
            self.logger.error(
                "Failed to add OTDS system attribute -> '%s' with value -> '%s'!",
                attribute_name,
                attribute_value,
            )
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._system_attributes,
    )

    return success

process_teams_m365(section_name='teamsM365')

Process groups in payload and create matching Teams in Microsoft 365.

We need to do this after the creation of the M365 users as we require Group Owners to create teams. These are NOT the teams for OTCS workspaces! Those are created by Scheduled Bots (Jobs) from OTCS!

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'teamsM365'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_teams_m365")
def process_teams_m365(self, section_name: str = "teamsM365") -> bool:
    """Process groups in payload and create matching Teams in Microsoft 365.

    We need to do this after the creation of the M365 users as we require
    Group Owners to create teams. These are NOT the teams for OTCS
    workspaces! Those are created by Scheduled Bots (Jobs) from OTCS!

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not isinstance(self._m365, M365):
        self.logger.error(
            "Microsoft 365 connection not setup properly. Skipping payload section -> '%s'...",
            section_name,
        )
        return False

    if not self._groups:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for group in self._groups:
        if "name" not in group:
            self.logger.error("Team needs a name. Skipping...")
            success = False
            continue
        group_name = group["name"]

        # Check if group has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not group.get("enabled", True):
            self.logger.info(
                "Payload for group -> '%s' is disabled. Skipping...",
                group_name,
            )
            continue

        if "enable_o365" not in group or not group["enable_o365"]:
            self.logger.info(
                "Microsoft 365 is not enabled in payload for group -> '%s'. Skipping...",
                group_name,
            )
            continue

        # Check if the M365 group does not exist (this should actually never happen at this point)
        m365_group_id = self.determine_group_id_m365(group=group)
        if not m365_group_id:
            # The "m365_id" value is set by the method process_groups_m365()
            self.logger.error(
                "No M365 group exist for group -> '%s' (M365 group creation may have failed). Skipping...",
                group_name,
            )
            success = False
            continue

        if self._m365.has_team(group_name=group_name):
            self.logger.info(
                "M365 group -> '%s' already has an MS Team connected. Skipping...",
                group_name,
            )
            continue

        self.logger.info(
            "Create M365 team -> '%s' for existing M365 group -> '%s'...",
            group_name,
            group_name,
        )
        # Now "upgrading" this group to a MS Team:
        new_team = self._m365.add_team(name=group_name)
        if not new_team:
            success = False
            continue

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._groups,
    )

    return success

process_teams_m365_apps(section_name='teamsM365Apps', tab_name='Extended ECM')

Process groups in payload and configure Extended ECM Teams Apps.

The app is exposed as a tab called "Extended ECM" in the "General" channel of the M365 Team.

We need to do this after the transports as we need top level folders we can point the Extended ECM teams app to.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'teamsM365Apps'
tab_name	str	Name of the Extended ECM tab. Default is "Extended ECM".	'Extended ECM'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_teams_m365_apps")
def process_teams_m365_apps(
    self,
    section_name: str = "teamsM365Apps",
    tab_name: str = "Extended ECM",
) -> bool:
    """Process groups in payload and configure Extended ECM Teams Apps.

    The app is exposed as a tab called "Extended ECM" in the "General"
    channel of the M365 Team.

    We need to do this after the transports as we need top level folders
    we can point the Extended ECM teams app to.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.
        tab_name (str, optional):
            Name of the Extended ECM tab. Default is "Extended ECM".

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not isinstance(self._m365, M365):
        self.logger.error(
            "Microsoft 365 connection not setup properly. Skipping payload section -> '%s'...",
            section_name,
        )
        return False

    if not self._groups:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # Determine the ID of the Extended ECM App:
    app_name = self._m365.config()["teamsAppName"]
    app_internal_id = self._m365.config()["teamsAppInternalId"]
    if not app_internal_id:
        response = self._m365.get_teams_apps(
            filter_expression="contains(displayName, '{}')".format(app_name),
        )
        # Get the App catalog ID:
        app_internal_id = self._m365.get_result_value(
            response=response,
            key="id",
            index=0,
        )
    if not app_internal_id:
        self.logger.error("M365 Teams app -> '%s' not found in catalog!", app_name)
        return False

    for group in self._groups:
        if "name" not in group:
            self.logger.error("Team needs a name. Skipping...")
            success = False
            continue
        group_name = group["name"]

        # Check if group has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not group.get("enabled", True):
            self.logger.info(
                "Payload for group -> '%s' is disabled. Skipping...",
                group_name,
            )
            continue

        if "enable_o365" not in group or not group["enable_o365"]:
            self.logger.info(
                "Microsoft 365 is not enabled in payload for group -> '%s'. Skipping...",
                group_name,
            )
            continue

        #
        # Now we create a tab in the "General" channel for the Extended ECM Teams App
        #

        # 1. Check if the tab is already assigned to the General channel.
        # This determines if we need to create or update the tab / app:
        app_name = self._m365.config()["teamsAppName"]
        response = self._m365.get_team_channel_tabs(
            team_name=group_name,
            channel_name="General",
        )
        # Check if tab is already there:
        if self._m365.exist_result_item(
            response=response,
            key="displayName",
            value=tab_name,
        ):
            self.logger.info(
                "M365 Teams app -> '%s' is already configured for M365 Team -> '%s' (tab -> '%s' does already exist). Updating it with new URLs and IDs...",
                app_name,
                group_name,
                tab_name,
            )
            update = True  # update existing tab
        else:
            self.logger.info(
                "Add tab -> '%s' with app -> '%s' to channel -> 'General' of M365 Team -> '%s' ",
                tab_name,
                app_name,
                group_name,
            )
            update = False  # create new tab

        # 2. Determine the M365 Team ID. If the team is not found then skip:
        response = self._m365.get_team(name=group_name)
        team_id = self._m365.get_result_value(response=response, key="id", index=0)
        if not team_id:
            self.logger.error("M365 Team -> '%s' not found!", group_name)
            success = False
            continue

        # 3. Install the App for the particular M365 Team (if it is not yet installed):
        response = self._m365.get_teams_apps_of_team(
            team_id=team_id,
            filter_expression="contains(teamsAppDefinition/displayName, '{}')".format(
                app_name,
            ),
        )
        if self._m365.exist_result_item(
            response=response,
            key="displayName",
            value=app_name,
            sub_dict_name="teamsAppDefinition",
        ):
            self.logger.info(
                "M365 Teams app -> '%s' is already installed for M365 Team -> '%s' (%s). Trying to upgrade app...",
                app_name,
                group_name,
                team_id,
            )
            response = self._m365.upgrade_teams_app_of_team(
                team_id=team_id,
                app_name=app_name,
            )
            if not response:
                self.logger.error(
                    "Failed to upgrade the existing app -> '%s' for the M365 Team -> '%s'!",
                    app_name,
                    group_name,
                )
                success = False
                continue
        else:
            self.logger.info(
                "Install M365 Teams app -> '%s' for M365 team -> '%s'...",
                app_name,
                group_name,
            )
            response = self._m365.assign_teams_app_to_team(
                team_id=team_id,
                app_id=app_internal_id,
            )
            if not response:
                self.logger.error(
                    "Failed to install app -> '%s' (%s) for M365 Team -> '%s'!",
                    app_name,
                    app_internal_id,
                    group_name,
                )
                success = False
                continue

        # 4. Create a Tab in the "General" channel of the M365 Team:
        if group_name == "Innovate":
            # Use the Enterprise Workspace for the
            # top-level group "Innovate":
            node_id = 2000
        else:
            # We assume the departmental group names are identical to
            # top-level folders in the Enterprise volume
            node = self._otcs.get_node_by_parent_and_name(
                parent_id=2000,
                name=group_name,
            )
            node_id = self._otcs.get_result_value(response=node, key="id")
            if not node_id:
                self.logger.info(
                    "Cannot find a top-level folder for group -> '%s'. Cannot configure M365 Teams app. Skipping...",
                    group_name,
                )
                continue

        app_url = self._otcs_frontend.cs_support_public_url()  # it is important to use the frontend pod URL here
        app_url += "/xecmoffice/teamsapp.html?nodeId="
        app_url += str(node_id) + "&type=container&parentId=2000&target=content&csurl="
        app_url += self._otcs_frontend.cs_public_url()
        app_url += "&appId=" + app_internal_id

        if update:
            # App / Tab exist but needs to be updated with new
            # IDs for the new deployment of Extended ECM
            # as the M365 Teams survive between Terrarium deployments:

            self.logger.info(
                "Updating tab -> '%s' of M365 Team channel -> 'General' for app -> '%s' (%s) with new URLs and node IDs...",
                tab_name,
                app_name,
                app_internal_id,
            )

            response = self._m365.update_teams_app_of_channel(
                team_name=group_name,
                channel_name="General",
                tab_name=tab_name,
                app_url=app_url,
                cs_node_id=node_id,
            )
        else:
            # Tab does not exist in "General" channel so we
            # add / configure it freshly:

            self.logger.info(
                "Adding tab -> '%s' with app -> '%s' (%s) in M365 Team channel -> 'General'...",
                tab_name,
                app_name,
                app_internal_id,
            )

            response = self._m365.add_teams_app_to_channel(
                team_name=group_name,
                channel_name="General",
                app_id=app_internal_id,
                tab_name=tab_name,
                app_url=app_url,
                cs_node_id=node_id,
            )
            if not response:
                self.logger.error(
                    "Failed to add tab -> '%s' with app -> '%s' (%s) to M365 Team channel -> 'General'!",
                    tab_name,
                    app_name,
                    app_internal_id,
                )

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._groups,
    )

    return success

process_transport_packages(transport_packages, section_name='transportPackages')

Process transport packages in payload and import them to Content Server.

Parameters:

Name	Type	Description	Default
transport_packages	list	A list of transport packages. As we have three different lists (transport, content_transport, transport_post) we need a parameter to pass it.	required
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'transportPackages'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_transport_packages")
def process_transport_packages(
    self,
    transport_packages: list,
    section_name: str = "transportPackages",
) -> bool:
    """Process transport packages in payload and import them to Content Server.

    Args:
        transport_packages (list):
            A list of transport packages. As we have three different lists (transport,
            content_transport, transport_post) we need a parameter to pass it.
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not transport_packages:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for transport_package in transport_packages:
        name = transport_package.get("name")
        if not name:
            self.logger.error(
                "Transport package needs a name! Skipping to next transport...",
            )
            success = False
            continue

        # Check if transport package has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not transport_package.get("enabled", True):
            self.logger.info(
                "Payload for transport package -> '%s' is disabled. Skipping...",
                name,
            )
            continue

        url = transport_package.get("url")
        if not url:
            self.logger.error(
                "Transport package -> '%s' needs a URL! Skipping to next transport...",
                name,
            )
            success = False
            continue

        description = transport_package.get("description", "")
        if not description:
            self.logger.warning(
                "Transport package -> '%s' is missing a description.",
                name,
            )

        # if the transport is not in the local file system
        # but given by an URL we download it first:
        if url.startswith("http"):
            package_path = self.download_transport_package(package_url=url)
            if not package_path:
                success = False
                continue
            url = package_path

        # For some transports there can be string replacements
        # configured:
        if "replacements" in transport_package:
            replacements = transport_package["replacements"]
            self.logger.info(
                "Transport -> '%s' has replacements -> %s.",
                name,
                str(replacements),
            )
        else:
            replacements = None

        # For some transports there can be data extractions
        # configured:
        if "extractions" in transport_package:
            extractions = transport_package["extractions"]
            self.logger.info(
                "Transport -> '%s' has extractions -> %s.",
                name,
                str(extractions),
            )
        else:
            extractions = None

        if description:
            self.logger.info("Deploy transport -> '%s' ('%s')...", name, description)
        else:
            self.logger.info("Deploy transport -> '%s'...", name)

        response = self._otcs.deploy_transport(
            package_url=url,
            package_name=name,
            package_description=description,
            replacements=replacements,
            extractions=extractions,
        )
        if response is None:
            self.logger.error("Failed to deploy transport -> '%s'!", name)
            success = False
            if self._stop_on_error:
                break  # stop the for loop
        else:
            self.logger.info("Successfully deployed transport -> '%s'.", name)
            # Save the extractions for later processing, e.g. in process_business_object_types()
            if extractions:
                self.add_transport_extractions(extractions=extractions)
    # end for transports

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=transport_packages,
    )
    self.write_status_file(
        success=success,
        payload_section_name=section_name + "Extractions",
        payload_section=self._transport_extractions,
    )

    # If stop_on_error is enabled we want to completely
    # stop the execution of the customizer and bail out:
    if not success and self._stop_on_error:
        raise StopOnError(message="STOP_ON_ERROR enabled -> Stopping execution")

    return success

process_trusted_sites(section_name='trustedSites')

Process OTDS trusted sites in payload and create them in OTDS.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'trustedSites'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_trusted_sites")
def process_trusted_sites(self, section_name: str = "trustedSites") -> bool:
    """Process OTDS trusted sites in payload and create them in OTDS.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise

    """

    if not self._trusted_sites:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for trusted_site in self._trusted_sites:
        # old payload may still have trusted sites as list of string
        # we changed also the trusted sites to dict with 23.3
        # We want to be backwards compatible so we handle both cases:
        url = None
        if isinstance(trusted_site, dict):
            url = trusted_site.get("url")
        elif isinstance(trusted_site, str):
            url = trusted_site

        # Check if element has been disabled in payload (enabled = false).
        # In this case we skip the element:
        if isinstance(trusted_site, dict) and "enabled" in trusted_site and not trusted_site["enabled"]:
            self.logger.info(
                "Payload for OTDS Trusted Site -> '%s' is disabled. Skipping...",
                url if url else "<undefined>",
            )
            continue

        if not url:
            self.logger.error("OTDS Trusted site does not have a URL! Skipping...")
            success = False
            continue

        response = self._otds.add_trusted_site(trusted_site=url)
        if response:
            self.logger.info("Added OTDS trusted site -> '%s'.", url)
        else:
            self.logger.error("Failed to add trusted site -> '%s'!", url)
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._trusted_sites,
    )

    return success

process_user_favorites_and_profiles(section_name='userFavoritesAndProfiles')

Process user favorites in payload and create them in Content Server.

This method also simulates browsing the favorites to populate the widgets on the landing pages and sets personal preferences.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'userFavoritesAndProfiles'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_favorites_and_profiles")
def process_user_favorites_and_profiles(
    self,
    section_name: str = "userFavoritesAndProfiles",
) -> bool:
    """Process user favorites in payload and create them in Content Server.

    This method also simulates browsing the favorites to populate the
    widgets on the landing pages and sets personal preferences.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._users:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    for dic in self._payload_sections:
        if dic.get("name") == "users":
            users_payload = dic
            break
    smartui_theme = "jato" if users_payload.get("jato_enabled", True) else "cf"

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # We can only set favorites if we impersonate / authenticate as the user.
    # The following code (for loop) will change the authenticated user - we need to
    # switch it back to admin user later so we safe the admin credentials for this:

    for user in self._users:
        with tracer.start_as_current_span("process_user_favorites_and_profiles-user") as t:
            t.set_attribute("class", "otcs")
            user_name = user.get("name")
            if not user_name:
                self.logger.error(
                    "Missing user name - cannot configure user favorites and profile. Skipping user...",
                )
                continue

            self._log_header_callback(
                text="Process Favorites and Profile for user -> '{}'".format(user_name),
                char="-",
            )

            # Check if user has been explicitly disabled in payload
            # (enabled = false). In this case we skip the element:
            if not user.get("enabled", True):
                self.logger.info(
                    "Payload for user -> '%s' is disabled. Skipping...",
                    user_name,
                )
                continue

            # We skip also user of type "ServiceUser":
            if user.get("type", "User") == "ServiceUser":
                self.logger.info("Skipping service user -> '%s'...", user_name)
                continue

            # Impersonate as the user:
            self.logger.info("Impersonate user -> '%s'...", user_name)
            result = self.start_impersonation(username=user_name)
            if not result:
                self.logger.error("Couldn't impersonate user -> '%s'!", user_name)
                success = False
                continue

            # Configure default Theme to be Jato if configured
            user_smartui_theme = user.get("smartui_theme", smartui_theme)
            if user_smartui_theme != "cf":
                response = self._otcs.update_user_profile(
                    config_section="SmartUI",
                    field="theme",
                    value=user_smartui_theme,
                )
                if response is None:
                    self.logger.warning(
                        "Profile for user -> '%s' couldn't be updated to Smart View theme -> '%s'!",
                        user_name,
                        user_smartui_theme,
                    )
                else:
                    self.logger.info(
                        "Profile for user -> '%s' has been updated to Smart View theme -> '%s'.",
                        user_name,
                        user_smartui_theme,
                    )

            # Update the user profile to activate responsive (dynamic) containers:
            response = self._otcs.update_user_profile(
                field="responsiveContainerMode",
                value=True,
                config_section="SmartUI",
            )
            if response is None:
                self.logger.warning(
                    "Profile for user -> '%s' couldn't be updated with responsive container mode!",
                    user_name,
                )
            else:
                self.logger.info(
                    "Profile for user -> '%s' has been updated to enable responsive container mode.",
                    user_name,
                )
            response = self._otcs.update_user_profile(
                field="responsiveContainerMessageMode",
                value=True,
                config_section="SmartUI",
            )
            if response is None:
                self.logger.warning(
                    "Profile for user -> '%s' couldn't be updated with responsive container message mode!",
                    user_name,
                )
            else:
                self.logger.info(
                    "Profile for user -> '%s' has been updated to enable messages for responsive container mode.",
                    user_name,
                )

            restrict_personal_workspace = user.get("restrict_personal_workspace", False)
            if restrict_personal_workspace:
                # Let the user restrict itself to have read-only access to its
                # personal workspace:
                node = self._otcs.get_node_by_volume_and_path(
                    volume_type=self._otcs.VOLUME_TYPE_PERSONAL_WORKSPACE,
                    path=[],
                )
                node_id = self._otcs.get_result_value(response=node, key="id")
                if node_id:
                    self.logger.info(
                        "Restricting Personal Workspace of user -> '%s' to read-only.",
                        user_name,
                    )
                    response = self._otcs.assign_permission(
                        node_id=int(node_id),
                        assignee_type="owner",
                        assignee=0,
                        permissions=["see", "see_contents"],
                        apply_to=2,
                    )

            # We work through the list of favorites defined for the user:
            favorites = user.get("favorites", [])
            for favorite in favorites:
                # We try three things to determine the favorite node:
                # 1. If the favorite is a path (a python list) then we try to resolve
                #    this path in the Enterprise Workspace.
                # 2. We try to find the item (string) as a logical
                #    workspace ID inside the payload.
                # 3. We try to find the item (string) as a nickname in OTCS.
                favorite_id = None
                is_workspace = False
                # 1. Check if the favorite item is itself a list,
                #    then we try to interpret it as a path in
                #    the enterprise workspace:
                if isinstance(favorite, list):
                    favorite_item = self._otcs.get_node_by_volume_and_path(
                        volume_type=self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE,
                        path=favorite,
                    )
                    if not favorite_item:
                        self.logger.error("Cannot find path -> %s for favorite item!", str(favorite))
                        success = False
                        continue
                    favorite_id = self._otcs.get_result_value(
                        response=favorite_item,
                        key="id",
                    )
                    favorite_name = self._otcs.get_result_value(
                        response=favorite_item,
                        key="name",
                    )
                    favorite_type = self._otcs.get_result_value(
                        response=favorite_item,
                        key="type",
                    )
                    if favorite_type == self._otcs.ITEM_TYPE_BUSINESS_WORKSPACE:
                        is_workspace = True
                # 2. Check if it a logical workspace identifier in the payload:
                if not favorite_id:
                    # check if favorite is a logical workspace name
                    favorite_item = next(
                        (item for item in self._workspaces if item["id"] == favorite),
                        None,
                    )
                    if favorite_item:
                        if favorite_item.get("enabled", True):
                            self.logger.debug(
                                "Found favorite item (workspace) -> '%s' in payload and it is enabled.",
                                favorite_item["name"],
                            )
                        else:
                            self.logger.info(
                                "Found favorite item (workspace) -> '%s' in payload but it is not enabled. Skipping...",
                                favorite_item["name"],
                            )
                            continue
                        favorite_id = self.determine_workspace_id(workspace=favorite_item)
                        if not favorite_id:
                            self.logger.warning(
                                "Workspace of type -> '%s' and name -> '%s' does not exist. Cannot create favorite. Skipping...",
                                favorite_item["type_name"],
                                favorite_item["name"],
                            )
                            continue
                        is_workspace = True
                        favorite_name = favorite_item["name"]
                # 3. Check if it is a nickname:
                if not favorite_id:
                    favorite_item = self._otcs.get_node_from_nickname(nickname=favorite)
                    favorite_id = self._otcs.get_result_value(
                        response=favorite_item,
                        key="id",
                    )
                    favorite_name = self._otcs.get_result_value(
                        response=favorite_item,
                        key="name",
                    )
                    favorite_type = self._otcs.get_result_value(
                        response=favorite_item,
                        key="type",
                    )
                    if favorite_type == self._otcs.ITEM_TYPE_BUSINESS_WORKSPACE:
                        is_workspace = True

                # If nothing has worked then skip this payload favorite.
                if not favorite_id:
                    self.logger.warning(
                        "Favorite -> '%s' neither found as workspace payload ID, not as a path, nor as nickname. Skipping to next favorite...",
                        favorite,
                    )
                    continue

                response = self._otcs.add_favorite(node_id=favorite_id)
                if response is None:
                    self.logger.warning(
                        "Favorite -> '%s' (%s) couldn't be added for user -> '%s'!",
                        favorite_name,
                        favorite_id,
                        user_name,
                    )
                else:
                    self.logger.info(
                        "Added favorite -> '%s' (%s) for user -> '%s'.",
                        favorite_name,
                        favorite_id,
                        user_name,
                    )
                    self.logger.info(
                        "Simulate user -> '%s' browsing node -> '%s' (%s)...",
                        user_name,
                        favorite_name,
                        favorite_id,
                    )
                    # simulate a browse by the user to populate recently accessed items
                    response = (
                        self._otcs.get_workspace(node_id=favorite_id, fields="properties{id}")
                        if is_workspace
                        else self._otcs.get_node(node_id=favorite_id, fields="properties{id}")
                    )
            # end for favorite in favorites

            # we work through the list of proxies defined for the user
            # (we need to consider that not all users have the proxies element):
            proxies = user.get("proxies", [])

            for proxy in proxies:
                proxy_user = next(
                    (item for item in self._users if item["name"] == proxy),
                    None,
                )
                if not proxy_user or "id" not in proxy_user:
                    self.logger.error(
                        "The proxy -> '%s' for user -> '%s' does not exist! Skipping proxy...",
                        proxy,
                        user_name,
                    )
                    success = False
                    continue
                proxy_user_id = proxy_user["id"]

                # Check if the proxy is already set:
                if not self._otcs.is_proxy(user_name=proxy):
                    self.logger.info(
                        "Set user -> '%s' (%s) as proxy for user -> '%s'.",
                        proxy,
                        proxy_user_id,
                        user_name,
                    )
                    # set the user proxy - currently we don't support time based proxies in payload.
                    # The called method is ready to support this.
                    response = self._otcs.add_user_proxy(proxy_user_id=proxy_user_id)
                else:
                    self.logger.info(
                        "User -> '%s' (%s) is already proxy for user -> '%s'. Skipping...",
                        proxy,
                        proxy_user_id,
                        user_name,
                    )
        # end for user in self._users

    if self._users:
        # Impersonate as the admin user:
        self.logger.info(
            "Impersonate as admin user -> '%s'...",
            self._otcs.credentials()["username"],
        )
        # Stop the impersonation as a user:
        result = self.stop_impersonation()
        if not result:
            self.logger.error(
                "Couldn't impersonate as admin user -> '%s'!",
                self._otcs.credentials()["username"],
            )
            success = False

    # Also for the admin user we want to update the user profile to activate responsive (dynamic) containers:
    response = self._otcs.update_user_profile(
        field="responsiveContainerMode",
        value=True,
        config_section="SmartUI",
    )
    if response is None:
        self.logger.warning(
            "Profile for 'admin' user couldn't be updated with responsive container mode!",
        )
    else:
        self.logger.info(
            "Profile for 'admin' user has been updated to enable responsive container mode.",
        )
    response = self._otcs.update_user_profile(
        field="responsiveContainerMessageMode",
        value=True,
        config_section="SmartUI",
    )
    if response is None:
        self.logger.warning(
            "Profile for 'admin' user couldn't be updated with responsive container message mode!",
        )
    else:
        self.logger.info(
            "Profile for 'admin' user has been updated to enable messages for responsive container mode.",
        )

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._users,
    )

    return success

process_user_licenses(resource_name, license_feature, license_name='EXTENDED_ECM', user_specific_payload_field='licenses', section_name='userLicenses')

Assign a specific OTDS license feature to all Content Server users.

This method is used for OTIV and OTCS licenses.

Parameters:

Name	Type	Description	Default
resource_name	str	The name of the OTDS resource.	required
license_feature	str	The license feature to assign to the user (product specific).	required
license_name	str	The name of the license Key (e.g. "EXTENDED_ECM" or "INTELLIGENT_VIEWING").	'EXTENDED_ECM'
user_specific_payload_field	str	The name of the user specific field in payload (if empty it will be ignored).	'licenses'
section_name	str	The name of the section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'userLicenses'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_licenses")
def process_user_licenses(
    self,
    resource_name: str,
    license_feature: str,
    license_name: str = "EXTENDED_ECM",
    user_specific_payload_field: str = "licenses",
    section_name: str = "userLicenses",
) -> bool:
    """Assign a specific OTDS license feature to all Content Server users.

    This method is used for OTIV and OTCS licenses.

    Args:
        resource_name (str):
            The name of the OTDS resource.
        license_feature (str):
            The license feature to assign to the user (product specific).
        license_name (str):
            The name of the license Key (e.g. "EXTENDED_ECM" or "INTELLIGENT_VIEWING").
        user_specific_payload_field (str, optional):
            The name of the user specific field in payload
            (if empty it will be ignored).
        section_name (str, optional):
            The name of the section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise.

    """

    if not self._users:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    otds_resource = self._otds.get_resource(resource_name)
    if not otds_resource:
        self.logger.error(
            "OTDS Resource -> '%s' not found. Cannot assign licenses to users.",
            resource_name,
        )
        return False

    user_partition = self._otcs.config()["partition"]
    if not user_partition:
        self.logger.error("OTCS user partition not found in OTDS!")
        return False

    for user in self._users:
        user_name = user.get("name")
        if not user_name:
            self.logger.error(
                "Missing user name - cannot assign license to user. Skipping user...",
            )
            continue

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not user.get("enabled", True):
            self.logger.info(
                "Payload for user -> '%s' is disabled. Skipping...",
                user_name,
            )
            continue

        if user_specific_payload_field and user_specific_payload_field in user:
            self.logger.info(
                "Found specific license feature -> %s for user -> '%s'. Overwriting default license feature -> '%s'.",
                user[user_specific_payload_field],
                user_name,
                license_feature,
            )
            user_license_features = user[user_specific_payload_field]
        else:  # use the default feature from the actual parameter
            user_license_features = [license_feature]

        for user_license_feature in user_license_features:
            if isinstance(user_license_feature, dict):
                user_license_feature_dict = user_license_feature
                user_license_feature = user_license_feature_dict.get("feature", license_feature)
                license_name = user_license_feature_dict.get("name", license_name)
                if "enabled" in user_license_feature_dict and not user_license_feature_dict["enabled"]:
                    self.logger.info(
                        "Payload for License '%s' -> '%s' is disabled. Skipping...",
                        license_name,
                        license_feature,
                    )
                    continue

                if "resource" in user_license_feature_dict:
                    try:
                        resource_id = self._otds.get_resource(name=user_license_feature_dict["resource"])[
                            "resourceID"
                        ]
                    except Exception:
                        self.logger.error(
                            "Error retrieving resourceID for -> %s", user_license_feature_dict["resource"]
                        )
                        continue
                        success = False
                else:
                    resource_id = otds_resource["resourceID"]

            elif isinstance(user_license_feature, str):
                resource_id = otds_resource["resourceID"]

            else:
                self.logger.error("Invalid License feature specified -> %s", user_license_feature)
                continue

            if self._otds.is_user_licensed(
                user_name=user_name,
                resource_id=resource_id,
                license_feature=user_license_feature,
                license_name=license_name,
            ):
                self.logger.info(
                    "User -> '%s' is already licensed for -> '%s' (%s). Skipping...",
                    user_name,
                    license_name,
                    user_license_feature,
                )
                continue
            assigned_license = self._otds.assign_user_to_license(
                partition=user_partition,
                user_id=user_name,  # we want the plain login name here
                resource_id=resource_id,
                license_feature=user_license_feature,
                license_name=license_name,
            )

            if not assigned_license:
                self.logger.error(
                    "Failed to assign license feature -> '%s' to user -> %s!",
                    user_license_feature,
                    user_name,
                )
                success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._users,
    )

    return success

process_user_photos(section_name='userPhotos')

Process user photos in payload and assign them to Content Server users.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'userPhotos'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_photos")
def process_user_photos(self, section_name: str = "userPhotos") -> bool:
    """Process user photos in payload and assign them to Content Server users.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._users:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # we assume the nickname of the photo item equals the login name of the user
    # we also assume that the photos have been uploaded / transported into the target system
    for user in self._users:
        user_name = user.get("name")
        if not user_name:
            self.logger.error("User is missing a login. Skipping to next user...")
            success = False
            continue

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not user.get("enabled", True):
            self.logger.info(
                "Payload for user -> '%s' is disabled. Skipping...",
                user_name,
            )
            continue

        # We skip also user of type "ServiceUser":
        if user.get("type", "User") == "ServiceUser":
            self.logger.info("Skipping service user -> '%s'...", user_name)
            continue

        user_id = user.get("id")
        if not user_id:
            self.logger.error(
                "User -> '%s' does not have an ID. The user creation may have failed before. Skipping...",
                user_name,
            )
            success = False
            continue

        response = self._otcs.get_node_from_nickname(nickname=user_name)
        if response is None:
            self.logger.warning(
                "Missing photo for user -> '%s' - nickname not found. Skipping...",
                user_name,
            )
            continue
        photo_id = self._otcs.get_result_value(response=response, key="id")
        response = self._otcs.update_user_photo(user_id=user_id, photo_id=photo_id)
        if not response:
            self.logger.error("Failed to add photo for user -> '%s'!", user_name)
            success = False
        else:
            self.logger.info("Successfully added photo for user -> '%s'.", user_name)

    # Check if Admin has a photo as well (nickname needs to be "admin"):
    response = self._otcs.get_node_from_nickname(nickname="admin")
    if response is None:
        self.logger.warning(
            "Missing photo for admin - nickname not found. Skipping...",
        )
    else:
        photo_id = self._otcs.get_result_value(response=response, key="id")
        response = self._otcs.update_user_photo(user_id=1000, photo_id=photo_id)
        if response is None:
            self.logger.warning("Failed to add photo for user -> 'admin'!")
        else:
            self.logger.info("Successfully added photo for user -> 'admin'.")

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._users,
    )

    return success

process_user_photos_core_share(section_name='userPhotosCoreShare')

Process user photos in payload and assign them to Core Share users.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'userPhotosCoreShare'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_photos_core_share")
def process_user_photos_core_share(
    self,
    section_name: str = "userPhotosCoreShare",
) -> bool:
    """Process user photos in payload and assign them to Core Share users.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not isinstance(self._core_share, CoreShare):
        self.logger.error(
            "Core Share connection not setup properly. Skipping payload section -> '%s'...",
            section_name,
        )
        return False

    if not self._users:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # we assume the nickname of the photo item equals the login name of the user
    # we also assume that the photos have been uploaded / transported into the target system
    for user in self._users:
        if "name" not in user:
            self.logger.error(
                "User is missing login name. Skipping to next user...",
            )
            success = False
            continue
        user_login = user["name"]
        user_last_name = user.get("lastname", "")
        user_first_name = user.get("firstname", "")
        user_name = "{} {}".format(user_first_name, user_last_name).strip()

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not user.get("enabled", True):
            self.logger.info(
                "Payload for user -> '%s' is disabled. Skipping...",
                user_login,
            )
            continue

        # Check if the user is enabled for Core Share.
        # Core Share is disabled per default for the users.
        # There needs to be "enable_core_share" in user payload
        # and it needs to be True:
        if not user.get("enable_core_share", False):
            self.logger.info(
                "User -> '%s' is not enabled for Core Share. Skipping...",
                user_login,
            )
            continue

        core_share_user_id = self.determine_user_id_core_share(user=user)

        if core_share_user_id is None:
            self.logger.error(
                "Failed to get ID of Core Share user -> '%s'!",
                user_name,
            )
            success = False
            continue

        response = self._otcs.get_node_from_nickname(nickname=user_login)
        if response is None:
            self.logger.warning(
                "Missing photo for user -> '%s' - nickname not found. Skipping...",
                user_login,
            )
            continue
        photo_id = self._otcs.get_result_value(response=response, key="id")
        photo_name = self._otcs.get_result_value(response=response, key="name")
        photo_path = os.path.join(tempfile.gettempdir(), photo_name)

        # Check if it is not yet downloaded:
        if not os.path.isfile(photo_path):
            # download the profile picture into the tmp directory:
            result = self._otcs.download_document(
                node_id=photo_id,
                file_path=photo_path,
            )
            # download_document() delivers a boolean result:
            if result is False:
                self.logger.warning(
                    "Failed to download photo for user -> '%s' from Content Server to file -> '%s'!",
                    user_name,
                    photo_path,
                )
                success = False
                continue
            self.logger.info(
                "Successfully downloaded photo for user -> '%s' from Content Server to file -> '%s'.",
                user_name,
                photo_path,
            )
        else:
            self.logger.info(
                "Reusing downloaded photo -> '%s' for Core Share user -> '%s' (%s).",
                photo_path,
                user_name,
                core_share_user_id,
            )

        response = self._core_share.update_user_photo(
            user_id=core_share_user_id,
            photo_path=photo_path,
        )
        if response:
            self.logger.info(
                "Successfully updated profile photo of Core Share user -> '%s' (%s).",
                user_name,
                core_share_user_id,
            )
        else:
            self.logger.error(
                "Failed to update profile photo of Core Share user -> '%s' (%s)! Skipping...",
                user_name,
                core_share_user_id,
            )
            success = False
            continue

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._users,
    )

    return success

process_user_photos_m365(section_name='userPhotosM365')

Process user photos in payload and assign them to Microsoft 365 users.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'userPhotosM365'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_photos_m365")
def process_user_photos_m365(self, section_name: str = "userPhotosM365") -> bool:
    """Process user photos in payload and assign them to Microsoft 365 users.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not isinstance(self._m365, M365):
        self.logger.error(
            "Microsoft 365 connection not setup properly. Skipping payload section -> '%s'...",
            section_name,
        )
        return False

    if not self._users:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # we assume the nickname of the photo item equals the login name of the user
    # we also assume that the photos have been uploaded / transported into the target system
    for user in self._users:
        user_name = user.get("name")
        if not user_name:
            self.logger.error("User is missing a login. Skipping to next user...")
            success = False
            continue

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not user.get("enabled", True):
            self.logger.info(
                "Payload for user -> '%s' is disabled. Skipping...",
                user_name,
            )
            continue

        if "id" not in user:
            self.logger.error(
                "User -> '%s' does not have an ID. The user creation may have failed before. Skipping...",
                user_name,
            )
            success = False
            continue

        if "enable_o365" not in user or not user["enable_o365"]:
            self.logger.info(
                "Microsoft 365 is not enabled in payload for user -> '%s'. Skipping...",
                user_name,
            )
            continue

        # If the customizer pod is restarted it may be that
        # the M365 user exists even if the M365 user ID is
        # not yet written back into the payload. So we use the
        # determine_user_id_m365() method that handles both cases
        # and updates the payload if the user exists in M365 already.
        user_m365_id = self.determine_user_id_m365(user=user)
        if not user_m365_id:
            self.logger.error(
                "M365 user -> '%s' does not exist. Skipping...",
                user_name,
            )
            success = False
            continue

        if self._m365.get_user_photo(user_id=user_m365_id, show_error=False):
            self.logger.info(
                "User -> '%s' (%s) has already a photo in Microsoft 365. Skipping...",
                user_name,
                user_m365_id,
            )
            continue
        self.logger.info(
            "User -> '%s' (%s) has not yet a photo in Microsoft 365. Uploading...",
            user_name,
            user_m365_id,
        )

        response = self._otcs.get_node_from_nickname(nickname=user_name)
        if response is None:
            self.logger.warning(
                "Missing photo for user -> '%s' - nickname not found. Skipping...",
                user_name,
            )
            continue
        photo_id = self._otcs.get_result_value(response=response, key="id")
        photo_name = self._otcs.get_result_value(response=response, key="name")

        photo_path = os.path.join(tempfile.gettempdir(), photo_name)
        result = self._otcs.download_document(
            node_id=photo_id,
            file_path=photo_path,
        )
        # download_document() delivers a boolean result:
        if result is False:
            self.logger.warning(
                "Failed to download photo for user -> '%s' from Content Server!",
                user_name,
            )
            success = False
            continue
        self.logger.info(
            "Successfully downloaded photo for user -> '%s' from Content Server to file -> '%s'.",
            user_name,
            photo_path,
        )

        # Upload photo to M365:
        response = self._m365.update_user_photo(user_m365_id, photo_path)
        if response is None:
            self.logger.error(
                "Failed to upload photo for user -> '%s' to Microsoft 365!",
                user_name,
            )
            success = False
        else:
            self.logger.info(
                "Successfully uploaded photo for user -> '%s' to Microsoft 365.",
                user_name,
            )
    # end for loop

    # Check if Admin has a photo as well (nickname needs to be "admin")
    # Then we want this to be applied in M365 as well:
    response = self._otcs.get_node_from_nickname(nickname="admin")
    if response is None:
        self.logger.warning(
            "Missing photo for admin - nickname not found. Skipping...",
        )
    else:
        photo_id = self._otcs.get_result_value(response=response, key="id")
        photo_name = self._otcs.get_result_value(response=response, key="name")
        photo_path = os.path.join(tempfile.gettempdir(), photo_name)
        result = self._otcs.download_document(
            node_id=photo_id,
            file_path=photo_path,
        )
        # download_document() delivers a boolean result:
        if result is False:
            self.logger.warning(
                "Failed to download photo for user -> 'admin' from Content Server!",
            )
            success = False
        else:
            self.logger.info(
                "Successfully downloaded photo for user -> 'admin' from Content Server to file -> '%s'.",
                photo_path,
            )
            m365_admin_email = "admin@" + self._m365.config()["domain"]
            response = self._m365.update_user_photo(
                user_id=m365_admin_email,
                photo_path=photo_path,
            )
            if response is None:
                self.logger.warning("Failed to add photo for Microsoft 365 user -> '%s'!", m365_admin_email)
            else:
                self.logger.info(
                    "Successfully added photo for Microsoft 365 user -> '%s'.",
                    m365_admin_email,
                )

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._users,
    )

    return success

process_user_photos_salesforce(section_name='userPhotosSalesforce')

Process user photos in payload and assign them to Salesforce users.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'userPhotosSalesforce'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_photos_salesforce")
def process_user_photos_salesforce(
    self,
    section_name: str = "userPhotosSalesforce",
) -> bool:
    """Process user photos in payload and assign them to Salesforce users.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not isinstance(self._salesforce, Salesforce):
        self.logger.error(
            "Salesforce connection not setup properly. Skipping payload section -> '%s'...",
            section_name,
        )
        return False

    if not self._users:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # we assume the nickname of the photo item equals the login name of the user
    # we also assume that the photos have been uploaded / transported into the target system
    for user in self._users:
        user_name = user.get("name")
        if not user_name:
            self.logger.error("User is missing a login. Skipping to next user...")
            success = False
            continue

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not user.get("enabled", True):
            self.logger.info(
                "Payload for user -> '%s' is disabled. Skipping...",
                user_name,
            )
            continue

        # Check if the user is explicitly enabled for Salesforce:
        if not user.get("enable_salesforce", False):
            self.logger.info(
                "User -> '%s' is not enabled for Salesforce. Skipping...",
                user_name,
            )
            continue

        extra_attributes = user.get("extra_attributes", None)
        if not extra_attributes or len(extra_attributes) == 0:
            self.logger.info(
                "User -> '%s' does not have the extra attributes for Salesforce. Skipping...",
                user_name,
            )
            continue
        user_login = extra_attributes[0].get("value", "")
        if not user_login:
            self.logger.info(
                "User -> '%s' does not have the extra attributes value for Salesforce. Skipping...",
                user_name,
            )
            continue

        user_id = self._salesforce.get_user_id(username=user_login)
        if user_id is None:
            self.logger.error(
                "Failed to get Salesforce user ID of user -> '%s'!",
                user_login,
            )
            success = False
            continue

        response = self._otcs.get_node_from_nickname(nickname=user_name)
        if response is None:
            self.logger.warning(
                "Missing photo for user -> '%s' - nickname not found. Skipping...",
                user_name,
            )
            continue
        photo_id = self._otcs.get_result_value(response=response, key="id")
        photo_name = self._otcs.get_result_value(response=response, key="name")
        photo_path = os.path.join(tempfile.gettempdir(), photo_name)

        # Check if it is not yet downloaded:
        if not os.path.isfile(photo_path):
            # download the profile picture into the tmp directory:
            result = self._otcs.download_document(
                node_id=photo_id,
                file_path=photo_path,
            )
            # download_document() delivers a boolean result:
            if result is False:
                self.logger.warning(
                    "Failed to download photo for user -> '%s' from Content Server to file -> '%s'!",
                    user_name,
                    photo_path,
                )
                success = False
                continue
            self.logger.info(
                "Successfully downloaded photo for user -> '%s' from Content Server to file -> '%s'.",
                user_name,
                photo_path,
            )
        else:
            self.logger.info(
                "Reusing downloaded photo -> '%s' for Salesforce user -> '%s' (%s).",
                photo_path,
                user_name,
                user_id,
            )

        response = self._salesforce.update_user_photo(
            user_id=user_id,
            photo_path=photo_path,
        )
        if response:
            self.logger.info(
                "Successfully updated profile photo of Salesforce user -> '%s' (%s).",
                user_login,
                user_id,
            )
        else:
            self.logger.error(
                "Failed to update profile photo of Salesforce user -> '%s' (%s)! Skipping...",
                user_login,
                user_id,
            )
            success = False
            continue

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._users,
    )

    return success

process_user_placeholders()

Replace a user placeholder (sourrounded by %%...%%) with the ID of the Content Server user.

For this we prepare a lookup dict. The dict self._placeholder_values already includes lookups for the OTCS and OTAWP OTDS resource IDs (see customizer.py).

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_placeholders")
def process_user_placeholders(self) -> None:
    """Replace a user placeholder (sourrounded by %%...%%) with the ID of the Content Server user.

    For this we prepare a lookup dict. The dict self._placeholder_values already includes
    lookups for the OTCS and OTAWP OTDS resource IDs (see customizer.py).
    """

    for user in self._users:
        user_name = user.get("name")
        if not user_name:
            self.logger.error(
                "User needs a name for placeholder definition in payload! Skipping...",
            )
            continue

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the payload element:
        if not user.get("enabled", True):
            self.logger.info(
                "Payload for user -> '%s' is disabled. Skipping...",
                user_name,
            )
            continue

        # Now we determine the ID. Either it is in the payload section from
        # the current customizer run or we try to look it up in the system.
        # The latter case may happen if the customizer pod got restarted.
        user_id = self.determine_user_id(user=user)
        if not user_id:
            self.logger.warning(
                "User needs an ID for placeholder definition. Skipping...",
            )
            continue

        # Add user with its ID to the dict self._placeholder_values:
        self._placeholder_values["OTCS_USER_ID_%s", user_name.upper()] = str(
            user_id,
        )

    self.logger.debug(
        "Placeholder values after user processing = %s",
        str(self._placeholder_values),
    )

process_user_security(section_name='userSecurity')

Process Security Clearance and Supplemental Markings for Content Server users.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'userSecurity'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_security")
def process_user_security(self, section_name: str = "userSecurity") -> bool:
    """Process Security Clearance and Supplemental Markings for Content Server users.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._users:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for user in self._users:
        user_name = user.get("name")

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not user.get("enabled", True):
            self.logger.info(
                "Payload for user -> '%s' is disabled. Skipping...",
                user_name,
            )
            continue

        user_id = user.get("id")
        if not user_id:
            self.logger.error(
                "User is missing an ID. It was likely not created. Skipping to next user...",
            )
            success = False
            continue

        # Read security clearance from user payload (it is optional!)
        user_security_clearance = user.get("security_clearance")
        if user_id and user_security_clearance:
            self._otcs.assign_user_security_clearance(
                user_id=user_id,
                security_clearance=user_security_clearance,
            )

        # Read supplemental markings from user payload (it is optional!)
        user_supplemental_markings = user.get("supplemental_markings")
        if user_id and user_supplemental_markings:
            self._otcs.assign_user_supplemental_markings(
                user_id=user_id,
                supplemental_markings=user_supplemental_markings,
            )

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._users,
    )

    return success

process_user_settings(section_name='userSettings')

Process user settings in payload and apply them in OTDS.

This includes password settings and user display settings.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'userSettings'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_user_settings")
def process_user_settings(self, section_name: str = "userSettings") -> bool:
    """Process user settings in payload and apply them in OTDS.

    This includes password settings and user display settings.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._users:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for user in self._users:
        user_name = user.get("name")
        if not user_name:
            self.logger.error(
                "Missing user name - cannot apply user settings. Skipping user...",
            )
            continue

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not user.get("enabled", True):
            self.logger.info(
                "Payload for user -> '%s' is disabled. Skipping...",
                user_name,
            )
            continue

        self._log_header_callback(
            text="Process settings for user -> '{}'".format(user_name),
            char="-",
        )

        user_partition = self._otcs.config().get("partition", None)
        if not user_partition:
            self.logger.error("User partition not found!")
            success = False
            continue

        # Set the OTDS display name. Content Server does not use this but
        # it makes AppWorks display users correctly (and it doesn't hurt)
        # We only set this if firstname _and_ last name are in the payload:
        if "firstname" in user and "lastname" in user:
            user_display_name = user["firstname"] + " " + user["lastname"]
            response = self._otds.update_user(
                partition=user_partition,
                user_id=user_name,
                attribute_name="displayName",
                attribute_value=user_display_name,
            )
            if response:
                self.logger.info(
                    "Successfully updated display name of user -> '%s' to -> '%s'.",
                    user_name,
                    user_display_name,
                )
            else:
                self.logger.error(
                    "Failed to update display name of user -> '%s' to -> '%s'!",
                    user_name,
                    user_display_name,
                )
                success = False

        # Don't enforce the user to reset password at first login (settings in OTDS):
        self.logger.info(
            "Don't enforce password change for user -> '%s'...",
            user_name,
        )
        response = self._otds.update_user(
            partition=user_partition,
            user_id=user_name,
            attribute_name="UserMustChangePasswordAtNextSignIn",
            attribute_value="False",
        )
        if not response:
            success = False

        response = self._otds.update_user(
            partition=user_partition,
            user_id=user_name,
            attribute_name="UserCannotChangePassword",
            attribute_value="True",
        )
        if not response:
            success = False

        # Set user password to never expire
        response = self._otds.update_user(
            partition=user_partition,
            user_id=user_name,
            attribute_name="PasswordNeverExpires",
            attribute_value="True",
        )
        if not response:
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._users,
    )

    return success

process_users(section_name='users')

Process users in payload and create them in Content Server.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'users'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Side Effects

The user items are modified by adding an "id" dict element that includes the technical ID of the user in Content Server

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_users")
def process_users(self, section_name: str = "users") -> bool:
    """Process users in payload and create them in Content Server.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    Side Effects:
        The user items are modified by adding an "id" dict element that
        includes the technical ID of the user in Content Server

    """

    if not self._users:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # Even if this payload section has been processed successfully before we
    # to process it once more. So we are NOT checking the status file.

    success: bool = True

    # Add all users in payload and establish membership in
    # specified groups:
    for user in self._users:
        user_name = user.get("name")
        if not user_name:
            self.logger.error("User is missing a login. Skipping to next user...")
            success = False
            continue

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not user.get("enabled", True):
            self.logger.info(
                "Payload for user -> '%s' is disabled. Skipping...",
                user_name,
            )
            continue

        # Check if the user does already exist (e.g. if job is restarted)
        # determine_user_id() also writes back the user ID into the payload
        # if it has gathered it from OTCS.
        user_id = self.determine_user_id(user=user)
        if user_id:
            self.logger.info(
                "Found existing user -> '%s' (%s). Skipping to next user...",
                user_name,
                user_id,
            )
            continue
        self.logger.info(
            "Cannot find an existing user -> '%s' - creating a new user...",
            user_name,
        )

        # Sanity checks:
        if "password" not in user or user["password"] is None or user["password"] == "":
            self.logger.info(
                "User -> '%s' no password defined in payload, generating random password...",
                user_name,
            )
            user["password"] = self.generate_password(
                length=10,
                use_special_chars=True,
            )

            description_attribue = {
                "name": "description",
                "value": "initial password: " + user["password"],
            }

            try:
                user["extra_attributes"].append(description_attribue)
            except KeyError:
                user["extra_attributes"] = [description_attribue]

        # Sanity checks:
        if "base_group" not in user or not user["base_group"]:
            self.logger.warning(
                "User -> '%s' is missing a base group - setting to default group.",
                user_name,
            )
            user["base_group"] = "DefaultGroup"

        # Find the base group of the user. Assume 'Default Group' (= 1001) if not found:
        base_group = next(
            (item["id"] for item in self._groups if item["name"] == user["base_group"] and item.get("id")),
            1001,
        )

        # Get type to be able to create ServiceUsers, default to User -> 0
        user_type = 17 if user.get("type", "User") == "ServiceUser" else 0

        # Now we know it is a new user...
        new_user = self._otcs.add_user(
            name=user_name,
            password=user["password"],
            first_name=user.get("firstname", ""),  # be careful - can be empty
            last_name=user.get("lastname", ""),  # be careful - can be empty
            email=user.get("email", ""),  # be careful - can be empty
            title=user.get("title", ""),  # be careful - can be empty
            base_group=base_group,
            phone=user.get("phone", ""),
            privileges=user.get("privileges", ["Login", "Public Access"]),
            user_type=user_type,
        )

        new_user_id = self._otcs.get_result_value(response=new_user, key="id")
        if not new_user_id:
            self.logger.error(
                "Failed to create new user -> '%s'!",
                user_name,
            )
            success = False
            continue

        self.logger.info(
            "Successfully created user -> '%s' with ID -> %s.",
            user_name,
            new_user_id,
        )
        # Write back user ID into payload
        user["id"] = new_user_id

        # Assign usage privileges to the new user:
        usage_privileges = user.get("usage_privileges", [])
        for usage_privilege in usage_privileges:
            response = self._otcs.assign_usage_privilege(
                usage_privilege=usage_privilege,
                member_id=new_user_id,
            )
            if response:
                self.logger.info(
                    "Assigned usage privilege -> '%s' to user -> '%s' (%s).",
                    usage_privilege,
                    user_name,
                    new_user_id,
                )
            else:
                self.logger.info(
                    "Failed to assign usage privilege -> '%s' to user -> '%s' (%s)!",
                    usage_privilege,
                    user_name,
                    new_user_id,
                )

        # Assign usage privileges to the new user:
        object_privileges = user.get("object_privileges", [])
        for object_type in object_privileges:
            response = self._otcs.assign_object_privilege(
                object_type=object_type,
                member_id=new_user_id,
            )
            if response:
                self.logger.info(
                    "Assigned object privilege -> '%s' to user -> '%s' (%s).",
                    object_type,
                    user_name,
                    new_user_id,
                )
            else:
                self.logger.info(
                    "Failed to assign object privilege -> '%s' to user -> '%s' (%s)!",
                    object_type,
                    user_name,
                    new_user_id,
                )

        # Process group memberships of new user:
        user_groups = user.get("groups", [])  # list of groups the user is in
        for user_group in user_groups:
            # Try to find the group dictionary item in the payload
            # for user group name:
            group = next(
                (item for item in self._groups if item["name"] == user_group),
                None,
            )
            if group:
                group_id = group.get("id")  # Careful ID may not exist
                group_name = group["name"]
            else:
                # if group is not in payload try to find group in OTCS
                # in case it is a pre-existing group:
                group = self._otcs.get_group(name=user_group)
                group_id = self._otcs.get_result_value(response=group, key="id")
                if group_id is None:
                    self.logger.error(
                        "Group -> '%s' not found. Skipping...",
                        user_group,
                    )
                    success = False
                    continue
                group_name = self._otcs.get_result_value(
                    response=group,
                    key="name",
                )

            if group_id is None:
                self.logger.error(
                    "Group -> '%s' does not have an ID. Cannot add user -> '%s' to this group. Skipping...",
                    group_name,
                    user["name"],
                )
                success = False
                continue

            self.logger.info(
                "Add user -> '%s' (%s) to group -> '%s' (%s)...",
                user["name"],
                user["id"],
                group_name,
                group_id,
            )
            response = self._otcs.add_group_member(
                member_id=user["id"],
                group_id=group_id,
            )
            if not response:
                success = False
        # end for user_group in user_groups:

        # For some unclear reason the user is not added to its base group in OTDS
        # so we do this explicitly:
        self.logger.info(
            "Add user -> '%s' to its base group -> '%s'...",
            user["name"],
            user["base_group"],
        )
        response = self._otds.add_user_to_group(
            user["name"],
            user["base_group"],
        )
        if not response:
            success = False

        # Extra OTDS attributes for the user can be provided in "extra_attributes"
        # as part of the user payload.
        if "extra_attributes" in user:
            extra_attributes = user["extra_attributes"]
            for extra_attribute in extra_attributes:
                attribute_name = extra_attribute.get("name")
                attribute_value = extra_attribute.get("value")
                if not attribute_name or not attribute_value:
                    self.logger.error(
                        "User attribute is missing a name or value. Skipping...",
                    )
                    success = False
                    continue
                if "password" in attribute_value:
                    self.logger.info(
                        "Set user attribute -> '%s' to -> '*******' (sensitive password)",
                        attribute_name,
                    )
                else:
                    self.logger.info(
                        "Set user attribute -> '%s' to -> '%s'.",
                        attribute_name,
                        attribute_value,
                    )
                user_partition = self._otcs.config()["partition"]
                if not user_partition:
                    self.logger.error("User partition not found!")
                    success = False
                    continue
                self._otds.update_user(
                    user_partition,
                    user["name"],
                    attribute_name,
                    attribute_value,
                )
        # end if "extra_attributes" in user

        # Assign application roles to the new user:
        application_roles = user.get("application_roles", [])

        for role in application_roles:
            user_partition = self._otcs.config()["partition"]
            if not user_partition:
                self.logger.error("User partition not found!")
                success = False
                continue

            # Split role at the @ sign to identify Partitions
            role_parts = role.split("@", 1)
            role_name = role_parts[0]
            role_partition = role_parts[1] if len(role_parts) > 1 else "OAuthClients"

            response = self._otds.assign_user_to_application_role(
                user_id=user["name"],
                user_partition=user_partition,
                role_name=role_name,
                role_partition=role_partition,
            )

            if response:
                self.logger.info(
                    "Successfully assigned application role -> '%s' (%s) to user -> '%s' (%s).",
                    role_name,
                    role_partition,
                    user_name,
                    user_partition,
                )
            else:
                self.logger.info(
                    "Failed to assign application role -> '%s' (%s) to user -> '%s' (%s) in OTDS!",
                    role_name,
                    role_partition,
                    user_name,
                    user_partition,
                )

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._users,
    )

    return success

process_users_core_share(section_name='usersCoreShare')

Process users in payload and sync them with Core Share (passwords and email).

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'usersCoreShare'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Side Effects

The user items are modified by adding "core_share_user_id" dict element that includes the ID of the user in Core Share. This will be written into the status file.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_users_core_share")
def process_users_core_share(
    self,
    section_name: str = "usersCoreShare",
) -> bool:
    """Process users in payload and sync them with Core Share (passwords and email).

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    Side Effects:
        The user items are modified by adding "core_share_user_id"
        dict element that includes the ID of the user in Core Share. This will be written
        into the status file.

    """

    if not self._users:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    if not self._core_share:
        self.logger.error(
            "Core Share connection is not initialized. Bailing out...",
        )
        return False

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # traverse all users in payload:
    for user in self._users:
        user_last_name = user.get("lastname", "")  # Default is important here
        user_first_name = user.get("firstname", "")
        user_name = " ".join(filter(None, [user_first_name, user_last_name]))
        user_enabled = user.get("enabled", True) and user.get("enable_core_share", False)
        if not user_name and user_enabled:  # Avoid a warning if not enbaled
            self.logger.error(
                "User is missing last name and first name. Skipping to next user...",
            )
            success = False
            continue

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not user.get("enabled", True):
            self.logger.info(
                "Payload for user -> '%s' is disabled. Skipping...",
                user_name,
            )
            continue

        # Check if the user is enabled for Core Share.
        # Core Share is disabled per default for the users.
        # There needs to be "enable_core_share" in user payload
        # and it needs to be True:
        if not user.get("enable_core_share", False):
            self.logger.info(
                "User -> '%s' is not enabled for Core Share. Skipping...",
                user_name,
            )
            continue

        user_email = user.get("email", "")
        user_password = user.get("password", "")

        # Initialize variables:
        need_email_verification = False
        subject = None
        url_search_pattern = None

        #
        # 1. Check if user does already exist in Core Share:
        #

        core_share_user_id = self.determine_user_id_core_share(user=user)

        #
        # 2: Create or Update user in Core Share:
        #

        # Check if we need to create a new Core Share user:
        if core_share_user_id is None:
            self.logger.info(
                "Core Share user -> '%s' does not exist. Creating a new Core Share user...",
                user_name,
            )
            response = self._core_share.add_user(
                first_name=user_first_name,
                last_name=user_last_name,
                email=user_email,
                title=user.get("title", None),
                company=user.get("company", "Innovate"),
                password=user.get("password", None),
            )
            core_share_user_id = self._core_share.get_result_value(
                response=response,
                key="id",
            )
            if not core_share_user_id:
                self.logger.error(
                    "Failed to create Core Share user -> '%s'! Skipping...",
                    user_name,
                )
                success = False
                continue
            self.logger.info(
                "Successfully created Core Share user -> '%s' (%s).",
                user_name,
                core_share_user_id,
            )

            # Check if the user is still in pending state:
            is_confirmed = self._core_share.get_result_value(
                response=response,
                key="isConfirmed",
            )
            # we need to differentiate False an None here - don't simplify to "if not is_confirmed"!
            if is_confirmed is False:
                self.logger.info(
                    "New Core Share user -> '%s' is not yet confirmed and in a 'pending' state!",
                    user_name,
                )
            elif is_confirmed is True:
                self.logger.info(
                    "New Core Share user -> '%s' is already confirmed!",
                    user_name,
                )

            # We write the user password in addition to the "Other" Address field
            # to determine in a subsequent deployment the "old" password:
            update_data = {
                "addresses": [
                    {"type": "other", "value": user.get("password", None)},
                ],
            }
            response = self._core_share.update_user(
                user_id=core_share_user_id,
                update_data=update_data,
            )

            # We need email verification for new users:
            need_email_verification = True
            url_search_pattern = "verify-email"
            subject = "Welcome to OpenText Core Share"

            # For new users the old password is equal to the new password:
            old_password = user.get("password", None)
        # The user does already exist in Core Share:
        else:
            update_data = {
                "firstName": user.get("firstname", ""),
                "lastName": user.get("lastname", ""),
                "title": user.get("title", ""),
                "company": user.get("company", ""),
            }
            self.logger.info(
                "Core Share user -> '%s' does already exist. Updating Core Share user with -> %s...",
                user_name,
                str(update_data),
            )

            # Fetch the existing user:
            core_share_user = self._core_share.get_user_by_id(
                user_id=core_share_user_id,
            )

            # Check if the user is still in pending state:
            is_confirmed = self._core_share.get_result_value(
                response=core_share_user,
                key="isConfirmed",
            )
            # we need to differentiate False an None here - don't simplify to "if not is_confirmed"!
            if is_confirmed is False:
                self.logger.warning(
                    "Core Share user -> '%s' has not yet confirmed the email invitation and is in 'pending' state! Resend invite...",
                    user_name,
                )
                # We try the email verification once more...
                self._core_share.resend_user_invite(core_share_user_id)
                need_email_verification = True
                url_search_pattern = "confirm-account"
                subject = "Invitation to OpenText™ Core Share"

            # Check if we have the old password of the user in the "Other" address field:
            core_share_user_addresses = self._core_share.get_result_value(
                core_share_user,
                "addresses",
            )
            if core_share_user_addresses and len(core_share_user_addresses) > 0:
                old_password = core_share_user_addresses[0]["value"]
                self.logger.info(
                    "Found old password for Core Share user -> '%s' (%s).",
                    user_name,
                    core_share_user_id,
                )
            else:
                self.logger.info(
                    "No old password found for Core Share user -> '%s'. Cannot set a new password.",
                    user_name,
                )
                old_password = ""

            # We store the current password into the address field (this adds to the update dictionary
            # defined above and used below): THIS IS CURRENTLY NOT WORKING!
            update_data["addresses"] = [{"type": "other", "value": user_password}]

            # Check if the mail address has really changed. Otherwise we
            # don't need to set it again and can avoid email verification:
            core_share_user_email = self._core_share.get_result_value(
                core_share_user,
                "email",
            )
            if user_email != core_share_user_email:
                self.logger.info(
                    "Email for Core Share user -> '%s' has changed from -> '%s' to -> '%s'. We need to verify this via email.",
                    user_name,
                    core_share_user_email,
                    user_email,
                )
                # Additional email payload for user update:
                update_data["email"] = user_email
                # If email is changed this needs to be confirmed by passing
                # the current (old) password:
                update_data["password"] = old_password if old_password else user_password
                # As email has changed - we need the email verification below...
                need_email_verification = True
                url_search_pattern = "verify-email"
                subject = "OpenText Core Share: Email Updated"

            # Update the existing Core Share user with new / changed data:
            response = self._core_share.update_user(
                user_id=core_share_user_id,
                update_data=update_data,
            )
            if not response:
                self.logger.error(
                    "Failed to update Core Share user -> '%s'! Skipping...",
                    user_name,
                )
                success = False
                continue
            self.logger.info(
                "Successfully updated Core Share user -> '%s'.",
                user_name,
            )

            # Now update the password:
            if user_password and old_password and user_password != old_password:
                response = self._core_share.update_user_password(
                    user_id=core_share_user_id,
                    password=old_password,
                    new_password=user_password,
                )
                if response:
                    self.logger.info(
                        "Successfully updated password of Core Share user -> '%s' (%s).",
                        user_name,
                        core_share_user_id,
                    )
                else:
                    self.logger.error(
                        "Failed to update Core Share password for user -> '%s' (%s)! Skipping...",
                        user_name,
                        core_share_user_id,
                    )
                    success = False
                    continue
            elif not old_password:
                self.logger.warning(
                    "Cannot change Core Share user password for -> '%s' (%s). Need both, old and new passwords.",
                    user_name,
                    core_share_user_id,
                )
            else:
                self.logger.info(
                    "Core Share user password for -> '%s' (%s) is unchanged.",
                    user_name,
                    core_share_user_id,
                )

            # For existing users we want to cleanup possible left-overs form old deployments
            self.logger.info(
                "Cleanup existing file shares of Core Share user -> '%s' (%s)...",
                user_name,
                core_share_user_id,
            )
            response = self._core_share.cleanup_user_files(
                user_id=core_share_user_id,
                user_login=core_share_user_email,
                user_password=user_password,
            )
            if not response:
                self.logger.error(
                    "Failed to cleanup user files for user -> '%s' (%s)!", user_name, core_share_user_id
                )

        # Save result for status file content
        user["core_share_user_id"] = core_share_user_id

        #
        # 3: Handle Email verification:
        #

        # We now need to wait for the verification mail from Core Share,
        # get it from the M365 Outlook inbox of the user (or the admin
        # if the user does not have its own inbox) and click the
        # verification link...

        if need_email_verification and user.get("enable_o365", False):
            self.logger.info(
                "Processing Email verification for user -> '%s' (%s). Wait 20 seconds to make sure verification mail in user's inbox...",
                user_name,
                user_email,
            )
            time.sleep(20)

            # Process verification mail sent by Core Share.
            # TODO: This has some hard-coded value. We may want to optimize it further in the future:
            result = self._m365.email_verification(
                user_email=user_email,
                sender="noreply@opentext.cloud",
                subject=subject,
                url_search_pattern=url_search_pattern,
                line_end_marker="=",
                multi_line=True,
                multi_line_end_marker="%3D",
                replacements=None,
                max_retries=6,
                use_browser_automation=True,
                password=user_password,
                password_field_id="passwordInput",
                password_confirmation_field_id="confirmResetPassword",
                password_submit_xpath="//button[@type='submit']",
                terms_of_service_xpath="//div[@id = 'termsOfService']//button[@type='submit']",
            )
            if not result:
                # Email verification was not successful
                self.logger.warning(
                    "Core Share email verification failed. No verification mail received in user's inbox.",
                )
                # don't treat as error nor do "continue" here - we still want to process the user groups...
            else:
                self.logger.info(
                    "Successfully verified new email address -> '%s'.",
                    user_email,
                )
        # end if need_email_verification

        #
        # 4: Add users into groups in Core Share:
        #

        self.logger.info(
            "Processing group memberships of Core Share user -> '%s' (%s)...",
            user_name,
            user_email,
        )
        user_groups = user.get("groups", [])
        base_group = user.get("base_group", None)
        if base_group and base_group not in user_groups:
            user_groups.append(base_group)  # list of groups the user is in

        for user_group in user_groups:
            # "Business Administrators" is a OTCS generated group that we won't find
            # in payload - skip this group.
            if user_group == "Business Administrators":
                # Users that are Business Administrators in Content Server
                # become Content Manager (role = 5) in Core Share:
                self.logger.info(
                    "User -> '%s' is a business administrator in Content Server and becomes a 'Content Manager' (access role 5) in Core Share",
                    user_name,
                )
                self._core_share.add_user_access_role(
                    user_id=core_share_user_id,
                    role_id=5,
                )
                continue
            # Try to find the group dictionary item in the payload
            # for user group name:
            group = next(
                (item for item in self._groups if item["name"] == user_group),
                None,
            )
            if not group:
                self.logger.error(
                    "Cannot find group -> '%s'. Cannot establish membership in Core Share. Skipping to next group...",
                    user_group,
                )
                success = False
                continue

            group_name = group["name"]
            group_id = self.determine_group_id_core_share(
                group=group,
            )  # Careful ID may not exist
            if group_id is None:
                self.logger.info(
                    "Group -> '%s' does not have a Core Share ID. Cannot add user -> '%s' to this Core Share group (group may not be enabled for Core Share). Skipping...",
                    group_name,
                    user_name,
                )
                # We don't treat this as an error - there may be payload groups which are not enabled for Core Share!
                continue

            existing_members = self._core_share.get_group_members(group_id)

            # Only add user as new member if not yet a member or a 'pending' member:
            is_member = self._core_share.exist_result_item(
                response=existing_members,
                key="id",
                value=core_share_user_id,
                results_marker="groupMembers",
            )
            is_pending_member = self._core_share.exist_result_item(
                response=existing_members,
                key="email",
                value=user_email,
                results_marker="pending",
            )

            if not is_member and not is_pending_member:
                self.logger.info(
                    "Add Core Share user -> '%s' (%s) to Core Share group -> '%s' (%s)...",
                    user_name,
                    core_share_user_id,
                    group_name,
                    group_id,
                )
                # We make users that have this group as base_group
                # to Admins of the Core Share group:
                is_group_admin = user_group == base_group
                response = self._core_share.add_group_member(
                    group_id=group_id,
                    user_id=core_share_user_id,
                    is_group_admin=is_group_admin,
                )
                # the add_group_member() has a special return value
                # which is a list (not a dict). It has mostly 1 element
                # which is a dict with a "success" item. This (and not response.ok)
                # determines if the call was successful!
                success: bool = self._core_share.get_result_value(
                    response,
                    "success",
                )
                if not success:
                    errors = self._core_share.get_result_value(
                        response=response,
                        key="errors",
                    )
                    self.logger.error(
                        "Failed to add Core Share user -> '%s' (%s) as member to Core Share group -> '%s' (%s)! Error -> %s",
                        user_name,
                        core_share_user_id,
                        group_name,
                        group_id,
                        str(errors),
                    )
                    success = False
                    continue
            else:
                self.logger.info(
                    "Core Share user -> '%s' (%s) is already a %s of Core Share group -> '%s' (%s). Skipping...",
                    user_name,
                    core_share_user_id,
                    "member" if is_member else "pending member",
                    group_name,
                    group_id,
                )
        # end for loop user groups
    # end for loop users

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._users,
    )

    return success

process_users_m365(section_name='usersM365')

Process users in payload and create them in Microsoft 365 via MS Graph API.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'usersM365'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_users_m365")
def process_users_m365(self, section_name: str = "usersM365") -> bool:
    """Process users in payload and create them in Microsoft 365 via MS Graph API.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not isinstance(self._m365, M365):
        self.logger.error(
            "Microsoft 365 connection not setup properly. Skipping payload section -> '%s'...",
            section_name,
        )
        return False

    if not self._users:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # Add all users in payload and establish membership in
    # specified groups:
    for user in self._users:
        user_name = user.get("name")
        if not user_name:
            self.logger.error("User is missing a login. Skipping to next user...")
            success = False
            continue

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not user.get("enabled", True):
            self.logger.info(
                "Payload for user -> '%s' is disabled. Skipping...",
                user_name,
            )
            continue

        # Check if the user is enabled for Microsoft 365.
        # M365 is disabled per default for the users.
        # There needs to be "enable_o365" in user payload
        # and it needs to be True:
        if not user.get("enable_o365", False):
            self.logger.info(
                "Microsoft 365 is not enabled in payload for user -> '%s'. Skipping...",
                user_name,
            )
            continue

        # Sanity checks:
        if "password" not in user:
            self.logger.error(
                "User -> '%s' is missing a password. Skipping to next user...",
                user_name,
            )
            success = False
            continue
        user_password = user["password"]
        # be careful with the following fields - they could be empty
        user_department = user.get("base_group", "")
        user_first_name = user.get("firstname", "")
        user_last_name = user.get("lastname", "")
        user_location = user.get("location", "US")
        user_email = user.get("email", user_name)

        # Check if the user does already exist in M365 (e.g. if job is restarted)
        m365_user_id = self.determine_user_id_m365(user=user)
        if not m365_user_id:
            self.logger.info(
                "Did not find existing Microsoft 365 user - creating user -> '%s'...",
                user_email,
            )

            # Now we know it is a new user...
            # We are not 1:1 using the email address from the
            # payload as this could by an alias address using the "+" syntax:
            m365_user_name = user_name + "@" + self._m365.config()["domain"]

            new_user = self._m365.add_user(
                email=m365_user_name,
                password=user_password,
                first_name=user_first_name,
                last_name=user_last_name,
                location=user_location,
                department=user_department,
            )
            if new_user is not None:
                # Store the Microsoft 365 user ID in payload:
                user["m365_id"] = new_user["id"]
                m365_user_id = new_user["id"]
                self.logger.info(
                    "New Microsoft 365 user -> '%s' with ID -> %s has been created.",
                    user_name,
                    m365_user_id,
                )
            else:
                self.logger.error(
                    "Failed to create new Microsoft 365 user -> '%s'! Skipping...",
                    user_name,
                )
                success = False
                continue
        else:
            # if the user exists we just set the password according
            # the the payload definition to allow to bulk
            # update existing M365 users with new passwords:
            self.logger.info(
                "Found existing Microsoft 365 user -> '%s' - updating password...",
                user_name,
            )
            new_password_settings = {
                "passwordProfile": {
                    "forceChangePasswordNextSignIn": False,
                    "password": user_password,
                },
            }
            response = self._m365.update_user(
                user_id=m365_user_id,
                updated_settings=new_password_settings,
            )
            if not response:
                self.logger.error(
                    "Failed to update password of M365 user -> '%s' (%s)!",
                    user_name,
                    m365_user_id,
                )

        # Now we assign a license to the new M365 user.
        # First we see if there's a M365 SKU list in user
        # payload - if not we wrap the default SKU configured
        # for the m365 object into a single item list:
        existing_user_licenses = self._m365.get_user_licenses(user_id=m365_user_id)
        sku_list = user.get("m365_skus", [self._m365.config()["skuId"]])
        for sku_id in sku_list:
            # Check if the M365 user already has this license:
            if not self._m365.exist_result_item(
                existing_user_licenses,
                "skuId",
                sku_id,
            ):
                response = self._m365.assign_license_to_user(m365_user_id, sku_id)
                if not response:
                    self.logger.error(
                        "Failed to assign license -> '%s' to Microsoft 365 user -> '%s'!",
                        sku_id,
                        user_name,
                    )
                    success = False
                else:
                    if (
                        "m365_skus" not in user
                    ):  # this is only True if the default license from the m365 object is taken
                        user["m365_skus"] = [sku_id]
                    self.logger.info(
                        "License -> '%s' has been assigned to Microsoft 365 user -> '%s'.",
                        sku_id,
                        user_name,
                    )
            else:
                self.logger.info(
                    "Microsoft 365 user -> '%s' already has the license -> '%s'.",
                    user_name,
                    sku_id,
                )

        # Now we assign the Content Server Teams App to the new M365 user.
        # First we check if the app is already assigned to the user.
        # If not we install / assign the app. If the user already has
        # the Content Server app we try to upgrade it:
        app_name = self._m365.config()["teamsAppName"]
        app_external_id = self._m365.config()["teamsAppExternalId"]
        app_internal_id = self._m365.config().get("teamsAppInternalId", None)
        response = self._m365.get_teams_apps_of_user(
            user_id=m365_user_id,
            filter_expression="contains(teamsAppDefinition/displayName, '{}')".format(
                app_name,
            ),
        )
        if self._m365.exist_result_item(
            response=response,
            key="displayName",
            value=app_name,
            sub_dict_name="teamsAppDefinition",
        ):
            self.logger.info(
                "M365 Teams app -> '%s' is already assigned to M365 user -> '%s' (%s). Trying to upgrade app...",
                app_name,
                user_name,
                m365_user_id,
            )
            response = self._m365.upgrade_teams_app_of_user(
                user_id=m365_user_id,
                app_name=app_name,
            )
        else:
            self.logger.info(
                "Assign M365 Teams app -> '%s' (%s) to M365 user -> '%s' (%s)...",
                app_name,
                app_external_id,
                user_name,
                m365_user_id,
            )
            # This can produce errors because the app may be assigned organization-wide.
            # So we don't treat it as an error and just show a warning.
            # We also try to use the internal app id instead of the name:
            if app_internal_id:
                response = self._m365.assign_teams_app_to_user(
                    user_id=m365_user_id,
                    app_name=app_name,
                    app_internal_id=app_internal_id,
                    show_error=False,
                )
            else:
                response = self._m365.assign_teams_app_to_user(
                    user_id=m365_user_id,
                    app_name=app_name,
                    show_error=False,
                )

        # Process Microsoft 365 group memberships of new user:
        # don't forget the base group (department) if it is not yet in groups!
        group_names = user.get("groups", [])
        if user_department and user_department not in group_names:
            group_names.append(user_department)
        self.logger.info(
            "User -> '%s' has these groups in payload -> %s (including base group -> %s). Checking if they are Microsoft 365 groups...",
            user_name,
            group_names,
            user_department,
        )

        # Go through all group names:
        for group_name in group_names:
            # Find the group payload item to the parent group name:
            group = next(
                (item for item in self._groups if item["name"] == group_name),
                None,
            )
            if not group:
                # if group is not in payload then this membership
                # is not relevant for Microsoft 365. This could be system generated
                # groups like "PageEdit" or "Business Administrators".
                # In this case we do "continue" as we can't process parent groups
                # either:
                self.logger.debug(
                    "No payload found for group -> '%s'. Skipping...",
                    group_name,
                )
                continue
            if "enable_o365" not in group or not group["enable_o365"]:
                # If Microsoft 365 is not enabled for this group in
                # the payload we don't create a M365 but we do NOT continue
                # as there may still be parent groups that are M365 enabled
                # we want to put the user in (see below):
                self.logger.info(
                    "Payload group -> '%s' is not enabled for M365.",
                    group_name,
                )
            else:
                response = self._m365.get_group(group_name=group_name)
                if response is None or "value" not in response or not response["value"]:
                    self.logger.error(
                        "Microsoft 365 group -> '%s' not found. Skipping...",
                        group_name,
                    )
                    success = False
                else:
                    group_id = response["value"][0]["id"]

                    # Check if user is already a member. We don't want
                    # to throw an error if the user is not found as a member
                    # so we pass show_error=False:
                    if self._m365.is_member(
                        group_id,
                        m365_user_id,
                        show_error=False,
                    ):
                        self.logger.info(
                            "Microsoft 365 user -> '%s' (%s) is already in Microsoft 365 group -> '%s' (%s).",
                            user["name"],
                            m365_user_id,
                            group_name,
                            group_id,
                        )
                    else:
                        self.logger.info(
                            "Add Microsoft 365 user -> '%s' (%s) to Microsoft 365 group -> '%s' (%s)...",
                            user["name"],
                            m365_user_id,
                            group_name,
                            group_id,
                        )
                        response = self._m365.add_group_member(
                            group_id=group_id,
                            member_id=m365_user_id,
                        )
                        if not response:
                            self.logger.error(
                                "Failed to add Microsoft 365 user -> '%s' (%s) to Microsoft 365 group -> '%s' (%s)!",
                                user["name"],
                                m365_user_id,
                                group_name,
                                group_id,
                            )
                            success = False

                        # As each group should have at least one owner in M365
                        # we set all users also as owners for now. Later we
                        # may want to configure this via payload:
                        self.logger.info(
                            "Make Microsoft 365 user -> '%s' (%s) owner of Microsoft 365 group -> '%s' (%s)...",
                            user["name"],
                            m365_user_id,
                            group_name,
                            group_id,
                        )
                        response = self._m365.add_group_owner(
                            group_id,
                            m365_user_id,
                        )
                        if not response:
                            self.logger.error(
                                "Failed to make Microsoft 365 user -> '%s' (%s) owner of Microsoft 365 group -> '%s' (%s)!",
                                user["name"],
                                m365_user_id,
                                group_name,
                                group_id,
                            )
                            success = False

            # As M365 groups are flat (not nested), we also add the
            # user as member to the parent groups of the current group
            # if the parent group is enabled for M365:
            parent_group_names = group.get("parent_groups")
            self.logger.info(
                "Group -> '%s' has the following parent groups -> %s",
                group_name,
                parent_group_names,
            )
            for parent_group_name in parent_group_names:
                # Find the group dictionary item to the parent group name:
                parent_group = next(
                    (item for item in self._groups if item["name"] == parent_group_name),
                    None,
                )
                if parent_group is None or "enable_o365" not in parent_group or not parent_group["enable_o365"]:
                    # if parent group is not in payload then this membership
                    # is not relevant for Microsoft 365.
                    # If Microsoft 365 is not enabled for this parent group in
                    # the payload we can also skip:
                    self.logger.info(
                        "Parent group -> '%s' is not enabled for M365. Skipping...",
                        group_name,
                    )
                    continue

                response = self._m365.get_group(group_name=parent_group_name)
                if response is None or "value" not in response or not response["value"]:
                    self.logger.error(
                        "Microsoft 365 group -> '%s' not found. Skipping...",
                        group_name,
                    )
                    success = False
                    continue
                parent_group_id = response["value"][0]["id"]

                # Check if user is already a member. We don't want
                # to throw an error if the user is not found as a member:
                if self._m365.is_member(
                    group_id=parent_group_id,
                    member_id=m365_user_id,
                    show_error=False,
                ):
                    self.logger.info(
                        "Microsoft 365 user -> '%s' (%s) is already in Microsoft 365 group -> '%s' (%s).",
                        user["name"],
                        m365_user_id,
                        parent_group_name,
                        parent_group_id,
                    )
                else:
                    self.logger.info(
                        "Add Microsoft 365 user -> '%s' (%s) to Microsoft 365 group -> '%s' (%s)...",
                        user["name"],
                        m365_user_id,
                        parent_group_name,
                        parent_group_id,
                    )
                    self._m365.add_group_member(
                        group_id=parent_group_id,
                        member_id=m365_user_id,
                    )
            # end for parent_group_name

            # Make this user follow the SharePoint site of his/her department.
            # We only do this for users that have a valid M365 license (SKU):
            if group_name == user_department and user["m365_skus"]:
                group = self._m365.get_group(group_name=group_name)
                group_id = self._m365.get_result_value(response=group, key="id")
                if group_id:
                    group_site = self._m365.get_sharepoint_site_for_group(group_id=group_id)
                    group_site_id = self._m365.get_result_value(response=group_site, key="id")
                    if group_site_id:
                        group_site_name = self._m365.get_result_value(response=group_site, key="name")
                        # Make sure the user's mysite (drive) has been provisioned.
                        # This is a prerequisite for a user being able to follow a
                        # SharePoint site. For this to work we need "Files.ReadWrite"
                        # as a delegated permission in the Azure AppRegistration!
                        self.logger.info(
                            "Make sure user -> '%s' has a drive (mySite) provisioned...",
                            user["email"],
                        )
                        # We need to have 'delegated' permissions for this so we authenticate
                        # as the user... The scope is important here - the user's drive can only
                        # be provisioned if "Files.ReadWrite" scope is provided:
                        response = self._m365.authenticate_user(
                            username=user["email"], password=user["password"], scope="Files.ReadWrite"
                        )
                        if not response:
                            self.logger.error(
                                "Couldn't authenticate as M365 user -> '%s' to provision user's drive!",
                                username=user["email"],
                            )
                            success = False
                            continue
                        # Retrieve the drive endpoint to trigger the drive provisioning. It is important
                        # to use the 'me=True' to make sure the request is done with the user credentials,
                        # not the app credentials (using purely client_id / client_secret):
                        response = self._m365.get_user_drive(user_id=user["email"], me=True)
                        if not response:
                            self.logger.error("Couldn't get M365 drive of user -> '%s'!", user["email"])
                            success = False
                            continue
                        self.logger.info(
                            "Make user -> '%s' follow departmental SharePoint site -> '%s'...",
                            user["email"],
                            group_site_name,
                        )
                        response = self._m365.follow_sharepoint_site(site_id=group_site_id, user_id=m365_user_id)
                        if not response:
                            self.logger.warning(
                                "User -> '%s' cannot follow SharePoint site -> '%s'!",
                                user["email"],
                                group_site_name,
                            )
                            success = False
                    # end if group_site_id:
                # end if group_id:
            # end if group_name == user_department and user["m365_skus"]:
        # end for group name in group_names:
    # end for user

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._users,
    )

    return success

process_users_salesforce(section_name='usersSalesforce')

Process users in payload and sync them with Salesforce (passwords and email).

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'usersSalesforce'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Side Effects

The user items are modified by adding "salesforce_user_id", "salesforce_user_login" dict element that includes the ID of the user in Salesforce. This will be written into the status file.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_users_salesforce")
def process_users_salesforce(
    self,
    section_name: str = "usersSalesforce",
) -> bool:
    """Process users in payload and sync them with Salesforce (passwords and email).

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    Side Effects:
        The user items are modified by adding "salesforce_user_id", "salesforce_user_login"
        dict element that includes the ID of the user in Salesforce. This will be written
        into the status file.

    """

    if not self._users:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    if not self._salesforce:
        self.logger.error(
            "Salesforce connection is not initialized. Bailing out...",
        )
        return False

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # traverse all users in payload:
    for user in self._users:
        user_name = user.get("name")
        if not user_name:
            self.logger.error("User is missing a login. Skipping to next user...")
            success = False
            continue

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not user.get("enabled", True):
            self.logger.info(
                "Payload for user -> '%s' is disabled. Skipping...",
                user_name,
            )
            continue

        # Check if the user is explicitly enabled for Salesforce:
        if not user.get("enable_salesforce", False):
            self.logger.info(
                "User -> '%s' is not enabled for Salesforce. Skipping...",
                user_name,
            )
            continue

        extra_attributes = user.get("extra_attributes", None)
        if not extra_attributes or len(extra_attributes) == 0:
            self.logger.info(
                "User -> '%s' does not have the extra attributes for Salesforce. Cannot determine the Salesforce login for user. Skipping...",
                user_name,
            )
            continue
        user_login = extra_attributes[0].get("value", "")
        if not user_login:
            self.logger.info(
                "User -> '%s' does not have the extra attributes value for Salesforce. Skipping...",
                user_name,
            )
            continue

        user_email = user.get("email", "")
        need_email_verification = False

        #
        # 1. Check if user does already exist in Salesforce:
        #

        salesforce_user_id = self._salesforce.get_user_id(username=user_login)

        #
        # 2: Create or Update user in Salesforce:
        #

        if salesforce_user_id is None:
            self.logger.info(
                "Salesforce user -> '%s' does not exist. Creating a new Salesforce user...",
                user_name,
            )
            response = self._salesforce.add_user(
                username=user_login,
                email=user.get("email", ""),
                firstname=user.get("firstname", ""),
                lastname=user.get("lastname", ""),
                department=user.get("base_group", ""),
                title=user.get("title", ""),
                company_name=user.get("company", "Innovate"),
            )
            salesforce_user_id = self._salesforce.get_result_value(
                response=response,
                key="id",
            )
            if not salesforce_user_id:
                self.logger.error(
                    "Failed to create Salesforce user -> '%s'! Skipping...",
                    user_name,
                )
                success = False
                continue
            self.logger.info(
                "Successfully created Salesforce user -> '%s' with ID -> %s.",
                user_name,
                salesforce_user_id,
            )
            # We need email verification for new users (unclear if this is really the case...)
            need_email_verification = True
        # The user does already exist in Salesforce and we just update it:
        else:
            update_data = {
                "FirstName": user.get("firstname", ""),
                "LastName": user.get("lastname", ""),
                "Department": user.get("base_group", ""),
                "Title": user.get("title", ""),
                "CompanyName": user.get("company", ""),
            }
            self.logger.info(
                "Salesforce user -> '%s' does already exist. Updating Salesforce user with -> %s...",
                user_name,
                str(update_data),
            )

            # Check if the mail address has really changed. Otherwise we
            # don't need to set it again and can avoid email verification:
            salesforce_user = self._salesforce.get_user(user_id=salesforce_user_id)
            salesforce_user_email = self._salesforce.get_result_value(
                response=salesforce_user,
                key="Email",
            )
            if user_email != salesforce_user_email:
                self.logger.info(
                    "Email for Salesforce user -> '%s' has changed from -> '%s' to -> '%s'.",
                    user_name,
                    salesforce_user_email,
                    user_email,
                )
                # Additional email payload for user update:
                update_data["Email"] = user_email
                # OK, email has changed - we need the email verification below...
                need_email_verification = True

            # Update the existing Salesforce user with new / changed data:
            response = self._salesforce.update_user(
                user_id=salesforce_user_id,
                update_data=update_data,
            )
            if not response:
                self.logger.error(
                    "Failed to update Salesforce user -> '%s'! Skipping...",
                    user_login,
                )
                success = False
                continue
            self.logger.info(
                "Successfully updated Salesforce user -> '%s'.",
                user_login,
            )

        # Save result for status file content
        user["salesforce_user_id"] = salesforce_user_id
        user["salesforce_user_login"] = user_login

        #
        # 3: Update user password in Salesforce (we need to do this also for new users!):
        #

        # Lookup password in payload:
        user_password = user.get("password", "")

        if user_password:
            response = self._salesforce.update_user_password(
                user_id=salesforce_user_id,
                password=user_password,
            )
            if response:
                self.logger.info(
                    "Successfully updated password of Salesforce user -> '%s' (%s).",
                    user_login,
                    salesforce_user_id,
                )
            else:
                self.logger.error(
                    "Failed to update Salesforce password for user -> '%s' (%s)! Skipping...",
                    user_login,
                    salesforce_user_id,
                )
                success = False
                continue

        #
        # 4: Handle Email verification:
        #

        # We now need to wait for the verification mail from Salesforce,
        # get it from the M365 Outlook inbox of the user (or the admin
        # if the user does not have its own inbox) and click the
        # verification link...

        if need_email_verification and user.get("enable_o365", False):
            self.logger.info(
                "Processing Email verification for user -> '%s' (%s). Wait a few seconds to make sure verification mail in user's inbox...",
                user_name,
                user_email,
            )
            time.sleep(20)

            # Process verification mail sent by Salesforce.
            # This has some hard-coded value. We may want to optimize it further in the future:
            result = self._m365.email_verification(
                user_email=user_email,
                sender="QA_SUPPORT@salesforce.com",
                subject="Finish changing your Salesforce",
                url_search_pattern="setup/emailverif",
            )
            if not result:
                # Email verification was not successful
                self.logger.warning(
                    "Salesforce email verification failed. No verification mail received in user's inbox using email address -> '%s'!",
                    user_email,
                )
                # don't treat as error nor do "continue" here - we still want to process the user groups...
            else:
                self.logger.info(
                    "Successfully verified new email address -> '%s'.",
                    user_email,
                )
        # end if need_email_verification

        #
        # 5: Add users into groups in Salesforce:
        #

        self.logger.info(
            "Processing group memberships of Salesforce user -> '%s' (%s)...",
            user_name,
            user_email,
        )
        user_groups = user.get("groups", [])
        base_group = user.get("base_group", None)
        if base_group and base_group not in user_groups:
            user_groups.append(base_group)  # list of groups the user is in

        for user_group in user_groups:
            # "Business Administrators" is a OTCS generated group that we won't find
            # in payload - skip this group.
            if user_group == "Business Administrators":
                continue
            # Try to find the group dictionary item in the payload
            # for user group name:
            group = next(
                (item for item in self._groups if item["name"] == user_group),
                None,
            )
            if not group:
                self.logger.error(
                    "Cannot find group -> '%s'. Cannot establish membership in Salesforce. Skipping to next group...",
                    user_group,
                )
                success = False
                continue

            group_id = group.get("salesforce_id")  # Careful ID may not exist
            group_name = group["name"]
            if group_id is None:
                self.logger.info(
                    "Group -> '%s' does not have a Salesforce ID. Cannot add user -> '%s' to this Salesforce group (group may not be enabled for Salesforce). Skipping...",
                    group_name,
                    user_name,
                )
                # We don't treat this as an error - there may be payload groups which are not enabled for Salesforce!
                continue

            existing_members = self._salesforce.get_group_members(group_id=group_id)
            if not existing_members or not self._salesforce.exist_result_item(
                response=existing_members,
                key="UserOrGroupId",
                value=salesforce_user_id,
            ):
                self.logger.info(
                    "Add Salesforce user -> '%s' (%s) to Salesforce group -> '%s' (%s)...",
                    user_name,
                    salesforce_user_id,
                    group_name,
                    group_id,
                )
                response = self._salesforce.add_group_member(
                    group_id=group_id,
                    member_id=salesforce_user_id,
                )
                member_id = self._salesforce.get_result_value(
                    response=response,
                    key="id",
                )
                if not member_id:
                    self.logger.error(
                        "Failed to add Salesforce user -> '%s' (%s) as member to Salesforce group -> '%s' (%s)!",
                        user_name,
                        salesforce_user_id,
                        group_name,
                        group_id,
                    )
                    success = False
                    continue
            else:
                self.logger.info(
                    "Salesforce user -> '%s' (%s) does already exist in Salesforce group -> '%s' (%s). Skipping...",
                    user_name,
                    salesforce_user_id,
                    group_name,
                    group_id,
                )
        # end for loop user groups
    # end for loop users

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._users,
    )

    return success

process_users_sap(section_name='usersSAP')

Process users in payload and sync them with SAP (passwords only for now).

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'usersSAP'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise

Side Effects

The user items are modified by adding an "sap_sync_result" dict element that documents if the user password was properly synced with SAP.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_users_sap")
def process_users_sap(self, section_name: str = "usersSAP") -> bool:
    """Process users in payload and sync them with SAP (passwords only for now).

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool: True if payload has been processed without errors, False otherwise

    Side Effects:
        The user items are modified by adding an "sap_sync_result" dict element
        that documents if the user password was properly synced with SAP.

    """

    if not self._users:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    if not self._sap:
        self.logger.error("SAP connection is not initialized. Bailing out...")
        return False

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    rfc_name = "ZFM_TERRA_RFC_CHNG_USR_PW"
    rfc_description = "RFC to update the SAP user password"
    rfc_call_options = ()

    # Update SAP password for all users in payload:
    for user in self._users:
        user_name = user.get("name")
        if not user_name:
            self.logger.error("User is missing a login. Skipping to next user...")
            success = False
            continue

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not user.get("enabled", True):
            self.logger.info(
                "Payload for user -> '%s' is disabled. Skipping...",
                user_name,
            )
            continue

        # Check if the user is explicitly enabled for SAP:
        if not user.get("enable_sap", False):
            self.logger.info(
                "User -> '%s' is not enabled for SAP. Skipping...",
                user_name,
            )
            continue

        # Lookup mandatory password in payload:
        user_password = user.get("password")
        if not user_password:
            self.logger.error(
                "User -> '%s' is missing a password. Cannot sync with SAP. Skipping to next user...",
                user_name,
            )
            success = False
            continue

        rfc_params = {
            "USERNAME": user_name,
            "PASSWORD": user_password,
        }

        self.logger.info(
            "Updating password of user -> '%s' in SAP. Calling SAP RFC -> '%s' (%s) with parameters -> %s ...",
            user_name,
            rfc_name,
            rfc_description,
            str(rfc_params),
        )

        result = self._sap.call(
            rfc_name=rfc_name,
            options=rfc_call_options,
            rfc_parameters=rfc_params,
        )
        if result is None:
            self.logger.error(
                "Failed to call SAP RFC -> '%s' to update password of user -> '%s'!",
                rfc_name,
                user_name,
            )
            success = False
        elif result.get("RESULT") != "OK":
            self.logger.error(
                "Result of SAP RFC -> '%s' is not OK, it returned -> %s failed items in result -> %s",
                rfc_name,
                str(result.get("FAILED")),
                str(result),
            )
            success = False
            # Save result for status file content
            user["sap_sync_result"] = result
        else:
            self.logger.info(
                "Successfully called RFC -> '%s'. Result -> %s.",
                rfc_name,
                str(result),
            )
            # Save result for status file content
            user["sap_sync_result"] = result

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._users,
    )

    return success

process_users_successfactors(section_name='usersSuccessFactors')

Process users in payload and sync them with SuccessFactors (passwords and email).

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'usersSuccessFactors'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Side Effects

The user items are modified by adding an "successfactors_user_id" dict element that includes the personIdExternal of the user in SuccessFactors

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_users_successfactors")
def process_users_successfactors(
    self,
    section_name: str = "usersSuccessFactors",
) -> bool:
    """Process users in payload and sync them with SuccessFactors (passwords and email).

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    Side Effects:
        The user items are modified by adding an "successfactors_user_id" dict element that
        includes the personIdExternal of the user in SuccessFactors

    """

    if not self._users:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    if not self._successfactors:
        self.logger.error(
            "SuccessFactors connection is not initialized. Bailing out...",
        )
        return False

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # traverse all users in payload:
    for user in self._users:
        user_name = user.get("name")
        if not user_name:
            self.logger.error("User is missing a login. Skipping to next user...")
            success = False
            continue

        # Check if user has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not user.get("enabled", True):
            self.logger.info(
                "Payload for user -> '%s' is disabled. Skipping...",
                user_name,
            )
            continue

        # Check if the user is explicitly enabled for SuccessFactors:
        if not user.get("enable_successfactors", False):
            self.logger.info(
                "User -> '%s' is not enabled for SuccessFactors. Skipping...",
                user_name,
            )
            continue

        # Lookup password and email in payload:
        user_password = user.get("password", "")
        user_email = user.get("email", "")

        # first we need to get the SuccessFactors user account object
        # to determine the personIdExternal that we need to update the
        # SuccessFactors user.
        response = self._successfactors.get_user_account(username=user_name)
        user_id = self._successfactors.get_result_value(
            response=response,
            key="personIdExternal",
        )
        if user_id is None:
            self.logger.error(
                "Failed to get personIDExternal of SuccessFactors user -> '%s'!",
                user_name,
            )
            success = False
            continue
        self.logger.info(
            "SuccessFactors user -> '%s' has external user ID -> %s.",
            user_name,
            str(user_id),
        )

        # Now let's update the user password and email address:
        update_data = {}
        if user_password:
            self.logger.info(
                "Updating password of SuccessFactors user -> '%s' (%s)...",
                user_name,
                str(user_id),
            )
            update_data["password"] = user_password
        if user_email:
            update_data["email"] = user_email

        response = self._successfactors.update_user(
            user_id=user_id,
            update_data=update_data,
        )
        if response:
            self.logger.info(
                "Successfully updated SuccessFactors user -> '%s'.",
                str(user_name),
            )
            # Save result for status file content
            user["successfactors_user_id"] = user_id
        else:
            self.logger.error(
                "Failed to update SuccessFactors user -> '%s'! Skipping...",
                user_name,
            )
            success = False
            continue

        if not user_email:
            continue

        self.logger.info(
            "Updating email of SuccessFactors user -> '%s' to -> %s...",
            user_name,
            user_email,
        )
        response = self._successfactors.update_user_email(
            user_id=user_id,
            email_address=user_email,
        )
        if response:
            self.logger.info(
                "Successfully updated email address of SuccessFactors user -> '%s' to -> '%s'.",
                user_name,
                user_email,
            )
        else:
            self.logger.error(
                "Failed to update email address of SuccessFactors user -> '%s' to -> '%s'!",
                user_name,
                user_email,
            )
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._users,
    )

    return success

process_web_hooks(webhooks, section_name='webHooks')

Process Web Hooks in payload and do HTTP requests.

Parameters:

Name	Type	Description	Default
webhooks	list	The list of web hook payload settings.	required
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'webHooks'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_web_hooks")
def process_web_hooks(self, webhooks: list, section_name: str = "webHooks") -> bool:
    """Process Web Hooks in payload and do HTTP requests.

    Args:
        webhooks (list):
            The list of web hook payload settings.
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise.

    """

    if not webhooks:
        self.logger.info(
            "Payload section -> %s is empty. Skipping...",
            section_name,
        )
        return True

    # Even if this payload section has been processed successfully before we
    # want to call the webhook.

    success: bool = True

    for webhook in webhooks:
        url = webhook.get("url")

        # Check if element has been disabled in payload (enabled = false).
        # In this case we skip the element:
        enabled = webhook.get("enabled", True)

        if not enabled and not url:
            self.logger.info("Payload for Web Hook is disabled. Skipping...")
            continue
        if not url:
            self.logger.info("Web Hook does not have a url. Skipping...")
            success = False
            continue
        if not enabled:
            self.logger.info(
                "Payload for Web Hook -> '%s' is disabled. Skipping...",
                url,
            )
            continue

        description = webhook.get("description")

        method = webhook.get("method", "POST")

        payload = webhook.get("payload", {})

        headers = webhook.get("headers", {})

        if description:
            self.logger.info(
                "Calling Web Hook -> %s: %s (%s)...",
                method,
                url,
                description,
            )
        else:
            self.logger.info("Calling Web Hook -> %s: %s...", method, url)

        response = self._http_object.http_request(
            url=url,
            method=method,
            payload=payload,
            headers=headers,
            retries=webhook.get("retries", 0),
            wait_time=webhook.get("wait_time", 0),
        )
        if not response or not response.ok:
            success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=webhooks,
    )

    return success

process_web_reports(web_reports, section_name='webReports')

Process web reports in payload and run them in Content Server.

Parameters:

Name	Type	Description	Default
web_reports	list	The payload list of web reports. As we have two different list (pre and post) we need to pass the actual list as parameter.	required
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'webReports'

Returns:

Name	Type	Description
bool	bool	True, if a restart of the OTCS pods is required. False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_web_reports")
def process_web_reports(
    self,
    web_reports: list,
    section_name: str = "webReports",
) -> bool:
    """Process web reports in payload and run them in Content Server.

    Args:
        web_reports (list):
            The payload list of web reports. As we have two different list (pre and post)
            we need to pass the actual list as parameter.
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if a restart of the OTCS pods is required. False otherwise.

    """

    if not web_reports:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return False  # important to return False here as otherwise we are triggering a restart of services!!

    # If this payload section has been processed successfully before we
    # can return False and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return False  # important to return False here as otherwise we are triggering a restart of services!!

    restart_required: bool = False
    success: bool = True

    for web_report in web_reports:
        nickname = web_report.get("nickname")
        if not nickname:
            self.logger.error("Web Report payload needs a nickname! Skipping...")
            continue

        # Check if web report has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not web_report.get("enabled", True):
            self.logger.info(
                "Payload for Web Report -> '%s' is disabled. Skipping...",
                nickname,
            )
            continue

        description = web_report.get("description", "")
        restart = web_report.get("restart", False)

        if not self._otcs.get_node_from_nickname(nickname=nickname):
            self.logger.error(
                "Web Report with nickname -> '%s' does not exist! Skipping...",
                nickname,
            )
            success = False
            continue

        # be careful to avoid key errors as Web Report parameters are optional:
        actual_params = web_report["parameters"] if web_report.get("parameters") else {}
        formal_params = self._otcs.get_web_report_parameters(nickname=nickname)
        if actual_params:
            self.logger.info(
                "Running Web Report -> '%s' (%s) with parameters -> %s ...",
                nickname,
                description,
                actual_params,
            )
            # Do some sanity checks to see if the formal and actual parameters are matching...
            # Check 1: are there formal parameters at all?
            if not formal_params:
                self.logger.error(
                    "Web Report -> '%s' is called with actual parameters but it does not expect parameters! Skipping...",
                    nickname,
                )
                success = False
                continue
            lets_continue = False
            # Check 2: Iterate through the actual parameters given in the payload
            # and see if there's a matching formal parameter expected by the Web Report:
            for key, value in actual_params.items():
                # Check if there's a matching formal parameter defined on the Web Report node:
                formal_param = next(
                    (item for item in formal_params if item["parm_name"] == key),
                    None,
                )
                if formal_param is None:
                    self.logger.error(
                        "Web Report -> '%s' is called with parameter -> '%s' that is not expected! Value: %s) Skipping...",
                        nickname,
                        key,
                        value,
                    )
                    success = False
                    lets_continue = True  # we cannot do a "continue" here directly as we are in an inner loop
            # Check 3: Iterate through the formal parameters and validate there's a matching
            # actual parameter defined in the payload for each mandatory formal parameter
            # that does not have a default value:
            for formal_param in formal_params:
                if (
                    (formal_param["mandatory"] is True)
                    and (formal_param["default_value"] is None)
                    and not actual_params.get(formal_param["parm_name"])
                ):
                    self.logger.error(
                        "Web Report -> '%s' is called without mandatory parameter -> %s! Skipping...",
                        nickname,
                        formal_param["parm_name"],
                    )
                    success = False
                    lets_continue = True  # we cannot do a "continue" here directly as we are in an inner loop
            # Did any of the checks fail?
            if lets_continue:
                continue
            # Actual parameters are validated, we can run the Web Report:
            response = self._otcs.run_web_report(
                nickname=nickname,
                web_report_parameters=actual_params,
            )
        # end if actual_params
        else:
            self.logger.info(
                "Running Web Report -> '%s' (%s) without parameters...",
                nickname,
                description,
            )
            # Check if there's a formal parameter that is mandatory but
            # does not have a default value:
            if formal_params:
                required_param = next(
                    (item for item in formal_params if (item["mandatory"] is True) and (not item["default_value"])),
                    None,
                )
                if required_param:
                    self.logger.error(
                        "Web Report -> '%s' is called without parameters but has a mandatory parameter -> '%s' without a default value! Skipping...",
                        nickname,
                        required_param["parm_name"],
                    )
                    success = False
                    continue
                # we are good to proceed!
                self.logger.debug(
                    "Web Report -> '%s' does not have a mandatory parameter without a default value!",
                    nickname,
                )
            response = self._otcs.run_web_report(nickname=nickname)
        # end else
        if response is None:
            self.logger.error(
                "Failed to run web report with nickname -> '%s'!",
                nickname,
            )
            success = False

        if restart:
            restart_required = True

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=web_reports,
    )

    return restart_required

process_workflow_attributes(attributes, workflow_attribute_definition)

Process the attributes in the workflow steps.

This method adds the IDs for the attribute to the payload dicts. The IDs are needed for the workflow REST API calls.

Parameters:

Name	Type	Description	Default
attributes	list	The list of attributes (payload) processed in the workflow step.	required
workflow_attribute_definition	dict	The workflow attribute definition.	required

Returns:

Type	Description
None	None. The mutable dictionary in the workflow_step is updated with the IDs.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workflow_attributes")
def process_workflow_attributes(
    self,
    attributes: list,
    workflow_attribute_definition: dict,
) -> None:
    """Process the attributes in the workflow steps.

    This method adds the IDs for the attribute to the payload dicts.
    The IDs are needed for the workflow REST API calls.

    Args:
        attributes (list):
            The list of attributes (payload) processed in the workflow step.
        workflow_attribute_definition (dict):
            The workflow attribute definition.

    Returns:
        None. The mutable dictionary in the workflow_step is updated with the IDs.

    """

    # now we need to get the technical attribute IDs from
    # the workflow definition and map them
    # with the attribute names from the payload:
    for attribute in attributes:
        attribute_name = attribute["name"]
        attribute_value = attribute["value"]
        attribute_type = attribute.get("type", None)

        # Special treatment for type user: determine the ID for the login name.
        # the ID is the actual value we have to put in the attribute:
        if attribute_type and attribute_type.lower() == "user":
            user = self._otcs.get_user(name=attribute_value, show_error=True)
            user_id = self._otcs.get_result_value(response=user, key="id")
            if not user_id:
                self.logger.error(
                    "Cannot find user with login name -> '%s'. Skipping...",
                    attribute_value,
                )
                continue
            attribute_value = user_id
            attribute["value"] = user_id

        attribute_definition = workflow_attribute_definition.get(attribute_name)
        if not attribute_definition:
            self.logger.error(
                "Cannot find the attribute -> '%s' in the workflow definition. Skipping...",
            )
            continue
        # Enrich the attribute dictionary with the attribute ID from the workflow definition:
        attribute["id"] = attribute_definition["id"]
        # Enrich the attribute dictionary with the attribute form ID from the workflow definition:
        attribute["form_id"] = attribute_definition["form_id"]

    if attributes:
        self.logger.info(
            "Updated workflow step attributes with IDs -> %s.",
            str(attributes),
        )

process_workflow_step(workflow_id, workflow_step, workflow_attribute_definition, documents=None, process_id=None)

Process a workflow step of a workflow.

Parameters:

Name	Type	Description	Default
workflow_id	int	The node ID of the workflow (the workflow map).	required
workflow_step	dict	Payload dictionary for a single workflow step. Keys: "action", "exec_as_user", "attributes"	required
workflow_attribute_definition	dict	The workflow attribute definition.	required
documents	list | None	The list of workflow documents (attachments9).	None
process_id	int | None	The process ID of the workflow.	None

Returns:

Name	Type	Description
bool	bool	True, if the workflow step has been processed successfully, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workflow_step")
def process_workflow_step(
    self,
    workflow_id: int,
    workflow_step: dict,
    workflow_attribute_definition: dict,
    documents: list | None = None,
    process_id: int | None = None,
) -> bool:
    """Process a workflow step of a workflow.

    Args:
        workflow_id (int):
            The node ID of the workflow (the workflow map).
        workflow_step (dict):
            Payload dictionary for a single workflow step.
            Keys: "action", "exec_as_user", "attributes"
        workflow_attribute_definition (dict):
            The workflow attribute definition.
        documents (list | None, optional):
            The list of workflow documents (attachments9).
        process_id (int | None, optional):
            The process ID of the workflow.

    Returns:
        bool:
            True, if the workflow step has been processed successfully, False otherwise.

    """

    if "action" not in workflow_step:
        self.logger.error("Missing workflow action in workflow step.")
        return False
    action = workflow_step["action"]

    if "exec_as_user" not in workflow_step:
        self.logger.error("Missing workflow user in workflow step.")
        return False
    exec_as_user = workflow_step["exec_as_user"]

    # Find the user in the users payload:
    exec_user = next(
        (item for item in self._users if item["name"] == exec_as_user),
        None,
    )
    # Have we found the user in the payload?
    if exec_user is None:
        self.logger.error(
            "Cannot find user with login name -> '%s' for workflow processing.",
            exec_as_user,
        )
        return False

    self.logger.info("Executing workflow step as user -> '%s'...", exec_as_user)

    # Impersonate as the user:
    self.logger.info("Impersonate user -> '%s'...", exec_as_user)
    # True = force new login with new user
    result = self.start_impersonation(username=exec_as_user)
    if not result:
        self.logger.error("Couldn't impersonate user -> '%s'", exec_as_user)
        return False

    # "attributes" is optional:
    if "attributes" not in workflow_step:
        self.logger.warning(
            "No workflow attributes specified in the payload for this workflow step.",
        )
        attributes = []
        workflow_step_values = None
    else:
        attributes = workflow_step["attributes"]
        self.logger.info(
            "Workflow step has attributes -> %s. Adding attribute IDs to the payload names...",
            str(attributes),
        )
        # Update / enrich the attributes in the workflow step with the IDs
        # from the workflow definition (this CHANGES the attributes!)
        self.process_workflow_attributes(
            attributes=attributes,
            workflow_attribute_definition=workflow_attribute_definition,
        )
        # Prepare the data for the REST call to
        # update the process:
        workflow_step_values = {
            attr["form_id"]: attr["value"] for attr in attributes if "form_id" in attr and "value" in attr
        }

    if action == "Initiate":
        # Create a draft process in preparation for the workflow initiation:
        response = self._otcs.create_draft_process(
            workflow_id=workflow_id,
            documents=documents,
            attach_documents=True,
        )
        draftprocess_id = self._otcs.get_result_value(
            response=response,
            key="draftprocess_id",
            property_name="",
        )
        if not draftprocess_id:
            self.logger.error(
                "Failed to create draft process for workflow ID -> %s as user -> '%s'!",
                str(workflow_id),
                exec_as_user,
            )
            # Stop the impersonation as a user:
            result = self.stop_impersonation()
            return False
        else:
            self.logger.info(
                "Successfully generated draft process with ID -> %s%s.",
                str(draftprocess_id),
                " attching document IDs -> " + str(documents) if documents else "",
            )
        workflow_step["draftprocess_id"] = draftprocess_id

        # Check if a due date is specified. The payload has
        # a relative offset in number of days that we add to
        # the current date:
        due_in_days = workflow_step.get("due_in_days")
        if due_in_days:
            due_date = datetime.now(UTC) + timedelta(days=int(due_in_days))
            due_date = due_date.strftime("%Y-%m-%d")
        else:
            due_date = None
        # Record the due date in the workflow step dictionary
        workflow_step["due_date"] = due_date

        # Update the draft process with title, due date
        # and workflow attribute values from the payload:
        response = self._otcs.update_draft_process(
            draftprocess_id=draftprocess_id,
            title=workflow_step.get("title"),
            due_date=due_date,
            values=workflow_step_values,
        )

        # Initiate the draft process. This creates
        # the running workflow instance:
        response = self._otcs.initiate_draft_process(
            draftprocess_id=draftprocess_id,
            comment=workflow_step.get("comment"),
        )
        process_id = self._otcs.get_result_value(
            response=response,
            key="process_id",
            property_name="",
        )
        if not process_id:
            self.logger.error(
                "Failed to initiate process for workflow with ID -> %s as user -> '%s'!",
                str(workflow_id),
                exec_as_user,
            )
            # Stop the impersonation as a user:
            result = self.stop_impersonation()
            return False
        self.logger.info(
            "Successfully initiated process with ID -> %s for workflow with ID -> %s as user -> '%s'.",
            str(process_id),
            str(workflow_id),
            exec_as_user,
        )
        workflow_step["process_id"] = process_id
    # end if action == "Initiate"
    else:
        if not process_id:
            self.logger.error(
                "Workflow step cannot be executed as process is not initiated (process ID not set)",
            )
            # Stop the impersonation as a user:
            result = self.stop_impersonation()
            return False
        response = self._otcs.get_process_task(
            process_id=process_id,
        )
        # Are there any workflow attributes to update with new values?
        if attributes:
            response = self._otcs.update_process_task(
                process_id=process_id,
                values=workflow_step_values,
            )
        # Execute the step action defined in the payload
        response = self._otcs.update_process_task(
            process_id=process_id,
            action=action,
        )

    # Impersonate as the admin user:
    self.logger.info(
        "Impersonate as admin user -> '%s'...",
        self._otcs.credentials()["username"],
    )
    # Stop the impersonation as a user:
    result = self.stop_impersonation()

    return True

process_workflows(section_name='workflows')

Initiate and process workflows for a defined workspace type and folder path.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'workflows'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workflows")
def process_workflows(self, section_name: str = "workflows") -> bool:
    """Initiate and process workflows for a defined workspace type and folder path.

    Args:
        section_name (str, optional):
            The name of the section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise.

    """

    if not self._workflows:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    # save admin credentials for later switch back to admin user:
    admin_credentials = self._otcs.credentials()

    for workflow in self._workflows:
        workflow_nickname = workflow.get("workflow_nickname")
        if not workflow_nickname:
            self.logger.error(
                "To initiate and process workflows for documents in workspaces the workflow nickname needs to be specified in the payload! Skipping to next workflow initiation...",
            )
            success = False
            continue
        workflow_node = self._otcs.get_node_from_nickname(
            nickname=workflow_nickname,
        )
        workflow_id = self._otcs.get_result_value(response=workflow_node, key="id")
        workflow_name = self._otcs.get_result_value(
            response=workflow_node,
            key="name",
        )
        if not workflow_id:
            self.logger.error(
                "Cannot find workflow by nickname -> '%s'! Skipping to next workflow...",
                workflow_nickname,
            )
            success = False
            continue

        workspace_type = workflow.get("workspace_type")
        if not workspace_type:
            self.logger.error(
                "To process workflow -> '%s' for documents in workspaces the workspace type needs to be specified in the payload! Skipping to next workflow...",
                workflow_name,
            )
            success = False
            continue

        # Check if workflow has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not workflow.get("enabled", True):
            self.logger.info(
                "Payload for workflow -> '%s' of workspace type -> '%s' is disabled. Skipping...",
                workflow_name,
                workspace_type,
            )
            continue

        # Find the workspace type with the name given in the _workspace_types
        # datastructure that has been generated by process_workspace_types() method before:
        workspace_type_id = next(
            (item["id"] for item in self._workspace_types if item["name"] == workspace_type),
            None,
        )

        workspace_folder_path = workflow.get("workspace_folder_path", [])
        if not workspace_folder_path:
            self.logger.info(
                "No workspace folder path defined for workspaces of type -> '%s'. Workflows will be started for documents in workspace root.",
                workspace_type,
            )

        workflow_steps = workflow.get("steps")
        if not workflow_steps:
            self.logger.error(
                "To process workflow -> '%s', workflow steps ('steps') needs to be specified in the payload! Skipping to next workflow initiation...",
                workflow_name,
            )
            success = False
            continue

        # Get the attribute details (name, ID, type) from the workflow definition:
        workflow_attribute_definition = self._otcs.get_workflow_attributes(
            workflow_id=workflow_id,
        )

        # The workspace type name is used as a "starts with" search on the
        # workspace type name. This can cause issues, so we prefer to use the type ID
        # if we have it. Otherwise the REST API prefers the name over the type:
        workspace_instances = self._otcs.get_workspace_instances_iterator(
            type_name=workspace_type if not workspace_type_id else None,
            type_id=workspace_type_id,
        )
        for workspace_instance in workspace_instances:
            workspace = workspace_instance.get("data").get("properties")
            workspace_id = workspace["id"]
            workspace_name = workspace["name"]

            if workspace_folder_path:
                workspace_folder = self._otcs.get_node_by_workspace_and_path(
                    workspace_id=workspace_id,
                    path=workspace_folder_path,
                )
                if workspace_folder:
                    workspace_folder_id = self._otcs.get_result_value(
                        response=workspace_folder,
                        key="id",
                    )
                else:
                    # If the workspace template is not matching
                    # the path we may have an error here. Then
                    # we fall back to workspace root level.
                    self.logger.warning(
                        "Folder path does not exist in workspace -> '%s'. Using workspace root level instead...",
                        workspace_name,
                    )
                    workspace_folder_id = workspace_id
            else:
                workspace_folder_id = workspace_id

            # Get all documents (-3 = non-container) from the defined folder:
            response = self._otcs.get_subnodes(
                parent_node_id=workspace_folder_id,
                filter_node_types=-3,
            )
            documents = self._otcs.get_result_values(response=response, key="id")

            process_id = None
            for workflow_step in workflow_steps:
                result = self.process_workflow_step(
                    workflow_id=workflow_id,
                    workflow_step=workflow_step,
                    workflow_attribute_definition=workflow_attribute_definition,
                    documents=documents,
                    process_id=process_id,
                )
                # If the step fails we are bailing out as it doesn't make
                # sense to continue with further steps:
                if not result:
                    success = False
                    break
                if "process_id" in workflow_step:
                    process_id = workflow_step["process_id"]
        # end for workspace_instance in workspace_instances:
    # end for workflow in self._workflows

    # Set back admin credentials:
    self._otcs.set_credentials(
        username=admin_credentials["username"],
        password=admin_credentials["password"],
    )
    # Authenticate back as the admin user:
    self.logger.info(
        "Authenticate as admin user -> '%s'...",
        admin_credentials["username"],
    )
    # True = force new login with new user
    self._otcs.authenticate(revalidate=True)

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._workflows,
    )

    return success

process_workspace(workspace)

Worker thread for workspace creation.

Parameters:

Name	Type	Description	Default
workspace	dict	Dictionary with payload of a single workspace.	required

Returns:

Name	Type	Description
bool	bool	True = Success, False = Failure

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace")
def process_workspace(
    self,
    workspace: dict,
) -> bool:
    """Worker thread for workspace creation.

    Args:
        workspace (dict):
            Dictionary with payload of a single workspace.

    Returns:
        bool:
            True = Success, False = Failure

    """

    # Read name from payload:
    if "name" not in workspace:
        self.logger.error("Workspace needs a name! Skipping to next workspace...")
        return False
    name = workspace["name"]

    # Check if workspace has been explicitly disabled in payload
    # (enabled = false). In this case we skip the element:
    if not workspace.get("enabled", True):
        self.logger.info(
            "Payload for workspace -> '%s' is disabled. Skipping...",
            name,
        )
        return True

    # Read optional description from payload:
    description = workspace.get("description", "")

    # Read Type Name from payload:
    if "type_name" not in workspace:
        self.logger.error(
            "Workspace -> '%s' needs a type name! Skipping to next workspace...",
            name,
        )
        return False
    type_name = workspace["type_name"]

    # We need to do this early to find out if we have a cross-application workspace
    # and need to continue even if the workspace does exist...
    if workspace.get("business_objects"):
        business_objects = workspace["business_objects"]

        business_object_list = self.prepare_workspace_business_objects(
            workspace=workspace,
            business_objects=business_objects,
        )
        # Check if any of the external systems are avaiable:
        if business_object_list:
            self.logger.info(
                "Workspace -> '%s' will be connected to -> %s business object%s.",
                name,
                str(len(business_object_list)),
                "s" if len(business_object_list) > 1 else "",
            )
    else:
        self.logger.debug(
            "Workspace -> '%s' is not connected to any business object.",
            name,
        )
        business_object_list = []

    # Intialize cross-application workspace to "off":
    ibo_workspace_id = None

    # check if the workspace has been created before (effort to make the customizing code idem-potent)
    self.logger.debug(
        "Check if workspace -> '%s' of type -> '%s' does already exist...",
        name,
        type_name,
    )
    # Check if workspace does already exist
    # In case the workspace exists, determine_workspace_id()
    # also stores the node ID into workspace["nodeId"]
    workspace_id = self.determine_workspace_id(workspace=workspace)
    if workspace_id:
        self.logger.info(
            "Workspace -> '%s' of type -> '%s' does already exist and has ID -> %d.",
            name,
            type_name,
            workspace_id,
        )
        # Check if we have an existing workspace that is cross-application.
        # In this case we cannot just skip (return).
        if len(business_object_list) > 1:
            ibo_workspace_id = workspace_id
            self.logger.info(
                "Workspace -> '%s' is a cross-application workspace so we cannot skip the creation...",
                name,
            )
        elif not business_object_list:
            self.logger.info(
                "Workspace -> '%s' does already exist and has no business object references to update - skipping the creation...",
                name,
            )
            return True

    # Parent ID is optional and only required if workspace type does not specify a create location.
    # This is typically the case if it is a nested workspace or workspaces of the same type can be created
    # in different locations in the Enterprise Workspace:
    parent_id = workspace.get("parent_id")

    if parent_id is not None:
        parent_workspace = next(
            (item for item in self._workspaces if item["id"] == parent_id),
            None,
        )
        if parent_workspace is None:
            self.logger.error(
                "Parent workspace with logical ID -> %s not found.",
                parent_id,
            )
            return False

        parent_workspace_node_id = self.determine_workspace_id(
            workspace=parent_workspace,
        )
        if not parent_workspace_node_id:
            self.logger.error(
                "Parent workspace without node ID (parent workspace creation may have failed). Skipping to next workspace...",
            )
            return False

        self.logger.debug(
            "Parent workspace with logical ID -> %s has node ID -> %s",
            parent_id,
            parent_workspace_node_id,
        )
    else:
        # Alternatively a path could be specified in the payload:
        parent_path = workspace.get("parent_path")
        if parent_path:
            self.logger.info(
                "Workspace -> '%s' has parent path -> %s specified in payload.",
                name,
                parent_path,
            )
            response = self._otcs.get_node_by_volume_and_path(
                volume_type=self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE,
                path=parent_path,
                create_path=True,
            )
            parent_workspace_node_id = self._otcs.get_result_value(
                response=response,
                key="id",
            )
        else:
            # if no parent_id is specified the workspace location is determined by the workspace type definition
            # and we pass None as parent ID to the get_workspace_create_form and create_workspace methods below:
            parent_workspace_node_id = None
            self.logger.info(
                "Workspace -> '%s' has no parent path specified in payload. Using the default defined in the workspace type...",
                name,
            )

    # Find the workspace type with the name given in the payload:
    workspace_type = next(
        (item for item in self._workspace_types if item["name"] == type_name),
        None,
    )
    if workspace_type is None:
        self.logger.error(
            "Workspace type -> '%s' not found. Skipping to next workspace...",
            type_name,
        )
        return False
    if workspace_type["templates"] == []:
        self.logger.error(
            "Workspace type -> '%s' does not have templates. Skipping to next workspace...",
            type_name,
        )
        return False

    # check if the template to be used is specified in the payload:
    if "template_name" in workspace:
        template_name = workspace["template_name"]
        workspace_template = next(
            (item for item in workspace_type["templates"] if item["name"] == template_name),
            None,
        )
        if workspace_template:  # does this template exist?
            self.logger.debug(
                "Workspace template -> '%s' has been specified in payload and it does exist.",
                template_name,
            )
        else:
            self.logger.error(
                "Workspace template -> '%s' has been specified in payload but it doesn't exist!",
                template_name,
            )
            self.logger.error(
                "Workspace type -> '%s' has only these templates -> %s",
                type_name,
                workspace_type["templates"],
            )
            return False
    # template to be used is NOT specified in the payload - then we just take the first one:
    else:
        workspace_template = workspace_type["templates"][0]
        self.logger.info(
            "Workspace template has not been specified in payload - taking the first one (%s)...",
            workspace_template,
        )

    template_id = workspace_template["id"]
    template_name = workspace_template["name"]
    workspace_type_id = workspace_type["id"]

    if not workspace_id:
        self.logger.info(
            "Create workspace -> '%s' (type -> '%s') from workspace template -> '%s' (%s)%s...",
            name,
            type_name,
            template_name,
            template_id,
            " with business object references -> {}".format(business_object_list) if business_object_list else "",
        )
    elif business_object_list:
        self.logger.info(
            "Update workspace -> '%s' (type -> '%s') business object references with -> %s...",
            name,
            type_name,
            str(business_object_list),
        )

    # Handle the case where the workspace is not connected
    # to any external system / business object:
    if not business_object_list:
        business_object_list.append(
            {
                "ext_system_id": None,
                "bo_type": None,
                "bo_id": None,
            },
        )

    for business_object in business_object_list:
        external_system_id = business_object["ext_system_id"]
        bo_type = business_object["bo_type"]
        bo_id = business_object["bo_id"]

        if workspace_id and not ibo_workspace_id and bo_id:
            # See if the existing workspace does not yet have a business object references
            # that is given in the payload:
            self.logger.info("Get existing workspace references for workspace -> '%s' (%d)...", name, workspace_id)
            workspace_references = self._otcs.get_workspace_references(node_id=workspace_id)
            workspace_reference = next(
                (
                    item
                    for item in workspace_references or []
                    if item["external_system_id"] == external_system_id
                    and item["business_object_id"] == bo_id
                    and item["business_object_type"].lower() == bo_type.lower()
                ),
                None,
            )
            if not workspace_reference:
                self.logger.info(
                    "Set workspace reference for workspace -> '%s' (%d) to business object -> ('%s', '%s', %s)...",
                    name,
                    workspace_id,
                    external_system_id,
                    bo_type,
                    bo_id,
                )
                response = self._otcs.set_workspace_reference(
                    workspace_id=workspace_id, external_system_id=external_system_id, bo_type=bo_type, bo_id=bo_id
                )
                if not response:
                    self.logger.error(
                        "Failed to set a reference for workspace -> '%s' (%d) to business object -> ('%s', '%s', %s)!",
                        name,
                        workspace_id,
                        external_system_id,
                        bo_type,
                        bo_id,
                    )
                    return False
            else:
                self.logger.info(
                    "Workspace -> '%s' (%d) has already a reference to business object -> ('%s', '%s', %s).",
                    name,
                    workspace_id,
                    external_system_id,
                    bo_type,
                    bo_id,
                )
            continue

        # Read categories from payload:
        if "categories" in workspace:
            categories = workspace["categories"]
            workspace_category_data = self.prepare_workspace_create_form(
                categories=categories,
                template_id=template_id,
                ext_system_id=business_object["ext_system_id"],
                bo_type=business_object["bo_type"],
                bo_id=business_object["bo_id"],
                parent_workspace_node_id=parent_workspace_node_id,
            )
            if not workspace_category_data:
                self.logger.error(
                    "Failed to prepare the category data for workspace -> '%s'!",
                    name,
                )
                return False
        else:
            self.logger.debug(
                "Workspace payload has no category data! Will leave category attributes empty...",
            )
            workspace_category_data = {}

        if ibo_workspace_id:
            self.logger.info(
                "Connect existing workspace -> '%s' to an additional business object of type -> '%s' and ID -> '%s' (IBO)...",
                name,
                business_object["bo_type"],
                business_object["bo_id"],
            )
        # Create the workspace with all provided information:
        response = self._otcs.create_workspace(
            workspace_template_id=template_id,
            workspace_name=name,
            workspace_description=description,
            workspace_type=workspace_type_id,
            category_data=workspace_category_data,
            external_system_id=business_object["ext_system_id"],
            bo_type=business_object["bo_type"],
            bo_id=business_object["bo_id"],
            parent_id=parent_workspace_node_id,
            ibo_workspace_id=ibo_workspace_id,
            show_error=(
                not self._sap
            ),  # if SAP is active it may produce workspaces concurrently (race condition). Then we don't want to issue errors.
        )
        if response is None:
            # Check if workspace has been concurrently created by some other
            # process (e.g. via SAP or Salesforce). This would be a race condition
            # that seems to really occur.

            # we wait a bit to give the concurrent thread the chance to fully finish the creation:
            time.sleep(5)

            workspace_id = self.determine_workspace_id(workspace=workspace)
            if workspace_id:
                self.logger.info(
                    "Workspace -> '%s' of type -> '%s' has been created by an external process and has ID -> %s.",
                    name,
                    type_name,
                    workspace_id,
                )
                workspace["nodeId"] = workspace_id
            else:
                self.logger.error(
                    "Failed to create workspace -> '%s' of type -> '%s'!",
                    name,
                    type_name,
                )
                return False
        # Now we add the node ID of the new workspace to the payload data structure
        # This will be reused when creating the workspace relationships!
        elif not ibo_workspace_id:
            workspace["nodeId"] = self._otcs.get_result_value(
                response=response,
                key="id",
            )
            workspace_id = workspace["nodeId"]
            ibo_workspace_id = workspace["nodeId"]

            # We also get the name the workspace was finally created with.
            # This can be different form the name in the payload as additional
            # naming conventions from the Workspace Type definitions may apply.
            # This is important to make the python container idem-potent.
            response = self._otcs.get_workspace(node_id=workspace["nodeId"])
            workspace["name"] = self._otcs.get_result_value(
                response=response,
                key="name",
            )
            # Also update the 'name' variable accordingly, as it is used below.
            name = workspace["name"]

            self.logger.info(
                "Successfully created workspace with final name -> '%s', type -> '%s', and node ID -> %s.",
                workspace["name"],
                type_name,
                workspace["nodeId"],
            )
    # end for business_object in business_object_list

    # if the workspace creation has failed - e.g. error in lookup of business
    # object in external system then we continue with the next workspace:
    if "nodeId" not in workspace:
        self.logger.error(
            "Couldn't create the workspace -> '%s'. Skipping to next workspace...",
            workspace["name"],
        )
        return False

    # Check if there's an workspace nickname configured:
    if "nickname" in workspace:
        nickname = workspace["nickname"]
        self.logger.info(
            "Assign nickname -> '%s' to workspace -> '%s' (%s)...",
            nickname,
            name,
            workspace["nodeId"],
        )
        response = self._otcs.set_node_nickname(
            node_id=workspace["nodeId"],
            nickname=nickname,
            show_error=True,
        )
        if not response:
            self.logger.error(
                "Failed to assign nickname -> '%s' to workspace -> '%s' (%s)!",
                nickname,
                name,
                workspace["nodeId"],
            )

    # Check if there's an workspace icon/image configured:
    if "image_nickname" in workspace:
        image_nickname = workspace["image_nickname"]

        response = self._otcs.get_node_from_nickname(nickname=image_nickname)
        node_id = self._otcs.get_result_value(response=response, key="id")
        if node_id:
            mime_type = self._otcs.get_result_value(
                response=response,
                key="mime_type",
            )
            if not mime_type:
                self.logger.warning(
                    "Missing mime type information - assuming 'image/png'...",
                )
                mime_type = "image/png"
            file_path = os.path.join(tempfile.gettempdir(), image_nickname)
            result = self._otcs.download_document(node_id=node_id, file_path=file_path)
            if not result:
                self.logger.error(
                    "Failed to download workspace image with nickname -> '%s' to file -> '%s'!",
                    image_nickname,
                    file_path,
                )
            else:
                response = self._otcs.update_workspace_icon(
                    workspace_id=workspace["nodeId"],
                    file_path=file_path,
                    file_mimetype=mime_type,
                )
                if not response:
                    self.logger.error(
                        "Failed to assign icon -> '%s' to workspace -> '%s' from file -> '%s'!",
                        image_nickname,
                        name,
                        file_path,
                    )
        else:
            self.logger.error(
                "Cannot find workspace image with nickname -> '%s' for workspace -> '%s'!",
                image_nickname,
                name,
            )

    # Check if an RM classification is specified for the workspace:
    # RM Classification is specified as list of path elements (top-down)
    if "rm_classification_path" in workspace and workspace["rm_classification_path"] != []:
        rm_class_node = self._otcs.get_node_by_volume_and_path(
            volume_type=self._otcs.VOLUME_TYPE_CLASSIFICATION_VOLUME,
            path=workspace["rm_classification_path"],
        )
        rm_class_node_id = self._otcs.get_result_value(
            response=rm_class_node,
            key="id",
        )
        if rm_class_node_id:
            response = self._otcs.assign_rm_classification(
                node_id=workspace["nodeId"],
                rm_classification=rm_class_node_id,
                apply_to_sub_items=False,
            )
            if response is None:
                self.logger.error(
                    "Failed to assign RM classification -> '%s' (%s) to workspace -> '%s'!",
                    workspace["rm_classification_path"][-1],
                    rm_class_node_id,
                    name,
                )
            else:
                self.logger.info(
                    "Assigned RM Classification -> '%s' to workspace -> '%s'.",
                    workspace["rm_classification_path"][-1],
                    name,
                )
    # Check if one or multiple classifications are specified for the workspace
    # Classifications are specified as list of path elements (top-down)
    if "classification_pathes" in workspace and workspace["classification_pathes"] != []:
        for classification_path in workspace["classification_pathes"]:
            class_node = self._otcs.get_node_by_volume_and_path(
                volume_type=self._otcs.VOLUME_TYPE_CLASSIFICATION_VOLUME,
                path=classification_path,
            )
            class_node_id = self._otcs.get_result_value(
                response=class_node,
                key="id",
            )
            if class_node_id:
                response = self._otcs.assign_classifications(
                    node_id=workspace["nodeId"],
                    classifications=[class_node_id],
                    apply_to_sub_items=False,
                )
                if response is None:
                    self.logger.error(
                        "Failed to assign classification -> '%s' to workspace -> '%s'!",
                        class_node_id,
                        name,
                    )
                else:
                    self.logger.info(
                        "Successfully assigned Classification -> '%s' to workspace -> '%s'.",
                        classification_path[-1],
                        name,
                    )

    return True

process_workspace_aviators(section_name='workspaceAviators')

Process workspaces Content Aviator settings in payload and enable Aviator for selected workspaces.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'workspaceAviators'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_aviators")
def process_workspace_aviators(
    self,
    section_name: str = "workspaceAviators",
) -> bool:
    """Process workspaces Content Aviator settings in payload and enable Aviator for selected workspaces.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._workspaces:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for workspace in self._workspaces:
        # Read name from payload (just for logging):
        workspace_name = workspace.get("name")
        if not workspace_name:
            continue

        # Check if workspace has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not workspace.get("enabled", True):
            self.logger.info(
                "Payload for workspace -> '%s' is disabled. Skipping...",
                workspace_name,
            )
            continue

        # Read Aviator setting from payload:
        if not workspace.get("enable_aviator", False):
            self.logger.info(
                "Aviator is not enabled for workspace -> '%s'. Skipping to next workspace...",
                workspace_name,
            )
            continue

        # We cannot just lookup with workspace.get("nodeId") as the customizer
        # may have been restarted inbetween - so we use our proper determine_workspace_id
        # here...
        workspace_id = self.determine_workspace_id(workspace=workspace)
        if not workspace_id:
            self.logger.error(
                "Cannot find node ID for workspace -> '%s'. Workspace creation may have failed. Skipping to next workspace...",
                workspace_name,
            )
            success = False
            continue

        # Make code idem-potent and check if Aviator is already enabled
        # for this workspace:
        if self._otcs.check_workspace_aviator(workspace_id=workspace_id):
            self.logger.info(
                "Skip workspace -> '%s' (%s) as Aviator is already enabled...",
                workspace_name,
                workspace_id,
            )
            continue

        # Now enable the Content Aviator for the workspace:
        response = self._otcs.update_workspace_aviator(
            workspace_id=workspace_id,
            status=True,
        )
        if not response:
            self.logger.error(
                "Failed to enable Content Aviator for workspace -> '%s' (%s)!",
                workspace_name,
                workspace_id,
            )
            success = False
            continue
        # Also embed the workspace metadata:
        if not self._otcs.aviator_embed_metadata(
            node_id=workspace_id,
            workspace_metadata=True,
            wait_for_completion=False,
        ):
            success = False
            continue
    # end for workspace in self._workspaces

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._workspaces,
    )

    return success

process_workspace_member_permissions(section_name='workspaceMemberPermissions')

Process workspaces members in payload and set their permissions.

We need this separate from process_workspace_members() which also sets permissions (if in payload) as we add documents to workspaces with content transports and these documents don't inherit role permissions (this is a transport limitation)

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'workspaceMemberPermissions'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_member_permissions")
def process_workspace_member_permissions(
    self,
    section_name: str = "workspaceMemberPermissions",
) -> bool:
    """Process workspaces members in payload and set their permissions.

    We need this separate from process_workspace_members() which also
    sets permissions (if in payload) as we add documents to workspaces with
    content transports and these documents don't inherit role permissions
    (this is a transport limitation)

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._workspaces:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for workspace in self._workspaces:
        # Read name from payload (just for logging):
        workspace_name = workspace.get("name")
        if not workspace_name:
            continue

        # Check if workspace has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not workspace.get("enabled", True):
            self.logger.info(
                "Payload for workspace -> '%s' is disabled. Skipping...",
                workspace_name,
            )
            continue

        # Read members from payload:
        members = workspace.get("members")
        if not members:
            self.logger.info(
                "Workspace -> '%s' has no members in payload. No need to update permissions. Skipping to next workspace...",
                workspace_name,
            )
            continue

        workspace_id = workspace["id"]
        workspace_node_id = int(self.determine_workspace_id(workspace=workspace))
        if not workspace_node_id:
            self.logger.warning(
                "Workspace without node ID cannot cannot get permission changes (workspaces creation may have failed). Skipping to next workspace...",
            )
            continue

        workspace_roles = self._otcs.get_workspace_roles(
            workspace_id=workspace_node_id,
        )
        if workspace_roles is None:
            self.logger.info(
                "Workspace with payload ID -> %s and node Id -> %s has no roles to update permissions. Skipping to next workspace...",
                workspace_id,
                workspace_node_id,
            )
            continue

        for member in members:
            # read user list and role name from payload:
            member_users = (
                member["users"] if member.get("users") else []
            )  # be careful to avoid key errors as users are optional
            member_groups = (
                member["groups"] if member.get("groups") else []
            )  # be careful to avoid key errors as groups are optional
            member_role_name = member["role"]

            if member_role_name == "":  # role name is required
                self.logger.error(
                    "Members of workspace -> '%s' is missing the role name.",
                    workspace_name,
                )
                success = False
                continue
            if member_users == [] and member_groups == []:  # we either need users or groups (or both)
                self.logger.debug(
                    "Role -> '%s' of workspace -> '%s' does not have any members (no users nor groups).",
                    member_role_name,
                    workspace_name,
                )
                continue

            role_id = self._otcs.lookup_result_value(
                response=workspace_roles,
                key="name",
                value=member_role_name,
                return_key="id",
            )
            if role_id is None:
                self.logger.error(
                    "Workspace -> '%s' does not have a role -> '%s'",
                    workspace_name,
                    member_role_name,
                )
                success = False
                continue
            self.logger.debug(
                "Role -> '%s' has ID -> %s",
                member_role_name,
                role_id,
            )

            member_permissions = member.get("permissions", [])
            if member_permissions == []:
                self.logger.debug(
                    "No permission change for workspace -> '%s' and role -> '%s'.",
                    workspace_name,
                    member_role_name,
                )
                continue

            self.logger.info(
                "Update permissions of workspace -> '%s' (%s) and role -> '%s' to -> %s...",
                workspace_name,
                str(workspace_node_id),
                member_role_name,
                str(member_permissions),
            )
            response = self._otcs.assign_permission(
                node_id=workspace_node_id,
                assignee_type="custom",
                assignee=role_id,
                permissions=member_permissions,
                apply_to=2,  # inherit to all sub-folders
            )
            if not response:
                self.logger.error(
                    "Failed to update permissions of workspace -> '%s' (%s) and role -> '%s' to -> %s!",
                    workspace_name,
                    str(workspace_node_id),
                    member_role_name,
                    str(member_permissions),
                )
                success = False

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._workspaces,
    )

    return success

process_workspace_members(section_name='workspaceMembers')

Process workspaces members in payload and create them in Content Server.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'workspaceMembers'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_members")
def process_workspace_members(self, section_name: str = "workspaceMembers") -> bool:
    """Process workspaces members in payload and create them in Content Server.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._workspaces:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for workspace in self._workspaces:
        # Read name from payload (just for logging):
        if "name" not in workspace:
            continue
        workspace_name = workspace["name"]

        # Check if workspace has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not workspace.get("enabled", True):
            self.logger.info(
                "Payload for workspace -> '%s' is disabled. Skipping...",
                workspace_name,
            )
            continue

        # Read members from payload:
        members = workspace.get("members")
        if not members:
            self.logger.info(
                "Workspace -> '%s' has no members in payload. Skipping to next workspace...",
                workspace_name,
            )
            continue

        workspace_id = workspace["id"]

        workspace_node_id = int(self.determine_workspace_id(workspace=workspace))
        if not workspace_node_id:
            self.logger.warning(
                "Workspace without node ID cannot have members (workspaces creation may have failed). Skipping to next workspace...",
            )
            continue

        self.logger.info(
            "Workspace -> '%s' (%s) has memberships in payload - establishing...", workspace_name, workspace_node_id
        )

        # now determine the actual node IDs of the workspaces (have been created by process_workspaces()):
        workspace_node = self._otcs.get_node(node_id=workspace_node_id)
        workspace_owner_id = self._otcs.get_result_value(
            response=workspace_node,
            key="owner_user_id",
        )
        workspace_owner_name = self._otcs.get_result_value(
            response=workspace_node,
            key="owner",
        )

        workspace_roles = self._otcs.get_workspace_roles(
            workspace_id=workspace_node_id,
        )
        if workspace_roles is None:
            self.logger.debug(
                "Workspace -> '%s' (%s) has no roles. Skipping to next workspace...",
                workspace_name,
                workspace_node_id,
            )
            continue

        # We don't want the workspace creator to be in the leader role
        # of automatically created workspaces - this can happen because the
        # creator gets added to the leader role automatically if
        # the workspace type advanved configuration setting
        # "Add the creator of a business workspace to the Lead role" is
        # enabled:
        roles_iterator = self._otcs.get_result_values_iterator(response=workspace_roles)
        for role in roles_iterator:
            # We can have two leader roles if in a sub-workspaces a leader
            # roles is inherited from the parent workspace. As we want
            # don't want to consider leader role of the parent workspace
            # we check that 'inherited_from_id' is not set:
            if role["leader"] and role["inherited_from_id"] is None:
                leader_role_id = role["id"]
                break
        else:
            leader_role_id = None

        if leader_role_id:
            leader_role_name = self._otcs.lookup_result_value(
                response=workspace_roles,
                key="leader",
                value=True,
                return_key="name",
            )
            response = self._otcs.remove_workspace_member(
                workspace_id=workspace_node_id,
                role_id=leader_role_id,
                member_id=workspace_owner_id,
                show_warning=False,
            )
            if response:
                self.logger.info(
                    "Removed creator user -> '%s' (%s) from leader role -> '%s' (%s) of workspace -> '%s' (%s).",
                    workspace_owner_name,
                    workspace_owner_id,
                    leader_role_name,
                    leader_role_id,
                    workspace_name,
                    workspace_node_id,
                )
            else:
                self.logger.info(
                    "Creator user -> '%s' (%s) is not in leader role -> '%s' (%s) of workspace -> '%s' (%s). No need to remove it.",
                    workspace_owner_name,
                    workspace_owner_id,
                    leader_role_name,
                    leader_role_id,
                    workspace_name,
                    workspace_node_id,
                )

        self.logger.info(
            "Adding members to workspace -> '%s' (%s) defined in payload...",
            workspace_name,
            workspace_node_id,
        )

        for member in members:
            # read user list and role name from payload:
            member_users = member.get("users", [])
            member_groups = member.get("groups", [])
            member_role_name = member.get("role", "")
            member_clear = member.get("clear", False)

            if member_role_name == "":  # role name is required
                self.logger.error(
                    "Members of workspace -> '%s' is missing the role name in the payload.",
                    workspace_name,
                )
                success = False
                continue

            role_id = self._otcs.lookup_result_value(
                response=workspace_roles,
                key="name",
                value=member_role_name,
                return_key="id",
            )
            if role_id is None:
                #    if member_role is None:
                self.logger.error(
                    "Workspace -> '%s' does not have a role -> '%s'",
                    workspace_name,
                    member_role_name,
                )
                success = False
                continue
            inherited_role_id = self._otcs.lookup_result_value(
                response=workspace_roles,
                key="name",
                value=member_role_name,
                return_key="inherited_from_id",
            )
            if inherited_role_id is not None:
                self.logger.error(
                    "The role -> '%s' (%s) of workspace -> '%s' (%s) is inherited from role with ID -> %d and members cannot be set in this sub-workspace.",
                    member_role_name,
                    role_id,
                    workspace_name,
                    workspace_node_id,
                    inherited_role_id,
                )
                success = False
                continue
            self.logger.debug(
                "Role -> '%s' has ID -> %s",
                member_role_name,
                role_id,
            )

            # check if we want to clear (remove) existing members of this role:
            if member_clear:
                self.logger.info(
                    "Clear existing members of role -> '%s' (%s) for workspace -> '%s' (%s)...",
                    member_role_name,
                    role_id,
                    workspace_name,
                    workspace_id,
                )
                self._otcs.remove_workspace_members(
                    workspace_id=workspace_node_id,
                    role_id=role_id,
                )

            if member_users == [] and member_groups == []:  # we either need users or groups (or both)
                self.logger.debug(
                    "Role -> '%s' of workspace -> '%s' does not have any members (no users nor groups).",
                    member_role_name,
                    workspace_name,
                )
                continue

            # Process users as workspaces members:
            for member_user in member_users:
                # find member user in current payload:
                member_user_id = next(
                    (item for item in self._users if item["name"] == member_user),
                    {"name": member_user},  # we construct a payload on the fly to make determine_user_id() work
                )
                user_id = self.determine_user_id(user=member_user_id)
                if not user_id:
                    self.logger.error(
                        "Cannot find member user with login -> '%s'. Skipping...",
                        member_user,
                    )
                    continue

                # Add member if it does not yet exists - suppress warning
                # message if user is already in role:
                response = self._otcs.add_workspace_member(
                    workspace_id=workspace_node_id,
                    role_id=int(role_id),
                    member_id=user_id,
                    show_warning=False,
                )
                if response is None:
                    self.logger.error(
                        "Failed to add user -> '%s' (%s) as member to role -> '%s' of workspace -> '%s' (%s)!",
                        member_user,
                        user_id,
                        member_role_name,
                        workspace_name,
                        workspace_node_id,
                    )
                    success = False
                else:
                    self.logger.info(
                        "Successfully added user -> '%s' (%s) as member to role -> '%s' of workspace -> '%s' (%s).",
                        member_user,
                        user_id,
                        member_role_name,
                        workspace_name,
                        workspace_node_id,
                    )

            # Process groups as workspaces members:
            for member_group in member_groups:
                member_group_id = next(
                    (item for item in self._groups if item["name"] == member_group),
                    None,
                )

                group_id = self.determine_group_id(group=member_group_id) if member_group_id else None
                if not group_id:
                    self.logger.error(
                        "Cannot find member group -> '%s'. Skipping...",
                        member_group,
                    )
                    success = False
                    continue

                response = self._otcs.add_workspace_member(
                    workspace_id=workspace_node_id,
                    role_id=int(role_id),
                    member_id=group_id,
                    show_warning=False,
                )
                if response is None:
                    self.logger.error(
                        "Failed to add group -> '%s' (%s) to role -> '%s' of workspace -> '%s'!",
                        member_group_id["name"],
                        group_id,
                        member_role_name,
                        workspace_name,
                    )
                    success = False
                else:
                    self.logger.info(
                        "Successfully added group -> '%s' (%s) to role -> '%s' of workspace -> '%s'.",
                        member_group_id["name"],
                        group_id,
                        member_role_name,
                        workspace_name,
                    )

            # Optionally the payload may have a permission list for the role
            # to change the default permission from the workspace template
            # to something more specific:
            member_permissions = member.get("permissions", [])
            if member_permissions == []:
                self.logger.debug(
                    "No permission change for workspace -> '%s' and role -> '%s'",
                    workspace_name,
                    member_role_name,
                )
                continue

            self.logger.info(
                "Update permissions of workspace -> '%s' (%s) and role -> '%s' to -> %s...",
                workspace_name,
                str(workspace_node_id),
                member_role_name,
                str(member_permissions),
            )
            response = self._otcs.assign_permission(
                node_id=workspace_node_id,
                assignee_type="custom",
                assignee=role_id,
                permissions=member_permissions,
                apply_to=2,
            )
            if not response:
                self.logger.error(
                    "Failed to update permissions of workspace -> '%s' (%s) and role -> '%s' to -> %s!",
                    workspace_name,
                    str(workspace_node_id),
                    member_role_name,
                    str(member_permissions),
                )
                success = False
        # end for member in members:
    # end for workspace in self._workspaces:

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._workspaces,
    )

    return success

process_workspace_permissions(section_name='workspacePermissions')

Process items specified in payload and upadate workspace permissions.

Parameters:

Name	Type	Description	Default
workspace_permissions	list	List of items to apply permissions to. Each list item in the payload is a dict with this structure: { workspace_type = "..." workspace_folder = "..." regex = True public_permissions = ["see", "see_content", ...] owner_permissions = [] owner_group_permissions = [] groups = [ { name = "..." permissions = [] } ] users = [ { name = "..." permissions = [] } ] roles = [ { name = "..." permissions = [] } ] apply_to = 2 }	required
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections like "permissionsPost"). This name is also used for the "success" status files written to the Admin Personal Workspace.	'workspacePermissions'

Returns:

Name	Type	Description
bool	bool	True if payload has been processed without errors, False otherwise

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_permissions")
def process_workspace_permissions(
    self,
    section_name: str = "workspacePermissions",
) -> bool:
    """Process items specified in payload and upadate workspace permissions.

    Args:
        workspace_permissions (list):
            List of items to apply permissions to.
            Each list item in the payload is a dict with this structure:
            {
                workspace_type = "..."
                workspace_folder = "..."
                regex = True
                public_permissions = ["see", "see_content", ...]
                owner_permissions = []
                owner_group_permissions = []
                groups = [
                    {
                        name = "..."
                        permissions = []
                    }
                ]
                users = [
                    {
                        name = "..."
                        permissions = []
                    }
                ]
                roles = [
                    {
                        name = "..."
                        permissions = []
                    }
                ]
                apply_to = 2
            }
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections like "permissionsPost").
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True if payload has been processed without errors, False otherwise

    """

    if not self._workspace_permissions:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for workspace_permission in self._workspace_permissions:
        workspace_type_name = workspace_permission.get("workspace_type")
        if not workspace_type_name:
            self.logger.error(
                "Workspaces type to change permissions is not specified. Skipping...",
            )
            success = False
            continue

        # Check if workspace permission has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not workspace_permission.get("enabled", True):
            self.logger.info(
                "Payload for workspace permission for workspace type -> '%s' is disabled. Skipping...",
                workspace_type_name,
            )
            continue

        workspace_folder_name = workspace_permission.get("workspace_folder", None)
        regex = workspace_permission.get("regex", False)

        # Make item + sub-items (2) the default:
        apply_to = workspace_permission.get("apply_to", 2)

        # Find the workspace type with the name given in the payload:
        workspace_type_id = next(
            (item["id"] for item in self._workspace_types if item["name"] == workspace_type_name),
            None,
        )
        if workspace_type_id is None:
            self.logger.error(
                "Workspace type -> '%s' not found. Skipping...",
                workspace_type_name,
            )
            success = False
            continue

        # The workspace type name is used as a "starts with" search on the
        # workspace type name. This can cause issues, so we prefer to use the type ID
        # if we have it. Otherwise the REST API prefers the name over the type:
        workspace_instances = self._otcs.get_workspace_instances_iterator(
            type_name=workspace_type_name if not workspace_type_id else None,
            type_id=workspace_type_id,
        )
        for workspace_instance in workspace_instances:
            workspace = workspace_instance.get("data").get("properties")
            workspace_id = workspace["id"]
            workspace_name = workspace["name"]
            if workspace_folder_name:
                if not regex:
                    workspace_folder = self._otcs.get_node_by_parent_and_name(
                        parent_id=workspace_id,
                        name=workspace_folder_name,
                    )
                else:
                    workspace_folder = self._otcs.lookup_node_by_regex(
                        parent_node_id=workspace_id,
                        regex_list=[workspace_folder_name],
                    )
                workspace_folder_id = self._otcs.get_result_value(
                    response=workspace_folder,
                    key="id",
                )
                if not workspace_folder_id:
                    self.logger.info(
                        "Workspace folder name -> '%s' was not found in workspace -> '%s' (%s). Skipping this workspace...",
                        workspace_folder_name,
                        workspace_name,
                        workspace_id,
                    )
                    continue
            else:
                # if no folder is specified we apply the permission on the workspace
                workspace_folder_id = workspace_id
                workspace_folder_name = workspace_name

            # Process a single workspace permission payload item:
            if not self.process_permission(
                node_id=workspace_folder_id,
                node_name=workspace_folder_name,
                permission=workspace_permission,
                apply_to=apply_to,
            ):
                success = False
                continue
        # end for workspace_instance in workspace_instances:

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._workspace_permissions,
    )

    return success

process_workspace_relationship(workspace)

Worker thread for workspace relationship creation.

Parameters:

Name	Type	Description	Default
workspace	dict	Dictionary with payload of a single workspace.	required

Returns:

Name	Type	Description
bool	bool	True = Success, False = Failure

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_relationship")
def process_workspace_relationship(self, workspace: dict) -> bool:
    """Worker thread for workspace relationship creation.

    Args:
        workspace (dict):
            Dictionary with payload of a single workspace.

    Returns:
        bool:
            True = Success, False = Failure

    """

    # Read name from payload. If it does not exist then bail out:
    name = workspace.get("name")
    if not name:
        return False

    # Check if element has been explicitly disabled in payload
    # (enabled = false). In this case we skip the element:
    if not workspace.get("enabled", True):
        self.logger.info(
            "Payload for workspace -> '%s' is disabled. Skipping...",
            name,
        )
        return True

    # Read relationships from payload:
    if "relationships" not in workspace:
        self.logger.debug(
            "Workspace -> '%s' has no relationships. Skipping to next workspace...",
            name,
        )
        return True

    # Check that workspaces actually have a logical ID -
    # otherwise we cannot establish the relationship:
    workspace_id = workspace.get("id")
    if not workspace_id:
        self.logger.warning(
            "Workspace without logical ID in payload cannot have a relationship. Skipping to next workspace...",
        )
        return False

    self.logger.info(
        "Workspace -> '%s' (type -> '%s') has relationships - creating...",
        name,
        workspace["type_name"],
    )

    # now determine the actual node ID of the workspace (which should have been created before):
    workspace_node_id = int(self.determine_workspace_id(workspace=workspace))
    if not workspace_node_id:
        self.logger.warning(
            "Workspace -> '%s' (type -> '%s') has no node ID and cannot have a relationship (workspace creation may have failed or final name is different from payload). Skipping to next workspace...",
            name,
            workspace["type_name"],
        )
        return False

    self.logger.debug(
        "Workspace with logical ID -> %s has node ID -> %s",
        str(workspace_id),
        str(workspace_node_id),
    )

    success: bool = True

    for related_workspace in workspace["relationships"]:
        # Initialize variable to determine if we found a related workspace:
        related_workspace_node_id = None
        found_by = ""

        if isinstance(related_workspace, (str, int)):
            #
            # 1. Option: Find the related workspace with the logical ID given in the payload:
            #
            related_workspace_payload = next(
                (item for item in self._workspaces if str(item["id"]) == str(related_workspace)),
                None,
            )
            if related_workspace_payload:
                if not related_workspace_payload.get("enabled", True):
                    self.logger.info(
                        "Payload for related workspace -> '%s' is disabled. Skipping...",
                        related_workspace_payload["name"],
                    )
                    continue

                related_workspace_node_id = self.determine_workspace_id(
                    workspace=related_workspace_payload,
                )
                if not related_workspace_node_id:
                    self.logger.warning(
                        "Related workspace -> '%s' (type -> '%s') has no node ID (workspaces creation may have failed or name is different from payload). Skipping to next workspace...",
                        related_workspace_payload["name"],
                        related_workspace_payload["type_name"],
                    )
                    continue
                found_by = "logical ID -> '{}' in payload".format(related_workspace)
            # end if related_workspace_payload:

            #
            # 2. Option: Find the related workspace with nickname:
            #
            else:
                # See if a nickname exists the the provided related_workspace:
                response = self._otcs.get_node_from_nickname(nickname=related_workspace)
                related_workspace_node_id = self._otcs.get_result_value(
                    response=response,
                    key="id",
                )
                if related_workspace_node_id:
                    found_by = "nickname -> '{}'".format(related_workspace)
        # end if isinstance(related_workspace_id, (str, int)):

        #
        # 3. Option: Find the related workspace type and name:
        #
        elif isinstance(related_workspace, dict):
            related_workspace_type = related_workspace.get("type", None)
            related_workspace_name = related_workspace.get("name", None)
            if related_workspace_type and related_workspace_name:
                response = self._otcs.get_workspace_by_type_and_name(
                    type_name=related_workspace_type, name=related_workspace_name
                )
                related_workspace_node_id = self._otcs.get_result_value(
                    response=response,
                    key="id",
                )
                if related_workspace_node_id:
                    found_by = "type -> '{}' and name -> '{}'".format(
                        related_workspace_type, related_workspace_name
                    )
        #
        # 4. Option: Find the related workspace volume and path:
        #
        elif isinstance(related_workspace, list):
            response = self._otcs.get_node_by_volume_and_path(
                volume_type=self._otcs.VOLUME_TYPE_ENTERPRISE_WORKSPACE, path=related_workspace
            )
            related_workspace_node_id = self._otcs.get_result_value(
                response=response,
                key="id",
            )
            if related_workspace_node_id:
                found_by = "path -> {}".format(related_workspace)

        if related_workspace_node_id is None:
            self.logger.error(
                "Related workspace -> %s not found.",
                related_workspace,
            )
            success = False
            continue

        self.logger.debug(
            "Related workspace with %s has node ID -> %s",
            found_by,
            related_workspace_node_id,
        )

        # Check if relationship does already exists:
        response = self._otcs.get_workspace_relationships(
            workspace_id=workspace_node_id,
        )

        existing_workspace_relationship = self._otcs.exist_result_item(
            response=response,
            key="id",
            value=related_workspace_node_id,
        )
        if existing_workspace_relationship:
            self.logger.info(
                "Workspace relationship between workspace ID -> %s and related workspace ID -> %s does already exist. Skipping...",
                str(workspace_node_id),
                related_workspace_node_id,
            )
            continue

        self.logger.info(
            "Create workspace relationship between workspace node ID -> %s and workspace node ID -> %s...",
            str(workspace_node_id),
            related_workspace_node_id,
        )

        response = self._otcs.create_workspace_relationship(
            workspace_id=workspace_node_id,
            related_workspace_id=related_workspace_node_id,
        )
        if not response:
            self.logger.error("Failed to create workspace relationship!")
            success = False
        else:
            self.logger.info("Successfully created workspace relationship.")

    # end for relationships

    return success

process_workspace_relationships(section_name='workspaceRelationships')

Process workspaces relationships in payload and create them in Content Server.

Relationships can only be created if all workspaces have been created before. Once a workspace got created, the node ID of that workspaces has been added to the payload["workspaces"] data structure (see process_workspaces()) Relationships are created between the node IDs of two business workspaces (and not the logical IDs in the inital payload specification)

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'workspaceRelationships'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_relationships")
def process_workspace_relationships(
    self,
    section_name: str = "workspaceRelationships",
) -> bool:
    """Process workspaces relationships in payload and create them in Content Server.

    Relationships can only be created if all workspaces have been created before.
    Once a workspace got created, the node ID of that workspaces has been added
    to the payload["workspaces"] data structure (see process_workspaces())
    Relationships are created between the node IDs of two business workspaces
    (and not the logical IDs in the inital payload specification)

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    if not self._workspaces:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    if ENABLE_MULTI_THREADING:
        # Create a list to hold the threads
        threads = []
        # And another list to collect the results
        results = []

        df = Data(self._workspaces, logger=self.logger)

        partitions: list = df.partitionate(number=THREAD_NUMBER)

        # Create and start a thread for each partition
        for index, partition in enumerate(partitions, start=1):
            # start a thread executing the process_workspace_relationships_worker() method below:
            thread = threading.Thread(
                name=f"{section_name}_{index:02}",
                target=self.thread_wrapper,
                args=(
                    self.process_workspace_relationships_worker,
                    partition,
                    results,
                ),
            )
            self.logger.info("Starting thread -> '%s'...", str(thread.name))
            thread.start()
            threads.append(thread)
            # Avoid that all threads start at the exact same time with
            # potentially expired cookies that cases race conditions:
            time.sleep(1)
        # end for index, partition in enumerate(partitions, start=1)

        # Wait for all threads to complete
        for thread in threads:
            self.logger.info(
                "Waiting for thread -> '%s' to complete...",
                str(thread.name),
            )
            thread.join()
            self.logger.info("Thread -> '%s' has completed.", str(thread.name))

        # Print statistics for each thread. In addition,
        # check if all threads have completed without error / failure.
        # If there's a single failure in on of the thread results we
        # set 'success' variable to False.
        results.sort(key=lambda x: x["thread_name"])
        for result in results:
            if not result["success"]:
                self.logger.info(
                    "Thread -> '%s' had %s failures and completed relationships for %s workspaces successfully.",
                    result["thread_name"],
                    result["failure_counter"],
                    result["success_counter"],
                )
                success = False  # mark the complete processing as "failure" for the status file.
            else:
                self.logger.info(
                    "Thread -> '%s' completed relationships for %s workspace successfully.",
                    result["thread_name"],
                    result["success_counter"],
                )
    else:  # no multi-threading
        for workspace in self._workspaces:
            result = self.process_workspace_relationship(workspace=workspace)
            success = (
                success and result
            )  # if a single result is False then the 'success' variable becomes 'False' as well.

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._workspaces,
    )

    return success

process_workspace_relationships_worker(partition, results)

Multi-threading worker method to process a partition of the workspaces to create workspace relationships.

Parameters:

Name	Type	Description	Default
partition	DataFrame	The partition of the workspace relationships to process by this thread.	required
results	list	A mutable (shared) list of all workers to collect the results.	required

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_relationships_worker")
def process_workspace_relationships_worker(
    self,
    partition: pd.DataFrame,
    results: list,
) -> None:
    """Multi-threading worker method to process a partition of the workspaces to create workspace relationships.

    Args:
        partition (pd.DataFrame):
            The partition of the workspace relationships to process by this thread.
        results (list):
            A mutable (shared) list of all workers to collect the results.

    """

    thread_id = threading.get_ident()
    thread_name = threading.current_thread().name

    result = {}
    result["thread_id"] = thread_id
    result["thread_name"] = thread_name
    result["success"] = True
    result["failure_counter"] = 0
    result["success_counter"] = 0

    total_rows = len(partition)

    # Process all datasets in the partion that was given to the thread:
    for index, row in partition.iterrows():
        # Calculate percentage of completion
        percent_complete = ((partition.index.get_loc(index) + 1) / total_rows) * 100

        self.logger.info(
            "Processing data row -> %s to create relationships for workspace -> '%s' (%.2f %% complete)...",
            str(index),
            row["name"],
            percent_complete,
        )
        success = self.process_workspace_relationship(
            workspace=row.dropna().to_dict(),
        )
        if success:
            result["success_counter"] += 1
        else:
            self.logger.error(
                "Failed to process row -> %s for relationships of workspace -> '%s'!",
                str(index),
                row["name"],
            )
            result["failure_counter"] += 1
            result["success"] = False

    results.append(result)

process_workspace_templates(section_name='workspaceTemplates')

Process workspace template playload.

This allows to define role members on template basis. This avoids having to "pollute" workspace templates with user or group information and instead controls this via payload.

This method also allows to assign (additional) categories to a workspace template controlled by payload.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'workspaceTemplates'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_templates")
def process_workspace_templates(
    self,
    section_name: str = "workspaceTemplates",
) -> bool:
    """Process workspace template playload.

    This allows to define role members on template basis.
    This avoids having to "pollute" workspace templates
    with user or group information and instead controls this via payload.

    This method also allows to assign (additional) categories to
    a workspace template controlled by payload.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    """

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        return True

    success: bool = True

    for workspace_template in self._workspace_templates:
        # Read Workspace Type Name from payload:
        type_name = workspace_template.get("type_name")
        if not type_name:
            self.logger.error(
                "Workspace template needs a type name! Skipping to next workspace template...",
            )
            success = False
            continue

        # Check if workspace template has been explicitly disabled in payload
        # (enabled = false). In this case we skip the element:
        if not workspace_template.get("enabled", True):
            self.logger.info(
                "Payload for workspace template -> '%s' is disabled. Skipping to next workspace template...",
                type_name,
            )
            continue

        # Read Workspace Template Name from payload:
        template_name = workspace_template.get("template_name")
        if not template_name:
            self.logger.error(
                "Workspace template for workspace type -> '%s' needs a template name! Skipping...",
                type_name,
            )
            success = False
            continue

        # Read members from payload:
        members = workspace_template.get("members", [])

        # Read categories from payload:
        categories = workspace_template.get("categories", [])

        # Find the workspace type with the name given in the _workspace_types
        # datastructure that has been generated by process_workspace_types() method before:
        workspace_type = next(
            (item for item in self._workspace_types if item["name"] == type_name),
            None,
        )
        if workspace_type is None:
            self.logger.error(
                "Workspace type -> '%s' not found. Skipping...",
                type_name,
            )
            success = False
            continue
        if workspace_type["templates"] == []:
            self.logger.error(
                "Workspace type -> '%s' does not have templates. Skipping...",
                type_name,
            )
            success = False
            continue

        workspace_template = next(
            (item for item in workspace_type["templates"] if item["name"] == template_name),
            None,
        )
        if workspace_template:  # does this template exist?
            self.logger.info(
                "Workspace template -> '%s' has been specified in payload and it does exist.",
                template_name,
            )
        else:
            self.logger.error(
                "Workspace template -> '%s' has been specified in payload but it doesn't exist!",
                template_name,
            )
            self.logger.error(
                "Workspace type -> '%s' has only these templates -> %s",
                type_name,
                workspace_type["templates"],
            )
            success = False
            continue

        template_id = workspace_template["id"]

        workspace_roles = self._otcs.get_workspace_roles(workspace_id=template_id)
        if workspace_roles is None:
            self.logger.info(
                "Workspace template -> '%s' with node Id -> %s has no roles. Skipping to next workspace...",
                template_name,
                template_id,
            )
            continue

        for member in members:
            # read user list, group list, and role name from payload:
            member_users = member.get("users", [])
            member_groups = member.get("groups", [])
            member_role_name = member.get("role", "")

            if not member_role_name:  # role name is required
                self.logger.error(
                    "Members of workspace template -> '%s' is missing the role name.",
                    template_name,
                )
                success = False
                continue
            if (
                member_users == [] and member_groups == []
            ):  # we either need users or groups (or both) in the payload
                self.logger.debug(
                    "Payload for workspace template -> '%s' and role -> '%s' does not have any members (no users nor groups).",
                    template_name,
                    member_role_name,
                )
                continue

            role_id = self._otcs.lookup_result_value(
                response=workspace_roles,
                key="name",
                value=member_role_name,
                return_key="id",
            )
            if role_id is None:
                #    if member_role is None:
                self.logger.error(
                    "Workspace template -> '%s' does not have a role -> '%s'",
                    template_name,
                    member_role_name,
                )
                success = False
                continue

            # Process users as workspace template members:
            for member_user in member_users:
                # find member user in current payload:
                member_user_id = next(
                    (item for item in self._users if item["name"] == member_user),
                    {"name": member_user},  # we construct a payload on the fly to make determine_user_id() work
                )
                user_id = self.determine_user_id(user=member_user_id) if member_user_id else None
                if not user_id:
                    self.logger.error(
                        "Cannot find member user with login -> '%s'. Skipping...",
                        member_user,
                    )
                    success = False
                    continue

                # Add member if it does not yet exists - suppress warning
                # message if user is already in role:
                response = self._otcs.add_workspace_member(
                    workspace_id=template_id,
                    role_id=int(role_id),
                    member_id=user_id,
                    show_warning=False,
                )
                if response is None:
                    self.logger.error(
                        "Failed to add user -> '%s' (%s) as member to role -> '%s' (%s) of workspace template -> '%s' (%s)!",
                        member_user,
                        user_id,
                        member_role_name,
                        role_id,
                        template_name,
                        template_id,
                    )
                    success = False
                else:
                    self.logger.info(
                        "Successfully added user -> '%s' (%s) as member to role -> '%s' (%s) of workspace template -> '%s' (%s).",
                        member_user,
                        user_id,
                        member_role_name,
                        role_id,
                        template_name,
                        template_id,
                    )
            # end for member_user in member_users:

            # Process groups as workspace template members:
            for member_group in member_groups:
                member_group_id = next(
                    (item for item in self._groups if item["name"] == member_group),
                    None,
                )

                group_id = self.determine_group_id(group=member_group_id) if member_group_id else None
                if not group_id:
                    self.logger.error(
                        "Cannot find member group -> '%s'. Skipping...",
                        member_group,
                    )
                    success = False
                    continue

                response = self._otcs.add_workspace_member(
                    workspace_id=template_id,
                    role_id=int(role_id),
                    member_id=group_id,
                    show_warning=False,
                )
                if response is None:
                    self.logger.error(
                        "Failed to add group -> '%s' (%s) as member to role -> '%s' (%s) of workspace template -> '%s' (%s)!",
                        member_group_id["name"],
                        group_id,
                        member_role_name,
                        role_id,
                        template_name,
                        template_id,
                    )
                    success = False
                else:
                    self.logger.info(
                        "Successfully added group -> '%s' (%s) as member to role -> '%s' (%s) of workspace template -> '%s' (%s).",
                        member_group_id["name"],
                        group_id,
                        member_role_name,
                        role_id,
                        template_name,
                        template_id,
                    )
            # end for member_group in member_groups:
        # end for member in members:

        existing_template_categories = None
        for category in categories:
            category_path = category.get("path")
            category_nickname = category.get("nickname")
            inheritance = category.get("inheritance")
            apply_to_sub_items = category.get("apply_to_sub_items")
            category_id = None

            if not category_nickname and not category_path:
                self.logger.error(
                    "Category assignment to workspace template needs eith the category nickname or the category path in the category volume!",
                )
                success = False
                continue
            if category_path:
                response = self._otcs.get_node_by_volume_and_path(
                    volume_type=self._otcs.VOLUME_TYPE_CATEGORIES_VOLUME,
                    path=category_path,
                )
                category_id = self._otcs.get_result_value(response=response, key="id")
            elif category_nickname:
                response = self._otcs.get_node_from_nickname(nickname=category_nickname, show_error=False)
                category_id = self._otcs.get_result_value(response=response, key="id")
            if not category_id:
                self.logger.error(
                    "Cannot find category for workspace template -> '%s' (%s). Tried category %s.",
                    template_name,
                    template_id,
                    "path {}".format(category_path) if category_path else "nickname {}".format(category_nickname),
                )
                success = False
                continue
            if existing_template_categories is None:
                existing_template_categories = self._otcs.get_node_category_ids(node_id=template_id)
            if category_id not in existing_template_categories:
                result = self._otcs.assign_category(
                    node_id=template_id,
                    category_id=category_id,
                    inheritance=inheritance,
                    apply_to_sub_items=apply_to_sub_items,
                )
                if not result:
                    self.logger.error(
                        "Cannot assign category with ID -> %s to workspace template -> '%s' (%s)!",
                        category_id,
                        template_name,
                        template_id,
                    )
                    success = False
                    continue
                self.logger.info(
                    "Successfully assigned category with ID -> %s to workspace template -> '%s' (%s).",
                    category_id,
                    template_name,
                    template_id,
                )
                # Write ID back into the payload dictionary:
                category["id"] = category_id
            else:
                self.logger.info(
                    "Category with ID -> %s is already assigned to workspace template -> '%s' (%s).",
                    category_id,
                    template_name,
                    template_id,
                )

        # end for category in categories:
    # end for workspace_template in self._workspace_templates:

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._workspace_templates,
    )

    return success

process_workspace_types(section_name='workspaceTypes')

Create a data structure for all workspace types in the OTCS.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'workspaceTypes'

Returns:

Name	Type	Description
list	list	A list of workspace types. Each list element is a dict with these values: - id (str) - name (str) - templates (list) + name (str) + id

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspace_types")
def process_workspace_types(self, section_name: str = "workspaceTypes") -> list:
    """Create a data structure for all workspace types in the OTCS.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        list:
            A list of workspace types. Each list element is a dict with these values:
            - id (str)
            - name (str)
            - templates (list)
                + name (str)
                + id

    """

    # If this payload section has been processed successfully before we
    # still need to read the data structure from the status file and
    # initialize self._workspace_types:
    if self.check_status_file(payload_section_name=section_name):
        # read the list from the json file in admin Home
        # this is important for restart of customizer pod
        # as this data structure is used later on for workspace processing
        self.logger.info(
            "Re-Initialize workspace types list from status file -> '%s' for later use...",
            self.get_status_file_name(payload_section_name=section_name),
        )
        self._workspace_types = self.get_status_file(
            payload_section_name=section_name,
        )
        if self._workspace_types:
            self.logger.info(
                "Found -> %s workspace types.",
                str(len(self._workspace_types)),
            )
            self.logger.debug("Workspace types -> %s", str(self._workspace_types))
            return self._workspace_types
        else:
            self.logger.error(
                "Couldn't read workspace types from status file -> '%s'. Regenerate list...",
                self.get_status_file_name(payload_section_name=section_name),
            )

    # Read payload_section "workspaceTypes" if available
    payload_section = {}
    for section in self._payload_sections:
        if section["name"] == "workspaceTypes":
            payload_section = section

    # get all workspace types (these have been created by the transports and are not in the payload!)
    # we need to do this each time as it needs to work across multiple payload files...
    response = self._otcs.get_workspace_types()
    if response is None:
        self.logger.error("No workspace types found!")
        self._workspace_types = []
    else:
        self._workspace_types = response["results"]
        self.logger.info(
            "Found -> %s workspace types.",
            str(len(self._workspace_types)),
        )
        self.logger.debug("Workspace types -> %s", str(self._workspace_types))

    # now we enrich the workspace_type list elments (which are dicts)
    # with additional dict elements for further processing:
    for workspace_type in self._workspace_types:
        workspace_type_id = workspace_type["data"]["properties"]["wksp_type_id"]
        self.logger.debug("Workspace Type ID -> %s", workspace_type_id)
        workspace_type["id"] = workspace_type_id
        workspace_type_name = workspace_type["data"]["properties"]["wksp_type_name"]
        workspace_type["name"] = workspace_type_name
        workspace_templates = workspace_type["data"]["properties"]["templates"]
        if not workspace_templates:
            self.logger.warning(
                "Workspace type -> '%s' has no workspace templates to process! Skipping...",
                workspace_type_name,
            )
            continue
        self.logger.info(
            "Workspace type -> '%s' has %d template%s to process...",
            workspace_type_name,
            len(workspace_templates),
            "s" if len(workspace_templates) > 1 else "",
        )

        # Create empty lists of dicts with template names and node IDs:
        workspace_type["templates"] = []
        # Determine available templates per workspace type (there can be multiple!)
        # and record them in a simplified data structure:
        for workspace_template in workspace_templates:
            workspace_template_id = workspace_template["id"]
            workspace_template_name = workspace_template["name"]
            self.logger.info(
                "Found workspace template -> '%s' (%s).",
                workspace_template_name,
                workspace_template_id,
            )
            template = {
                "name": workspace_template_name,
                "id": workspace_template_id,
            }
            workspace_type["templates"].append(template)

            if payload_section.get("inherit_permissions", False):
                # TODO: Workaround for problem with workspace role inheritance
                # which may be related to Transport or REST API: to work-around this we
                # push down the workspace roles to the workspace folders explicitly:
                response = self._otcs.get_workspace_roles(
                    workspace_id=workspace_template_id,
                )
                for roles in response["results"]:
                    role_name = roles["data"]["properties"]["name"]
                    role_id = roles["data"]["properties"]["id"]
                    permissions = roles["data"]["properties"]["perms"]
                    # as get_workspace_roles() delivers permissions as a value (bit encoded)
                    # we need to convert it to a permissions string list:
                    permission_string_list = self._otcs.convert_permission_value_to_permission_string(
                        permission_value=permissions,
                    )

                    self.logger.info(
                        "Inherit permissions of workspace template -> '%s' (%s) and role -> '%s' (%s) to workspace folders...",
                        workspace_template_name,
                        workspace_template_id,
                        role_name,
                        role_id,
                    )

                    # Inherit permissions to folders of workspace template:
                    response = self._otcs.assign_workspace_permissions(
                        workspace_id=workspace_template_id,
                        role_id=role_id,
                        permissions=permission_string_list,
                        apply_to=1,  # 1 = only sub items - workspace node itself is OK
                    )
                # end for roles in response["results"]:
            # end if payload_section.get("inherit_permissions", False):
        # end for workspace_template in workspace_templates:
    # end for workspace_type in self._workspace_types:

    self.write_status_file(success=True, payload_section_name=section_name, payload_section=self._workspace_types)

    return self._workspace_types

process_workspaces(section_name='workspaces')

Process workspaces in payload and create them in Content Server.

Parameters:

Name	Type	Description	Default
section_name	str	The name of the payload section. It can be overridden for cases where multiple sections of same type are used (e.g. the "Post" sections). This name is also used for the "success" status files written to the Admin Personal Workspace.	'workspaces'

Returns:

Name	Type	Description
bool	bool	True, if payload has been processed without errors, False otherwise.

Side Effects

Set workspace["nodeId"] to the node ID of the created workspace and update the workspace["name"] to the final name of the workspaces (which may be different from the ones in the payload depending on workspace type configutrations)

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspaces")
def process_workspaces(self, section_name: str = "workspaces") -> bool:
    """Process workspaces in payload and create them in Content Server.

    Args:
        section_name (str, optional):
            The name of the payload section. It can be overridden
            for cases where multiple sections of same type
            are used (e.g. the "Post" sections).
            This name is also used for the "success" status
            files written to the Admin Personal Workspace.

    Returns:
        bool:
            True, if payload has been processed without errors, False otherwise.

    Side Effects:
        Set workspace["nodeId"] to the node ID of the created workspace and update
        the workspace["name"] to the final name of the workspaces (which may be different
        from the ones in the payload depending on workspace type configutrations)

    """

    if not self._workspaces:
        self.logger.info(
            "Payload section -> '%s' is empty. Skipping...",
            section_name,
        )
        return True

    # If this payload section has been processed successfully before we
    # can return True and skip processing it once more:
    if self.check_status_file(payload_section_name=section_name):
        # Read the list of created workspaces from the json file in admin Home
        # This is important in case of restart / rerun of customizer pod
        # as this data structure is used later on for workspace relationship
        # processing (and other) and the workspaces dictionaries have been
        # updated with "nodeId" and "name" (the final names of the workspaces
        # that can be different from the names in the payload)
        self.logger.info(
            "Re-Initialize workspace list from status file -> '%s' to have final names and node IDs...",
            self.get_status_file_name(payload_section_name=section_name),
        )
        self._workspaces = self.get_status_file(payload_section_name=section_name)

        return True

    success: bool = True

    if ENABLE_MULTI_THREADING:
        # Create a list to hold the threads
        threads = []
        # And another list to collect the results
        results = []

        df = Data(self._workspaces, logger=self.logger)

        # Add empty column for "nodeId" so that the worker threads can properly fill it:
        df.get_data_frame()["nodeId"] = None

        self.logger.info(
            "Created a data frame with -> %s rows from the workspaces list with -> %s elements.",
            str(len(df)),
            str(len(self._workspaces)),
        )
        df.print_info()

        partitions = df.partitionate(THREAD_NUMBER)

        # Create and start a thread for each partition
        for index, partition in enumerate(partitions, start=1):
            # start a thread executing the process_workspaces_worker() method below:
            thread = threading.Thread(
                name=f"{section_name}_{index:02}",
                target=self.thread_wrapper,
                args=(self.process_workspaces_worker, partition, results),
            )
            self.logger.info("Starting thread -> '%s'...", str(thread.name))
            thread.start()
            threads.append(thread)
            # Avoid that all threads start at the exact same time with
            # potentially expired cookies that cases race conditions:
            time.sleep(1)
        # end for index, partition in enumerate(partitions, start=1)

        # Wait for all threads to complete
        for thread in threads:
            self.logger.info(
                "Waiting for thread -> '%s' to complete...",
                str(thread.name),
            )
            thread.join()
            self.logger.info("Thread -> '%s' has completed.", str(thread.name))

        # As we have basically created a copy of self._workspaces into the Pandas
        # data frame (see df = Data(...) above) and the workspace processing
        # updates the workspaces data with "nodeID" and the final
        # workspace names, we need to write the Pandas Data frame
        # back into the self._workspaces data structure for further processing
        # e.g. in the process_workspace_relationships. Otherwise the
        # changes to "nodeId" or "name" would be lost. We need to do it
        # in 2 steps as we want to avoid to have NaN values in the resulting dicts:
        # 1. Convert the data frame back to a list of dictionaries:
        updated_workspaces = df.get_data_frame().to_dict(orient="records")
        # 2. Remove any dictionary item that has a "NaN" scalar value
        # (pd.notna() only works on scalar values, not on lists!):
        self._workspaces = [
            #                {k: v for k, v in w.items() if pd.notna(v)} for w in updated_workspaces
            {
                key: value
                for key, value in updated_workspace.items()
                if not pd.api.types.is_scalar(value) or pd.notna(value)
            }
            for updated_workspace in updated_workspaces
        ]

        # Print statistics for each thread. In addition,
        # check if all threads have completed without error / failure.
        # If there's a single failure in on of the thread results we
        # set 'success' variable to False.
        results.sort(key=lambda x: x["thread_name"])
        for result in results:
            if not result["success"]:
                self.logger.info(
                    "Thread -> '%s' had %s failures and completed %s workspaces successfully.",
                    result["thread_name"],
                    result["failure_counter"],
                    result["success_counter"],
                )
                success = False  # mark the complete processing as "failure" for the status file.
            else:
                self.logger.info(
                    "Thread -> '%s' completed %s workspaces successfully.",
                    result["thread_name"],
                    result["success_counter"],
                )
    else:  # no multi-threading
        for workspace in self._workspaces:
            try:
                result = self.process_workspace(workspace=workspace)
            except Exception:
                self.logger.exception("Failed process workspace -> %s", workspace)
                result = False
            success = success and result  # if a single result is False then mark this in 'success' variable.

    self.write_status_file(
        success=success,
        payload_section_name=section_name,
        payload_section=self._workspaces,
    )

    return success

process_workspaces_worker(partition, results)

Multi-threading worker method to process a partition of the workspaces.

Parameters:

Name	Type	Description	Default
partition	DataFrame	The partition of the workspaces to process by this thread.	required
results	list	A mutable (shared) list of all workers to collect the results.	required

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="process_workspaces_worker")
def process_workspaces_worker(
    self,
    partition: pd.DataFrame,
    results: list,
) -> None:
    """Multi-threading worker method to process a partition of the workspaces.

    Args:
        partition (pd.DataFrame):
            The partition of the workspaces to process by this thread.
        results (list):
            A mutable (shared) list of all workers to collect the results.

    """

    thread_id = threading.get_ident()
    thread_name = threading.current_thread().name

    t = trace.get_current_span()
    t.set_attributes(
        {
            "thread.id": thread_id,
            "thread.name": thread_name,
        }
    )

    result = {}
    result["thread_id"] = thread_id
    result["thread_name"] = thread_name
    result["success"] = True
    result["failure_counter"] = 0
    result["success_counter"] = 0

    total_rows = len(partition)

    # Process all datasets in the partion that was given to the thread:
    for index, row in partition.iterrows():
        # Calculate percentage of completion
        percent_complete = ((partition.index.get_loc(index) + 1) / total_rows) * 100

        self.logger.info(
            "Processing data row -> %s to create workspace -> '%s' (%.2f %% complete)...",
            str(index),
            row["name"],
            percent_complete,
        )
        # Convert the row to a dictionary - omitting any empty column:
        workspace = row.dropna().to_dict()
        # workspace is a mutable dictionary that may be updated
        # by process_workspace():
        try:
            success = self.process_workspace(workspace=workspace)
        except Exception:
            self.logger.exception("Failed process workspace -> %s", workspace)
            success = False
        # We need to make sure the row (and the whole data frame)
        # gets these updates back (and adds new columns such as "nodeId"):
        for key, value in workspace.items():
            row[key] = value  # This will update existing keys and add new ones
        self.logger.debug("Final values of row %s -> %s", str(index), str(row))

        # As iterrows() creates a copy of the data we need to
        # write the changes back into the partition
        partition.loc[index] = row

        if success:
            result["success_counter"] += 1
        else:
            self.logger.error(
                "Failed to process row -> %s for workspace -> '%s'!",
                str(index),
                row["name"],
            )
            result["failure_counter"] += 1
            result["success"] = False

    results.append(result)

read_data_source_file(data_source, compression=True)

Read a data source file from the Admin Personal Workspace in Content Server.

Parameters:

Name	Type	Description	Default
data_source	dict	Data source dictionary with embedded "data" Data object.	required
compression	bool	Use True, if the data source file is compressed.	True

Returns:

Name	Type	Description
bool	bool	True if the data source file (JSON) as been loaded from Content Server successfully, False otherwise

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="read_data_source_file")
def read_data_source_file(
    self,
    data_source: dict,
    compression: bool = True,
) -> bool:
    """Read a data source file from the Admin Personal Workspace in Content Server.

    Args:
        data_source (dict):
            Data source dictionary with embedded "data" Data object.
        compression (bool, optional):
            Use True, if the data source file is compressed.

    Returns:
        bool:
            True if the data source file (JSON) as been loaded from
            Content Server successfully, False otherwise

    """

    data_source_name = data_source.get("name", "")
    if not data_source_name:
        self.logger.error("Missing data source name!")
        return False

    response = self._otcs.get_node_by_volume_and_path(
        volume_type=self._otcs.VOLUME_TYPE_PERSONAL_WORKSPACE,
    )  # write to Personal Workspace of Admin (with Volume Type ID = 142)
    target_folder_id = self._otcs.get_result_value(response=response, key="id")
    if not target_folder_id:
        target_folder_id = 2004  # use Personal Workspace of Admin as fallback

    if self._payload_source:
        payload_file_name = os.path.basename(
            self._payload_source,
        )  # remove directories
        # Split once at the first occurance of a dot
        # as the _payload_source may have multiple suffixes
        # such as .yml.gz.b64:
        payload_file_name = payload_file_name.split(".", 1)[0] + "_"
    else:
        payload_file_name = ""

    file_name = "data_source_" + payload_file_name + data_source_name + ".json"
    if compression:
        file_name += ".zip"
    full_path = os.path.join(tempfile.gettempdir(), file_name)

    # Check if the data source file has been uploaded before.
    # This can happen if we re-run the python container.
    # In this case we add a version to the existing document:
    response = self._otcs.get_node_by_parent_and_name(
        parent_id=int(target_folder_id),
        name=file_name,
        show_error=False,
    )
    data_source_node_id = self._otcs.get_result_value(response=response, key="id")
    if not data_source_node_id:
        self.logger.warning(
            "Data source -> '%s' file -> '%s' not found!",
            data_source_name,
            file_name,
        )
        return False

    self.logger.info(
        "Download data source to temporary file -> '%s'...",
        full_path,
    )
    if not self._otcs.download_document(
        node_id=int(data_source_node_id),
        file_path=full_path,
    ):
        self.logger.error(
            "Error downloading data source -> '%s' to file -> '%s'!",
            data_source_name,
            full_path,
        )
        return False

    self.logger.info("Load data source file -> '%s' into data frame...", full_path)
    data = Data()
    if not data.load_json_data(
        json_path=full_path,
        index_column="row",
        compression="zip" if compression else None,
    ):
        self.logger.error(
            "Data source -> '%s' could not be loaded!",
            data_source_name,
        )
        return False

    self.logger.info(
        "Data source file -> '%s' has been loaded from Personal Workspace of admin user into the data frame.",
        file_name,
    )

    data_source["data"] = data

    # Don't leave stuff behind:
    os.remove(full_path)

    return True

replace_bulk_placeholders(input_string, row, index=0, replacements=None, additional_regex_list=None)

Replace placeholders like "{variable.subvalue}" in payload of bulk processing.

Parameters:

Name	Type	Description	Default
input_string	str	The string to replace placeholders in.	required
row	Series	Curent row in the data frame.	required
index	int | None	The index for use if we encounter a list value. If index is "None" then we return the complete list as value. Otherwise we return the list item with the given index (0 = first element is the default). IMPORTANT: None and 0 have different effects!	0
replacements	dict	The replacements to apply to given fields (dictionary key = field name)	None
additional_regex_list	list	These are not coming from the payload but dynamically added for special needs like determining the nicknames.	None

Returns:

Name	Type	Description
str	str	Updated string with replacements.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="replace_bulk_placeholders")
def replace_bulk_placeholders(
    self,
    input_string: str,
    row: pd.Series,
    index: int | None = 0,  # don't use None as the default here!
    replacements: dict | None = None,
    additional_regex_list: list | None = None,
) -> str:
    """Replace placeholders like "{variable.subvalue}" in payload of bulk processing.

    Args:
        input_string (str):
            The string to replace placeholders in.
        row (pd.Series):
            Curent row in the data frame.
        index (int | None, optional):
            The index for use if we encounter a list value.
            If index is "None" then we return the complete list as value.
            Otherwise we return the list item with the given index (0 = first element is the default).
            IMPORTANT: None and 0 have different effects!
        replacements (dict, optional):
            The replacements to apply to given fields (dictionary key = field name)
        additional_regex_list (list, optional):
            These are not coming from the payload but dynamically
            added for special needs like determining the nicknames.

    Returns:
        str:
            Updated string with replacements.

    """

    r"""
    XML data sources may include "@" in Pandas column names as well!
    This happens if the XML elements have attributes.
           pattern = r"\{([\w@]+(\.[\w@]+)*)\}"
           pattern = r"\{(\w+(\.\w+)*)\}"
    Adjust Pattern to allow any sequence of characters withint the {...}
       pattern = r"\{([^}]*)\}"
    """

    # non-greedy match of placeholders that are surrounded by curly braces:
    pattern = r"\{(.*?)\}"

    had_lookup_error = False

    # Define a function to replace placeholders. This
    # function is called by re.sub() for each pattern match below.
    def replace(match: re.Match) -> str:
        # we want to change the variable of the main method
        nonlocal had_lookup_error

        # Extract the first captured group from the regex match, which corresponds
        # to the placeholder name. In the regex pattern r"\{(.*?)\}", the parentheses
        # define a capturing group that matches any content inside curly braces `{}`.
        # For example, in the string "Hello {user.name}", the match would capture "user.name".
        # `match.group(1)` retrieves this captured text (the placeholder name),
        # which will later be used to look up corresponding values in the provided data frame
        # row or replacement dictionary.
        field_name = match.group(1)
        # split up the field name into keys at ".", e.g. cm_vehicles.make
        keys = field_name.split(".")  # Splitting the variable and sub-value
        # we initialize value with the data frame row (pd.Series):
        value = row
        # Walk through the list of keys:
        for key in keys:
            # first we access the field in the row and handle the
            # exception that key may not be a valid column (KeyError):
            try:
                # read the value of the data frame column defined by key
                value = value[key]
            except KeyError as e:
                self.logger.warning(
                    "KeyError: Cannot replace field -> '%s'%s as the data frame row does not have a column called -> '%s': %s",
                    field_name,
                    " (sub-key -> '{}')".format(key) if key != field_name else "",
                    field_name,
                    str(e),
                )
                had_lookup_error = True
                return ""
            except TypeError:
                # If we get an error with (value type -> <class 'float'>)
                # this is actually indicating we have a NaN value!
                self.logger.error(
                    "TypeError: Cannot replace field -> '%s'%s (value type -> %s). Expecting a dictionary-like value.",
                    field_name,
                    " (sub-key -> '{}')".format(key) if key != field_name else "",
                    str(type(value)),
                )
                had_lookup_error = True
                return ""

            # if the returned value is a list we use the index parameter
            # to select the item in the list according to the given index
            # We handle the exception that index may be out of range for
            # the list (IndexError).
            # If the given index is None we return the whole list. This
            # is required for multi-value attributes.
            if isinstance(value, list) and index is not None and len(value) > 0:
                try:
                    value = value[index]
                except IndexError:
                    self.logger.error(
                        "Index error in replacement of list field -> '%s' using index -> %s.",
                        field_name,
                        str(index),
                    )
                    had_lookup_error = True
                    return ""

            # Check if the column or sub-field value is NaN or None. Then we should not
            # continue to process the keys and return "" (it is important to do this
            # after the test for a list above otherwise we may get an error
            # "The truth value of an array with more than one element is ambiguous."):
            if not isinstance(value, list) and pd.isna(value):
                self.logger.debug(
                    "Cannot replace field -> '%s' as %s has no value in the current row!",
                    field_name,
                    (
                        "sub-key -> '{}'".format(key) if key != keys[0] else "column -> '{}'".format(key)
                    ),  # the first key is the column
                )
                had_lookup_error = True
                return ""

        # end for key in keys

        if isinstance(value, list):
            if value == []:
                had_lookup_error = True
                return ""
        else:
            if pd.isna(value):
                had_lookup_error = True
                return ""
            value = str(value)

        if replacements and field_name in replacements:
            # replacements is a dictionary that is defined
            # in the payload. Each item is a dictionary
            # that can be looked up by the field name
            field_replacements = replacements[field_name]
            upper = field_replacements.get("upper_case", False)
            lower = field_replacements.get("lower_case", False)
            regex_list = field_replacements.get("regex_list", [])
        else:
            regex_list = []
            upper = False
            lower = False

        if additional_regex_list:
            regex_list = (
                regex_list + additional_regex_list
            )  # don't do an append here as it would modify the original list

        if regex_list or upper or lower:
            if not isinstance(value, list):
                value = self.cleanup_value(
                    cleanup_value=value,
                    regex_list=regex_list,
                    upper=upper,
                    lower=lower,
                )
            else:  # we have a list, so we need to iterate
                for v in value:
                    v = self.cleanup_value(
                        cleanup_value=v,
                        regex_list=regex_list,
                        upper=upper,
                        lower=lower,
                    )

        value = str(value)

        return value

    # end sub-method replace()

    if not input_string:
        return ""

    # Use re.sub() to replace placeholders using the defined function
    # replace() - see above.
    result_string = re.sub(pattern, replace, input_string)

    if had_lookup_error:
        return ""

    return result_string

replace_bulk_placeholders_list(input_list, row, index=0, replacements=None, additional_regex_list=None)

Process list of payload strings and replace placeholders (see next method).

Parameters:

Name	Type	Description	Default
input_list	list	A list of strings that contain placeholders.	required
row	Series	The curent data frame row.	required
index	int	Index for use if we encounter a list value. Pass None here if you want alle elements of the list to be parsed and placeholders replaced.	0
replacements	dict	The replacements to apply to given fields (dictionary key = field name)	None
additional_regex_list	list	These are not coming from the payload but dynamically added for special needs like determining the nicknames.	None

Returns:

Name	Type	Description
bool	bool	True = all replacements worked, False = some replacements had lookup errors.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="replace_bulk_placeholders_list")
def replace_bulk_placeholders_list(
    self,
    input_list: list,
    row: pd.Series,
    index: int = 0,
    replacements: dict | None = None,
    additional_regex_list: list | None = None,
) -> bool:
    """Process list of payload strings and replace placeholders (see next method).

    Args:
        input_list (list):
            A list of strings that contain placeholders.
        row (pd.Series):
            The curent data frame row.
        index (int, optional):
            Index for use if we encounter a list value.
            Pass None here if you want alle elements of
            the list to be parsed and placeholders replaced.
        replacements (dict, optional):
            The replacements to apply to given fields (dictionary key = field name)
        additional_regex_list (list, optional):
            These are not coming from the payload but dynamically
            added for special needs like determining the nicknames.

    Returns:
        bool:
            True = all replacements worked, False = some replacements had lookup errors.

    """

    success = True

    if not input_list:
        return False

    for i, value in enumerate(input_list):
        input_list[i] = self.replace_bulk_placeholders(
            input_string=value,
            row=row,
            index=index,
            replacements=replacements,
            additional_regex_list=additional_regex_list,
        )
        if not input_list[i]:
            success = False

    return success

replace_placeholders(content)

Replace placeholders in file content.

The content of the file is provided via a parameter. The replacements are defined in a object variable _placeholder_values (type = dictionary) The placeholder values are supposed to be surrounded by double % signs like %%OTAWP_RESOURCE_ID%%

Parameters:

Name	Type	Description	Default
content	str	The file content to replace placeholders in.	required

Returns:

Name	Type	Description
str	str	Updated content with all defined replacements.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

def replace_placeholders(self, content: str) -> str:
    """Replace placeholders in file content.

    The content of the file is provided via a parameter.
    The replacements are defined in a object variable
    _placeholder_values (type = dictionary)
    The placeholder values are supposed to be surrounded by
    double % signs like %%OTAWP_RESOURCE_ID%%

    Args:
        content (str):
            The file content to replace placeholders in.

    Returns:
        str:
            Updated content with all defined replacements.

    """
    # https://stackoverflow.com/questions/63502218/replacing-placeholders-in-a-text-file-with-python

    # if no placeholders are defined we can return the
    # initial value:
    if not self._placeholder_values:
        return content

    try:
        # We do a dynamic replacement here. The replacement is calculated
        # by the lambda function that is basically a lookup of the replacement
        # key found in the settings file with the value defined in the list
        # of replacement values in self._placeholder_values
        return re.sub(
            r"%%(\w+?)%%",
            lambda match: self._placeholder_values[match.group(1)],
            content,
        )
    except KeyError:
        self.logger.error(
            "Found placeholder in settings file without a defined value!",
        )
        return content
    except re.error:
        self.logger.error("Regex substitution error!")
        return content

resolve_attribute_values(attribute_name, attribute_type, attribute_values)

Resolve user logins or group names to user / group IDs.

This method handles the "special" cases where the actual attribute values need to be IDs but the payload has names.

This method is called for all attributes. If no special handling is required it just return an unmodified attribute value.

Parameters:

Name	Type	Description	Default
attribute_name	str	The name of the attribute.	required
attribute_type	str	The type of the attribute.	required
attribute_values	str | list[str]	The value of the attribute. A single value or a list of values.	required

Returns:

Type	Description
str | list[str]	str | list[str]: A resolved attribute value or a list of resolved attribute values.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="resolve_attribute_values")
def resolve_attribute_values(
    self,
    attribute_name: str,
    attribute_type: str,
    attribute_values: str | list[str],
) -> str | list[str]:
    """Resolve user logins or group names to user / group IDs.

    This method handles the "special" cases where the actual
    attribute values need to be IDs but the payload has names.

    This method is called for all attributes. If no special handling
    is required it just return an unmodified attribute value.

    Args:
        attribute_name (str):
            The name of the attribute.
        attribute_type (str):
            The type of the attribute.
        attribute_values (str | list[str]):
            The value of the attribute. A single value or a list of values.

    Returns:
        str | list[str]:
            A resolved attribute value or a list of resolved attribute values.

    """

    # Special Case 1: handle special case where attribute type is a user picker.
    # we expect that the payload includes the login name for this
    # (as user IDs are not stable across systems) but then we need
    # to lookup the real user ID here:
    if attribute_type in ("otcs_user_picker", "otcs_member_picker"):
        self.logger.debug(
            "Attribute -> '%s' is is of type -> 'User Picker' (%s). Looking up user ID for user login name -> %s",
            attribute_name,
            attribute_type,
            attribute_values,
        )

        # Do we have a single value (not a list)?
        if not isinstance(attribute_values, list):
            user = self._otcs_frontend.get_user(name=attribute_values)
            user_id = self._otcs_frontend.lookup_result_value(
                response=user,
                key="name",
                value=attribute_values,
                return_key="id",
            )
            if user_id:
                # User has been found - determine ID:
                self.logger.debug(
                    "User -> '%s' has ID -> %s",
                    attribute_values,
                    user_id,
                )
                # Put the user ID into data structure
                return str(user_id)
            else:
                self.logger.error(
                    "User with login name -> '%s' does not exist!",
                    attribute_values,
                )
                # Clear the value to avoid workspace create failure
                return ""
        # Multi-value user attribute:
        else:
            user_ids = []
            for value in attribute_values:
                user = self._otcs_frontend.get_user(name=value)
                user_id = self._otcs_frontend.lookup_result_value(
                    response=user,
                    key="name",
                    value=value,
                    return_key="id",
                )
                if user_id:
                    # User has been found - determine ID:
                    self.logger.debug(
                        "User -> '%s' has ID -> %s",
                        value,
                        user_id,
                    )
                    # Put the user ID into the result list:
                    user_ids.append(str(user_id))
                else:
                    self.logger.error(
                        "User with login name -> '%s' does not exist!",
                        value,
                    )
            return user_ids

    # Special Case 2: handle Extended ECM for Government attribute type "Organizational Unit" (OU).
    # This is referring to a group ID which is not stable across deployments. So we need to lookup
    # the Group ID and add it to the data structure. This expects that the payload has the
    # group name and not the group ID:
    elif attribute_type == str(11480):
        self.logger.debug(
            "Attribute -> '%s' is is of type -> 'Organizational Unit' (%s). Looking up group ID for group name -> %s",
            attribute_name,
            attribute_type,
            attribute_values,
        )
        # Do we have a single value (not a list)?
        if not isinstance(attribute_values, list):
            group = self._otcs_frontend.get_group(
                name=attribute_values,
            )
            group_id = self._otcs_frontend.lookup_result_value(
                response=group,
                key="name",
                value=attribute_values,
                return_key="id",
            )

            if group_id:
                self.logger.debug(
                    "Group for Organizational Unit -> '%s' has ID -> %s",
                    attribute_values,
                    group_id,
                )
                # Put the group ID as a string:
                return str(group_id)
            else:
                self.logger.error(
                    "Group for Organizational Unit -> '%s' does not exist!",
                    attribute_values,
                )
                # Return an empty value string:
                return ""
        # Multi-value org group attribute:
        else:
            group_ids = []
            for value in attribute_values:
                group = self._otcs_frontend.get_group(
                    name=value,
                )
                group_id = self._otcs_frontend.lookup_result_value(
                    response=group,
                    key="name",
                    value=value,
                    return_key="id",
                )

                if group_id:
                    self.logger.debug(
                        "Group for Organizational Unit -> '%s' has ID -> %s",
                        value,
                        group_id,
                    )
                    # Put the group ID into the result list:
                    group_ids.append(str(group_id))
                else:
                    self.logger.error(
                        "Group for Organizational Unit -> '%s' does not exist!",
                        value,
                    )
            return group_ids

    # This is the default case - we return the unchanged attribute values:
    return attribute_values

start_impersonation(username, otcs_object=None)

Impersonate to a defined user.

Parameters:

Name	Type	Description	Default
username	str	The user to impersonate.	required
otcs_object	OTCS | None	The OTCS object to use. If not provided then the _otcs object in the Payload object is used.	None

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="start_impersonation")
def start_impersonation(self, username: str, otcs_object: OTCS | None = None) -> bool:
    """Impersonate to a defined user.

    Args:
        username (str):
            The user to impersonate.
        otcs_object (OTCS | None, optional):
            The OTCS object to use. If not provided then the _otcs object
            in the Payload object is used.

    """

    if not otcs_object:
        otcs_object = self._otcs

    # Depending on the authentication type used with OTDS (token or ticket based)
    # the response structure is different:
    response = self._otds.impersonate_user(user_id=username)
    if not response:
        return False

    if "ticket" in response:
        otds_ticket = response.get("ticket", None)
        otcs_object.set_otds_ticket(ticket=otds_ticket)
    elif "access_token" in response:
        access_token = response.get("access_token", None)
        otcs_object.set_otds_token(token=access_token)
    else:
        self.logger.error("Impersonation response does not contain ticket or access_token!")
        return False

    otcs_object.invalidate_authentication_ticket()
    response = otcs_object.authenticate(revalidate=False)

    return bool(response)

stop_impersonation(otcs_object=None)

Impersonate back to admin user.

Parameters:

Name	Type	Description	Default
username	str	The user to impersonate.	required
otcs_object	OTCS | None	The OTCS object to use. If not provided then the _otcs object in the Payload object is used.	None

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="stop_impersonation")
def stop_impersonation(self, otcs_object: OTCS | None = None) -> bool:
    """Impersonate back to admin user.

    Args:
        username (str):
            The user to impersonate.
        otcs_object (OTCS | None, optional):
            The OTCS object to use. If not provided then the _otcs object
            in the Payload object is used.

    """

    if not otcs_object:
        otcs_object = self._otcs

    # Clear OTDS ticket and token:
    otcs_object.set_otds_ticket(None)
    otcs_object.set_otds_token(None)

    # Clear OTCS ticket:
    otcs_object.invalidate_authentication_ticket()

    # Reauthenticate as admin (using username / password of OTCS object):
    response = otcs_object.authenticate(revalidate=True)

    return bool(response)

thread_wrapper(target, *args, **kwargs)

Wrap around threads to catch exceptions during exection.

Parameters:

Name	Type	Description	Default
target	callable	The method (callable) the Thread should run.	required
args	tuple	The arguments for the method.	()
kwargs	dict	Keyword arguments for the method.	{}

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

def thread_wrapper(self, target: Callable, *args: tuple, **kwargs: dict) -> None:
    """Wrap around threads to catch exceptions during exection.

    Args:
        target (callable):
            The method (callable) the Thread should run.
        args (tuple):
            The arguments for the method.
        kwargs (dict):
            Keyword arguments for the method.

    """

    try:
        target(*args, **kwargs)
    except Exception:
        thread_name = threading.current_thread().name
        self.logger.error(
            "Thread '%s' failed!",
            thread_name,
        )
        self.logger.error(traceback.format_exc())

write_data_source_file(data_source, compression=True)

Write a data source file as JSON into the Admin Personal Workspace in Content Server.

This allows for further analysis or to reload it in succeeding customizer runs.

Parameters:

Name	Type	Description	Default
data_source	dict	Data source dictionary with embedded "data" Data object.	required
compression	bool	If True, a compressed JSON file gets saved. If False then an uncompressed JSON is saved.	True

Returns:

Name	Type	Description
bool	bool	True if the data source file (JSON) as been upladed to Content Server successfully, False otherwise.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="write_data_source_file")
def write_data_source_file(
    self,
    data_source: dict,
    compression: bool = True,
) -> bool:
    """Write a data source file as JSON into the Admin Personal Workspace in Content Server.

    This allows for further analysis or to reload it in succeeding customizer runs.

    Args:
        data_source (dict):
            Data source dictionary with embedded "data" Data object.
        compression (bool):
            If True, a compressed JSON file gets saved. If False then
            an uncompressed JSON is saved.

    Returns:
        bool:
            True if the data source file (JSON) as been upladed to
            Content Server successfully, False otherwise.

    """

    data_source_name = data_source.get("name", "")
    if not data_source_name:
        self.logger.error("Missing data source name!")
        return False

    data: Data = data_source.get("data")
    if not data:
        self.logger.error(
            "Data source -> '%s' does not have data! Cannot save it.",
            data_source_name,
        )
        return False

    response = self._otcs.get_node_by_volume_and_path(
        volume_type=self._otcs.VOLUME_TYPE_PERSONAL_WORKSPACE,
    )  # write to Personal Workspace of Admin (with Volume Type ID = 142)
    target_folder_id = self._otcs.get_result_value(response=response, key="id")
    if not target_folder_id:
        target_folder_id = 2004  # use Personal Workspace of Admin as fallback

    if self._payload_source:
        payload_file_name = os.path.basename(
            self._payload_source,
        )  # remove directories
        # Split once at the first occurance of a dot
        # as the _payload_source may have multiple suffixes
        # such as .yml.gz.b64:
        payload_file_name = payload_file_name.split(".", 1)[0] + "_"
    else:
        payload_file_name = ""

    file_name = "data_source_" + payload_file_name + data_source_name + ".json"
    if compression:
        file_name += ".zip"
    full_path = os.path.join(tempfile.gettempdir(), file_name)

    # We also want to keep the row numbers (index):
    if not data.save_json_data(
        json_path=full_path,
        preserve_index=True,
        index_column="row",
        compression="zip" if compression else None,
    ):
        self.logger.error(
            "Data source -> '%s' could not be saved!",
            data_source_name,
        )
        return False

    # Check if the status file has been uploaded before.
    # This can happen if we re-run the python container.
    # In this case we add a version to the existing document:
    response = self._otcs.get_node_by_parent_and_name(
        parent_id=int(target_folder_id),
        name=file_name,
        show_error=False,
    )
    data_source_node_id = self._otcs.get_result_value(response=response, key="id")
    if data_source_node_id:
        response = self._otcs.add_document_version(
            node_id=int(data_source_node_id),
            file_url=full_path,
            file_name=file_name,
            mime_type="application/zip" if compression else "application/json",
            description="Updated data source file after re-run of customization",
        )
    else:
        response = self._otcs.upload_file_to_parent(
            file_url=full_path,
            file_name=file_name,
            mime_type="application/zip" if compression else "application/json",
            parent_id=int(target_folder_id),
        )

    if response:
        self.logger.info(
            "Data source file -> '%s' has been written to Personal Workspace of admin user.",
            file_name,
        )
        # Don't leave stuff behind:
        os.remove(full_path)

        return True

    self.logger.error(
        "Failed to write data source file -> '%s' to Personal Workspace of admin user!",
        file_name,
    )

    return False

write_status_file(success, payload_section_name, payload_section, payload_specific=True)

Write a status file into the Admin Personal Workspace in Content Server.

This is to indicate that the payload section has been deployed successfully. This speeds up the customizing process in case the customizer pod is restarted.

Parameters:

Name	Type	Description	Default
success	bool	True, if the section was processed successful, False otherwise.	required
payload_section_name	str	The name of the payload section.	required
payload_section	list	The payload section content - this is written as JSon into the file.	required
payload_specific	bool	Whether or not the success should be specific for each payload file or if success is "global" - like for the deletion of the existing M365 teams (which we don't want to execute per payload file)	True

Returns:

Name	Type	Description
bool	bool	True if the status file as been upladed to Content Server successfully, False otherwise

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

@tracer.start_as_current_span(attributes=OTEL_TRACING_ATTRIBUTES, name="write_status_file")
def write_status_file(
    self,
    success: bool,
    payload_section_name: str,
    payload_section: list,
    payload_specific: bool = True,
) -> bool:
    """Write a status file into the Admin Personal Workspace in Content Server.

    This is to indicate that the payload section has been deployed successfully.
    This speeds up the customizing process in case the customizer pod
    is restarted.

    Args:
        success (bool):
            True, if the section was processed successful, False otherwise.
        payload_section_name (str):
            The name of the payload section.
        payload_section (list):
            The payload section content - this is written as JSon into the file.
        payload_specific (bool, optional):
            Whether or not the success should be specific for
            each payload file or if success is "global" - like for the deletion
            of the existing M365 teams (which we don't want to execute per
            payload file)

    Returns:
        bool:
            True if the status file as been upladed to Content Server successfully, False otherwise

    """

    # Do we want status files to be uploaded?
    if not self.upload_status_files:
        return True

    if success:
        self.logger.info(
            "Payload section -> '%s' has been completed successfully.",
            payload_section_name,
        )
        prefix = "success_"
    else:
        self.logger.error(
            "Payload section -> '%s' had failures!",
            payload_section_name,
        )
        prefix = "failure_"

    while not self._otcs.is_ready():
        self.logger.info(
            "OTCS is not ready. Cannot write status file for -> '%s' to OTCS. Waiting 30 seconds and retry...",
            payload_section_name,
        )
        time.sleep(30)

    response = self._otcs.get_node_by_volume_and_path(
        volume_type=self._otcs.VOLUME_TYPE_PERSONAL_WORKSPACE,
    )  # write to Personal Workspace of Admin (with Volume Type ID = 142)
    target_folder_id = self._otcs.get_result_value(response=response, key="id")
    if not target_folder_id:
        target_folder_id = 2004  # use Personal Workspace of Admin as fallback

    file_name = self.get_status_file_name(
        payload_section_name=payload_section_name,
        payload_specific=payload_specific,
        prefix=prefix,
    )

    full_path = os.path.join(tempfile.gettempdir(), file_name)

    with open(full_path, mode="w", encoding="utf-8") as localfile:
        localfile.write(json.dumps(payload_section, indent=2))

    # Check if the status file has been uploaded before.
    # This can happen if we re-run the python container.
    # In this case we add a version to the existing document:
    response = self._otcs.get_node_by_parent_and_name(
        parent_id=int(target_folder_id),
        name=file_name,
        show_error=False,
    )
    target_document_id = self._otcs.get_result_value(response=response, key="id")
    if target_document_id:
        response = self._otcs.add_document_version(
            node_id=int(target_document_id),
            file_url=full_path,
            file_name=file_name,
            mime_type="text/plain",
            description="Updated status file after re-run of customization",
        )
    else:
        response = self._otcs.upload_file_to_parent(
            file_url=full_path,
            file_name=file_name,
            mime_type="text/plain",
            parent_id=int(target_folder_id),
        )

    if response:
        self.logger.info(
            "Status file -> '%s' has been written to Personal Workspace of admin user.",
            file_name,
        )
        return True

    self.logger.error(
        "Failed to write status file -> '%s' to Personal Workspace of admin user!",
        file_name,
    )

    return False

load_payload(payload_source, logger=default_logger)

Load payload file.

We don't want to have this inside the class to allow it to be used also indepent (see api.py)

Parameters:

Name	Type	Description	Default
payload_source	str	The path to the payload file.	required
logger	Logger	The logger object. Defaults to default_logger.	default_logger

Returns:

Type	Description
dict | None	dict | None: The payload dictionary or None in case of an error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload.py

def load_payload(
    payload_source: str,
    logger: logging.Logger = default_logger,
) -> dict | None:
    """Load payload file.

    We don't want to have this inside the class
    to allow it to be used also indepent (see api.py)

    Args:
        payload_source (str):
            The path to the payload file.
        logger (logging.Logger, optional):
            The logger object. Defaults to default_logger.

    Returns:
        dict | None:
            The payload dictionary or None in case of an error.

    """

    if not os.path.exists(payload_source):
        logger.error("Cannot access payload file -> '%s'!", payload_source)
        return None

    # Is it a YAML file?
    if payload_source.endswith(".yaml"):
        logger.info("Open payload from YAML file -> '%s'...", payload_source)
        try:
            with open(payload_source, encoding="utf-8") as stream:
                payload_data = stream.read()
            return yaml.safe_load(payload_data)
        except yaml.YAMLError:
            logger.error(
                "Error while reading YAML payload file -> '%s'!",
                payload_source,
            )
            payload = {}
    # Or is it a Terraform HCL file?
    elif payload_source.endswith((".tf", ".tfvars")):
        logger.info("Open payload from Terraform file -> '%s'...", payload_source)
        try:
            with open(payload_source, encoding="utf-8") as stream:
                payload = hcl2.api.load(stream)
            # If payload is wrapped into "external_payload" we unwrap it:
            if payload.get("external_payload"):
                payload = payload["external_payload"]

        except (
            lark_exceptions.UnexpectedToken,
            lark_exceptions.UnexpectedCharacters,
        ) as exc:
            exception = f"Syntax error while reading Terraform payload file -> '{payload_source}'! --> {traceback.format_exception_only(exc)}"
            raise PayloadImportError(exception) from exc

        except (
            FileNotFoundError,
            ImportError,
            ValueError,
            SyntaxError,
        ) as exc:
            exception = f"Error while reading Terraform payload file -> '{payload_source}'! --> {traceback.format_exception_only(exc)}"
            raise PayloadImportError(exception) from exc

    elif payload_source.endswith(".yml.gz.b64"):
        logger.info("Open payload from base64-gz-YAML file -> '%s'...", payload_source)
        try:
            with open(payload_source, encoding="utf-8") as stream:
                content = base64.b64decode(stream.read())
                decoded_data = gzip.decompress(content)

                payload = yaml.safe_load(decoded_data)

        except yaml.YAMLError:
            logger.error(
                "Error while reading YAML payload file -> '%s'!",
                payload_source,
            )
            payload = {}

    # If not, it is an unsupported type:
    else:
        logger.error(
            "Payload file -> '%s' has unsupported file type!",
            payload_source,
        )
        payload = {}

    if not payload:
        logger.error(
            "Payload -> '%s' is undefined or empty! Skipping...",
            payload_source,
        )
        return None

    return payload

Payload List Module to implement methods to maintain and process a list of payload files.

This code typically runs in a container as part of the cloud automation.

PayloadList

Manage a sorted list of payload items using a pandas data frame.

Each payload item with metadata such as name, filename, dependency (referencing another item by index), logfile, and status. Provides list-like functionality with additional methods for adding, removing, and reordering items.

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

class PayloadList:
    """Manage a sorted list of payload items using a pandas data frame.

    Each payload item with metadata such as name, filename, dependency (referencing another item by index),
    logfile, and status. Provides list-like functionality with additional methods
    for adding, removing, and reordering items.
    """

    logger: logging.Logger = default_logger

    _stopped: bool = True
    payload_items: pd.DataFrame

    def __init__(self, logger: logging.Logger = default_logger) -> None:
        """Initialize the Payload List object.

        Args:
            logger (logging.Logger, optional):
                The logging object to use for all log messages. Defaults to default_logger.

        """
        if logger != default_logger:
            self.logger = logging.getLogger(f"{logger.name}.payload_list")

        self.payload_items = pd.DataFrame(
            columns=[
                "name",
                "filename",
                "dependencies",
                "logfile",
                "status",
                "enabled",
                "git_url",
                "loglevel",
                "start_time",
                "stop_time",
                "duration",
                "log_debug",
                "log_info",
                "log_warning",
                "log_error",
                "log_critical",
                "customizer_settings",
            ],
        )

    # end method definition

    def calculate_payload_item_duration(self) -> None:
        """Update the dataframe column "duration" for all running items."""

        def calculate_duration(row: pd.Series) -> str:
            if row["status"] == "running":
                now = datetime.now(UTC)
                start_time = pd.to_datetime(row["start_time"])

                duration = now - start_time
                hours, remainder = divmod(duration.total_seconds(), 3600)
                minutes, seconds = divmod(remainder, 60)
                formatted_duration = f"{int(hours):02}:{int(minutes):02}:{int(seconds):02}"

                return formatted_duration
            else:
                return str(row["duration"])  # or whatever the original value should be

        # updates the "duration" column of the DataFrame self.payload_items
        # by applying the method calculate_duration() to each row:
        self.payload_items["duration"] = self.payload_items.apply(
            calculate_duration,
            axis=1,
        )

    # end method definition

    def get_payload_items(self) -> pd.DataFrame:
        """Get the payload items in their current order in the PayloadList.

        Returns:
            pd.DataFrame:
                A data frame containing all items in their current order.

        """

        self.calculate_payload_item_duration()

        return self.payload_items

    # end method definition

    def get_payload_item(self, index: int) -> pd.Series:
        """Get the payload item by index if it exists, otherwise return None.

        Args:
            index (int): index of the row

        Returns:
            pd.Series: row with the matching index or None if there is no row with that index

        """

        self.calculate_payload_item_duration()

        if index not in self.payload_items.index:
            self.logger.error("Index -> %s is out of range", str(index))
            return None

        return self.payload_items.loc[index]

    # end method definition

    def get_payload_item_by_name(self, name: str) -> pd.Series:
        """Get the payload item by name if it exists, otherwise return None.

        Args:
            name (str):
                The name of the payload.

        Returns:
            pd.Series:
                Row with the matching name or None if there is no row with that index

        """

        self.calculate_payload_item_duration()

        df = self.get_payload_items()
        data = [{"index": idx, **row} for idx, row in df.iterrows()]

        return next((item for item in data if item.get("name") == name), None)

    # end method definition

    def get_payload_items_by_value(
        self,
        column: str,
        value: str,
    ) -> pd.DataFrame | None:
        """Filter the PayloadList by a given value in a specific column.

        Args:
            column (str):
                The column to filter by.
            value (str):
                The value to match in the specified column.

        Returns:
            pd.DataFrame: A DataFrame containing rows where the given column matches the value.

        Example:
            >>> payload_list = PayloadList()
            >>> payload_list.add_item("Task1", "task1.txt", status="running")
            >>> payload_list.add_item("Task2", "task2.txt", status="completed")
            >>> payload_list.add_item("Task3", "task3.txt", status="running")
            >>> payload_list.get_payload_items_by_value(column="status", value="running")
                name         file    dependencies  logfile   status    enabled
            0   Task1    task1.txt        NaN        None   running     True
            2   Task3    task3.txt        NaN        None   running     True

        """

        if column not in self.payload_items.columns:
            self.logger.error(
                "Column -> '%s' does not exist in the payload list!",
                str(column),
            )
            return None

        filtered_items = self.payload_items[self.payload_items[column] == value]

        return filtered_items

    # end method definition

    def add_payload_item(
        self,
        name: str,
        filename: str,
        logfile: str,
        dependencies: list | None = None,
        status: str = "pending",
        enabled: bool = True,
        git_url: str | None = None,
        loglevel: str = "INFO",
        customizer_settings: dict | None = None,
    ) -> dict:
        """Add a new item to the PayloadList.

        Args:
            name (str):
                The name of the item.
            filename (str):
                The file associated with the item.
            logfile (str):
                Log file information for the item. Defaults to None.
            dependencies (list):
                The index of another item this item depends on. Defaults to None.
            status (str):
                The status of the item. Must be one of 'planned', 'running',
                'completed', or 'failed'. Defaults to 'planned'.
            enabled (bool):
                True if the payload is enabled. False otherwise.
            git_url (str):
                Link to the payload in the GIT repository.
            loglevel (str):
                The log level for processing the payload. Either "INFO" or "DEBUG".
            customizer_settings (dict):
                Customizer settings for the payload. Defaults to None.

        """

        new_item = {
            "name": name if name else filename,
            "filename": filename,
            "dependencies": dependencies if dependencies else [],
            "logfile": logfile,
            "status": status,
            "enabled": enabled,
            "git_url": git_url,
            "loglevel": loglevel,
            "log_debug": 0,
            "log_info": 0,
            "log_warning": 0,
            "log_error": 0,
            "log_critical": 0,
            "customizer_settings": customizer_settings if customizer_settings else {},
        }
        self.payload_items = pd.concat(
            [self.payload_items, pd.DataFrame([new_item])],
            ignore_index=True,
        )

        new_item = self.payload_items.tail(1).to_dict(orient="records")[0]
        new_item["index"] = self.payload_items.index[-1]

        return new_item

    # end method definition

    def update_payload_item(
        self,
        index: int,
        update_data: dict,
    ) -> bool:
        """Update an existing item in the PayloadList.

        Args:
            index (int):
                The position of the payload.
            update_data (str):
                The data of the item.

        Returns:
            bool:
                True = success, False = error.

        """

        if index not in self.payload_items.index:
            self.logger.error("Illegal index -> %s for payload update!", index)
            return False

        for column in self.payload_items.columns:
            if column in update_data:
                tmp = self.payload_items.loc[index].astype(object)
                tmp[column] = update_data[column]
                self.payload_items.loc[index] = tmp

        return True

    # end method definition

    def remove_payload_item(self, index: int) -> bool:
        """Remove an item by its index from the PayloadList.

        Args:
            index (int):
                The index of the item to remove.

        Returns:
            bool:
                True = success. False = failure.

        Raises:
            IndexError: If the index is out of range.

        """

        if index not in self.payload_items.index:
            self.logger.error("Index -> %s is out of range!", index)
            return False

        self.payload_items.drop(index, inplace=True)

        return True

    # end method definition

    def move_payload_item_up(self, index: int) -> int | None:
        """Move an item up by one position in the PayloadList.

        Args:
            index (int): The index of the item to move up.

        Results:
            bool: False, if the index is out of range or the item is already at the top.
                  True otherwise

        """

        if index <= 0 or index >= len(self.payload_items):
            self.logger.error(
                "Index -> %s is out of range or already at the top!",
                str(index),
            )
            return None

        self.payload_items.iloc[[index - 1, index]] = self.payload_items.iloc[[index, index - 1]].to_numpy()

        new_postion = self.payload_items.index.get_loc(index)

        return new_postion

    # end method definition

    def move_payload_item_down(self, index: int) -> int | None:
        """Move an item down by one position in the PayloadList.

        Args:
            index (int):
                The index of the item to move down.

        Returns:
            int:
                The new position of the payload item.

        """

        if index < 0 or index >= len(self.payload_items) - 1:
            self.logger.error(
                "Index -> %s is out of range or already at the bottom!",
                str(index),
            )
            return None

        self.payload_items.iloc[[index, index + 1]] = self.payload_items.iloc[[index + 1, index]].to_numpy()

        new_postion = self.payload_items.index.get_loc(index)

        return new_postion

    # end method definition

    def __len__(self) -> int:
        """Return the number of items in the PayloadList.

        Returns:
            int:
                The count of items in the list.

        """

        return len(self.payload_items)

    # end method definition

    def __getitem__(self, index: int) -> pd.Series:
        """Get an item by its index using the "[index]" syntax.

        Args:
            index (int):
                The index of the item to retrieve.

        Returns:
            pd.Series:
                The item at the specified index as a Series.

        Raises:
            IndexError: If the index is out of range.

        Example:
            >>> payload_list = PayloadList()
            >>> payload_list.add_item("Task1", "task1.txt")
            >>> payload_list[0]
            name        Task1
            file    task1.txt
            dependencies    NaN
            logfile      None
            status    planned
            Name: 0, dtype: object

        """

        if index not in self.payload_items.index:
            exception = "Index -> {} is out of range".format(index)
            raise IndexError(exception)

        return self.payload_items.loc[index]

    # end method definition

    def __setitem__(self, index: int, value: dict) -> None:
        """Set an item at the specified index using the "[index]" syntax.

        Args:
            index (int): The index to set the item at.
            value (dict): The item dictionary to set, which must include 'name' and 'file' keys.

        Raises:
            IndexError: If the index is out of range.
            ValueError: If the provided value is not a valid item dictionary.

        Example:
            >>> payload_list = PayloadList()
            >>> payload_list.add_item("Task1", "task1.txt")
            >>> payload_list[0]
            name        Task1
            filename    task1.txt
            dependencies    NaN
            logfile      None
            status    planned
            Name: 0, dtype: object
            >>> payload_list[0] = {"name": "Updated Task1", "file": "updated_task1.txt", "status": "completed"}
            >>> payload_list[0]
            name        Updated Task1
            filename    updated_task1.txt
            dependencies    NaN
            logfile      None
            status    completed
            Name: 0, dtype: object

        """

        if not {"name", "filename"}.issubset(value):
            exception = ("Value must be a dictionary with at least 'name' and 'filename' keys",)
            raise ValueError(
                exception,
            )

        if index not in self.payload_items.index:
            exception = "Index -> {} is out of range".format(index)
            raise IndexError(exception)

        self.payload_items.loc[index] = value

    # end method definition

    def __delitem__(self, index: int) -> None:
        """Delete an item by its index.

        Args:
            index (int): The index of the item to delete.

        Raises:
            IndexError: If the index is out of range.

        """

        self.remove_item(index=index)

    # end method definition

    def __getattr__(self, attribute: str) -> pd.Series:
        """Provide dynamic access to columns using the "." syntax.

        For example, `payload_list.name` will return the 'name' column values.

        Args:
            attribute (str): The column name to retrieve.

        Returns:
            pd.Series: The specified column as a pandas Series.

        Example:
            >>> payload_list = PayloadList()
            >>> payload_list.add_item("Task1", "task1.txt")
            >>> payload_list.name
            0    Task1
            Name: name, dtype: object

        """

        if attribute in self.payload_items.columns:
            return self.payload_items[attribute]

        self.logger.error("Payload list has no attribute -> '%s'", attribute)
        return None

    # end method definition

    def __repr__(self) -> str:
        """Return a string representation of the PayloadList for logging and debugging.

        Returns:
            str:
                A string representing the items in the DataFrame.

        """

        return self.payload_items.to_string(index=True)

    # end method definition

    def __iter__(self) -> iter:
        """Iterate over the rows of the PayloadList.

        Returns:
            iterator: An iterator over the rows of the payload_items DataFrame.

        Example:
            >>> payload_list = PayloadList()
            >>> payload_list.add_item("Task1", "task1.txt")
            >>> payload_list.add_item("Task2", "task2.txt")
            >>> for payload in payload_list:
            >>>     print(payload)
            name        Task1
            filename    task1.txt
            dependencies    NaN
            logfile      None
            status    planned
            Name: 0, dtype: object
            name        Task2
            file    task2.txt
            dependencies    NaN
            logfile      None
            status    planned
            Name: 1, dtype: object

        """

        # Return an iterator for the rows of the DataFrame
        for _, row in self.payload_items.iterrows():
            yield row

    # end method definition

    def pick_runnables(self) -> pd.DataFrame:
        """Pick all PayloadItems with status "planned" and no dependencies on items that are not in status "completed".

        Returns:
            pd.DataFrame:
                A list of runnable payload items.

        """

        def is_runnable(row: pd.Series) -> bool:
            # Check if item is enabled:
            if not row["enabled"]:
                return False

            # Check if all dependencies have been completed
            dependencies: list[int] = row["dependencies"]

            return all(self.payload_items.loc[dep, "status"] == "completed" for dep in dependencies or [])

        # end sub-method definition

        if self.payload_items.empty:
            return None

        # Filter payload items to find runnable items
        runnable_df: pd.DataFrame = self.payload_items[
            (self.payload_items["status"] == "planned") & self.payload_items.apply(is_runnable, axis=1)
        ].copy()

        # Add index as a column to the resulting DataFrame
        runnable_df["index"] = runnable_df.index

        # Log each runnable item
        for _, row in runnable_df.iterrows():
            self.logger.debug(
                "Added payload file -> '%s' with index -> %s to runnable queue.",
                row["name"] if row["name"] else row["filename"],
                row["index"],
            )

        return runnable_df

    # end method definition

    def pick_running(self) -> int:
        """Pick all PayloadItems with status "running".

        Returns:
            pd.DataFrame:
                A list of running payload items.

        """

        if self.payload_items.empty:
            return 0

        all_status = self.payload_items["status"].value_counts().to_dict()

        return all_status.get("running", 0)

    # end method definition

    def process_payload_list(self, concurrent: int | None = None) -> None:
        """Process runnable payloads.

        Args:
            concurrent (int | None, optional):
                The maximum number of concurrent payloads to run at any given time.

        Continuously checks for runnable payload items and starts their
        "process_payload" method in separate threads.
        Runs as a daemon until the customizer ends.

        """

        @tracer.start_as_current_span("run_and_complete_payload")
        def run_and_complete_payload(payload_item: pd.Series) -> None:
            """Run the payload's process_payload method and marks the status as completed afterward."""

            t = trace.get_current_span()
            t.set_attributes({"payload_id": payload_item["index"], "payload_name": payload_item["name"]})

            start_time = datetime.now(UTC)
            self.update_payload_item(payload_item["index"], {"start_time": start_time})

            # Create a logger with thread_id:
            thread_logger = logging.getLogger(
                name="Payload_{}".format(payload_item["index"]),
            )

            thread_logger.setLevel(level=payload_item["loglevel"])

            # Check if the logger already has handlers. If it does, they are removed before creating new ones.
            if thread_logger.hasHandlers():
                thread_logger.handlers.clear()

            # Create a handler for the logger:
            handler = logging.FileHandler(filename=payload_item.logfile)

            # Create a formatter:
            formatter = logging.Formatter(
                fmt="%(asctime)s %(levelname)s [%(name)s] [%(threadName)s] %(message)s",
                datefmt="%d-%b-%Y %H:%M:%S",
            )
            # Add the formatter to the handler
            handler.setFormatter(fmt=formatter)
            thread_logger.addHandler(hdlr=handler)

            if len(thread_logger.filters) == 0:
                thread_logger.debug("Adding log count filter to logger")
                thread_logger.addFilter(
                    LogCountFilter(
                        payload_items=self.payload_items,
                        index=payload_item["index"],
                    ),
                )

            thread_logger.info(
                "Start processing of payload -> '%s' (%s) from filename -> '%s'",
                payload_item["name"],
                payload_item["index"],
                payload_item["filename"],
            )

            local = threading.local()

            # Read customizer Settings from customizerSettings in the payload:
            payload = load_payload(payload_item["filename"])

            if not payload:
                success = False

            if payload:
                customizer_settings = payload_item["customizer_settings"]

                # Overwrite the customizer settings with the payload specific ones:
                customizer_settings.update(
                    {
                        "cust_payload": payload_item["filename"],
                        "cust_payload_gz": "",
                        "cust_payload_external": "",
                        "cust_log_file": payload_item.logfile,
                    },
                )

                try:
                    local.customizer_thread_object = Customizer(
                        settings=customizer_settings,
                        logger=thread_logger,
                    )
                    thread_logger.info("Customizer initialized successfully.")

                    thread_logger.debug(
                        "Customizer Settings -> \n %s",
                        pprint.pformat(
                            local.customizer_thread_object.settings.model_dump(),
                        ),
                    )

                    if customizer_settings.get("profiling", False):
                        import cProfile
                        import pstats

                        cprofiler = cProfile.Profile()
                        cprofiler.enable()

                    success = local.customizer_thread_object.customization_run()

                    if customizer_settings.get("profiling", False):
                        cprofiler.disable()

                    now = datetime.now(UTC)
                    log_path = os.path.dirname(payload_item.logfile)
                    profile_log_prefix = (
                        f"{log_path}/{payload_item['index']}_{payload_item['name']}_{now.strftime('%Y-%m-%d_%H-%M-%S')}"
                    )

                    if customizer_settings.get("profiling", False):
                        import io

                        s = io.StringIO()
                        stats = pstats.Stats(cprofiler, stream=s).sort_stats("cumtime")
                        stats.print_stats()
                        with open(f"{profile_log_prefix}.log", "w+") as f:
                            f.write(s.getvalue())
                        stats.dump_stats(filename=f"{profile_log_prefix}.cprof")

                except ValidationError:
                    thread_logger.error("Validation error!")
                    success = False

                except StopOnError:
                    success = False
                    thread_logger.error(
                        "StopOnErrorException occurred. Stopping payload processing...",
                    )

                except Exception:
                    success = False
                    thread_logger.error(
                        "An exception occurred: \n%s",
                        traceback.format_exc(),
                    )

            if not success:
                thread_logger.error(
                    "Failed to initialize payload -> '%s'!",
                    payload_item["filename"],
                )
                # Update the status to "failed" in the DataFrame after processing finishes
                self.update_payload_item(payload_item["index"], {"status": "failed"})

            else:
                # Update the status to "completed" in the DataFrame after processing finishes
                self.update_payload_item(payload_item["index"], {"status": "completed"})

            stop_time = datetime.now(UTC)
            duration = stop_time - start_time

            # Format duration in hh:mm:ss
            hours, remainder = divmod(duration.total_seconds(), 3600)
            minutes, seconds = divmod(remainder, 60)
            formatted_duration = f"{int(hours):02}:{int(minutes):02}:{int(seconds):02}"

            self.update_payload_item(
                payload_item["index"],
                {"stop_time": stop_time, "duration": formatted_duration},
            )

        # end  def run_and_complete_payload()

        # add delay here to allow for logging to work reliably for the the first payload
        time.sleep(10)

        while not self._stopped:
            # Get runnable items as subset of the initial data frame:
            runnable_items: pd.DataFrame = self.pick_runnables()

            # Start a thread for each runnable item (item is a pd.Series)
            if runnable_items is not None:
                for _, item in runnable_items.iterrows():
                    if concurrent and self.pick_running() >= concurrent:
                        self.logger.debug(
                            "Reached concurrency limit of %s payloads. Waiting for one to finish.",
                        )
                        break

                    self.logger.info(
                        "Added payload file -> '%s' with index -> %s to runnable queue.",
                        item["name"] if item["name"] else item["filename"],
                        item["index"],
                    )

                    # Update the status to "running" in the data frame to prevent re-processing
                    self.payload_items.loc[
                        self.payload_items["name"] == item["name"],
                        "status",
                    ] = "running"

                    # Start the process_payload method in a new thread
                    thread = threading.Thread(
                        target=run_and_complete_payload,
                        args=(item,),
                        name=item["name"],
                    )
                    thread.start()
                    break

            # Sleep briefly to avoid a busy wait loop
            time.sleep(1)

    # end method definition

    def run_payload_processing(self, concurrent: int | None = None) -> None:
        """Start the `process_payload_list` method in a daemon thread."""

        scheduler_thread = threading.Thread(
            target=self.process_payload_list,
            daemon=True,
            name="Scheduler",
            kwargs={"concurrent": concurrent},
        )

        self.logger.info(
            "Starting '%s' thread for payload list processing...",
            str(scheduler_thread.name),
        )
        self._stopped = False
        scheduler_thread.start()

    # end method definition

    def stop_payload_processing(self) -> None:
        """Set a stop flag which triggers the stopping of further payload processing."""

        self._stopped = True

__delitem__(index)

Delete an item by its index.

Parameters:

Name	Type	Description	Default
index	int	The index of the item to delete.	required

Raises:

Type	Description
IndexError	If the index is out of range.

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def __delitem__(self, index: int) -> None:
    """Delete an item by its index.

    Args:
        index (int): The index of the item to delete.

    Raises:
        IndexError: If the index is out of range.

    """

    self.remove_item(index=index)

__getattr__(attribute)

Provide dynamic access to columns using the "." syntax.

For example, payload_list.name will return the 'name' column values.

Parameters:

Name	Type	Description	Default
attribute	str	The column name to retrieve.	required

Returns:

Type	Description
Series	pd.Series: The specified column as a pandas Series.

Example

payload_list = PayloadList() payload_list.add_item("Task1", "task1.txt") payload_list.name 0 Task1 Name: name, dtype: object

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def __getattr__(self, attribute: str) -> pd.Series:
    """Provide dynamic access to columns using the "." syntax.

    For example, `payload_list.name` will return the 'name' column values.

    Args:
        attribute (str): The column name to retrieve.

    Returns:
        pd.Series: The specified column as a pandas Series.

    Example:
        >>> payload_list = PayloadList()
        >>> payload_list.add_item("Task1", "task1.txt")
        >>> payload_list.name
        0    Task1
        Name: name, dtype: object

    """

    if attribute in self.payload_items.columns:
        return self.payload_items[attribute]

    self.logger.error("Payload list has no attribute -> '%s'", attribute)
    return None

__getitem__(index)

Get an item by its index using the "[index]" syntax.

Parameters:

Name	Type	Description	Default
index	int	The index of the item to retrieve.	required

Returns:

Type	Description
Series	pd.Series: The item at the specified index as a Series.

Raises:

Type	Description
IndexError	If the index is out of range.

Example

payload_list = PayloadList() payload_list.add_item("Task1", "task1.txt") payload_list[0] name Task1 file task1.txt dependencies NaN logfile None status planned Name: 0, dtype: object

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def __getitem__(self, index: int) -> pd.Series:
    """Get an item by its index using the "[index]" syntax.

    Args:
        index (int):
            The index of the item to retrieve.

    Returns:
        pd.Series:
            The item at the specified index as a Series.

    Raises:
        IndexError: If the index is out of range.

    Example:
        >>> payload_list = PayloadList()
        >>> payload_list.add_item("Task1", "task1.txt")
        >>> payload_list[0]
        name        Task1
        file    task1.txt
        dependencies    NaN
        logfile      None
        status    planned
        Name: 0, dtype: object

    """

    if index not in self.payload_items.index:
        exception = "Index -> {} is out of range".format(index)
        raise IndexError(exception)

    return self.payload_items.loc[index]

__init__(logger=default_logger)

Initialize the Payload List object.

Parameters:

Name	Type	Description	Default
logger	Logger	The logging object to use for all log messages. Defaults to default_logger.	default_logger

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def __init__(self, logger: logging.Logger = default_logger) -> None:
    """Initialize the Payload List object.

    Args:
        logger (logging.Logger, optional):
            The logging object to use for all log messages. Defaults to default_logger.

    """
    if logger != default_logger:
        self.logger = logging.getLogger(f"{logger.name}.payload_list")

    self.payload_items = pd.DataFrame(
        columns=[
            "name",
            "filename",
            "dependencies",
            "logfile",
            "status",
            "enabled",
            "git_url",
            "loglevel",
            "start_time",
            "stop_time",
            "duration",
            "log_debug",
            "log_info",
            "log_warning",
            "log_error",
            "log_critical",
            "customizer_settings",
        ],
    )

__iter__()

Iterate over the rows of the PayloadList.

Returns:

Name	Type	Description
iterator	iter	An iterator over the rows of the payload_items DataFrame.

Example

payload_list = PayloadList() payload_list.add_item("Task1", "task1.txt") payload_list.add_item("Task2", "task2.txt") for payload in payload_list: print(payload) name Task1 filename task1.txt dependencies NaN logfile None status planned Name: 0, dtype: object name Task2 file task2.txt dependencies NaN logfile None status planned Name: 1, dtype: object

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def __iter__(self) -> iter:
    """Iterate over the rows of the PayloadList.

    Returns:
        iterator: An iterator over the rows of the payload_items DataFrame.

    Example:
        >>> payload_list = PayloadList()
        >>> payload_list.add_item("Task1", "task1.txt")
        >>> payload_list.add_item("Task2", "task2.txt")
        >>> for payload in payload_list:
        >>>     print(payload)
        name        Task1
        filename    task1.txt
        dependencies    NaN
        logfile      None
        status    planned
        Name: 0, dtype: object
        name        Task2
        file    task2.txt
        dependencies    NaN
        logfile      None
        status    planned
        Name: 1, dtype: object

    """

    # Return an iterator for the rows of the DataFrame
    for _, row in self.payload_items.iterrows():
        yield row

__len__()

Return the number of items in the PayloadList.

Returns:

Name	Type	Description
int	int	The count of items in the list.

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def __len__(self) -> int:
    """Return the number of items in the PayloadList.

    Returns:
        int:
            The count of items in the list.

    """

    return len(self.payload_items)

__repr__()

Return a string representation of the PayloadList for logging and debugging.

Returns:

Name	Type	Description
str	str	A string representing the items in the DataFrame.

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def __repr__(self) -> str:
    """Return a string representation of the PayloadList for logging and debugging.

    Returns:
        str:
            A string representing the items in the DataFrame.

    """

    return self.payload_items.to_string(index=True)

__setitem__(index, value)

Set an item at the specified index using the "[index]" syntax.

Parameters:

Name	Type	Description	Default
index	int	The index to set the item at.	required
value	dict	The item dictionary to set, which must include 'name' and 'file' keys.	required

Raises:

Type	Description
IndexError	If the index is out of range.
ValueError	If the provided value is not a valid item dictionary.

Example

payload_list = PayloadList() payload_list.add_item("Task1", "task1.txt") payload_list[0] name Task1 filename task1.txt dependencies NaN logfile None status planned Name: 0, dtype: object payload_list[0] = {"name": "Updated Task1", "file": "updated_task1.txt", "status": "completed"} payload_list[0] name Updated Task1 filename updated_task1.txt dependencies NaN logfile None status completed Name: 0, dtype: object

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def __setitem__(self, index: int, value: dict) -> None:
    """Set an item at the specified index using the "[index]" syntax.

    Args:
        index (int): The index to set the item at.
        value (dict): The item dictionary to set, which must include 'name' and 'file' keys.

    Raises:
        IndexError: If the index is out of range.
        ValueError: If the provided value is not a valid item dictionary.

    Example:
        >>> payload_list = PayloadList()
        >>> payload_list.add_item("Task1", "task1.txt")
        >>> payload_list[0]
        name        Task1
        filename    task1.txt
        dependencies    NaN
        logfile      None
        status    planned
        Name: 0, dtype: object
        >>> payload_list[0] = {"name": "Updated Task1", "file": "updated_task1.txt", "status": "completed"}
        >>> payload_list[0]
        name        Updated Task1
        filename    updated_task1.txt
        dependencies    NaN
        logfile      None
        status    completed
        Name: 0, dtype: object

    """

    if not {"name", "filename"}.issubset(value):
        exception = ("Value must be a dictionary with at least 'name' and 'filename' keys",)
        raise ValueError(
            exception,
        )

    if index not in self.payload_items.index:
        exception = "Index -> {} is out of range".format(index)
        raise IndexError(exception)

    self.payload_items.loc[index] = value

add_payload_item(name, filename, logfile, dependencies=None, status='pending', enabled=True, git_url=None, loglevel='INFO', customizer_settings=None)

Add a new item to the PayloadList.

Parameters:

Name	Type	Description	Default
name	str	The name of the item.	required
filename	str	The file associated with the item.	required
logfile	str	Log file information for the item. Defaults to None.	required
dependencies	list	The index of another item this item depends on. Defaults to None.	None
status	str	The status of the item. Must be one of 'planned', 'running', 'completed', or 'failed'. Defaults to 'planned'.	'pending'
enabled	bool	True if the payload is enabled. False otherwise.	True
git_url	str	Link to the payload in the GIT repository.	None
loglevel	str	The log level for processing the payload. Either "INFO" or "DEBUG".	'INFO'
customizer_settings	dict	Customizer settings for the payload. Defaults to None.	None

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def add_payload_item(
    self,
    name: str,
    filename: str,
    logfile: str,
    dependencies: list | None = None,
    status: str = "pending",
    enabled: bool = True,
    git_url: str | None = None,
    loglevel: str = "INFO",
    customizer_settings: dict | None = None,
) -> dict:
    """Add a new item to the PayloadList.

    Args:
        name (str):
            The name of the item.
        filename (str):
            The file associated with the item.
        logfile (str):
            Log file information for the item. Defaults to None.
        dependencies (list):
            The index of another item this item depends on. Defaults to None.
        status (str):
            The status of the item. Must be one of 'planned', 'running',
            'completed', or 'failed'. Defaults to 'planned'.
        enabled (bool):
            True if the payload is enabled. False otherwise.
        git_url (str):
            Link to the payload in the GIT repository.
        loglevel (str):
            The log level for processing the payload. Either "INFO" or "DEBUG".
        customizer_settings (dict):
            Customizer settings for the payload. Defaults to None.

    """

    new_item = {
        "name": name if name else filename,
        "filename": filename,
        "dependencies": dependencies if dependencies else [],
        "logfile": logfile,
        "status": status,
        "enabled": enabled,
        "git_url": git_url,
        "loglevel": loglevel,
        "log_debug": 0,
        "log_info": 0,
        "log_warning": 0,
        "log_error": 0,
        "log_critical": 0,
        "customizer_settings": customizer_settings if customizer_settings else {},
    }
    self.payload_items = pd.concat(
        [self.payload_items, pd.DataFrame([new_item])],
        ignore_index=True,
    )

    new_item = self.payload_items.tail(1).to_dict(orient="records")[0]
    new_item["index"] = self.payload_items.index[-1]

    return new_item

calculate_payload_item_duration()

Update the dataframe column "duration" for all running items.

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def calculate_payload_item_duration(self) -> None:
    """Update the dataframe column "duration" for all running items."""

    def calculate_duration(row: pd.Series) -> str:
        if row["status"] == "running":
            now = datetime.now(UTC)
            start_time = pd.to_datetime(row["start_time"])

            duration = now - start_time
            hours, remainder = divmod(duration.total_seconds(), 3600)
            minutes, seconds = divmod(remainder, 60)
            formatted_duration = f"{int(hours):02}:{int(minutes):02}:{int(seconds):02}"

            return formatted_duration
        else:
            return str(row["duration"])  # or whatever the original value should be

    # updates the "duration" column of the DataFrame self.payload_items
    # by applying the method calculate_duration() to each row:
    self.payload_items["duration"] = self.payload_items.apply(
        calculate_duration,
        axis=1,
    )

get_payload_item(index)

Get the payload item by index if it exists, otherwise return None.

Parameters:

Name	Type	Description	Default
index	int	index of the row	required

Returns:

Type	Description
Series	pd.Series: row with the matching index or None if there is no row with that index

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def get_payload_item(self, index: int) -> pd.Series:
    """Get the payload item by index if it exists, otherwise return None.

    Args:
        index (int): index of the row

    Returns:
        pd.Series: row with the matching index or None if there is no row with that index

    """

    self.calculate_payload_item_duration()

    if index not in self.payload_items.index:
        self.logger.error("Index -> %s is out of range", str(index))
        return None

    return self.payload_items.loc[index]

get_payload_item_by_name(name)

Get the payload item by name if it exists, otherwise return None.

Parameters:

Name	Type	Description	Default
name	str	The name of the payload.	required

Returns:

Type	Description
Series	pd.Series: Row with the matching name or None if there is no row with that index

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def get_payload_item_by_name(self, name: str) -> pd.Series:
    """Get the payload item by name if it exists, otherwise return None.

    Args:
        name (str):
            The name of the payload.

    Returns:
        pd.Series:
            Row with the matching name or None if there is no row with that index

    """

    self.calculate_payload_item_duration()

    df = self.get_payload_items()
    data = [{"index": idx, **row} for idx, row in df.iterrows()]

    return next((item for item in data if item.get("name") == name), None)

get_payload_items()

Get the payload items in their current order in the PayloadList.

Returns:

Type	Description
DataFrame	pd.DataFrame: A data frame containing all items in their current order.

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def get_payload_items(self) -> pd.DataFrame:
    """Get the payload items in their current order in the PayloadList.

    Returns:
        pd.DataFrame:
            A data frame containing all items in their current order.

    """

    self.calculate_payload_item_duration()

    return self.payload_items

get_payload_items_by_value(column, value)

Filter the PayloadList by a given value in a specific column.

Parameters:

Name	Type	Description	Default
column	str	The column to filter by.	required
value	str	The value to match in the specified column.	required

Returns:

Type	Description
DataFrame | None	pd.DataFrame: A DataFrame containing rows where the given column matches the value.

Example

payload_list = PayloadList() payload_list.add_item("Task1", "task1.txt", status="running") payload_list.add_item("Task2", "task2.txt", status="completed") payload_list.add_item("Task3", "task3.txt", status="running") payload_list.get_payload_items_by_value(column="status", value="running") name file dependencies logfile status enabled 0 Task1 task1.txt NaN None running True 2 Task3 task3.txt NaN None running True

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def get_payload_items_by_value(
    self,
    column: str,
    value: str,
) -> pd.DataFrame | None:
    """Filter the PayloadList by a given value in a specific column.

    Args:
        column (str):
            The column to filter by.
        value (str):
            The value to match in the specified column.

    Returns:
        pd.DataFrame: A DataFrame containing rows where the given column matches the value.

    Example:
        >>> payload_list = PayloadList()
        >>> payload_list.add_item("Task1", "task1.txt", status="running")
        >>> payload_list.add_item("Task2", "task2.txt", status="completed")
        >>> payload_list.add_item("Task3", "task3.txt", status="running")
        >>> payload_list.get_payload_items_by_value(column="status", value="running")
            name         file    dependencies  logfile   status    enabled
        0   Task1    task1.txt        NaN        None   running     True
        2   Task3    task3.txt        NaN        None   running     True

    """

    if column not in self.payload_items.columns:
        self.logger.error(
            "Column -> '%s' does not exist in the payload list!",
            str(column),
        )
        return None

    filtered_items = self.payload_items[self.payload_items[column] == value]

    return filtered_items

move_payload_item_down(index)

Move an item down by one position in the PayloadList.

Parameters:

Name	Type	Description	Default
index	int	The index of the item to move down.	required

Returns:

Name	Type	Description
int	int | None	The new position of the payload item.

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def move_payload_item_down(self, index: int) -> int | None:
    """Move an item down by one position in the PayloadList.

    Args:
        index (int):
            The index of the item to move down.

    Returns:
        int:
            The new position of the payload item.

    """

    if index < 0 or index >= len(self.payload_items) - 1:
        self.logger.error(
            "Index -> %s is out of range or already at the bottom!",
            str(index),
        )
        return None

    self.payload_items.iloc[[index, index + 1]] = self.payload_items.iloc[[index + 1, index]].to_numpy()

    new_postion = self.payload_items.index.get_loc(index)

    return new_postion

move_payload_item_up(index)

Move an item up by one position in the PayloadList.

Parameters:

Name	Type	Description	Default
index	int	The index of the item to move up.	required

Results

bool: False, if the index is out of range or the item is already at the top. True otherwise

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def move_payload_item_up(self, index: int) -> int | None:
    """Move an item up by one position in the PayloadList.

    Args:
        index (int): The index of the item to move up.

    Results:
        bool: False, if the index is out of range or the item is already at the top.
              True otherwise

    """

    if index <= 0 or index >= len(self.payload_items):
        self.logger.error(
            "Index -> %s is out of range or already at the top!",
            str(index),
        )
        return None

    self.payload_items.iloc[[index - 1, index]] = self.payload_items.iloc[[index, index - 1]].to_numpy()

    new_postion = self.payload_items.index.get_loc(index)

    return new_postion

pick_runnables()

Pick all PayloadItems with status "planned" and no dependencies on items that are not in status "completed".

Returns:

Type	Description
DataFrame	pd.DataFrame: A list of runnable payload items.

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def pick_runnables(self) -> pd.DataFrame:
    """Pick all PayloadItems with status "planned" and no dependencies on items that are not in status "completed".

    Returns:
        pd.DataFrame:
            A list of runnable payload items.

    """

    def is_runnable(row: pd.Series) -> bool:
        # Check if item is enabled:
        if not row["enabled"]:
            return False

        # Check if all dependencies have been completed
        dependencies: list[int] = row["dependencies"]

        return all(self.payload_items.loc[dep, "status"] == "completed" for dep in dependencies or [])

    # end sub-method definition

    if self.payload_items.empty:
        return None

    # Filter payload items to find runnable items
    runnable_df: pd.DataFrame = self.payload_items[
        (self.payload_items["status"] == "planned") & self.payload_items.apply(is_runnable, axis=1)
    ].copy()

    # Add index as a column to the resulting DataFrame
    runnable_df["index"] = runnable_df.index

    # Log each runnable item
    for _, row in runnable_df.iterrows():
        self.logger.debug(
            "Added payload file -> '%s' with index -> %s to runnable queue.",
            row["name"] if row["name"] else row["filename"],
            row["index"],
        )

    return runnable_df

pick_running()

Pick all PayloadItems with status "running".

Returns:

Type	Description
int	pd.DataFrame: A list of running payload items.

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def pick_running(self) -> int:
    """Pick all PayloadItems with status "running".

    Returns:
        pd.DataFrame:
            A list of running payload items.

    """

    if self.payload_items.empty:
        return 0

    all_status = self.payload_items["status"].value_counts().to_dict()

    return all_status.get("running", 0)

process_payload_list(concurrent=None)

Process runnable payloads.

Parameters:

Name	Type	Description	Default
concurrent	int | None	The maximum number of concurrent payloads to run at any given time.	None

Continuously checks for runnable payload items and starts their "process_payload" method in separate threads. Runs as a daemon until the customizer ends.

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def process_payload_list(self, concurrent: int | None = None) -> None:
    """Process runnable payloads.

    Args:
        concurrent (int | None, optional):
            The maximum number of concurrent payloads to run at any given time.

    Continuously checks for runnable payload items and starts their
    "process_payload" method in separate threads.
    Runs as a daemon until the customizer ends.

    """

    @tracer.start_as_current_span("run_and_complete_payload")
    def run_and_complete_payload(payload_item: pd.Series) -> None:
        """Run the payload's process_payload method and marks the status as completed afterward."""

        t = trace.get_current_span()
        t.set_attributes({"payload_id": payload_item["index"], "payload_name": payload_item["name"]})

        start_time = datetime.now(UTC)
        self.update_payload_item(payload_item["index"], {"start_time": start_time})

        # Create a logger with thread_id:
        thread_logger = logging.getLogger(
            name="Payload_{}".format(payload_item["index"]),
        )

        thread_logger.setLevel(level=payload_item["loglevel"])

        # Check if the logger already has handlers. If it does, they are removed before creating new ones.
        if thread_logger.hasHandlers():
            thread_logger.handlers.clear()

        # Create a handler for the logger:
        handler = logging.FileHandler(filename=payload_item.logfile)

        # Create a formatter:
        formatter = logging.Formatter(
            fmt="%(asctime)s %(levelname)s [%(name)s] [%(threadName)s] %(message)s",
            datefmt="%d-%b-%Y %H:%M:%S",
        )
        # Add the formatter to the handler
        handler.setFormatter(fmt=formatter)
        thread_logger.addHandler(hdlr=handler)

        if len(thread_logger.filters) == 0:
            thread_logger.debug("Adding log count filter to logger")
            thread_logger.addFilter(
                LogCountFilter(
                    payload_items=self.payload_items,
                    index=payload_item["index"],
                ),
            )

        thread_logger.info(
            "Start processing of payload -> '%s' (%s) from filename -> '%s'",
            payload_item["name"],
            payload_item["index"],
            payload_item["filename"],
        )

        local = threading.local()

        # Read customizer Settings from customizerSettings in the payload:
        payload = load_payload(payload_item["filename"])

        if not payload:
            success = False

        if payload:
            customizer_settings = payload_item["customizer_settings"]

            # Overwrite the customizer settings with the payload specific ones:
            customizer_settings.update(
                {
                    "cust_payload": payload_item["filename"],
                    "cust_payload_gz": "",
                    "cust_payload_external": "",
                    "cust_log_file": payload_item.logfile,
                },
            )

            try:
                local.customizer_thread_object = Customizer(
                    settings=customizer_settings,
                    logger=thread_logger,
                )
                thread_logger.info("Customizer initialized successfully.")

                thread_logger.debug(
                    "Customizer Settings -> \n %s",
                    pprint.pformat(
                        local.customizer_thread_object.settings.model_dump(),
                    ),
                )

                if customizer_settings.get("profiling", False):
                    import cProfile
                    import pstats

                    cprofiler = cProfile.Profile()
                    cprofiler.enable()

                success = local.customizer_thread_object.customization_run()

                if customizer_settings.get("profiling", False):
                    cprofiler.disable()

                now = datetime.now(UTC)
                log_path = os.path.dirname(payload_item.logfile)
                profile_log_prefix = (
                    f"{log_path}/{payload_item['index']}_{payload_item['name']}_{now.strftime('%Y-%m-%d_%H-%M-%S')}"
                )

                if customizer_settings.get("profiling", False):
                    import io

                    s = io.StringIO()
                    stats = pstats.Stats(cprofiler, stream=s).sort_stats("cumtime")
                    stats.print_stats()
                    with open(f"{profile_log_prefix}.log", "w+") as f:
                        f.write(s.getvalue())
                    stats.dump_stats(filename=f"{profile_log_prefix}.cprof")

            except ValidationError:
                thread_logger.error("Validation error!")
                success = False

            except StopOnError:
                success = False
                thread_logger.error(
                    "StopOnErrorException occurred. Stopping payload processing...",
                )

            except Exception:
                success = False
                thread_logger.error(
                    "An exception occurred: \n%s",
                    traceback.format_exc(),
                )

        if not success:
            thread_logger.error(
                "Failed to initialize payload -> '%s'!",
                payload_item["filename"],
            )
            # Update the status to "failed" in the DataFrame after processing finishes
            self.update_payload_item(payload_item["index"], {"status": "failed"})

        else:
            # Update the status to "completed" in the DataFrame after processing finishes
            self.update_payload_item(payload_item["index"], {"status": "completed"})

        stop_time = datetime.now(UTC)
        duration = stop_time - start_time

        # Format duration in hh:mm:ss
        hours, remainder = divmod(duration.total_seconds(), 3600)
        minutes, seconds = divmod(remainder, 60)
        formatted_duration = f"{int(hours):02}:{int(minutes):02}:{int(seconds):02}"

        self.update_payload_item(
            payload_item["index"],
            {"stop_time": stop_time, "duration": formatted_duration},
        )

    # end  def run_and_complete_payload()

    # add delay here to allow for logging to work reliably for the the first payload
    time.sleep(10)

    while not self._stopped:
        # Get runnable items as subset of the initial data frame:
        runnable_items: pd.DataFrame = self.pick_runnables()

        # Start a thread for each runnable item (item is a pd.Series)
        if runnable_items is not None:
            for _, item in runnable_items.iterrows():
                if concurrent and self.pick_running() >= concurrent:
                    self.logger.debug(
                        "Reached concurrency limit of %s payloads. Waiting for one to finish.",
                    )
                    break

                self.logger.info(
                    "Added payload file -> '%s' with index -> %s to runnable queue.",
                    item["name"] if item["name"] else item["filename"],
                    item["index"],
                )

                # Update the status to "running" in the data frame to prevent re-processing
                self.payload_items.loc[
                    self.payload_items["name"] == item["name"],
                    "status",
                ] = "running"

                # Start the process_payload method in a new thread
                thread = threading.Thread(
                    target=run_and_complete_payload,
                    args=(item,),
                    name=item["name"],
                )
                thread.start()
                break

        # Sleep briefly to avoid a busy wait loop
        time.sleep(1)

remove_payload_item(index)

Remove an item by its index from the PayloadList.

Parameters:

Name	Type	Description	Default
index	int	The index of the item to remove.	required

Returns:

Name	Type	Description
bool	bool	True = success. False = failure.

Raises:

Type	Description
IndexError	If the index is out of range.

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def remove_payload_item(self, index: int) -> bool:
    """Remove an item by its index from the PayloadList.

    Args:
        index (int):
            The index of the item to remove.

    Returns:
        bool:
            True = success. False = failure.

    Raises:
        IndexError: If the index is out of range.

    """

    if index not in self.payload_items.index:
        self.logger.error("Index -> %s is out of range!", index)
        return False

    self.payload_items.drop(index, inplace=True)

    return True

run_payload_processing(concurrent=None)

Start the process_payload_list method in a daemon thread.

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def run_payload_processing(self, concurrent: int | None = None) -> None:
    """Start the `process_payload_list` method in a daemon thread."""

    scheduler_thread = threading.Thread(
        target=self.process_payload_list,
        daemon=True,
        name="Scheduler",
        kwargs={"concurrent": concurrent},
    )

    self.logger.info(
        "Starting '%s' thread for payload list processing...",
        str(scheduler_thread.name),
    )
    self._stopped = False
    scheduler_thread.start()

stop_payload_processing()

Set a stop flag which triggers the stopping of further payload processing.

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def stop_payload_processing(self) -> None:
    """Set a stop flag which triggers the stopping of further payload processing."""

    self._stopped = True

update_payload_item(index, update_data)

Update an existing item in the PayloadList.

Parameters:

Name	Type	Description	Default
index	int	The position of the payload.	required
update_data	str	The data of the item.	required

Returns:

Name	Type	Description
bool	bool	True = success, False = error.

Source code in packages/pyxecm/src/pyxecm_customizer/payload_list.py

def update_payload_item(
    self,
    index: int,
    update_data: dict,
) -> bool:
    """Update an existing item in the PayloadList.

    Args:
        index (int):
            The position of the payload.
        update_data (str):
            The data of the item.

    Returns:
        bool:
            True = success, False = error.

    """

    if index not in self.payload_items.index:
        self.logger.error("Illegal index -> %s for payload update!", index)
        return False

    for column in self.payload_items.columns:
        if column in update_data:
            tmp = self.payload_items.loc[index].astype(object)
            tmp[column] = update_data[column]
            self.payload_items.loc[index] = tmp

    return True