Synchronization
Synchronization Service is responsible for synchronizing data between resources.
- OSGi service class
-
org.forgerock.openidm.sync.impl.SynchronizationService
- OSGi persistent identifier
-
org.forgerock.openidm.synchronization
- Router mapping
-
/sync/*
Synchronization between two resources is defined by mapping in the project file conf/sync.json
.
Each mapping defines the transformation of a source object into a target object, as well as connected lifecycle rules.
- OSGi service class
-
org.forgerock.openidm.sync.impl.SyncMappings
- OSGi persistent identifier
-
org.forgerock.openidm.sync
- Configuration file
-
sync.json
The following chapters describe the details of the mapping definition and connected lifecycle policies.
Types of synchronization
There are several types of synchronization. Changes from the external source system are synchronized using reconciliation or live-sync. Changes in the Wren:IdM repository are propagated to the external target systems using implicit synchonization.
Reconciliation
Reconciliation is a synchronization process that ensures that the data in the target system (e.g. accounts) matches the data in the source system. Reconciliation is always performed for a specific mapping, i.e. between a source and target system.
During reconciliation, the IdM iterates over the entire dataset of the source and target systems to calculate the changes that need to be propagated to the target system. This can take a long time if there are many objects in the external system or if the read / update operation in the external system is slow.
Reconciliation is a fundamental process for compliance enforcement because it can detect invalid data states in the target system (e.g. orphan accounts).
Live-sync
Live-sync is a synchronization process that synchronizes data changes from the external source system to the IdM. Unlike reconciliation, live-sync synchronizes only changes that have been made since the last synchronization. These changes are detected by a system-specific mechanism:
-
retrieve changes based on the modification timestamp
-
retrieve changes based on the sequence number
-
any other supported mechanism
This type of synchronization is used to quickly propagate data changes from the external source system to the IdM.
Synchronization mapping
Mapping defines the transformation from source object to target object, including the definition of a synchronized object lifecycle. There are two types of objects:
-
managed objects
– objects maintained in Wren:IDM repository (e.g. managed user representing user identity) -
system objects
– objects maintained in external systems (e.g. account in external LDAP)
Both can be either source or target objects. Managed user will be the source object when creating accounts the in target system (e.g. LDAP). Managed user will be the target object when synchronizing users from the source personal system.
Mappings are defined inside conf/sync.json
project file.
{
"mappings" : [
{
// First mapping
},
{
// Second mapping
},
// ...
]
}
Each mapping consists of the identification of the source and target resources, validation and correlation rules, mapping properties and handling of synchronization situations.
{
"name" : "csvEmployee_managedUser",
"source" : "system/csv/employee",
"target" : "managed/user",
"validSource" : {
"type" : "text/javascript",
"source" : "!!source.uid && !!source.firstName && !!source.lastName"
},
"correlationQuery" : {
"type" : "text/javascript",
"source" : "var query = { _queryFilter: '_id eq \"' + source.uid + '\"' }; query"
},
"onCreate" : {
"type" : "text/javascript",
"source" : "target.userName = source.lastName.toLowerCase() + source.uid;"
},
"properties" : [
{
"source" : "uid",
"target" : "_id"
},
{
"source" : "firstName",
"target" : "givenName"
},
{
"source" : "lastName",
"target" : "sn"
},
{
"source" : "",
"transform" : {
"type" : "text/javascript",
"source" : "source.firstName.toLowerCase() + '.' + source.lastName.toLowerCase() + '@wrensecurity.org';"
},
"target" : "mail"
},
{
"default" : "FooBar123",
"target" : "password"
}
],
"policies" : [
{
"situation" : "ABSENT",
"action" : "CREATE"
},
{
"situation" : "CONFIRMED",
"action" : "UPDATE"
},
{
"situation" : "UNQUALIFIED",
"action" : "DELETE"
}
]
}
The previous example shows the mapping between HR system (represented by a CSV file) and Wren:IDM (managed object representing user). The mapping source is the employee record in the source CSV file, the mapping target is the managed user in the IdM.
Mapping is defined by the following attributes:
-
name
– name of the mapping -
source
– mapping source (managed
orsystem
resource) -
target
– mapping target (managed
orsystem
resource) -
enableSync
– flag indicating whether implicit synchonization is enabled -
sourceCondition
– script to filter the source dataset to be mapped -
validSource
– script that determines whether the source object is a valid to be mapped -
validTarget
– script that determines whether the target object is a valid to be mapped -
correlationScript
– script to perform correlation between source and target objects -
{synchronization script hooks}
– see Synchronization script hooks -
{target object script hooks}
– see Target object script hooks -
properties
– objects that describe the mapping of attributes between source and target objects (see Properties) -
policies
– definition of actions for synchronization situations (see Policies)
Properties
Each property object defines how the value of a target attribute is determined. There are a number of ways how to map the target attribute value. The value can simply be taken directly from the source attribute, or it can be calculated using non-trivial transformation logic.
{
...
"properties" : [
{
"source" : "lastName",
"target" : "sn"
},
{
"source" : "",
"transform" : {
"type" : "text/javascript",
"source" : "source.lastName.toLowerCase() + '@wrensecurity.org';"
},
"target" : "mail"
},
{
"default" : "FooBar123",
"target" : "password"
}
],
...
}
Property object is defined by the following attributes:
-
source
– source attribute to get value (can be empty, then the value is calculated using the transform script or the default value) -
target
– target attribute to map value -
transform
– script to calculate the target attribute value (global propertysource
can be used to retrieve source object properties) -
default
– default value to be used when no value is calculated from thesource
ortransform
-
condition
– script that determines whether or not to map the target attribute
Policies
Each policy object defines actions to be performed for specific synchronization situations. Each synchronization situation has a default action associated with it (see Synchonization situations and actions).
{
...
"policies" : [
{
"situation" : "ABSENT",
"action" : "CREATE",
"postAction" : {
"type" : "text/javascript",
"source" : "// Send email notification ..."
}
},
{
"situation" : "UNQUALIFIED",
"action" : "DELETE"
}
]
...
}
Property object is defined by the following attributes:
-
situation
– synchronization situation to associate action -
action
– action to be performed for the specified synchronization situation -
postAction
– script to be executed after the action is completed
Synchonization situations and actions
There are several situations that can occur during synchronization. Synchronization situation is determined using the following procedure:
-
Is source object valid?
The source object is valid if it satisfies the valid source requirements. Source objects that have been filtered out by the
validSource
orsourceCondition
scripts are invalid. -
Is source object linked?
Check if there is a link in the IdM for the source object.
-
Is there a valid target object?
Use correlation logic to check whether the valid target object exists for the source object.
Situation | Source valid? | Link? | Target object? | Default action |
---|---|---|---|---|
ABSENT |
yes |
no |
no |
CREATE |
ALL_GONE |
no |
no |
no |
NOREPORT |
AMBIGUOUS |
yes |
no |
yes (>1) |
EXCEPTION |
CONFIRMED |
yes |
yes |
yes |
UPDATE |
FOUND_ALREADY_LINKED |
yes |
no[1] |
yes |
EXCEPTION |
FOUND |
yes |
no |
yes |
UPDATE |
LINK_ONLY |
no |
yes |
no |
EXCEPTION |
MISSING |
yes |
yes |
no |
EXCEPTION |
SOURCE_IGNORED |
no |
no |
no |
REPORT |
SOURCE_MISSING |
no |
yes |
yes |
EXCEPTION |
TARGET_IGNORED |
no |
yes/no |
no[2] |
REPORT |
UNASSIGNED |
? |
no |
yes |
EXCEPTION |
UNQUALIFIED |
no |
yes/no |
yes |
DELETE |
The following situations can only be detected during reconciliation:
-
SOURCE_MISSING
-
TARGET_IGNORED
-
UNASSIGNED
The following situations can only be detected during source object changes (delete):
-
ALL_GONE
-
LINK_ONLY
Action | Description |
---|---|
ASYNC |
Indicates asynchronous process, ignore action or reporting |
CREATE |
Target object should be created and linked |
DELETE |
Target object should be deleted and unlinked |
EXCEPTION |
Throw an exception for the situation |
IGNORE |
Do not perform any action |
LINK |
Correlated target object should be linked |
NOREPORT |
Do not perform action, do not report the situation |
REPORT |
Do not perform action, report the situation |
UNLINK |
Linked target object should be unlinked |
UPDATE |
Target object should be linked and updated |
Synchronization script hooks
Synchronization script hooks provide extension points that allow invoking some logic during various synchronization events.
Script Hook | When Executed | Global Variables |
---|---|---|
postMapping |
Script to execute after implicit synchronization has been performed |
|
onRecon |
Script to execute at the beginning of the reconciliation |
|
result |
Script to execute for each mapping after successful reconciliation |
|
The following global properties are available to all script hooks:
-
context
– execution context (e.g. ReconContext)
onRecon
script hook{
"name" : "csvEmployee_managedUser",
// ...
"onRecon" : {
"type" : "text/javascript",
"source" : "logger.info('Reconciliation for mapping ' + mappingConfig.name + ' started...')"
},
// ...
}
Target object script hooks
Target object script hooks provide extension points that allow invoking some logic during various lifecycle events.
Script Hook | When Executed | Global Variables |
---|---|---|
onCreate |
Target object is being created |
|
onUpdate |
Target object is being updated |
|
onDelete |
Target object is being deleted |
|
onLink |
Source object is being linked to the target object |
|
onUnlink |
Link between source and target objects is being removed |
|
The following global properties are available to all script hooks:
-
context
– execution context (e.g. ReconContext)
onCreate
script hook{
"name" : "csvEmployee_managedUser",
// ...
"onCreate" : {
"type" : "text/javascript",
"source" : "target.userName = source.lastName.toLowerCase() + source.uid;"
},
// ...
}
Links
Link objects represent relationships between source objects and target objects (e.g. link between managed user in IdM and account in LDAP).
The links are used to search for objects in the source / target system and enable the detection of non-standard synchronization situations (e.g. MISSING
situation when linked target object has been deleted).
Links are managed (created / updated / deleted) during the synchronization process and are stored in the IdM database table links
with the following structure:
-
_id
– unique identifier -
_rev
– link object’s revision -
linkType
– mapping name -
firstId
– identifier in the source system -
secondId
– identifier in the target system -
linkQualifier
– link qualifier used for the mapping (default
if no qualifier is specified)