Overview

The Normalisation step in Supplier Data Manager (SDM) is a data clean-up phase that prepares supplier-provided product data for integration into the target system (for example, Akeneo PIM). It ensures that product data is correctly formatted, validated, and ready for subsequent processing steps.

During normalisation, four operations are carried out in sequence:

1. Type checking: Ensures that each field contains the correct data type. For example, if a field is expected to be a number but the value is the string "19", SDM converts it to a number. If the value cannot be converted (for example, "AA"), an error is raised.
2. Requirement level verification: Validates that mandatory fields are filled in according to their configured requirement level. If a field marked as required is empty, an error is raised.
3. Transformations: Executes predefined data transformations to modify or standardise field values.
4. Rule and warning validation: Checks that fields comply with configured business rules and warnings, raising errors or alerts where violations are detected.

This step ensures the integrity and quality of supplier data before it progresses to subsequent steps such as classification or export.

Use cases

The Normalisation module in SDM supports the following scenarios:

1. Basic data normalisation: Validates that product data has the expected types, rules, and requirement levels before categorisation or export.
2. Error and warning handling: SDM automatically flags errors or warnings for missing required fields or incorrect data types, allowing users to identify and fix issues before proceeding.
3. Transformation execution: When a user updates a field, the system re-runs all configured transformations for that field automatically.
4. Selective normalisation: Administrators can specify which fields to include or exclude from the normalisation process using the exclude_fields or include_fields parameters, depending on the data requirements for each product group.

Interface

The Normalisation step in Supplier Data Manager presents three tabs:

• Rows to Check: Displays all rows containing errors or warnings that require user attention.
• Rows Checked by You: Shows rows that have been reviewed and corrected manually by a user.
• Rows Checked by the AI: Displays rows that were processed automatically with no issues detected.

When an error is detected, a counter in the first column indicates how many fields have issues. An error or warning icon appears next to each affected field.

The screenshot above shows the Normalisation step interface with the Rows to Check tab active, displaying product rows with error counters in the first column and warning/error icons next to affected fields.

When editing a row in the Normalisation step:

• By default, only fields with errors are shown. Click Show all the fields in the row to display all fields for that row.
• If you update a field that has a transformation rule, all transformations for that field re-execute automatically. An information icon alerts you when this occurs.
• If a row has only warnings (no critical errors), you can bypass them by clicking Ignore Warnings.

The screenshot above shows the row editing side panel in the Normalisation step, with error descriptions displayed under each affected field and the Ignore Warnings and Show all the fields in the row buttons visible.

Configuration

Run mode

The Normalisation step can be run in one of two modes, configured via the run_mode parameter:

• fast: If the normalisation step completes without any errors or warnings, SDM automatically advances the job to the next step.
• manual (default): The user must review the normalisation results and confirm before advancing, regardless of whether errors or warnings are present.

The screenshot above shows the run mode configuration panel for the Normalisation step, with the fast mode option selected.

The screenshot above shows the run mode configuration panel for the Normalisation step, with the manual mode option selected.

Field inclusion/exclusion

• exclude_fields and include_fields: Specify which fields to include or exclude from the normalisation process. Each entry is an object with:

  • name: The name of the field. If null, the rule applies to all fields.
  • groups[]: A list of groups. If null, all groups are included. [null] selects fields in the null group (which applies to all products).

If both parameters are used, SDM first applies include_fields to select a subset, then removes any fields listed in exclude_fields from that subset.

• groups: null — includes all fields.
• groups: [null] — includes only fields from the null (default) group, which contains all fields not assigned to a specific group.
• groups: [] — includes no fields (equivalent to not specifying the field at all).

Fields configuration

• fields: Overrides for specific fields, allowing rule removal for this normalisation step only. Fields or groups can only be overridden if they already exist in the project — new fields cannot be added here. All field rules must be reset at once; partial overrides are not supported.

Additional parameters

• keep_excluded_fields: Default is true. Determines whether fields excluded from the normalisation process are still retained in the output. For example, if a field such as rpm is populated for a t-shirt but the attribute is inactive, setting this to true retains it in the output. Set to false to discard excluded fields.
• max_requirement_level: Caps the highest requirement level enforced during this normalisation step. For example, if a field is configured as required but max_requirement_level is set to optional, the field becomes optional for this step and no error is raised if it is empty. Default is required, meaning all fields are validated at their full configured requirement level.
• sources: A list of field names to display (but not allow editing) in the side panel during normalisation. Use this to surface reference data that helps users fill in error-prone fields.

Transformations

The Normalisation step supports configurable transformations that clean and standardise field values before further processing.

Rules and warnings

The Normalisation step evaluates configured field rules and warnings:

• A rule is a hard requirement. Fields that fail a rule block the job and prevent the user from advancing past the Normalisation step.
• A warning is a soft requirement. Users can acknowledge that a field does not meet the configured rule and proceed to the next step.

Limitations and known issues

1. Partial overrides not supported: If you need to override rules for a field in the fields parameter, you must re-specify all rules for that field. Modifying only specific rules is not supported.
2. Exclusion behaviour: By default, excluded fields are retained in the output (keep_excluded_fields: true). Configure this explicitly if you want excluded fields to be discarded.
3. Requirement level override: When max_requirement_level is used to lower the enforced level, fields originally marked as required will no longer raise an error if left empty. Ensure this is intentional to avoid incomplete data in downstream steps.
4. Automatic transformation re-execution: When a user modifies a field in the UI, all configured transformations for that field re-execute automatically. This can produce unexpected results if transformation rules are not clearly understood by the user making corrections.