[FEATURE] Data validator: Multiple ids in a scenario should be caught

[nitbharambe]

Problem

Currently we don't support checking multiple ids update within same scenario in the data validator. (See example below)

• The PGM core's behaviour is known and it is undefined behaviour: What ends up happening is that the last update gets applied.
• Checking it is also known and unsupported at data validator: What happens in this case is that data validator applies updates as per search of _get_indexer in the validation util. Hence the duplicate id data gets ignored.

Adding this in the current implementation of validator is possible but not very simple. One motivation of this issue is to make users aware of this behaviour as it is easy to miss. Hence It would be beneficial to support this in the validator sometime.
Another is that we are thinking of migrating data validator to C++ core soon. And maybe we don't want to have large changes in python data validator right now. But lets remember to support this after migration.

Example

from power_grid_model import AttributeType, CalculationType, ComponentType, initialize_array, DatasetType, LoadGenType, PowerGridModel
from power_grid_model.validation import assert_valid_batch_data, assert_valid_input_data
    
# node
node = initialize_array(DatasetType.input, ComponentType.node, 1)
node[AttributeType.id] = [1]
node[AttributeType.u_rated] = [10.5e3]
# source
source = initialize_array(DatasetType.input, ComponentType.source, 1)
source[AttributeType.id] = [10]
source[AttributeType.node] = [1]
source[AttributeType.status] = [1]
source[AttributeType.u_ref] = [1.0]
# load
sym_load = initialize_array(DatasetType.input, ComponentType.sym_load, 2)
sym_load[AttributeType.id] = [4, 7]
sym_load[AttributeType.node] = [1, 1]
sym_load[AttributeType.status] = [1, 1]
sym_load[AttributeType.type] = [LoadGenType.const_power, LoadGenType.const_power]
sym_load[AttributeType.p_specified] = [0e6, 0e6]
sym_load[AttributeType.q_specified] = [5e6, 2e6]

input_data = {
    ComponentType.node: node,
    ComponentType.source: source,
    ComponentType.sym_load: sym_load,
}

# load
sym_load = initialize_array(DatasetType.update, ComponentType.sym_load, (2, 2))
sym_load[AttributeType.id] = [[4, 4], [7, 7]]
sym_load[AttributeType.p_specified] = [[10e6, 20e6], [30e6, 40e6]]

update_data = {
    ComponentType.sym_load: sym_load,
}

assert_valid_input_data(input_data=input_data, calculation_type=CalculationType.power_flow)
assert_valid_batch_data(input_data=input_data, update_data=update_data, calculation_type=CalculationType.power_flow)

model = PowerGridModel(input_data=input_data)
output_data = model.calculate_power_flow(update_data=update_data)

print(f"load output: {output_data[ComponentType.sym_load]['p']}")

Gives

load output: [[20000000.        0.]
 [       0. 40000000.]]

Community Meeting

• I wish to discuss this feature in the bi-monthly PGM community meeting. Please join the PGM community meeting.

[appleweiping]

Hi, I am interested in working on this if a Python-side fix is still wanted before the validator migration.

I reproduced the issue on current main. The per-scenario validation checks whether update IDs exist in the input, but does not check their uniqueness. The update is then merged using the repeated indexer positions, so the later value overwrites the earlier one before the merged data is validated and the duplicate information is lost.

A focused implementation could reuse the existing NotUniqueError machinery to validate explicit update IDs within each scenario before merging. I would cover row and columnar data, duplicates within one scenario, the same ID reused across different scenarios, optional-ID positional updates, and zero-width updates.

Because the issue explicitly notes the planned move of validation to the C++ core, would you prefer this focused Python fix now, or should implementation wait for the migration?

I use OpenAI Codex interactively for repository navigation, reproduction, and small edits. I will personally review, understand, and test every change and remain responsible for the contribution.

[nitbharambe]

Hi @appleweiping, The plan for #1421 is to fully migrate the logic to C++ side. And we are (tentatively) planning this ticket in next 2 3 months. So I imagine implementing on python side at the moment is inefficient. So no need to fix this at the moment.