Skip to content

convertTo schema fillWhenMissing option #1337

@Jolanrensen

Description

@Jolanrensen

Discussed in #1336

Originally posted by hantsy July 22, 2025
How to apply the fill to the condition that the fields is missing the csv but defind in the DataSchema.

df.convertTo<MyData> {
     fill { col("NewCol") }.with{ "default" }
}

Currently, I faced the situation like this, we updated the Data Schema and add some new columns.

Some users use the new schema fields in the CSV file, and other not update and still used the old schema which lacked the new col here.

The fill is good to apply the default to old format CSV if they do not provide the new col. But the fill will override the value in the new format CSV file.

Is possible to only apply the fill to the old format here, and in the new format, use the value in CSV directly.

Adding a fillWhenMissing {} option would be a good solution for this

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions