-
Notifications
You must be signed in to change notification settings - Fork 76
Open
Labels
enhancementNew feature or requestNew feature or request
Milestone
Description
Discussed in #1336
Originally posted by hantsy July 22, 2025
How to apply thefill
to the condition that the fields is missing the csv but defind in the DataSchema.df.convertTo<MyData> { fill { col("NewCol") }.with{ "default" } }Currently, I faced the situation like this, we updated the Data Schema and add some new columns.
Some users use the new schema fields in the CSV file, and other not update and still used the old schema which lacked the new col here.
The
fill
is good to apply the default to old format CSV if they do not provide the new col. But the fill will override the value in the new format CSV file.Is possible to only apply the
fill
to the old format here, and in the new format, use the value in CSV directly.
Adding a fillWhenMissing {}
option would be a good solution for this
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request