Skip to content

Commit 49de2b1

Browse files
committed
#424 Add one more corner case
1 parent 577422d commit 49de2b1

File tree

2 files changed

+22
-0
lines changed

2 files changed

+22
-0
lines changed

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1333,6 +1333,7 @@ For multisegment variable lengths tests:
13331333
- #### 2.4.1 to be released soon.
13341334
- [#420](https://github.com/AbsaOSS/cobrix/issues/420) Add _experimental_ support for [fixed blocked (FB)](https://www.ibm.com/docs/en/zos/2.3.0?topic=sets-fixed-length-record-formats) record format.
13351335
- [#422](https://github.com/AbsaOSS/cobrix/issues/422) Fixed decoding of 'broken pipe' (`¦`) character from EBCDIC.
1336+
- [#424](https://github.com/AbsaOSS/cobrix/issues/424) Fixed an ASCII reader corner case.
13361337

13371338
- #### 2.4.0 released 7 September 2021.
13381339
- [#412](https://github.com/AbsaOSS/cobrix/issues/412) Add support for [variable block (VB aka VBVR)](https://www.ibm.com/docs/en/zos/2.3.0?topic=formats-format-v-records) record format.

spark-cobol/src/test/scala/za/co/absa/cobrix/spark/cobol/source/regression/Test13AsciiCrLfText.scala

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -190,4 +190,25 @@ class Test13AsciiCrLfText extends WordSpec with SparkTestBase with BinaryFileFix
190190
assertEqualsMultiline(actual, expected)
191191
}
192192
}
193+
194+
"correctly read text files with a double EOL characters and the last record is too short" in {
195+
val text = "AA\r\nBBC"
196+
withTempBinFile("crlf_empty", ".dat", text.getBytes()) { tmpFileName =>
197+
val df = spark
198+
.read
199+
.format("cobol")
200+
.option("copybook_contents", copybook)
201+
.option("pedantic", "true")
202+
.option("record_format", "D")
203+
.load(tmpFileName)
204+
205+
val expected = """[{"A":"AA"},{"A":"BB"},{"A":"C"}]"""
206+
207+
val count = df.count()
208+
val actual = df.toJSON.collect().mkString("[", ",", "]")
209+
210+
assert(count == 3)
211+
assertEqualsMultiline(actual, expected)
212+
}
213+
}
193214
}

0 commit comments

Comments
 (0)