From 122caa1338515cec3b40098cd028a64e18a2f0fa Mon Sep 17 00:00:00 2001 From: Steve Suh Date: Mon, 22 Jun 2020 14:28:06 -0700 Subject: [PATCH 1/6] initial commit --- README.md | 2 +- benchmark/scala/pom.xml | 2 +- docs/release-notes/0.12/release-0.12.md | 109 ++++++++++++++++++++++++ eng/Versions.props | 2 +- src/scala/pom.xml | 2 +- 5 files changed, 113 insertions(+), 4 deletions(-) create mode 100644 docs/release-notes/0.12/release-0.12.md diff --git a/README.md b/README.md index 2d5638a97..5b8647ca0 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ 2.3.* - v0.11.0 + v0.12.0 2.4.0 diff --git a/benchmark/scala/pom.xml b/benchmark/scala/pom.xml index 56b8dc1ea..54608d6ef 100644 --- a/benchmark/scala/pom.xml +++ b/benchmark/scala/pom.xml @@ -3,7 +3,7 @@ 4.0.0 com.microsoft.spark microsoft-spark-benchmark - 0.11.0 + 0.12.0 2019 UTF-8 diff --git a/docs/release-notes/0.12/release-0.12.md b/docs/release-notes/0.12/release-0.12.md new file mode 100644 index 000000000..50717915e --- /dev/null +++ b/docs/release-notes/0.12/release-0.12.md @@ -0,0 +1,109 @@ +# .NET for Apache Spark 0.11 Release Notes + +### New Features and Improvements + +* Improve build pipeline ([#510](https://github.com/dotnet/spark/pull/510)) ([#511](https://github.com/dotnet/spark/pull/511)) ([#512](https://github.com/dotnet/spark/pull/512)) ([#513](https://github.com/dotnet/spark/pull/513)) ([#524](https://github.com/dotnet/spark/pull/524)) ([]()) ([]()) +* Support for [dotnet-interactive](https://github.com/dotnet/interactive) ([#515](https://github.com/dotnet/spark/pull/515)) ([#517](https://github.com/dotnet/spark/pull/517)) ([#554](https://github.com/dotnet/spark/pull/554)) +* Update `Microsoft.Data.Analysis` to v`0.4.0` ([#528](https://github.com/dotnet/spark/pull/528)) +* Support for Spark 2.4.6 ([#547](https://github.com/dotnet/spark/pull/547)) +* Update AppName for the C# Spark Examples ([548](https://github.com/dotnet/spark/pull/548)) +* Support for [Hyperspace v0.1.0](https://github.com/microsoft/hyperspace) APIs ([555](https://github.com/dotnet/spark/pull/555)) +* Update maven links in build documentation ([558](https://github.com/dotnet/spark/pull/558)) ([560](https://github.com/dotnet/spark/pull/560)) +* Bug fix caused by `BroadcastVariablesRegistry` ([551](https://github.com/dotnet/spark/pull/551)) + +### Breaking Changes + +* None + +### Compatibility + +#### Backward compatibility + +The following table describes the oldest version of the worker that the current version is compatible with, along with new features that are incompatible with the worker. + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Oldest compatible Microsoft.Spark.Worker versionIncompatible features
v0.9.0DataFrame with Grouped Map UDF (#277)
DataFrame with Vector UDF (#277)
Support for Broadcast Variables (#414)
Support for TimestampType (#428)
v0.12.0Support for dotnet-interactive (#515) (#517) (#554)
+ +#### Forward compatibility + +The following table describes the oldest version of .NET for Apache Spark release that the current worker is compatible with. + + + + + + + + + + + + +
Oldest compatible .NET for Apache Spark release version
v0.9.0
+ +### Supported Spark Versions + +The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Spark Versionmicrosoft-spark JAR
2.3.*microsoft-spark-2.3.x-0.12.0.jar
2.4.0microsoft-spark-2.4.x-0.12.0.jar
2.4.1
2.4.3
2.4.4
2.4.5
2.4.6
2.4.2Not supported
diff --git a/eng/Versions.props b/eng/Versions.props index dc954bcc5..b1d1c2efd 100644 --- a/eng/Versions.props +++ b/eng/Versions.props @@ -1,7 +1,7 @@ - 0.11.0 + 0.12.0 prerelease $(RestoreSources); diff --git a/src/scala/pom.xml b/src/scala/pom.xml index 34ee5c338..aacc2da49 100644 --- a/src/scala/pom.xml +++ b/src/scala/pom.xml @@ -7,7 +7,7 @@ ${microsoft-spark.version} UTF-8 - 0.11.0 + 0.12.0 From 1d0861c0989b47b2e7bf5ef29bdea722195acdd9 Mon Sep 17 00:00:00 2001 From: Steve Suh Date: Mon, 22 Jun 2020 14:29:35 -0700 Subject: [PATCH 2/6] remove extra paranthesis --- docs/release-notes/0.12/release-0.12.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/0.12/release-0.12.md b/docs/release-notes/0.12/release-0.12.md index 50717915e..e5b81a1c1 100644 --- a/docs/release-notes/0.12/release-0.12.md +++ b/docs/release-notes/0.12/release-0.12.md @@ -2,7 +2,7 @@ ### New Features and Improvements -* Improve build pipeline ([#510](https://github.com/dotnet/spark/pull/510)) ([#511](https://github.com/dotnet/spark/pull/511)) ([#512](https://github.com/dotnet/spark/pull/512)) ([#513](https://github.com/dotnet/spark/pull/513)) ([#524](https://github.com/dotnet/spark/pull/524)) ([]()) ([]()) +* Improve build pipeline ([#510](https://github.com/dotnet/spark/pull/510)) ([#511](https://github.com/dotnet/spark/pull/511)) ([#512](https://github.com/dotnet/spark/pull/512)) ([#513](https://github.com/dotnet/spark/pull/513)) ([#524](https://github.com/dotnet/spark/pull/524)) * Support for [dotnet-interactive](https://github.com/dotnet/interactive) ([#515](https://github.com/dotnet/spark/pull/515)) ([#517](https://github.com/dotnet/spark/pull/517)) ([#554](https://github.com/dotnet/spark/pull/554)) * Update `Microsoft.Data.Analysis` to v`0.4.0` ([#528](https://github.com/dotnet/spark/pull/528)) * Support for Spark 2.4.6 ([#547](https://github.com/dotnet/spark/pull/547)) From 064fcdc4debb400e3f485e99cda60d3c7cde483a Mon Sep 17 00:00:00 2001 From: Steve Suh Date: Mon, 22 Jun 2020 16:07:01 -0700 Subject: [PATCH 3/6] PR comments --- docs/release-notes/0.12/release-0.12.md | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/docs/release-notes/0.12/release-0.12.md b/docs/release-notes/0.12/release-0.12.md index e5b81a1c1..2c195476c 100644 --- a/docs/release-notes/0.12/release-0.12.md +++ b/docs/release-notes/0.12/release-0.12.md @@ -1,4 +1,4 @@ -# .NET for Apache Spark 0.11 Release Notes +# .NET for Apache Spark 0.12 Release Notes ### New Features and Improvements @@ -6,10 +6,11 @@ * Support for [dotnet-interactive](https://github.com/dotnet/interactive) ([#515](https://github.com/dotnet/spark/pull/515)) ([#517](https://github.com/dotnet/spark/pull/517)) ([#554](https://github.com/dotnet/spark/pull/554)) * Update `Microsoft.Data.Analysis` to v`0.4.0` ([#528](https://github.com/dotnet/spark/pull/528)) * Support for Spark 2.4.6 ([#547](https://github.com/dotnet/spark/pull/547)) -* Update AppName for the C# Spark Examples ([548](https://github.com/dotnet/spark/pull/548)) -* Support for [Hyperspace v0.1.0](https://github.com/microsoft/hyperspace) APIs ([555](https://github.com/dotnet/spark/pull/555)) -* Update maven links in build documentation ([558](https://github.com/dotnet/spark/pull/558)) ([560](https://github.com/dotnet/spark/pull/560)) -* Bug fix caused by `BroadcastVariablesRegistry` ([551](https://github.com/dotnet/spark/pull/551)) +* Update AppName for the C# Spark Examples ([#548](https://github.com/dotnet/spark/pull/548)) +* Support for [Hyperspace v0.1.0](https://github.com/microsoft/hyperspace) APIs ([#555](https://github.com/dotnet/spark/pull/555)) +* Update maven links in build documentation ([#558](https://github.com/dotnet/spark/pull/558)) ([#560](https://github.com/dotnet/spark/pull/560)) +* Bug fix caused by `BroadcastVariablesRegistry` ([#551](https://github.com/dotnet/spark/pull/551)) +* Expose `DataStreamWriter.ForeachBatch` API ([#549](https://github.com/dotnet/spark/pull/549)) ### Breaking Changes @@ -42,10 +43,6 @@ The following table describes the oldest version of the worker that the current Support for TimestampType (#428) - - v0.12.0 - Support for dotnet-interactive (#515) (#517) (#554) - From 8d1fccb455689969a7e298bfc4666f170436361e Mon Sep 17 00:00:00 2001 From: Steve Suh Date: Mon, 22 Jun 2020 20:57:25 -0700 Subject: [PATCH 4/6] PR comments --- docs/release-notes/0.12/release-0.12.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/release-notes/0.12/release-0.12.md b/docs/release-notes/0.12/release-0.12.md index 2c195476c..073a46ec5 100644 --- a/docs/release-notes/0.12/release-0.12.md +++ b/docs/release-notes/0.12/release-0.12.md @@ -1,16 +1,19 @@ # .NET for Apache Spark 0.12 Release Notes -### New Features and Improvements +### New Features/Improvements and Bug Fixes -* Improve build pipeline ([#510](https://github.com/dotnet/spark/pull/510)) ([#511](https://github.com/dotnet/spark/pull/511)) ([#512](https://github.com/dotnet/spark/pull/512)) ([#513](https://github.com/dotnet/spark/pull/513)) ([#524](https://github.com/dotnet/spark/pull/524)) +* Expose `DataStreamWriter.ForeachBatch` API ([#549](https://github.com/dotnet/spark/pull/549)) * Support for [dotnet-interactive](https://github.com/dotnet/interactive) ([#515](https://github.com/dotnet/spark/pull/515)) ([#517](https://github.com/dotnet/spark/pull/517)) ([#554](https://github.com/dotnet/spark/pull/554)) -* Update `Microsoft.Data.Analysis` to v`0.4.0` ([#528](https://github.com/dotnet/spark/pull/528)) +* Support for [Hyperspace v0.1.0](https://github.com/microsoft/hyperspace) APIs ([#555](https://github.com/dotnet/spark/pull/555)) * Support for Spark 2.4.6 ([#547](https://github.com/dotnet/spark/pull/547)) +* Bug fix caused by `BroadcastVariablesRegistry` ([#551](https://github.com/dotnet/spark/pull/551)) +* Update `Microsoft.Data.Analysis` to v`0.4.0` ([#528](https://github.com/dotnet/spark/pull/528)) + +### Infrastructure / Documentation / Etc. + +* Improve build pipeline ([#510](https://github.com/dotnet/spark/pull/510)) ([#511](https://github.com/dotnet/spark/pull/511)) ([#512](https://github.com/dotnet/spark/pull/512)) ([#513](https://github.com/dotnet/spark/pull/513)) ([#524](https://github.com/dotnet/spark/pull/524)) * Update AppName for the C# Spark Examples ([#548](https://github.com/dotnet/spark/pull/548)) -* Support for [Hyperspace v0.1.0](https://github.com/microsoft/hyperspace) APIs ([#555](https://github.com/dotnet/spark/pull/555)) * Update maven links in build documentation ([#558](https://github.com/dotnet/spark/pull/558)) ([#560](https://github.com/dotnet/spark/pull/560)) -* Bug fix caused by `BroadcastVariablesRegistry` ([#551](https://github.com/dotnet/spark/pull/551)) -* Expose `DataStreamWriter.ForeachBatch` API ([#549](https://github.com/dotnet/spark/pull/549)) ### Breaking Changes From 1e1011029374e1f039762e3bce06eebd02529284 Mon Sep 17 00:00:00 2001 From: Steve Suh Date: Mon, 22 Jun 2020 23:07:27 -0700 Subject: [PATCH 5/6] add known issues section --- docs/release-notes/0.12/release-0.12.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/release-notes/0.12/release-0.12.md b/docs/release-notes/0.12/release-0.12.md index 073a46ec5..2dc938b87 100644 --- a/docs/release-notes/0.12/release-0.12.md +++ b/docs/release-notes/0.12/release-0.12.md @@ -19,6 +19,10 @@ * None +### Known Issues + +* Broadcast variables do not work with [dotnet-interactive](https://github.com/dotnet/interactive) ([#561](https://github.com/dotnet/spark/pull/561)) + ### Compatibility #### Backward compatibility From 4540df2155efa53c37607752e277105619d9bd7c Mon Sep 17 00:00:00 2001 From: Steve Suh Date: Tue, 23 Jun 2020 10:54:18 -0700 Subject: [PATCH 6/6] add 530 --- docs/release-notes/0.12/release-0.12.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/release-notes/0.12/release-0.12.md b/docs/release-notes/0.12/release-0.12.md index 2dc938b87..7000299ea 100644 --- a/docs/release-notes/0.12/release-0.12.md +++ b/docs/release-notes/0.12/release-0.12.md @@ -6,7 +6,9 @@ * Support for [dotnet-interactive](https://github.com/dotnet/interactive) ([#515](https://github.com/dotnet/spark/pull/515)) ([#517](https://github.com/dotnet/spark/pull/517)) ([#554](https://github.com/dotnet/spark/pull/554)) * Support for [Hyperspace v0.1.0](https://github.com/microsoft/hyperspace) APIs ([#555](https://github.com/dotnet/spark/pull/555)) * Support for Spark 2.4.6 ([#547](https://github.com/dotnet/spark/pull/547)) -* Bug fix caused by `BroadcastVariablesRegistry` ([#551](https://github.com/dotnet/spark/pull/551)) +* Bug fixes: + * Udf bug caused by `BroadcastVariablesRegistry` ([#551](https://github.com/dotnet/spark/pull/551)) + * Null checks for `TimestampType` and `DateType` ([#530](https://github.com/dotnet/spark/pull/530)) * Update `Microsoft.Data.Analysis` to v`0.4.0` ([#528](https://github.com/dotnet/spark/pull/528)) ### Infrastructure / Documentation / Etc.