v0.3.0 release prep

README.md

@@ -36,7 +36,7 @@
     <tbody align="center">
         <tr>
             <td >2.3.*</td>
-            <td rowspan=3><a href="https://github.com/dotnet/spark/releases/tag/v0.2.0">v0.2.0</a></td>
+            <td rowspan=4><a href="https://github.com/dotnet/spark/releases/tag/v0.3.0">v0.3.0</a></td>
         </tr>
         <tr>
             <td>2.4.0</td>
@@ -45,12 +45,11 @@
             <td>2.4.1</td>
         </tr>
         <tr>
-            <td>2.4.2</td>
-            <td><a href="https://github.com/dotnet/spark/issues/60">Not supported</a></td>
+            <td>2.4.3</td>
         </tr>
         <tr>
-            <td>2.4.3</td>
-            <td>master branch</td>
+            <td>2.4.2</td>
+            <td><a href="https://github.com/dotnet/spark/issues/60">Not supported</a></td>
         </tr>
     </tbody>
 </table>

benchmark/scala/pom.xml

@@ -3,7 +3,7 @@
     <modelVersion>4.0.0</modelVersion>
     <groupId>com.microsoft.spark</groupId>
     <artifactId>microsoft-spark-benchmark</artifactId>
-    <version>0.2.0</version>
+    <version>0.3.0</version>
     <inceptionYear>2019</inceptionYear>
     <properties>
         <encoding>UTF-8</encoding>

docs/getting-started/ubuntu-instructions.md

@@ -7,7 +7,7 @@ These instructions will show you how to run a .NET for Apache Spark app using .N
 - Download and install the following: **[.NET Core 2.1 SDK](https://dotnet.microsoft.com/download/dotnet-core/2.1)** | **[OpenJDK 8](https://openjdk.java.net/install/)** | **[Apache Spark 2.4.1](https://archive.apache.org/dist/spark/spark-2.4.1/spark-2.4.1-bin-hadoop2.7.tgz)**
 - Download and install **[Microsoft.Spark.Worker](https://github.com/dotnet/spark/releases)** release:
     - Select a **[Microsoft.Spark.Worker](https://github.com/dotnet/spark/releases)** release from .NET for Apache Spark GitHub Releases page and download into your local machine (e.g., `~/bin/Microsoft.Spark.Worker`).
-    - **IMPORTANT** Create a [new environment variable](https://help.ubuntu.com/community/EnvironmentVariables) `DotnetWorkerPath` and set it to the directory where you downloaded and extracted the Microsoft.Spark.Worker (e.g., `~/bin/Microsoft.Spark.Worker`).
+    - **IMPORTANT** Create a [new environment variable](https://help.ubuntu.com/community/EnvironmentVariables) `DOTNET_WORKER_DIR` and set it to the directory where you downloaded and extracted the Microsoft.Spark.Worker (e.g., `~/bin/Microsoft.Spark.Worker`).
 
 For detailed instructions, you can see [Building .NET for Apache Spark from Source on Ubuntu](../building/ubuntu-instructions.md).
 

docs/getting-started/windows-instructions.md

@@ -7,7 +7,7 @@ These instructions will show you how to run a .NET for Apache Spark app using .N
 - Download and install the following: **[.NET Core 2.1 SDK](https://dotnet.microsoft.com/download/dotnet-core/2.1)** | **[Visual Studio 2019](https://www.visualstudio.com/downloads/)** | **[Java 1.8](https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)** | **[Apache Spark 2.4.1](https://archive.apache.org/dist/spark/spark-2.4.1/spark-2.4.1-bin-hadoop2.7.tgz)**
 - Download and install **[Microsoft.Spark.Worker](https://github.com/dotnet/spark/releases)** release:
     - Select a **[Microsoft.Spark.Worker](https://github.com/dotnet/spark/releases)** release from .NET for Apache Spark GitHub Releases page and download into your local machine (e.g., `c:\bin\Microsoft.Spark.Worker\`).
-    - **IMPORTANT** Create a [new environment variable](https://www.java.com/en/download/help/path.xml) `DotnetWorkerPath` and set it to the directory where you downloaded and extracted the Microsoft.Spark.Worker (e.g., `c:\bin\Microsoft.Spark.Worker`).
+    - **IMPORTANT** Create a [new environment variable](https://www.java.com/en/download/help/path.xml) `DOTNET_WORKER_DIR` and set it to the directory where you downloaded and extracted the Microsoft.Spark.Worker (e.g., `c:\bin\Microsoft.Spark.Worker`).
 
 For detailed instructions, you can see [Building .NET for Apache Spark from Source on Windows](../building/windows-instructions.md).
 

docs/release-notes/0.3/release-0.3.md

@@ -0,0 +1,46 @@
+# .NET for Apache Spark 0.3 Release Notes
+
+### Release Notes
+
+Below are some of the highlights from this release.
+
+* [Apache Spark 2.4.3](https://spark.apache.org/news/spark-2-4-3-released.html) support ([#118](https://github.com/dotnet/spark/pull/108))
+* dotnet/spark is now using [dotnet/arcade](https://github.com/dotnet/arcade) as the build infrastructure ([#113](https://github.com/dotnet/spark/pull/113))
+    * [Source Link](https://github.com/dotnet/sourcelink) is now supported for the Nuget package ([#40](https://github.com/dotnet/spark/issues/40)).
+    * Fixed the issue where Microsoft.Spark.dll is not signed ([#119](https://github.com/dotnet/spark/issues/119)).
+* Pickling performance is improved ([#111](https://github.com/dotnet/spark/pull/111)).
+    * Related PRs in the Pickling Library: [irmen/Pyrolite#64](https://github.com/irmen/Pyrolite/pull/64), [irmen/Pyrolite#67](https://github.com/irmen/Pyrolite/pull/67)
+* ArrayType and MapType are supported as UDF return types ([#114](https://github.com/dotnet/spark/pull/114))
+
+### Supported Spark Versions
+
+The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:
+
+<table>
+    <thead>
+        <tr>
+            <th>Spark Version</th>
+            <th>microsoft-spark JAR</th>
+        </tr>
+    </thead>
+    <tbody align="center">
+        <tr>
+            <td>2.3.*</td>
+            <td>microsoft-spark-2.3.x-0.2.0.jar</td>
+        </tr>
+        <tr>
+            <td>2.4.0</td>
+            <td rowspan=3>microsoft-spark-2.4.x-0.2.0.jar</td>
+        </tr>
+        <tr>
+            <td>2.4.1</td>
+        </tr>
+        <tr>
+            <td>2.4.3</td>
+        </tr>
+        <tr>
+            <td>2.4.2</td>
+            <td><a href="https://github.com/dotnet/spark/issues/60">Not supported</a></td>
+        </tr>
+    </tbody>
+</table>

eng/Versions.props

@@ -1,7 +1,7 @@
 <?xml version="1.0" encoding="utf-8"?>
 <Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
   <PropertyGroup>
-    <VersionPrefix>0.2.0</VersionPrefix>
+    <VersionPrefix>0.3.0</VersionPrefix>
     <PreReleaseVersionLabel>prerelease</PreReleaseVersionLabel>
     <RestoreSources>
       $(RestoreSources);

src/csharp/Microsoft.Spark.E2ETest/SparkFixture.cs

@@ -36,7 +36,7 @@ public SparkFixture()
                 AppDomain.CurrentDomain.BaseDirectory);
 #elif NETCOREAPP2_1
             // For .NET Core, the user must have published the worker as a standalone
-            // executable and set DotnetWorkerPath to the published directory.
+            // executable and set the worker path to the published directory.
             if (string.IsNullOrEmpty(Environment.GetEnvironmentVariable(workerDirEnvVarName)))
             {
                 throw new Exception(

src/csharp/Microsoft.Spark/Sql/SparkSession.cs

@@ -65,6 +65,22 @@ public void Dispose()
         public SparkSession NewSession() =>
             new SparkSession((JvmObjectReference)_jvmObject.Invoke("newSession"));
 
+        /// <summary>
+        /// Returns the specified table/view as a DataFrame.
+        /// </summary>
+        /// <param name="tableName">Name of a table or view</param>
+        /// <returns>DataFrame object</returns>
+        public DataFrame Table(string tableName)
+            => new DataFrame((JvmObjectReference)_jvmObject.Invoke("table", tableName));
+
+        /// <summary>
+        /// Executes a SQL query using Spark, returning the result as a DataFrame.
+        /// </summary>
+        /// <param name="sqlText">SQL query text</param>
+        /// <returns>DataFrame object</returns>
+        public DataFrame Sql(string sqlText)
+            => new DataFrame((JvmObjectReference)_jvmObject.Invoke("sql", sqlText));
+
         /// <summary>
         /// Returns a DataFrameReader that can be used to read non-streaming data in
         /// as a DataFrame.
@@ -80,14 +96,6 @@ public DataFrameReader Read() =>
         public DataStreamReader ReadStream() =>
             new DataStreamReader((JvmObjectReference)_jvmObject.Invoke("readStream"));
 
-        /// <summary>
-        /// Executes a SQL query using Spark, returning the result as a DataFrame.
-        /// </summary>
-        /// <param name="sqlText">SQL query text</param>
-        /// <returns>DataFrame object</returns>
-        public DataFrame Sql(string sqlText)
-            => new DataFrame((JvmObjectReference)_jvmObject.Invoke("sql", sqlText));
-
         /// <summary>
         /// Returns UDFRegistraion object with which user-defined functions (UDF) can 
         /// be registered.

src/scala/microsoft-spark-2.3.x/pom.xml

@@ -4,7 +4,7 @@
   <parent>
     <groupId>com.microsoft.scala</groupId>
     <artifactId>microsoft-spark</artifactId>
-    <version>0.2.0</version>
+    <version>0.3.0</version>
   </parent>
   <artifactId>microsoft-spark-2.3.x</artifactId>
   <inceptionYear>2019</inceptionYear>

src/scala/microsoft-spark-2.4.x/pom.xml

@@ -4,7 +4,7 @@
   <parent>
     <groupId>com.microsoft.scala</groupId>
     <artifactId>microsoft-spark</artifactId>
-    <version>0.2.0</version>
+    <version>0.3.0</version>
   </parent>
   <artifactId>microsoft-spark-2.4.x</artifactId>
   <inceptionYear>2019</inceptionYear>

src/scala/pom.xml

@@ -4,7 +4,7 @@
   <groupId>com.microsoft.scala</groupId>
   <artifactId>microsoft-spark</artifactId>
   <packaging>pom</packaging>
-  <version>0.2.0</version>
+  <version>0.3.0</version>
   <properties>
     <encoding>UTF-8</encoding>
   </properties>