[FEATURE REQUEST]: Implement complete Spark bindings using Autogen Technology

[rrekapalli]

Dear Team:

Is there a possibility to implement complete .Net bindings (including core and MLLib etc) for Spark using Autogen technology? .Net for Spark .Net for Spark seems to only have core bindings and doesn't seem to implement MLLib from long time! Just wondering if it would be easy with Autogen technology to convert/port the whole Spark library to .Net.

Thanks & Regards.

[github-actions]

Hey @rrekapalli 👋!
Thank you so much for reporting the issue/feature request 🚨.
Someone from SynapseML Team will be looking to triage this issue soon.
We appreciate your patience.

[mhamilton723]

@serena-ruan is working on this item and will follow up when she has updates to share

[rrekapalli]

This is great news! Can't wait for the implementation!

[serena-ruan]

@rrekapalli Sorry for the late reply! It's definitely on the roadmap, and there is an initial PR dotnet/spark#1096 that uses our existing codegen system to auto generate the code. I have talked to the owner that they're super interested. But we might need to wait as the dotnet spark team is quite busy recently, and we will work on polishing the existing .NET codegen bindings to make it usable outside of SynapseML :D

[rrekapalli]

Thank you for the update @serena-ruan !

[Vislesha]

Hi team, any update/guidance on this feature implementation!

[rrekapalli]

Dear @serena-ruan,
As .Net for Spark is not being active, is there any chance that a full Spark implementation be part of this library anytime soon? I think this is a great library that brings all pieces together and Spark should be part of this library as well. Greatly appreciate if you can share the future direction and/or any plans for full Spark implementation!

[serena-ruan]

Hi @rrekapalli Sorry for the inconvenience, .NET team has been real busy and hasn't actively responded to our feature requests. We'll reach out to them again offline and see what's in their roadmap. dotnet/spark repo is the official repository for support Spark with .NET, so there's no way for us to bringing the spark implementation directly into us. But you could use all the existing features with C# in SynapseML library now. @mhamilton723 if you have more comments :)

[Vislesha]

Hi @serena-ruan, @mhamilton723,
Could you please let me know if there is any chance that we see a complete Spark binding whether in .Net for Spark or SynapseML? There hasn't been much traction on the .Net for Spark library from longtime and have been hearing there won't be support to C# (.Net) for SynapseML. Could you please provide some guidance on the support of .Net for these libraries.