Feedback on Growing the .NET Ecosystem Document

[sbwalker]

Some personal thoughts/feedback on the initial Growing the .NET Ecosystem document...

Observation: The document appears to have a high level hypothesis and a number of supporting arguments…

Hypothesis: Customers demand features from Microsoft because they do not trust third party solutions. This results in negative consequences to the .NET ecosystem.

Supporting Arguments ( ie. reasons why customers do not trust third party solutions ):

• Product quality
• Long term stability
• Lack of support
• Low visibility in official channels
• Provenance between source code and binary
• Cross platform support

Some of the topics above do resonate with me; however, I do question if the hypothesis is in fact the root problem. I question it because there is plenty of evidence from the past 2 decades that customers do trust third party solutions.

• customers trust commercial vendors ( ie. Progress Telerik, Infragistics, DevExpress, etc… )
• customers trust front-end open source frameworks ( React, Angular )
• customers trust many open source third party libraries ( Automapper, Serilog, Castle, Moq, Xunit )
• customers trust open source libraries officially endorsed by Microsoft ( JSON.NET, ImageSharp, IdentityServer )

So if customers do trust third party solutions, then what is hampering the growth and innovation in the third party .NET ecosystem? I believe the document touches on some of these issues already – especially in the “Critical opinions on the Microsoft OSS ecosystem” section. As an independent software developer who created a commercial open source company, I am intimately familiar with some of these challenges. I personally believe the bigger issue is not a lack of trust by customers of third party solutions, but a lack of trust of Microsoft by the potential creators of third party solutions in the ecosystem. The fear of Microsoft “eating its own young”, employing the 3E strategy ( Embrace, Extend, Extinguish ), and creating downward pricing pressure in the market are all barriers to entry for the third party ecosystem.

The other problem when it comes to trust that I have seen is Microsoft’s unwillingness to trust third party solutions and make official endorsements or recommendations to customers. Perhaps this is for reasons of legal liability. And some of the arguments in this document seem to be solutions specifically designed to try and solve this problem. Improving product quality, providing support, ensuring provenance between source code and binary, and ensuring cross platform support all appear to be items that are more important for Microsoft to reduce risk than they are for actual customers. I am sure customers would appreciate these items to some degree as well, but third party solution maintainers will view these as heavy handed requirements being imposed upon them. This was partly the problem with the original Maturity Ladder as well… and anything which appears to reduce freedom is not going to be well accepted.

The following diagram attempts to explain the challenges I outlined above. Each arrow represents a trust relationship and the color signifies its strength ( ie. green = strong, yellow = neutral, red = weak ). And like I said earlier, this is my personal opinion on the current state of .NET. Other folks will certainly have different perspectives. This is exactly why collaboration will be crucial in solving some of these challenges.

[gortok]

First off, I want to say that I'm gratified to see this document meet the light of day. There are lots of corporate environments where the mere existence of that document would be threatening, and I'm glad to see that isn't the case at Microsoft.

I'm also glad to see that people at Microsoft recognizes that the number one problem facing a robust .NET eco-system is Microsoft themselves.

This is not a value judgment. Lots of companies have embraced a closed-ecosystem model and it has done well for them. Apple, to name a very large example. Adobe has also done this, and failed in quite a few different ways; with ColdFusion and ActionScript, most notably, and Adobe Flash not far behind.

Developer ecosystems need the same three things that a fire needs:

• Oxygen (the ability to flourish)
• Heat (momentum)
• Fuel (contributors)

If you take away any one of those; the eco-system starts to falter; and before long, it will fail.

But let's take a step back and describe what I mean when I say a "robust .NET eco-system".

An eco-system is robust when it is self-sustaining; when it is viable for new life; and when and when there is balance to the forces within it. (Not being an ecologist I'm probably missing some key terminology; but for our purposes this will do).

For .NET, this means that there are competing options for potential adopters to chose (whether those adopters are enterprises, SMBs, or individual developers) the libraries and frameworks are able to sustain themselves; whether that's through Open Source and its succession model; or a commercial model, or a combination of the two (I maintain that 'sponsorship' by a singular corporate entity is not 'self-sustaining'), and the general attitude of the community contributes to growth instead of decline.

There are commercial .NET libraries and frameworks; though far less on the framework side than the library side. Open source wise, it is common for Microsoft (by their own admission) to rewrite instead of contribute. The rewrite effectively becomes a 'replace' once the library is released, as is the case with Newtonsoft.Json.

Newtonsoft.Json is a good representative example of an issue with taking the oxygen out of the room. James Newton-King started at Microsoft in March of 2018. In October of 2018, System.Text.Json (hereafter STJ) was announced. By November of 2019, the latest (and possibly last) release of NewtonSoft.Json (NJ) was published, and there haven't been any releases since then, and only 9 commits.

I think it's wonderful that JNK works at Microsoft and that Microsoft hired him. I also think it's wonderful that System.Text.Json exists; but the dual effect of hiring JNK and announcing STJ caused an undesirable side effect: the appearance that NJ is done.

Now, that may be true; but commercial entities do not like it when software has no momentum, because that means they aren't likely to get any support; but since STJ is new and shiny and from Microsoft, they'll just ditch NJ for STJ. This may or may not be a problem; but what it will do is lend further credence to the "Embrace Extend Extinguish" perception Microsoft has, even if it's not warranted.

There are two other ways to handle this that build momentum in the .NET ecosystem; the path taken does not build momentum or give the ecosystem the ability to flourish.

1. Acquire Newtonsoft.Json from JNK and make it available under the .NET foundation
2. JNK passes on the torch to another maintainer

Given the lack of momentum on NJ, I wouldn't recommend that any developer or business adopt it. (I'm ignoring turning it into a commercial product; since this isn't the place for that discussion; but there are other avenues open)

Some teams from Microsoft may rightfully claim that they do not have the ability to acquire software; and you're right, you probably don't -- but people outside of Microsoft don't make the distinction between the teams at Microsoft; and that means that someone in Microsoft needs to be at a high enough level to sign off on an acquisition and to 'mind the store' when it comes to being empowered to 'do the right thing' to maintain a healthy eco-system. If someone at Microsoft doesn't already have that job and the requisite budget authority, they should -- especially since it's a 1.5 Trillion dollar company.

There are other examples, of course; like AppGet/Winget debacle, or further back OpenWrap and Nuget clashed. And there are likely others I've forgotten about. I'm still bummed out about XNA though, and glad Monogame picked it up.

In order for there to be a healthy eco-system, Microsoft needs to add to the momentum, and not use its status to suck the oxygen out of the room. Not every problem needs to be solved by Microsoft; and Microsoft solving every problem instead of contributing to solutions that already solve that problem hurts the community and hurts an already fragile ecosystem.

This may mean that Microsoft points its customers to third party solutions; or that Microsoft acquires those third party solutions instead of competing at the cost of free; which sends a giant signal that says "Don't develop anything that Microsoft has the manpower and time to write themselves", which makes it difficult for both open source and commercial software to flourish on .NET.

Bottom line: Whatever actions you take as "Microsoft" will either improve the eco-system or hurt it. Remember the three things an eco-system needs, and act accordingly.

[Aaronontheweb]

@sbwalker your link is broken at the top of your post https://github.com/microsoft/dotnet/blob/master/docs/ecosystem-issues.md

[ericsampson]

@gortok this is a total side note, but AFAICT Newtonsoft.Json is effectively unmaintained, and unfortunately I've come to treat it as such. There have been pretty significant regression bugs with no comments from the owners. Just take a look at the number of issues/bugs filed over the past year with roughly zero response. She's dead Jim, until new maintainers can be found/accepted (if JNK wants that).

[ericsampson]

Regarding the below portion of the document, going forward I would encourage the group to consider finding a different term than "quality" to describe the aspects that you are trying to encourage/shepherd, because this is a triggering word for many people who take it as being a reflection of their code/design quality. And it is very easy for it to come across as condescending.
In general, I think the language of these types of communications should be carefully crafted to help avoid these type of miscommunications and misunderstandings by the readers. Thankfully, Microsoft has a lot of great professional tech writers that can help review communications for tone etc. Cheers!

[ericsampson]

Re the document:

• Microsoft IMO needs to work culturally on being OK with relying on using top-shelf OSS projects as part of its frameworks etc, and being OK exposing those to consumers. Instead of rebuilding them in-house and/or providing (sometimes leaky or not-great) abstractions that then force the original projects to retcon their libraries under the new 'blessed' abstractions, causing extra work and making existing features hard or impossible to support.
• There are recent examples of both bad behavior in this regard, and some small examples of good - like the EF Core team deciding to implement their Lazy Loading support using OSS (Castle DynamicProxy) instead of reinventing the wheel internally.
• The other thing I think of is to make sure to leverage existing internal champions in these discussions that you are trying to have, such as @jeffwilcox.

[cartermp]

I think there is some good feedback here and it'll take a while for everyone to find the right balance as stated in the document. To that end, here's my perspective - as an MS internal but not an author or co-author of the document - about what we can do. This is based on my own engagement with the F# OSS community and helping various people in our own sub-ecosystem while mostly focusing efforts on F# itself.

I think there are three primary things we, Microsoft, can do to help our OSS partners to address the notion of "quality" that is mentioned in the document:

1. Contribute test coverage for existing features if there is a need.

I'm sure most OSS authors would be happy to have a larger regression suite, especially for high-use, high-churn areas. But it's time-consuming work and most OSS authors are doing this in their spare time. This is quite literally a way to address quality gaps that may exist, especially since the MS definition of quality is usually highly associated with a robust test suite.

2. Make infrastructure contributions - moving off of a flaky CI provider to github actions, dealing with CI churn, etc.

This is always a big time sink for OSS authors, and it's very easy to slip into a state where CI appears unreliable. That's inevitable, but it's best to just have someone keeping watch on things from time to time to make sure it doesn't get stuck there forever. The document alludes to the fact that we built some custom infrastructure that OSS partners could benefit from. It's worth talking to library authors to see if it would help them out.

3. Contributions to help deal with .NET product churn.

Another big time sink for OSS library authors is dealing with .NET product churn. Of course the best thing we can do here is not churn things, but the reality is that we do and it's just part and parcel of evolving an ecosystem. Some libraries, such as those that built against ASP.NET Core primitives, are more impacted by this than others. I think we, Microsoft, can identify some of the libraries that are impacted by this the most and make contributions to help authors deal with the churn that comes with .NET releases.

This doesn't come close to "solving the problem" (for some fuzzy definition of "the problem"). But I think it can help further the goals of the document in a generally uncontroversial way. I would certainly love to spend some of my work time doing this kind of work.

[ericsampson]

Re @cartermp's list, one thing I've always wondered if it would help take some burden off OSS projects is to have access/time from a tech writer available through the DNF? Good docs are important for product adoption, but they can be a real time suck.

[NickCraver]

It's worth noting that in @cartermp's points 1, 2, and 3: GitHub Actions is great, but it also has very poor support for tests (e.g. presenting them). I totally agree with the infrastructure/CI burden, and it being in better shape makes taking contributions much easier as well, but we have a big tooling gap that Microsoft/GitHub can help with here: tests in Actions. We feel this most when there's a .NET release and we have to go do all the builds, so it relates to point 3 in waves, but is ever present as an issue. Lack of any decent testing solution we've thus far found here is the blocker to us moving everything to GitHub Actions for CI.

I don't believe this particular thing is a problem third parties can solve well, it needs to be baked in. Pluggable, using others, etc. is great and in the spirit here, but developers also need that "just works" that's very accessible for something so concretely basic. If they want to divert from that later and anyone wants to build it: yeah sure, have at it.

[sbwalker]

@sbwalker your link is broken at the top of your post https://github.com/microsoft/dotnet/blob/master/docs/ecosystem-issues.md

@Aaronontheweb thank you for letting me know - I fixed the link

[sbwalker]

A few articles have been published on this topic in the past week...

https://medium.com/young-coder/net-has-a-third-party-software-problem-45d24cdc30c9

https://myetsoftware.com/microsoft-seeks-to-build-trust-in-third-party-net-libraries/

https://www.infoworld.com/article/3601748/microsoft-seeks-to-build-trust-in-third-party-net-libraries.html

[sbwalker]

Another article which references the Growing the .NET Ecosystem document:

https://medium.com/young-coder/net-has-a-third-party-software-problem-45d24cdc30c9

The author of this article appears to be in agreement with my original post in this thread:

"That’s the real trust gap. It isn’t between .NET developers and open-source software, but between the community and Microsoft."

[RLittlesII]

I realize I am late to the game. I agree with almost all of the points made, I do see some different angles.

This is not me advocating or defending any behavior taken by Microsoft

The only trust that I have is Microsoft will act in the best interest of its companies future. That's as far as I am willing to trust any entity.

You touched on the word I think should be replaced with trust. Liability. Microsoft has a liability issue to its customers. If they recommend a framework that goes away, there may be legal departments that would sue Microsoft. So I trust Microsoft to insulate itself from that.

Open source wise, it is common for Microsoft (by their own admission) to rewrite instead of contribute.

How many large corporations contribute to small .NET projects? The company, not individual devs who work for the company acting on their own time and hardware? I would argue not many. I have yet to work for a company that invested in my interest in Open Source, let alone encouraged me to contribute to packages we were consuming.

All my open-source contributions to date have not been at the behest of the Customer, it's been because I want to be involved in the .NET Community. Me. Not a Microsoft Customer paying me to write their software.

It is a common practice for any shop delivering software to do what was outlined in the document. When you don't find a solution that fits your requirements, roll your own. Are we scrutinizing Microsoft for being a victim to requirements (assuming everything is above board)?

Roll your own is the preferred way of a lot of developers and is honestly how we get so many cool things into the ecosystem. Sadly, when Microsoft does it, it ends up in the core framework for the world to consume. Although it does not excuse them from what appears to be malicious violations, like AppGet, it is reasonable this practice would happen. No different than any other entity evaluating a software project and technology stack.

So our trust as authors is non-existent because Microsoft has shown us, they are not above doing what any other developer would do. Taking the code that they got from someone else and shipping it.

I agree there are many examples over two decades of good Open Source Projects. There are plenty that did not fair as well. The .NET Foundation currently has two projects that are no longer supported, that I personally have used in the past 3 years. Sustainability is a valid concern that the community has not found a solution to.

Growth

The .NET Framework should not concern itself with consuming libraries in the .NET Eco System.
I believe if Microsoft wants to truly grow the .NET ecosystem there should be a more target focus in the following areas.

.NET should be Open for Extension

Make it easy for consumers of the framework to extend it and make cool new things. I believe this is already happening, I feel this needs to be the focus. If Microsoft realizes, they can't make everything, they should focus on making our ability to make their framework extensions easily.

.NET should provide guidance for how to Extend and Request extensions

As a library maintainer, I should be encouraged to petition for extension points. When the framework is easier for people to extend and build on, the barrier to attempt is lower, and you potentially get more growth.

Help ecosystem maintainers with the pace of .NET

There are several paces that matter here. The important point is, not everyone can consume the latest .NET bits all the time. It's considered a bug when a library that predates a .NET feature doesn't conform to the feature. The strain that puts on maintainers is ignored, by Microsoft and the community. How do we make it easier for library producers to keep up with the pace of the .NET team? How do we ensure backward compatibility Microsoft Customers have grown to depend on? Should library maintainers have to? These are real things that are not standard in the .NET Ecosystem, as a result, you have to evaluate the third-party dependencies per package you include in your application.

Give more weight to community-driven concerns

This is self-explanatory, while the reality of it is difficult. The biggest barrier I see to this is long-term support. We need to find an effective model to handle long-term support of community-driven efforts that won't result in Microsoft taking on more burden than it had planned.

We should revisit design decisions from the earlier days of the framework, and upgrade them with modern concepts

- Expressions
- Events

The points made about what the Ecosystem needs are spot on. We need extension points provided by the framework and room to play! We do not need to be concerned that Microsoft will strangle out our usefulness.

Lastly, any Ecosystem will have growth and declines simultaneously. Let's appreciate that every library won't survive. Growth takes a lot of forms. Growth is important, health is important too. We need to make sure the growth is healthy and sustainable for a well-balanced Eco System.