Here are two things I think should be opened up to third party developers:

• External services. Mathematica comes with built-in support for some popular APIs like Google Analytics, Dropbox etc. but what if I want to use another API? I would like to be able to write a package that I could share with others that adds a new API to the list of $Services. It would work just like an external service with built-in support.
• I would like to be able to provide my own data source for entities. Imagine collaborating on some project where you need some data that suits the entity framework, then being able to set up your own data source, a database in the cloud say, would be very nice. I should have to write a lot of code myself to get it working, I should just have to configure the endpoint and where to find the entities etc.

In reply to George Woodrow I have been documenting a rather large application, Grassmann Calculus, with a beta version of Workbench 3. And another developmental application also. It basically works, but it required a fair amount of effort, back and forth with support, and installing of obscure file types to get it working.

In Mathematica 5 it was possible for every user to do documentation. One just wrote a text file. Yes, the new paclet documentation is much better but now nobody except those who pay extra through Premier Service or otherwise can write documentation. And there is not even a standard facility for doing it. WRI should consider a possible application or program specialized for user documentation - either outside of Workbench or maybe a stripped down version of Workbench. It should be available to EVERY user. And they should listen carefully to users and third party developers because their needs are not exactly coincident with internal documentation. One important requirement is that user produced documentation should have the same 'look and feel' as standard Wolfram documentation.

The fact that ordinary users have been cut off from documentation since version 5 looks as if this is by deliberate decision. It is a HUGE design and business mistake.

People in the community should think more in terms of "applications" and a little less in terms of "packages". An application may or may not include packages. It may not initially include a package but add one later. It is just a good way to develop, organize, preserve and communicate your work around some particular subject matter. If you start out the right way it's easy to grow into the more advanced features as you are ready for them. This is discussed in my essay A Mathematica Style. It is something everybody should consider, not just "developers". Except for the ability to document it is already designed and provided by WRI.

It's basically easy to write packages and, if Wolfram would provide the facility, to write documentation. Writing documentation is not just some irksome chore. It's part of developing and preserving your own work. If it's difficult to document how your routine is used then maybe it's badly designed and you should change it. A year from now will you remember how some routine was used or where there were some examples? Documentation could also include test examples. The examples would probably be written at the time you developed the routine so why not just copy them to a documentation page?

Many more users should be writing packages and documentation and consider it as just part of standard Mathematica usage.

I completely agree with most of what you've said. Not so sure how I feel about the System shadowing issue, but I see your point. (It's just that excessive namespacing drives me nuts. I don't want to declare or load a module to use something as basic as sine and pi.)

Anyway, a somewhat related point: PackageData.net is a nifty site, but a proper repository for packages would be really cool. (As in, a place where the packages can actually be hosted and automatically be installed from within a WL system, possibly even loaded directly from the cloud without a local installation.) It doesn't even have to have version control and other development features (not right away anyway), as long as it's a good place to put finished releases.

namespacing

Yes, I figured that that would be the kind of namespacing you had in mind, I just wanted to caution against overshooting the target. There's probably a host of reasons why Python does this (I suspect the collaborative nature of development will be at the top of the list). It may very well be a reasonable path for many languages, and Python is a lovely language in the greater scheme of things; I'd just hate to see WL turn to this type of namespacing, not that I sense any immediate danger of that happening.

As you describe it though, I have to agree that there are portions of functionality that look as though they could be a clearly defined group of things. That might very well be feasible and would serve as a way to structure things a bit.

On the other hand, the result would be far from clash-proof, since any package can still be broken by any of the namespaces it uses. For instance, a package that is likely to be broken by new functions in the ImageProcessing namespace is also a package that very likely uses ImageProcessing already. Unless of course you want to get rid of $ContextPath altogether and force fully-qualified names for everything that's not System or Global. (I'm against that idea!)

repository

Regarding the package repository: I'm aware of most of the discussion (although I didn't participate), and I appreciate the difficulties. Of course something like this shouldn't be a community effort, that's just not how the Wolfram ecosystem is designed (and that's totally fine). My suggestion was really geared towards WRI, because honestly, the Library Archive doesn't cut it as a code repository.

In the meantime, you guys did a great job on PackageData and I'm definitely using it. I haven't tried the automatic installer solutions that have appeared recently because I have a peculiar setup and don't really use $UserBaseDirectory, but for the majority of use cases I can see the benefit of a unified installer.

prevention of extension

You also mentioned that a lot of functions are easy to use in their default mode, but really hard to extend (your examples were NIntegrate and Dendrogram). I just wanted to add that this is indeed also a pet peeve of mine, although I usually run into this with Plot and friends. I recall a talk by Stephen Wolfram where he said that people always ask for more customizability, but then a survey of real usage always shows that almost everyone uses the default even in places where it's easy enough to change (such as ViewPoint in Graphics3D). So maybe this reduction in extendability is actually a conscious decision.

On the name conflict with System. Two things: If one does not use UpperCase first letter for functions and symbols names in the package (as they should ;) then the chance of conflict is almost zero. Second: I always think it is best to call a function in a package using the syntax

  packageName`foo

After loading the package even though adding the context name is not needed. But suppose a package now had a function called NDSolve and one used

    packageName`NDSolve 

To call it, then there is no conflict, right? Would this not resolved the issue of name conflict? May be Mathematica should make this a requirement for calling functions in non-system context, i.e. one must always add the context name. I do this myself all the time, just to make it clear in the code, where a function came from, unless it is from the System of course. This way one can write

      packageOne`foo 
      packageTwo`foo 

in the same code and know from which package each function comes from. Do you think there will still be problem if these two things are followed by package developers and users?

Nice points!

Actually I think a `DistanceMatrix would suffice instead of a fully qualified name. It shouldn't be such a breaking change to get used to adding a heading `.

I agree completely. Support for 3rd party packages used to be done by a group of 3 people. Now it is a collateral duty for one person.