Message Boards Message Boards

11
|
19228 Views
|
21 Replies
|
49 Total Likes
View groups...
Share
Share this post:

Verify documentation example before the release of a new version?

My question is based on the observation that when a new Mathematica version is released (this time being 12.2) some of the functions that don't even receive an update from the previous version often get broken.

How is it so that the examples in the documentation are not checked for consistency of operation/results before a new Mathematica version is shipped? I believe that even with a team of 5-6 people in the quality control department (assuming there is any) it is possible to check the examples for all listed functions in Mathematica in about a week.

Why is such a simple task and simple expectation so hard to meet? The issue at the end is that a buggy release will cause many of the mission critical functions for say an end user to break.

It is a pity that I have to install not one version (that I can fully trust) but keep four versions of Mathematica (11.3, 12.0,12.1 and 12.2) simultaneously (grabbing over 50 GB). Reason: because some things work in one version and the others in another version. Please do enlighten me how hard are the user expectations to be met. We do not want more and more functions; we just want the functions to work reliably and correctly. There seems to be issues with the quality control here that needs to be sorted before releasing future versions.

Take this simple example:

ReplacePixelValue did not receive any update since 2014 (version 10). Then how come the second example in its documentation (in the "Applications" subsection) cease to function properly. Furthermore, I am attaching a notebook with another example for this particular function to demonstrate that something that works perfectly in 12.0 and 12.1 fails in the 12.2 release.

Attachments:
BUG.nb
POSTED BY: Ali Hashmi
21 Replies
Sort By:
Hans Milton
Hans Milton
Posted 3 years ago

Mathematica 12.3 just came out. The second example in the documentation reference for Manipulate shows this:

enter image description here

??

Here is the the same in 12.2:

enter image description here

I am using Windows 10. Was the Manipulate documentation tested, on Windows, before release of 12.3?
POSTED BY: Hans Milton

Without seeing anything else, I'm going to go out on a limb that the Preferences dialog also looks weird and you probably are using an older Intel integrated GPU. If that is the case, I would evaluate CurrentValue[$FrontEnd, {RenderingOptions, "PreferredGPU"}] = "Software" and restart the app.
POSTED BY: Ian Hojnicki
Hans Milton
Hans Milton
Posted 3 years ago

Thanks Ian. Setting "PreferredGPU" to "Software" solved it.

enter image description here

Btw, I did not see anything wrong in the Preferences dialog
POSTED BY: Hans Milton

Great. If you use 3D graphics a lot, you might want to consider setting 3DRenderingEngine to OpenGL while PreferredGPU is set to Software.
POSTED BY: Ian Hojnicki
Hans Milton
Hans Milton
Posted 3 years ago

It looks like it is more than Manipulate that is affected by the GPU setup, in 12.3.

I have two PC's with Mathematica 12.3. I will label them as A and B:

A. Has an Intel CPU with integrated GPU. It had the Manipulate problem, fixed by the "Software" workaround as discussed.

B. Has an AMD CPU with Radeon GPU. Manipulate works fine out of the box, without need for any workaround.

The new feature of auto replacements, replacing [[ with \[LeftDoubleBracket] etc, works fine on B. But on A there are no replacements done. Not even the familiar -> \[Rule] or :>\[RuleDelayed]

I will take this up with Technical Support.
POSTED BY: Hans Milton
Hans Milton
Hans Milton
Posted 3 years ago

Resetting Mathematica on A solved this problem. Input auto replacements now works.
POSTED BY: Hans Milton

Perhaps this might be useful somehow: https://redmine.wolfram.com
POSTED BY: Kapio Letto
Hans Milton
Hans Milton
Posted 4 years ago

Quoting Stefan:

We do in fact check documentation examples before a release. However, things do sometimes get missed from time to time, or a fix was not ready in time.

Locator has a documented option AutoAction. But it has not worked after version 10.4. That is more than 4 years ago.
POSTED BY: Hans Milton

We have been aware of this issue for quite some time. I cannot give any estimate on a time frame for a fix, though.

POSTED BY: Stefan Ragnarsson

Hi Stefan ! this is precisely what we end users are requesting with every minor or major release. If the same developers who have created old functions are busy creating new and new functionality then not much time or resources can be invested to fix the old issues. Additionally, new bugs are incorporated along the way either because the new features rely on older functionality/infrastructure under the hood or possibly because new bugs are inevitably incorporated while developing newer functions. What would be considerably better is to put a great deal of effort to get either 12.3 or possibly 13.0 free of critical bugs and old lingering issues in the system. Thanks !
POSTED BY: Ali Hashmi

I completely agree with this statement. The idea of a development cycle for just fixing bugs like it was done in 12.1.1. was great. Hopefully it will also be done in the future. But as far as I know it is unfortunately not planned for 12.2.

It is also a big problem for a user that there is no clear time frame when a bug will be fixed. It could be 3 month when you are lucky, three years or probably never. For some issues it might not be easy and thats understandable but for regression bugs it is not acceptable. Regressions should be fixed short term. Waiting a full development cycle which can last almost a year is simply too long. And then it is by far not clear if the issue gets fixed. If Wolfram wants to get a foot in the enterprise market for production use (Wolfram Engine, Application Server, ...) in my opinion thats absolutely necessary. I wouldn‘t take the risk and bet on a tool for critical applications when there is no clear roadmap for bug fixing. Matlab for example release a major version twice a year and a bug fix update every month. Another example is Visual Studio with regular bug fix updates.
POSTED BY: Philipp Winkler

The fact that 12.1.1 was an official bug-fix release and 12.2 was not in no way implies that bugs were not fixed for 12.2. On my own part, I had no bug fixes go into 12.1.1 and several dozen go into 12.2. On a related note, I also did sweeps through some categories of open bug reports. I found that over time more than 30% of the bugs from Integrate were apparently fixed. This can happen due to continued code overhaul wherein "collateral damage repair" occurs (one bug fix actually fixing more than one problem, with the developer unaware of this).

As it happens, I would also like to see more effort put into fixing open issues. I am simply trying to point out that the ongoing efforts in that direction are appreciably greater than what is being claimed in this thread. I will also state, as others have done, that this cannot happen at the expense of all new development, as that would sacrifice the future for the present.
POSTED BY: Daniel Lichtblau

I know that a lot of stuff has been fixed in 12.2 and that there is on going effort in that direction which is highly appreciated. But you confirmed my posting by this:

I found that over time more than 30% of the bugs from Integrate were apparently fixed. This can happen due to continued code overhaul wherein "collateral damage repair" occurs (one bug fix actually fixing more than one problem, with the developer unaware of this).

That is exactly the problem I am talking about. Depending on getting a bug fixed over time by „accident“ is at least in enterprise deployment not acceptable. I am explicitly not talking about some edge cases in mathematical functions which are not supported and you can discuss if it is a bug or a feature. I am mainly talking about examples in the documentation which are not working (there some long standing issues reported multiple times, eg. FindMaximumFlow), functions silently giving wrong results (eg. Batch Prediction for Classify with certain methods, ...), functions which are just not working for real world use cases (I am aware that this can end again in a „is it a bug or feature“ argumentation but often it is pretty obvious) and regressions, which occur in a new version and do not get a high enough priority assigned for being fixed in a reasonable time frame.

I am also excited for new features being added but I am also pretty sure that a lot of long time users of Mathematica would appreciate focusing on polishing and bug fixing instead of new features for at least a few development cycles and in no way the future is being sacrificed for the present by this. Indeed I think exactly the opposite would happen as I know users which canceled their subscription for issues not getting corrected over multiple releases. What is the benefit of a new feature which is put in and therefore breaks stuff that already worked? Such things are actually very frustrating. And as I said before: for production deployment in enterprise applications there has to be some kind of a more predictable roadmap after an issue has been reported when a fix will be available.
POSTED BY: Philipp Winkler

I fail to see how the quoted remark in any way indicates a failure to check for bugs and prioritize fixing them. First, it referred rather specifically to one function, Integrate. As it happens, this one does not lend itself to easy fixes, and its peculiar features in no way extrapolate to other areas such as graphs and networks or machine learning. Whatever you might want to think about prioritizing work on this one (admittedly important) function, forget about it. The knowledge for how to do this simply does not exist. Some of it will come into existence, and some will in fact be invented by my Wolfram colleagues. But it is a process, and not a fast one. On the brighter side, it has begun-- there is a project in early stages to overhaul some of that functionality.

I have no particular dispute with your second paragraph, or indeed with much of the first one. These involve matters of prioritization and I neither speak for my employer nor in all cases agree with the choices that emerge. (I will state that they are reasonable choices given the competing factors both commercial and otherwise.) But the first part of your first paragraph made a leap from handling of very specific functionality to our entire quality assurance efforts, and that leap is simply unfounded.

One final remark is that we do offer enterprise support. While I have not had much involvement, my limited intersection with that sphere of the business leads me to believe the problems encountered by that set of customers do get high prioritization. Often enough it is not (entirely) because they pay more, but because they put stress on the software in ways we rarely see, and thus uncover issues that otherwise might be missed. And of course these will be important to other enterprise customers. What we fail to do is offer bug prioritization or fix roadmaps to all on a bug-by-bug basis. Should we do this? That question is way above my pay grade.
POSTED BY: Daniel Lichtblau

Hi Daniel, while I understand your point, I should point out that what we and many end users here are referring to are the longstanding issues that are overdue. Take for instance the case of FindMaximumFlow. I think for the past four years or so at least three bug reports were filed by myself alone, hoping that it might get fixed in the next update. It just never gets fixed.

The problem is that the bugs are not weeded out quickly and a lot of bug-baggage has built up over the years. From the user's perspective this can be seen (misconstrued) as Wolfram being nonchalant about longstanding bugs.

Don't you consider that bug in FindMaximumFlow and many others functions should be dealt with irrespective whether the enterprise users need it or not. The function is important for many people doing research or working with graphs and networks. Besides the inclusion of the function in the language itself mandates that any bugs should be sorted out, especially ones that are over 5 years old. A bug is a bug and needs to be fixed to maintain the userbase.
POSTED BY: Ali Hashmi

I agree bugs that affect multiple users, especially ones doing serious R&D or related work, should get priority. I cannot comment specifically on FindMaximumFlow because I am unfamiliar with that area of the code base. Possibly it is simply difficult to fix?

As for general factors that go into prioritization, I will disagree on the "bug is a bug" point. Some are simply more important than others. As for weighing in enterprise customer needs, I see no reason why business considerations should not be taken into account. But again, those also get balanced against questions like "Is this really a bug vs feature?" or "Is there a simple workaround?" or "Is this user error?" Enterprise customers do not get a free pass on these, any more than others.
POSTED BY: Daniel Lichtblau

Ok I agree with your point, sometimes it may not be obvious that a particular issue is a bug or an intended feature. However, I don't agree on the workaround part. A workaround should be temporary in my opinion. If say algorithm A or a built in Function A does not work and algorithm B or a composition of other builtin functions is a workaround, then still an effort should be made to make 'A' work. Why? Because Mathematica is an expressive language and should allow all possible ways to work.

FindMaximumFlow type functionality exists elsewhere and there are cookbook procedures for it. Even the Wikipedia entry for Maximum flow problems has listed more than 10 published algorithms for solving such problems. Plus there is a section on 'Maximum Flow with vertex capacities' (which is the buggy part of the function in Mathematica) which leads me to believe that there are established procedures for solving these problems. An open source implementation can be translated for a fix.
POSTED BY: Ali Hashmi

Please file a report to support@wolfram.com. They usually monitor the community, but not always.

Documentation can be updated in paclet updates, which are (mostly) silent, but they need to know about the problem.
POSTED BY: George Woodrow III

Hi Ali,

Thank you for bringing this issue to our attention. We do in fact check documentation examples before a release. However, things do sometimes get missed from time to time, or a fix was not ready in time.

Please note that the Wolfram Language is highly interconnected. Many functions internally use other WL functions as part of their implementation, or share a codebase with several related functions. So even though a particular function did not get a specific update in a release, something it depends on might have been changed.

POSTED BY: Stefan Ragnarsson

Here is another example in the Documentation which used to work in 12.1.1 but not anymore in 12.2:

GeoSmoothHistogram[
WeightedData[
CountryData["UnitedStates", 
"LargestCities"], #["Population"] &], 20, PlotLegends -> Automatic]
POSTED BY: Philipp Winkler
Hans Milton
Hans Milton
Posted 4 years ago

Another case when documentation examples are broken is Locator with option AutoAction set to True. The documentation example works in 10.4 but not in 11.3, 12.0, 12.1 or 12.2.

There is a change between 12.1 and 12.2, but only for the worse. In 12.1 the locator with AutoAction can be moved if you click on it and drag. But in 12.2 it will not move at all, clicked or not.
POSTED BY: Hans Milton
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract