Community RSS Feed
http://community.wolfram.com
RSS Feed for Wolfram Community showing any discussions in tag Import and Export sorted by activeHow to get time series data from FitBit service connection?
http://community.wolfram.com/groups/-/m/t/1189146
Working through the basic examples in the Wolfram documentation related to the FitBit service connection. I'm able to get a proper ServiceObject from:
**fitbit = ServiceConnect["Fitbit", "New"]**
Running ServiceExecute for the "ActivityData" requests as shown in the example returns results as expected. However, running a similar ServiceExecute on "StepsPlot" does not return any results. I've gone through several other available requests (e.g., "SleepDate", "UserDate", "CaloriesTimeSeries", etc.) and what I'm finding is that requests that take "StartDate" and "EndDate" parameters don't seem to work for me.
I'm running Mathematica "11.2.0 for Mac OS X x86 (64-bit) (September 11, 2017)". Again, not really doing anything fancy other than copy-n-pasting the examples from the documentation and slightly adjusting date ranges to reflect "real" data for my case. BTW, leaving the exact dates from the example seems to make no difference in behavior (e.g., still getting back symbolic result).
Any ideas on what I'm doing wrong or missing here?
Below is a screen shot of my notebook for reference.
![FitBit Example Failing][1]
[1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fitbit_fail.png&userId=1189127macrod2017-09-21T03:42:42Z[✓] Calculate area of a maple leaf image?
http://community.wolfram.com/groups/-/m/t/1183671
Hi,
I want to calculate the area of the maple leaf in cm^2.
![enter image description here][1]
I have tried this
img = Binarize@
Import["http://i37.ltalk.ru/22/85/298522/92/9427492/leave2.png"]~
Erosion~1;
(m = MorphologicalComponents[img]) // Colorize
linecount =
2 /. (Binarize@
Import["http://i37.ltalk.ru/22/85/298522/92/9427492/leave2.\
png"] // ColorNegate // Thinning //
ComponentMeasurements[#, "PerimeterCount"] &);
areacount =
2 /. ComponentMeasurements[MorphologicalComponents[img], "Count"];
areacount/linecount^2 // N
but I'm not sure which region is calculated and the unit given.
Can somebody help me? Thank you.
[1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=maple.png&userId=862195Nurul Ainina2017-09-13T07:22:20ZFalling back to Mathematica 11.1. Imports of CSV files are 100 times slower
http://community.wolfram.com/groups/-/m/t/1186345
Hi,
I originally posted a problem I was encountering using Mathematica 11.2 under:
[Mathematica 11.2 Import Issues][1]
After implementing the recommended workaround that eliminated the Import CSV row truncation issue,
I had to give up, and fall back to Mathematica 11.1.
Working with over 100,000 rows of downloaded Wolfram StarData, I had a minimum of 11 files, with 10,000 rows per file.
The import times under Mathematic 11.1 were around 2 seconds for each CSV file.
Under Mathematica 11.2, Imports of the same files are over 250 seconds for each file, making using Mathematica11.2 to analyze StarData unworkable. Falling back to Mathematica 11.1 is my only solution until this problem is fixed.
See attached Import image file.
I've also included the sample CSV file.
![enter image description here][2]
[1]: http://community.wolfram.com/groups/-/m/t/1186112
[2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fallback.jpeg&userId=885550Joseph Karpinski2017-09-17T16:37:27ZAvoid MMA 11.2 Import Statement Produces Errors & Truncates CSV Data?
http://community.wolfram.com/groups/-/m/t/1186112
https://mathematica.stackexchange.com/questions/155908/mathematica-11-2-import-statement-produces-errors-truncates-csv-data-created-i
In Mathematica 11.1 I've created many CSV files from StarData:
sunMass = StarData["Sun", "Mass"];
sunLuminosity = StarData["Sun", "Luminosity"];
sunTemperature = StarData["Sun", "EffectiveTemperature"];
sunGravity = StarData["Sun", "Gravity"];
sunDensity = StarData["Sun", "Density"];
sunVolume = StarData["Sun", "Volume"];
sunDiameter = StarData["Sun", "Diameter"];
SetDirectory[$UserDocumentsDirectory]
listData1 = Take[StarData[EntityClass["Star", All]], {1, 10000}];
CloseKernels[]; LaunchKernels[4]
AbsoluteTiming[
Length[
data =
Transpose[
ParallelMap[
StarData[listData1, #] &,
{"Name", "Metallicity", "SpectralClass", "BVColorIndex",
"EffectiveTemperature",
"Mass", "Luminosity", "AbsoluteMagnitude", "Gravity", "Density",
"Diameter",
"DistanceFromEarth", "MainSequenceLifetime", "Parallax",
"RadialVelocity", "Radius", "StarEndState", "StarType",
"SurfaceArea",
"VariablePeriod", "Volume", "HDName"}]]]]
zeroData = data /. {Missing["NotAvailable"] -> 0};
noUnitsData =
zeroData /. {c1_, c2_, c3_, c4_, c5_, c6_, c7_, c8_, c9_, c10_,
c11_, c12_, c13_, c14_, c15_, c16_, c17_, c18_, c19_, c20_, c21_,
c22_} -> {c1, c2, c3, c4, QuantityMagnitude[c5],
QuantityMagnitude[c6/sunMass],
QuantityMagnitude[c7/sunLuminosity], c8,
QuantityMagnitude[c9/sunGravity], QuantityMagnitude[c10],
QuantityMagnitude[c11/sunDiameter], QuantityMagnitude[c12],
QuantityMagnitude[c13], QuantityMagnitude[c14]
, QuantityMagnitude[c15], QuantityMagnitude[c16], c17, c18,
QuantityMagnitude[c19], QuantityMagnitude[c20],
QuantityMagnitude[c21/sunVolume], c22};
Length[noUnitsData]
prePendData =
Prepend[noUnitsData, {"Name", "Metallicity", "SpectralClass",
"BVColorIndex", "EffectiveTemperature",
"Mass", "Luminosity", "AbsoluteMagnitude", "Gravity", "Density",
"Diameter",
"DistanceFromEarth", "MainSequenceLifetime", "Parallax",
"RadialVelocity", "Radius", "StarEndState", "StarType",
"SurfaceArea",
"VariablePeriod", "Volume", "HDName"}];
TableForm[Take[prePendData, 5]]
Export["allStars1.csv", prePendData, "CSV"]
This creates a CSV file with 10,000 rows of comma separated data. Works great for all 108,939 rows of StarData, by creating 11 CSV files.
Importing each CSV file in a new notebook is pretty straight forward
Length[data1 =
Import["allStarData1.csv", {"Data", {All}},
"HeaderLines" -> 1] /. {c1_, c2_, c3_, c4_, c5_, c6_, c7_, c8_,
c9_, c10_, c11_, c12_, c13_, c14_, c15_, c16_, c17_, c18_, c19_,
c20_, c21_, c22_} -> {c1, c2, c3, c4, c5, c6, c7, c8, c9, c10,
c11, c12, c13, c14
, c15, c16, c17, c18, c19, c20, c21, c22}
]
This worked fine across different releases of Mathematica 11 for a large number of CSV files.
Upgraded today to Mathematica 11.2.
None of the existing notebooks work.
Import statements now take forever, generating error messages, and truncating large numbers of rows from existing CSV files.
One workaround that I'm currently testing, is running the StarData extract code listed above under Mathematica 11.2 creating new CSV files, and then importing the new CSV files.
This worked for the first 10,000 row StarData extract. No errors, no truncation. But still runs very slow. Will have to run the other 10 extracts and create new 10,000 row CSV files for each.
Feels like this is a bug in Mathematica 11.2 Import statement internal code. Where new internal data verification checks are incompatible to previously created CSV files.
Anyone else run into this issue?
Also I turned off the error messages to try to get through existing code, but don't know how to turn the error messages back on, so that I can include them in this post. Anyone know how?
Thanks
Including JPEG of Import errors:
![enter image description here][1]
Mathematica 11.2 documentation points to updates in CSV Import & Export functions:
![enter image description here][2]
Sample 10,000 record CSV file Export created in Mathematica 11.1 that is truncated when Import under Mathematica 11.2
Same files created as Export under Mathematica 11.2 and Import under Mathematica 11.2 are not truncated, and have full 10,000 records per file.
![enter image description here][3]
"TextDelimiters"->"" fixed the problem. Eliminated row truncation. Import is still horribly slow under Mathematica 11.2 for a 10,000 row CSV file, taking over 250 seconds. Workaround I tested was to create all files under Mathematica 11.2 Export and then Import. No Truncation. The Part::partw warning messages are new under 11.2. They did not show up under Mathematica 11.1. Turning them off via Off[Part::part] did not improve the performance elapsed time of the Import.
![enter image description here][4]
[TextDelimiters Fixed the Problem][5]
Fixed
Falling back to Mathematica 11.1. Importing CSV files under Mathematica 11.2 is 100 times slower.
![enter image description here][6]
[1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=error_snapshot.jpeg&userId=885550
[2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=IMG_0104.PNG&userId=885550
[3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=full_file_on_import.jpeg&userId=885550
[4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fixed.jpeg&userId=885550
[5]: https://mathematica.stackexchange.com/questions/155908/mathematica-11-2-import-statement-produces-errors-truncates-csv-data-created-i
[6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fallback.jpeg&userId=885550Joseph Karpinski2017-09-17T03:53:04ZThoughts on a Python interface, and why ExternalEvaluate is just not enough
http://community.wolfram.com/groups/-/m/t/1185247
`ExternalEvaluate`, introduced in M11.2, is a nice initiative. It enables limited communication with multiple languages, including Python, and appears to be designed to be relatively easily extensible (see ``ExternalEvaluate`AddHeuristic`` if you want to investigate, though I wouldn't invest in this until it becomes documented).
**My great fear, however, is that with `ExternalEvaluate` Wolfram will consider the question of a Python interface settled.**
This would be a big mistake. A *general* framework, like `ExternalEvaluate`, that aims to work with *any* language and relies on passing code (contained in a string) to an evaluator and getting JSON back, will never be fast enough or flexible enough for *practical scientific computing*.
Consider a task as simple as computing the inverse of a $100\times100$ Mathematica matrix using Python (using [`numpy.linalg.inv`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.inv.html)).
I challenge people to implement this with `ExternalEvaluate`. It's not possible to do it *in a practically useful way*. The matrix has to be sent *as code*, and piecing together code from strings just can't replace structured communication. The result will need to be received as something encodable to JSON. This has terrible performance due to multiple conversions, and even risks losing numerical precision.
Just sending and receiving a tiny list of 10000 integers takes half a second (!)
In[6]:= ExternalEvaluate[py, "range(10000)"]; // AbsoluteTiming
Out[6]= {0.52292, Null}
Since I am primarily interested in scientific and numerical computing (as I believe most M users are), I simply won't use `ExternalEvaluate` much, as it's not suitable for this purpose. What if we need to do a [mesh transformation](https://mathematica.stackexchange.com/q/155484/12) that Mathematica can't currently handle, but there's a Python package for it? It's exactly the kind of problem I am looking to apply Python for. I have in fact done mesh transformations using MATLAB toolboxes directly from within Mathematica, using [MATLink][1], while doing the rest of the processing in Mathematica. But I couldn't do this with ExternalEvaluate/Python in a reasonable way.
In 2017, any scientific computing system *needs* to have a Python interface to be taken seriously. [MATLAB has one][2], and it *is* practically usable for numerical/scientific problems.
----
## A Python interface
I envision a Python interface which works like this:
- The MathLink/WSTP API is exposed to Python, and serves as the basis of the system. MathLink is good at transferring large numerical arrays efficiently.
- Fundamental data types (lists, dictionaries, bignums, etc.) as well as datatypes critical for numerical computing (numpy arrays) can be transferred *efficiently* and *bidirectionally*. Numpy arrays in particular must translate to/from packed arrays in Mathematica with the lowest possible overhead.
- Python functions can be set up to be called from within Mathematica, with automatic argument translation and return type translation. E.g.,
PyFun["myfun"][ (* myfun is a function defined in Python *)
{1,2,3} (* a list *),
PyNum[{1,2,3}] (* cast to numpy array, since the interpretation of {1,2,3} is ambiguous *),
PySet[{1,2,3}] (* cast to a set *)
]
- The system should be user-extensible to add translations for new datatypes, e.g. a Python class that is needed frequently for some application.
- The primary mode of operation should be that Python is run as a slave (subprocess) of Mathematica. But there should be a second mode of operation where both Mathematica and Python are being used interactively, and they are able to send/receive structured data to/from each other on demand.
- As a bonus: Python can also call back to Mathematica, so e.g. we can use a numerical optimizer available in Python to find the minimum of a function defined in Mathematica
- An interface whose primary purpose is to call Mathematica from Python is a different topic, but can be built on the same data translation framework described above.
The development of such an interface should be driven by real use cases. Ideally, Wolfram should talk to users who use Mathematica for more than fun and games, and do scientific computing as part of their daily work, with multiple tools (not just M). Start with a number of realistic problems, and make sure the interface can help in solving them. As a non-trivial test case for the datatype-extension framework, make sure people can set up auto-translation for [SymPy objects][3], or a [Pandas dataframe][4], or a [networkx graph][5]. Run `FindMinimum` on a Python function and make sure it performs well. (In a practical scenario this could be a function implementing a physics simulation rather than a simple formula.) As a performance stress test, run `Plot3D` (which triggers a very high number of evaluations) on a Python function. Performance and usability problems will be exposed by such testing early, and then the interface can be *designed* in such a way as to make these problems at least solvable (if not immediately solved in the first version). I do not believe that they are solvable with the `ExternalEvaluate` design.
Of course, this is not the only possible design for an interface. J/Link works differently: it has handles to Java-side objects. But it also has a different goal. Based on my experience with MATLink and RLink, I believe that *for practical scientific/numerical computing*, the right approach is what I outlined above, and that the performance of data structre translation is critical.
----
## ExternalEvaluate
Don't get me wrong, I do think that the `ExternalEvaluate` framework is a very useful initiative, and it has its place. I am saying this because I looked at its source code and it appears to be easily extensible. R has zeromq and JSON capabilities, and it looks like one could set it up to work with `ExternalEvaluate` in a day or so. So does Perl, anyone want to give it a try? `ExternalEvaluate` is great because it is simple to use and works (or can be made to work) with just about any interpreted language that speaks JSON and zeromq. But it is also, in essence, a quick and dirty hack (that's extensible in a quick and dirty way), and won't be able to scale to the types of problems I mentioned above.
----
## MathLink/WSTP
Let me finally say a few words about why MathLink/WSTP are critical for Mathematica, and what should be improved about them.
I believe that any serious interface should be built on top of MathLink. Since Mathematica already has a good interface capable of inter-process communication, that is designed to work well with Mathematica, and designed to handle numerical and symbolic data efficiently, use it!!
Two things are missing:
- Better documentation and example programs, so more people will learn MathLink
- If the MathLink library (not Mathematica!) were open source, people would be able to use it to link to libraries [which are licensed under the GPL][6]. Even a separate open source implementation that only supports shared memory passing would be sufficient—no need to publish the currently used code in full. Many scientific libraries are licensed under the GPL, often without their authors even realizing that they are practically preventing them from being used from closed source systems like Mathematica (due to the need to link to the MathLink libraries). To be precise, GPL licensed code *can* be linked with Mathematica, but the result cannot be shared with anyone. I have personally requested the author of a certain library to grant an exception for linking to Mathematica, and they did not grant it. Even worse, I am not sure they understood the issue. The authors of other libraries *cannot* grant such a permission because they themselves are using yet other GPL's libraries.
[MathLink already has a more permissive license than Mathematica.][7] Why not go all the way and publish an open source implementation?
I am hoping that Wolfram will fix these two problems, and encourage people to create MathLink-based interfaces to other systems. (However, I also hope that Wolfram will create a high-quality Python link themselves instead of relying on the community.)
I have talked about the potential of Mathematica as a glue-language at some Wolfram events in France, and I believe that the capability to interface external libraries/systems easily is critical for Mathematica's future, and so is a healthy third-party package ecosystem.
[1]: http://matlink.org/
[2]: https://www.mathworks.com/help/matlab/matlab-engine-for-python.html
[3]: http://www.sympy.org/
[4]: http://pandas.pydata.org/
[5]: https://networkx.github.io/
[6]: https://en.wikipedia.org/wiki/Copyleft
[7]: https://www.wolfram.com/legal/agreements/mathlink.htmlSzabolcs Horvát2017-09-15T12:33:04ZHas anyone gotten ExternalEvaluate to work with Anaconda Python on OS X?
http://community.wolfram.com/groups/-/m/t/1185221
Has anyone gotten ExternalEvaluate to work with Anaconda Python on OS X? It used to work for me in the prerelease, but it's not working in the final release.
In[2]:= py = StartExternalSession["Python"]
Out[2]= ExternalSessionObject["3dac9d41-1400-48be-83ea-9b6012316ff4"]
In[3]:= ExternalEvaluate[py, "1+1"]
During evaluation of In[3]:= Import::nopythonevals: No Python external evaluator found. Use RegisterExternalEvaluator to register an external evaluator.
During evaluation of In[3]:= ExternalEvaluate::interpFail: The result 2 failed to be interpreted as a WL expression. Use "ReturnType"->"String" instead.
Out[3]= $Failed
![enter image description here][1]
Notice that Python did return the result `2`. What fails is only the translation of this result to Mathematica.
I would like to know if this is broken for everyone or if my system is misconfigured.
----
After this problem is solved, I would also like to know if it is possible to use different [virtualenv][2]s. Can I have multiple external evaluators registered, each corresponding to a specific virtualenv? **Update:** It seems that virtualenv creates a copy of the python interpreter, thus passing this to `RegisterExternalEvaluator` should work. But I can't try until ExternalEvaluate starts working on my system ...
[1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-09-1511.01.00.png&userId=38370
[2]: https://virtualenv.pypa.io/en/stable/Szabolcs Horvát2017-09-15T09:04:31Z[✓] Avoid importing existing data as missing data?
http://community.wolfram.com/groups/-/m/t/1185797
Hello, Community
I have imported data from a csv file using the following code:
*data = SemanticImport[ "C:\\\Users\\\Thadeu\\\Documents\\\Kaggle\\\train.csv"]*
When I investigate the imported data, I see missing values on the DataSet (*house$size* for example)
![enter image description here][1]
that are not missing from the original data (notice *house$size* and *nbaths* as well)
![enter image description here][2]
What's going on?
Is it possible to fix it?
Thanks,
Thad
[1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=reply1.JPG&userId=529007
[2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=reply2.JPG&userId=529007Thadeu Freitas Filho2017-09-17T03:14:04Z