https://community.wolfram.com RSS Feed for Wolfram Community showing any discussions tagged with Data Science sorted by most likes. https://community.wolfram.com/groups/-/m/t/2593151 **Introducing our brand new YouTube channel, [Wolfram R&D][1]! Our channel features livestreams, behind-the-scenes creator presentations, insider videos, and more.** ---------- Join us for the unique Wolfram R&D livestreams on [Twitch][2] and [YouTube][3] led by our developers! You will see **LIVE** stream indicators on these channels on the dates listed below. The live streams provide tutorials and behind the scenes look at Mathematica and the Wolfram Language directly from developers. Join our livestreams every Wednesday at 11 AM CST and interact with developers who work on data science, machine learning, image processing, visualization, geometry, and other areas. ---------- ⭕ **UPCOMING** EVENTS - Jan 29 -- Reinforcement Learning Applied to Feedback Control with [Suba Thomas][61] ---------- ✅ **PAST** EVENTS - April 24 -- [FeynCalc][60] - April 3 -- [Explore the Total Solar Eclipse of April 2024][59] - Mar 22 -- [20 Years of xAct Tensor Computer Algebra][58] - Feb 28 -- [Zero Knowledge Authentication][57] - Jan 17 -- [Nutrients by the Numbers][56] - Dec 13 -- [Understanding Graphics][55] - Oct 18 -- [Overview of Number Theory][54] - Sep 27 -- [QMRITools: Processing Quantitative MRI Data][52] - Sep 13 -- [Make High Quality Graph Visualization][51] - Sep 6 -- [Insider's View of Graphs & Networks][53] - Aug 30 -- [Labeling Everywhere][49] - Aug 22 -- [Equation Generator for Equation-of-Motion Coupled Cluster Assisted by CAS][48] - Aug 16 -- [Foreign Function Interface][4] - July 26 -- [Modeling Fluid Circuits][6] - July 19 -- [Geocomputation][5] - July 5 -- [Protein Visualization][7] - Jun 14 -- [Chat Notebooks bring the power of Notebooks to LLMs][8] - May 31-- [Probability and Statistics: Random Sampling][9] - May 24 -- [Problem Solving][10] - May 17 -- [The state of Optimization][11] - May 10 -- [Building a video game with Wolfram notebooks][12] - April 26 -- [Control Systems: An Overview][13] - April 19 -- [MaXrd: A crystallography package developed for research support][14] - April 5th -- [Relational database in the Wolfram Language][15] - Mar 29th -- [Build your first game in the Wolfram Language with Unity game engine] [16] - Mar 22nd -- [Everything to know about Mellin-Barnes Integrals - Part II][17] - Mar 15th -- [Building your own Shakespearean GPT - a ChatGPT like GPT model][18] - Mar 8th -- [Understand Time, Date and Calendars][19] - Mar 1st -- [Introducing Astro Computation][20] - Feb 22nd -- [Latest features in System Modeler][21] - Feb 15th -- [Everything to know about Mellin-Barnes Integrals][22] - Feb 8th -- [Dive into Video Processing][23] - Feb 1st -- [PDE Modeling][24] - Jan. 25th -- [Ask Integration Questions to Oleg Marichev][25] - Jan. 18th -- [My Developer Tools][26] - Jan. 11th -- [Principles of Dynamic Interfaces][27] - Dec. 14th -- [Wolfram Resource System: Repositories & Archives][28] - Dec. 7th -- [Inner Workings of ImageStitch: Image Registration, Projection and Blending][29] - Nov. 30th -- [Q&A for Calculus and Algebra][30] - Nov. 23rd -- [xAct: Efficient Tensor Computer Algebra][31] - Nov. 16th -- [Latest in Machine Learning][32] - Nov. 9th -- [Computational Geology][33] - Nov. 2nd -- [Behind the Scenes at the Wolfram Technology Conference 2022][34] - Oct 26th -- [Group Theory Package (GTPack) and Symmetry Principles in Condensed Matter][35] - Oct 12th -- [Tree Representation for XML, JSON and Symbolic Expressions][36] - Oct. 5th -- [A Computational Exploration of Alcoholic Beverages][37] - Sept. 28th -- [Q&A with Visualization & Graphics Developers][38] - Sept. 14th -- [Paclet Development][39] - Sept. 7th -- [Overview of Chemistry][40] - Aug. 24th -- [Dive into Visualization][41] - Aug. 17th -- [Latest in Graphics & Shaders][42] - Aug. 10th -- [What's new in Calculus & Algebra][43] > **What are your interests? Leave a comment here on this post to share your favorite topic suggestions for our livestreams.** **Follow us on our live broadcasting channels [Twitch][44] and [YouTube][45] and for the up-to-date announcements on our social media: [Facebook][46] and [Twitter][47].** [1]: https://wolfr.am/1eatWLcDA [2]: https://www.twitch.tv/wolfram [3]: https://wolfr.am/1eatWLcDA [4]: https://www.youtube.com/watch?v=C82NHpy7D6k [5]: https://community.wolfram.com/groups/-/m/t/2985580 [6]: https://community.wolfram.com/groups/-/m/t/2982197 [7]: https://community.wolfram.com/groups/-/m/t/2982114 [8]: https://youtu.be/ZqawtrWwE0c [9]: https://community.wolfram.com/groups/-/m/t/2946101 [10]: https://community.wolfram.com/groups/-/m/t/2925156 [11]: https://community.wolfram.com/groups/-/m/t/2921756 [12]: https://community.wolfram.com/groups/-/m/t/2918746 [13]: https://community.wolfram.com/groups/-/m/t/2917597 [14]: https://community.wolfram.com/groups/-/m/t/2911327 [15]: https://community.wolfram.com/groups/-/m/t/2907390 [16]: https://community.wolfram.com/groups/-/m/t/2921593 [17]: https://community.wolfram.com/groups/-/m/t/2861119 [18]: https://community.wolfram.com/groups/-/m/t/2847286 [19]: https://community.wolfram.com/groups/-/m/t/2851575 [20]: https://community.wolfram.com/groups/-/m/t/2852934 [21]: https://community.wolfram.com/groups/-/m/t/2842136 [22]: https://community.wolfram.com/groups/-/m/t/2838335 [23]: https://community.wolfram.com/groups/-/m/t/2827166 [24]: https://community.wolfram.com/groups/-/m/t/2823264 [25]: https://community.wolfram.com/groups/-/m/t/2821053 [26]: https://youtu.be/istKGqpDUsw [27]: https://community.wolfram.com/groups/-/m/t/2777853 [28]: https://youtu.be/roCkXVkDuLA [29]: https://youtu.be/pYHAz-NatXI [30]: https://youtu.be/r7Hjdr_D7c4 [31]: https://community.wolfram.com/groups/-/m/t/2713818 [32]: https://community.wolfram.com/groups/-/m/t/2705779 [33]: https://community.wolfram.com/groups/-/m/t/2701172 [34]: https://youtu.be/UrM-OBu3H9o [35]: https://community.wolfram.com/groups/-/m/t/2678940 [36]: https://community.wolfram.com/groups/-/m/t/2649407 [37]: https://community.wolfram.com/groups/-/m/t/2635049 [38]: https://community.wolfram.com/groups/-/m/t/2618033 [39]: https://community.wolfram.com/groups/-/m/t/2616863 [40]: https://community.wolfram.com/groups/-/m/t/2613617 [41]: https://community.wolfram.com/groups/-/m/t/2605432 [42]: https://community.wolfram.com/groups/-/m/t/2600997 [43]: https://community.wolfram.com/groups/-/m/t/2596451 [44]: https://www.twitch.tv/wolfram [45]: https://wolfr.am/1eatWLcDA [46]: https://www.facebook.com/wolframresearch [47]: https://twitter.com/WolframResearch [48]: https://www.youtube.com/live/ElP55ZILxPw?si=nsAPOQ3u-RbvuGKX [49]: https://community.wolfram.com/groups/-/m/t/3007543 [50]: https://community.wolfram.com/web/charlesp [51]: https://community.wolfram.com/groups/-/m/t/3019288 [52]: https://www.youtube.com/live/KM1yWHRrF2k?si=g2R7rHB2IinVRpo6 [53]: https://community.wolfram.com/groups/-/m/t/3009184 [54]: https://community.wolfram.com/groups/-/m/t/3064700 [55]: https://community.wolfram.com/groups/-/m/t/3084291 [56]: https://community.wolfram.com/groups/-/m/t/3104670 [57]: https://community.wolfram.com/groups/-/m/t/3164204 [58]: https://youtube.com/playlist?list=PLdIcYTEZ4S8TSEk7YmJMvyECtF-KA1SQ2&si=paXZHs0ZzGdB7y1y [59]: https://youtube.com/playlist?list=PLdIcYTEZ4S8RyjEB7JSAsGerbYHl5xXeJ&si=xkNtkIDvKHFWHVmD [60]: https://youtu.be/KUWK19Gx2LE?si=qbKISbL8FtvweSWo [61]: https://community.wolfram.com/web/subat Charles Pooh 2022-08-05T21:37:19Z https://community.wolfram.com/groups/-/m/t/1872608 *Short URL to share this post*: https://wolfr.am/coronavirus ***JOIN*** *our* ***Medical Sciences*** *group for the latest updates & best networking*: https://wolfr.am/MedicalSciences ---------- This post is intended to be the hub for Wolfram resources related to novel coronavirus disease COVID-19 that originated in Wuhan, China. The larger aim is to provide a forum for disseminating ways in which Wolfram technologies and coding can be utilized to shed light on the virus and pandemic. Possibilities include using the Wolfram Language for data-mining, modeling, analysis, visualizations, and so forth. Among other things, we encourage comments and feedback on these resources. Please note that this is intended for technical analysis and discussion supported by computation. Aspects outside this scope and better suited for different forums should be avoided. Thank you for your contribution! ## ________________________________________ ## FEATURED CONTENT - [COVID-19 Livestream Notebook March 24][8] by Stephen Wolfram - [Agent-Based Networks Models for COVID-19][9] by Christopher Wolfram - [Live-Stream: Exploring Pandemic Data][10] by Stephen & Christopher Wolfram + guests - [Live-Stream: Exploring and Explaining Epidemic Modeling][11] by Stephen & Christopher Wolfram + guests ## ________________________________________ ## [CALL for Making COVID-19 Data Computable (*link*)][12] More pandemic-related information and data sets emerging every day. We invite people in the community to contribute to making more data surrounding this topic computable. Here is a call to action with some recommendations for people who want to do more, whether it's just pointing out relevant data sources, or taking the time to make some of that data computable and more instantly ready for other people to explore: https://wolfr.am/COVID-19-DATA . ## ________________________________________ ## [Curated Computable Data (*link*)][13] [FOLLOW THIS LINK][14] to see all available COVID-19 data repositories ready for computation in the Wolfram Language . [Changes in Updates to SARS-CoV-2 Sequences in the Wolfram Data Repository][16] We have published and are continuously updating the Wolfram Data Repository entries. Below are a few key ones. Follow the link above to browse all repositories. We encourage you to make [*your own contributions*][15] of curated data relevant to COVID-19. > **Pandemic Data for Novel Coronavirus COVID-19** > https://www.wolframcloud.com/obj/resourcesystem/published/DataRepository/resources/Epidemic-Data-for-Novel-Coronavirus-COVID-19 > **Genetic Sequences for the SARS-CoV-2 Coronavirus** > https://datarepository.wolframcloud.com/resources/Genetic-Sequences-for-the-SARS-CoV-2-Coronavirus > **Patient Medical Data for Novel Coronavirus COVID-19** > https://datarepository.wolframcloud.com/resources/Patient-Medical-Data-for-Novel-Coronavirus-COVID-19 > **COVID-19 Hospital Resource Use Projections** > https://datarepository.wolframcloud.com/resources/COVID-19-Hospital-Resource-Use-Projections > **OECD Data: Hospital Beds Per Country** > https://datarepository.wolframcloud.com/resources/OECD-Data-Hospital-Beds-Per-Country > **Hospital Beds Per US State** > https://datarepository.wolframcloud.com/resources/Hospital-Beds-Per-US-State ## ________________________________________ ## [Computational Publications (*link*)][17] We encourage you to share your computational explorations relevant to coronavirus on Wolfram Community as stand-alone articles and then comment with their URL links on this discussion thread. We will summarize these articles in the following list: ### ________________________________ ###FEATURED > **COVID-19 Livestream Notebook March 24** by Stephen Wolfram > https://www.wolframcloud.com/obj/s.wolfram/Published/COVID-19-Livestream-March-24.nb > **Agent-Based Networks Models for COVID-19** by Christopher Wolfram > https://community.wolfram.com/groups/-/m/t/1907703 > **Epidemiological Models for Influenza and COVID-19** by Robert Nachbar > https://community.wolfram.com/groups/-/m/t/1896178 > **Epidemic simulation with a polygon container** by Francisco Rodríguez > https://community.wolfram.com/groups/-/m/t/1901002 > **Distance to nearest confirmed US COVID-19 case** by Chip Hurst > https://community.wolfram.com/groups/-/m/t/1911583 ### ________________________________ ### EPIDEMIC MODELING: SIMULATION > **Epidemic simulation with a polygon container** by Francisco Rodríguez > https://community.wolfram.com/groups/-/m/t/1901002 > **Agent based epidemic simulation** by Jon McLoone > https://community.wolfram.com/groups/-/m/t/1900481 > **Modeling the spatial spread of infection diseases in the US** by Diego Zviovich > https://community.wolfram.com/groups/-/m/t/1889072 > **Geo-spatial-temporal COVID-19 simulations and visualizations over USA** by Diego Zviovich > https://community.wolfram.com/groups/-/m/t/1900514 > **Life, Liberty, and Lockdowns: cellular automaton approach** by Philip Maymin > https://community.wolfram.com/groups/-/m/t/2181433 ### ________________________________ ### EPIDEMIC MODELING: COMPARTMENTAL > **Teaching notebook on disease models** by Gareth Russell > https://community.wolfram.com/groups/-/m/t/2694698 > **Stochastic Epidemiology Models with Applications to the COVID-19** by Robert Nachbar > https://community.wolfram.com/groups/-/m/t/1980051 > **COVID19: Italian SIRD estimates and prediction** by Christos Papahristodoulou > https://community.wolfram.com/groups/-/m/t/1984320 > **Solver for COVID-19 epidemic model with the Caputo fractional derivatives** by Alexander Trounev > https://community.wolfram.com/groups/-/m/t/1976589 > **EpiPlay: using Mathematica to gamify education in epidemiology** by Rui Alves > https://community.wolfram.com/groups/-/m/t/2535927 > **Epidemiological Model for repetitive rapid testing for COVID-19** by Diego Zviovich > https://community.wolfram.com/groups/-/m/t/2075883 > **Phase transition of a SIR agent-based models** by Diego Zviovich > https://community.wolfram.com/groups/-/m/t/1977230 > **A simple estimate of covid-19 fatalities based on past data** by Kay Herbert > https://community.wolfram.com/groups/-/m/t/1959438 > **SIR Model with Log-normal infected periods** by Diego Zviovich > https://community.wolfram.com/groups/-/m/t/1946292 > **SEI2HR-Econ model with quarantine and supplies scenarios** by Anton Antonov > https://community.wolfram.com/groups/-/m/t/1937880 > **COVID-19 - Policy Simulator - Can you find the perfect policy?** by Jan Brugard > https://community.wolfram.com/groups/-/m/t/1931352 > **Epidemiological Models for Influenza and COVID-19** by Robert Nachbar > https://community.wolfram.com/groups/-/m/t/1896178 > **Exploring Epidemiological Modeling** by Jordan Hasler > https://community.wolfram.com/groups/-/m/t/1920119 > **SEI2HR model with quarantine scenarios** by Anton Antonov > https://community.wolfram.com/groups/-/m/t/1926505 > **The SIR Model for Spread of Disease** by Arnoud Buzing > https://community.wolfram.com/groups/-/m/t/1903289 > **COVID-19 - R0 and Herd Immunity - are we getting closer?** by Jan Brugard > https://community.wolfram.com/groups/-/m/t/1911422 > **Basic experiments workflow for simple epidemiological models** by Anton Antonov > https://community.wolfram.com/groups/-/m/t/1895675 > **Scaling of epidemiology models with multi-site compartments** by Anton Antonov > https://community.wolfram.com/groups/-/m/t/1897377 > **WirVsVirus 2020 hackathon participation** by Anton Antonov > https://community.wolfram.com/groups/-/m/t/1907256 > **An SEIR like model that fits the coronavirus infection data** by Enrique Garcia Moreno > https://community.wolfram.com/groups/-/m/t/1888335 > **A SEIRD Model For COVID-19 Using DDEs** by Luis Borgonovo > https://community.wolfram.com/groups/-/m/t/1996374 > **A Neat Package for Compartmental Model Diagrams** by Hamza Alsamraee > https://community.wolfram.com/groups/-/m/t/2078640 > **Redesign of didactics of S(E)IR(D) -> SI(EY)A(CD) models of epidemics** by Thomas Colignatus > https://community.wolfram.com/groups/-/m/t/2004784 > **COVID-19 SIR models: transmission, vaccination, herd immunity dynamics revealed** by Athanasios Paraskevopoulos > https://community.wolfram.com/groups/-/m/t/3008488 ### ________________________________ ### EPIDEMIC MODELING: LOGISTIC > **COVID-19 pandemic data in Italy** by Riccardo Fantoni > https://community.wolfram.com/groups/-/m/t/1909687 > **Predicting Coronavirus Epidemic in United States** by Robert Rimmer >https://community.wolfram.com/groups/-/m/t/1906954 > **Tracking Coronavirus Testing in the United States** by Robert Rimmer > https://community.wolfram.com/groups/-/m/t/1902302 > **Logistic Model for Quarantine Controlled Epidemics** by Robert Rimmer > https://community.wolfram.com/groups/-/m/t/1900530 > **Updated: coronavirus logistic growth model: China** by Robert Rimmer > https://community.wolfram.com/groups/-/m/t/1890271 > **Coronavirus logistic growth model: China** by Robert Rimmer > https://community.wolfram.com/groups/-/m/t/1887435 > **Coronavirus logistic growth model: Italy and South Korea** by Robert Rimmer > https://community.wolfram.com/groups/-/m/t/1887823 > **Coronavirus logistic growth model: South Korea** by Robert Rimmer > https://community.wolfram.com/groups/-/m/t/1894561 > **Logistic growth model for epidemic Covid-19 in Colombia** by Diego Ramos > https://community.wolfram.com/groups/-/m/t/2092786 ### ________________________________ ### GENOMICS > **Analyzing the spread of SARS-CoV-2 variants in California** by Daniel Lichtblau > https://community.wolfram.com/groups/-/m/t/2205357 > **Analyzing the spread of SARS-CoV-2 variants in Florida** by Daniel Lichtblau > https://community.wolfram.com/groups/-/m/t/2206874 > **Analyzing Nextstrain Data with WFR Newick Functions (COVID-19/SARS-CoV-2)** by John Cassel > https://community.wolfram.com/groups/-/m/t/1958952 > **Finding and analyzing a COVID subvariant in Australia** by Daniel Lichtblau > https://community.wolfram.com/groups/-/m/t/2342489 > **Analyzing SARS-CoV-2 Genetic Sequences** by John Cassel & Daniel Lichtblau > https://blog.wolfram.com/2021/08/19/newick-trees-proximity-resources-and-accessions-analyzing-sars-cov-2-genetic-sequences/ > **Estimating the number of times the SARS CoV-2 virus has replicated** by Carlos Munoz > https://community.wolfram.com/groups/-/m/t/1943243 >**From sequenced SARS-CoV-2 genomes to a phylogenetic tree** by Daniel Lichtblau >https://community.wolfram.com/groups/-/m/t/1961461 > **Genome analysis and the SARS-nCoV-2** by Daniel Lichtblau > https://community.wolfram.com/groups/-/m/t/1874816 > **Visualizing Sequence Alignments from the COVID-19** by Jessica Shi > https://community.wolfram.com/groups/-/m/t/1875352 > **A walk-through of the SARS-CoV-2 nucleotide Wolfram resource** by John Cassel > https://community.wolfram.com/groups/-/m/t/1887456 > **Geometrical analysis of genome for COVID-19 vs SARS-like viruses** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/1878824 > **Chaos Game For Clustering of Novel Coronavirus COVID-19** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/1875994 ### ________________________________ ### DATA ANALYSIS > **Optimal Annual COVID-19 Vaccine Boosting Dates Following Previous Booster Vaccination or Breakthrough Infection** by Jeffrey Townsend > https://community.wolfram.com/groups/-/m/t/3341399 > **Probability of early infection extinction depends linearly on the virus clearance rate** by Nóra Juhász > https://community.wolfram.com/groups/-/m/t/3307502 > **Detecting Global Community Structure in a COVID-19 Activity Correlation Network** by Hiroki Sayama > https://community.wolfram.com/groups/-/m/t/3056172 > **Analyzing trends of COVID-19 through public news feeds** by Silvia Hao > https://community.wolfram.com/groups/-/m/t/2569395 > **Deep neural network detection & clinical staging of COVID-19 chest X-rays** by Peter Riley > https://community.wolfram.com/groups/-/m/t/2389110 > **COVID-19 - The Swedish Experiment - Is it working?** by Jan Brugard > https://community.wolfram.com/groups/-/m/t/1974412 > **A simple COVID-19 spread model** by Daniel Lichtblau > https://community.wolfram.com/groups/-/m/t/1945196 > **COVID19: The performance of the Swedish strategy** by Christos Papahristodoulou > https://community.wolfram.com/groups/-/m/t/1990972 > **Exploring social trends on Covid-19 pandemic using WikipediaData** by Jofre Espigule-Pons > https://community.wolfram.com/groups/-/m/t/1931508 > **Google Mobility Data** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/1946686 > **Understanding Aggregate COVID Curves** by Christopher Wolfram > https://community.wolfram.com/groups/-/m/t/2068457 > **Apple mobility trends data visualization** by Anton Antonov > https://community.wolfram.com/groups/-/m/t/1942813 > **Computing COVID-19 Spread Rates in US Cities** by Daniel Lichtblau > https://community.wolfram.com/groups/-/m/t/1930261 > **COVID-19 data and the Newcomb Benford Distribution** by Gustavo Delfino > https://community.wolfram.com/groups/-/m/t/1913908 > **Short-time trends for COVID-19**, by Fabian Wenger > https://community.wolfram.com/groups/-/m/t/1912710 > **What countries are hit hard by COVID19 outbreak?** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/1904507 > **COVID19 in Iran: under-diagnosis issue** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/1891140 > **Coronavirus analysis: descriptive statistics with SQL functions** by Damian Calin > https://community.wolfram.com/groups/-/m/t/2206078 > **Covid-19 vaccine campaigns efficacy analysis** by Damian Calin > https://community.wolfram.com/groups/-/m/t/2383314 > **Argentina: COVID-19 Data Analysis** by Tobias Canavesi > https://community.wolfram.com/groups/-/m/t/1932910 > **Analysis of the Change in Phillips Curve After COVID-19 with Regression** by Seojin Yoon > https://community.wolfram.com/groups/-/m/t/2055704 > **COVID wave alert: statistical analysis and visualization** by Antonio Neves > https://community.wolfram.com/groups/-/m/t/2115658 > **Predicting COVID-19 using cough sounds classification** by Siria Sadeddin > https://community.wolfram.com/groups/-/m/t/2166833 > **Covid-19 vaccination data analysis using SQL functions** by Damian Calin > https://community.wolfram.com/groups/-/m/t/2324474 > **Analyzing COVID-19 vaccine sentiment over time** by Arshaan Sayed > https://community.wolfram.com/groups/-/m/t/2317293 > **VAERS data analysis using SQL functions** by Damian Calin > https://community.wolfram.com/groups/-/m/t/2351726 > **Correlating COVID-19 government measures to biweekly/daily outbreaks** by Arshaan Sayed > https://community.wolfram.com/groups/-/m/t/2362327 > **Plotting Covid19 sentiment in different regions of Chennai** by Aditya Sairam Prakash > https://community.wolfram.com/groups/-/m/t/2388236 ### ________________________________ ### DATA VISUALIZATIONS > **CDC COVID19 vaccination data across US counties** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/2282418 > **Top 20 COVID countries HeatMap by absolute death and death in ppm** by Rodrigo Murta > https://community.wolfram.com/groups/-/m/t/2004800 > **COVIDWORLD app: current data and visualizations for SARS-CoV2 pandemic** by Rui Alves > https://community.wolfram.com/groups/-/m/t/2473065 > **US Counties COVID-19 confirmed cases by population density timelines** by Bob Sandheinrich > https://community.wolfram.com/groups/-/m/t/1992898 > **3D Modeling of the SARS-CoV-2 Virus in the Wolfram Language** by Jeff Bryant > https://community.wolfram.com/groups/-/m/t/1989540 > **California COVID19 Data** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/2132204 > **COVID-19 progress in Peru macro regions: coast vs mountain vs jungle** by Francisco Rodríguez > https://community.wolfram.com/groups/-/m/t/1965079 > **COVID-19 reopening criterion: a simple visualization** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/1962615 > **100 Days of COVID19 Over US Counties** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/1956368 > **Population Density Map** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/1955760 > **Google Mobility Data** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/1946686 > **COVID19 Case-Fatality Ratio, Income, and Age: Simple Visualization** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/1939045 > **Data Analysis of Coronavirus in Mexico** by Ivan Martinez > https://community.wolfram.com/groups/-/m/t/1927657 > **Confirmed COVID-19 Cases in Catalonia** by Bernat Espigulé Pons > https://community.wolfram.com/groups/-/m/t/1919468 > **Distance to nearest confirmed US COVID-19 case** by Chip Hurst > https://community.wolfram.com/groups/-/m/t/1911583 > **COVID19 Confirmed Cases: US Counties** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/1950980 > **COVID19 data visualization across US counties** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/2119049 > **Maps for Visualizing Covid-19's Effect** by Eric Mockensturm > https://community.wolfram.com/groups/-/m/t/1934457 > **US Counties COVID-19 deaths plot** by Bob Sandheinrich > https://community.wolfram.com/groups/-/m/t/1918332 > **Comparing the spread of COVID-19 between countries**, Jan Brugard > https://community.wolfram.com/groups/-/m/t/1905992 > **NY Times COVID-19 data visualization** by Anton Antonov > https://community.wolfram.com/groups/-/m/t/1911668 > **COVID-19 cases for each administrative division in Spain** by Bernat Espigulé Pons > https://community.wolfram.com/groups/-/m/t/1910116 > **Propagation risk of COVID-19 by local contact in Spain (10 - 14 March)** by Bernat Espigulé Pons > https://community.wolfram.com/groups/-/m/t/1898126 > **Visualizing the Pandemic Data COVID-19** by Martijn Froeling > https://community.wolfram.com/groups/-/m/t/1899870 > **COVID-19 visualization of turning point** by Isao Maruyama > https://community.wolfram.com/groups/-/m/t/1899911 > **Mapping "Live" COVID Data on a Globe** by Gabriel Lemieux > https://community.wolfram.com/groups/-/m/t/1902102 > **Novel Coronavirus COVID-19 in Brazil** by Estevao Teixeira > https://community.wolfram.com/groups/-/m/t/1905950 > **Mapping Novel Coronavirus COVID-19 Outbreak** by Jofre Espigule-Pons > https://community.wolfram.com/groups/-/m/t/1868945 > **Ways to visualize COVID-19 simulation results?** by Kyle Keane > https://community.wolfram.com/groups/-/m/t/1962739 > **General and COVID-19 deaths in Sweden** by Oscar Rodriguez > https://community.wolfram.com/groups/-/m/t/2006377 > **COVID19 Tokyo per days of the week** Isao Maruyama > https://community.wolfram.com/groups/-/m/t/2133807 ### ________________________________ ### DATA PROCESSING > **Cov-Tell: Daily COVID-19 Updates with Alexa (made with Wolfram APIFunction)** by Jessica Shi > https://community.wolfram.com/groups/-/m/t/1958307 > **Build a COVID-19 Chest X-Ray Image Uploader with Cloud & Data Drop** by Jofre Espigule-Pons > https://community.wolfram.com/groups/-/m/t/1919770 > **Scraping OpenTable's "State of the Industry" page** by Aaron Enright > https://community.wolfram.com/groups/-/m/t/1911043 > **City-level Search Tool for Coronavirus (COVID-19) Confirmed Cases** by David Lomiashvili > https://community.wolfram.com/groups/-/m/t/1913247 > **Web Scraper: New York Times Coronavirus Data** by Robert Rimmer > https://community.wolfram.com/groups/-/m/t/1894426 > **TraCOV: Personalized COVID-19 Risk Analysis Tool** by Jessica Shi > https://community.wolfram.com/groups/-/m/t/1977700 > **Mobility changes data: transforming to Wolfram Language dataset** by Mads Bahrami > https://community.wolfram.com/groups/-/m/t/2160386 ### ________________________________ ### MASKS > **Effect of mandatory mask usage in COVID cases** by Diego Zviovich > https://community.wolfram.com/groups/-/m/t/1919060 > **Face mask detection: classifying image data** by Siria Sadeddin > https://community.wolfram.com/groups/-/m/t/2139499 ## ________________________________________ ## [Livestream Archives (*link*)][18] - Stephen & Christopher Wolfram + guests [Exploring Pandemic Data][19] - Stephen & Christopher Wolfram + guests [Exploring and Explaining Epidemic Modeling][20] - Robert Nachbar - [Epidemiological Models for Influenza and COVID-19][21] - Brian Wood - [COVID-19 Dashboard Visualizations][22] - John Cassel - [Behind the Genetic Sequences for Novel Coronavirus SARS-CoV-2][23] - Keiko Hirayama - [Patient Data Exploration for the Novel Coronavirus COVID-19][24] - Keiko Hirayama - [Pandemic Data Exploration for the Novel Coronavirus COVID-19][25] - Diego Zviovich - [Geo-spatial-temporal COVID-19 Simulations and Visualizations Over USA][26] - Anton Antonov - [COVID19 Epidemic Modeling: Compartmental Models][27] - Anton Antonov - [Scaling of Epidemiology Models with Multi-site Compartments][28] - Anton Antonov - [Simple Economic Extension of Compartmental Epidemiological Models][29] - Juan Klopper - [Coronavirus medical data analysis][30] - Juan Klopper - [Coronavrirus epidemiological data analysis][31] - Rory Foulger - [Coronavirus Data Exploration - Wolfram Livecoding with Students][32] ## ________________________________________ ## Other useful resources - Arnoud Buzing [GitHub][33] repository and [Notebook Gallery][34] for coronavirus - [Modeling a Pandemic like Ebola with the Wolfram Language](https://blog.wolfram.com/2014/11/04/modeling-a-pandemic-like-ebola-with-the-wolfram-language) - [Epidemics at Wolfram Demonstrations](https://demonstrations.wolfram.com/search.html?query=epidemic) - [IGSIRProcess - IGraph Epidemic models][35] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=1China_c.png&userId=1624544 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=1.5US_c.png&userId=1624544 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=2World_c.png&userId=1624544 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=3Genetic_c.png&userId=1624544 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=4Patient_c.png&userId=1624544 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=5Resources_c.png&userId=1624544 [7]: https://www.wolframcloud.com/obj/examples/COVID19Preview.png [8]: https://www.wolframcloud.com/obj/s.wolfram/Published/COVID-19-Livestream-March-24.nb [9]: https://community.wolfram.com/groups/-/m/t/1907703 [10]: https://youtu.be/Vs5APySGYnk [11]: https://youtu.be/kC6LHAv_lx0 [12]: https://community.wolfram.com/groups/-/m/t/1908923 [13]: https://datarepository.wolframcloud.com/search/?i=COVID-19 [14]: https://datarepository.wolframcloud.com/search/?i=COVID-19 [15]: https://reference.wolfram.com/language/workflow/SubmitToTheWolframDataRepository.html [16]: https://community.wolfram.com/groups/-/m/t/2238214 [17]: http://wolfr.am/StaffPicks [18]: https://www.youtube.com/playlist?list=PLxn-kpJHbPx3_hUbroRYC_7NxcOwZ1SWa [19]: https://youtu.be/Vs5APySGYnk [20]: https://youtu.be/kC6LHAv_lx0 [21]: https://youtu.be/pcFB6_yrxGE [22]: https://youtu.be/vUq8qx7kTYA [23]: https://youtu.be/HCJgv3N_kDo [24]: https://youtu.be/MlI_8o4A3BA [25]: https://youtu.be/P86ZY-znE64 [26]: https://youtu.be/Kjk-sYlg-U0 [27]: https://youtu.be/LRs9rYCXIzs [28]: https://youtu.be/b8oCNjRI0gY [29]: https://youtu.be/C-sjXQiPE7s [30]: https://youtu.be/gA0TPQZgNY0 [31]: https://youtu.be/I-n3zN4aU6c [32]: https://youtu.be/4xCfPIiredM [33]: https://github.com/arnoudbuzing/wolfram-coronavirus [34]: https://wolfr.am/JZNRriEE [35]: http://szhorvat.net/mathematica/IGDocumentation/#epidemic-models Vitaliy Kaurov 2020-02-04T15:18:14Z https://community.wolfram.com/groups/-/m/t/1432072 *NOTE: This is a long page with many images. Scroll through to find some gems.* ---------- ![enter image description here][1] [WordFrequencyData][4] is a nifty instrument for mining oceans of texts and discovering wonderful historical semantic curiosities. **This post is a call for you to share your discoveries of interesting word histories**. Rules are very simple. - Post you discovery as a comment on this thread - Your discovery should be curious histories of some words that can be seen in their WordFrequencyData - Start your comment with a title clearly indicating the meaning of your discovery (use # as the first character to make a title) - Your comment must contain a plot WordFrequencyData of your terms. You can use the function I provide below. Alternatively you can use your own what to visualize WordFrequencyData. - Your comment must contain Wolfram Language code you use to make the plot - Your comment must contain some text explaining why you think the words you found are curious and interesting in your opinion - *If you want to comment on someone's work please click REPLY to his/her specific post so it is clear to what you refer and nested structure of comments is preserved.* Please see comment below for good examples. ---------- ### FUNCTION for PLOTs ---------- Feel free to use this function for your visualizations and change or improve it if you wish. Note what kind of options you can provide to this plot. I tried to limit those options to only very important once, fixing other options to make a nice plot. ClearAll@WordFrequencyPlot; Options[WordFrequencyPlot]= {"YearStart"->1800,"YearEnd"->Now,"Case"->True, "Smooth"->3,"Scaling"->None,"Style"->Automatic}; WordFrequencyPlot[words_,OptionsPattern[]]:= With[{ $data=WordFrequencyData[words,"TimeSeries", {OptionValue["YearStart"],OptionValue["YearEnd"]}, IgnoreCase->OptionValue["Case"]]}, DateListPlot[ MapThread[Callout, {MeanFilter[#,Quantity[OptionValue["Smooth"],"Years"]]&/@ Values[$data],words}], ScalingFunctions->OptionValue["Scaling"], PlotRange->All, PlotTheme->"Detailed", PlotStyle->OptionValue["Style"], FrameTicks->{Automatic,None}, ImageSize->Large, FrameLabel->{"YEAR","FREQUENCY in TEXT"}] ] [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-09-04at5.02.04PM.png&userId=11733 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-08-30at8.19.02PM.png&userId=20103 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-09-04at12.48.22PM.png&userId=11733 [4]: http://reference.wolfram.com/language/ref/WordFrequencyData.html Vitaliy Kaurov 2018-08-30T21:41:25Z https://community.wolfram.com/groups/-/m/t/1868945 *MODERATOR NOTE: coronavirus resources & updates:* https://wolfr.am/coronavirus ---------- ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=4686Wuhan_Coronavirus_Outbreak_Jan29.gif&userId=95400 [2]: https://www.wolframcloud.com/obj/c2eafc69-4016-4e05-a8ad-b1d22a37379f Jofre Espigule-Pons 2020-01-29T02:18:52Z https://community.wolfram.com/groups/-/m/t/842698 NOTE: the actual APP that does some analysis of Shakespeare's "Romeo and Juliet" is located [**HERE**][1]. Please wait through a potential little load or evaluation times, it is computing ! ;-) Read below and through comments for many ideas on Shakespeare's data mining. [![enter image description here][2]][3] I also highly recommend reading recent related [blog by Jofre Espigule][4] and checking out his [Wolfram Cloud app][5] that does some social and linguistic visualizations of the Shakespeare's texts: [![enter image description here][6]][5] April 23, 2016 marks 400th anniversary of Shakespeare’s death. Just a few decades of life's work produced texts that fascinate humanity for already 400 years. This centuries-old fascination tells us Shakespeare's works highlight the perpetual social and cultural phenomena. And also that Shakespeare is a seldom genius, a true master of the written and spoken word. But have you ever thought that Shakespeare's texts can be deemed as data? Perhaps Emerging filed of digital humanities can tell us what to read between the lines. Modern technologies can provide a new insight into social networks of characters, semantic, statistical and other properties of corpus that is usually considered of only high artistic value. Is there a pattern in the art? **Could you think of data mining analysis or visualizations to apply to Shakespeare's works? Please share your thoughts! Dive with Wolfram technologies into infinite depths of Shakespeare's data.** EXAMPLE: Storyline ------- Imagine I would like to see in a few quick pictures how the dramatic development of events propagates through a play. I consider "Romeo and Juliet" and download full text as a string (lower-casing all words): romeojuliet = ToLowerCase[Import["http://shakespeare.mit.edu/romeo_juliet/full.html"]]; Now I will write a function `drama` that displays the **density** of a specific word in a play. It is done by indexing positions of words in the text and then running [SmoothKernelDistribution][7] algorithm hidden inside SmoothHistogram function that also plots the density: drama[keywords_List] := With[ {pos = StringPosition[romeojuliet, #][[All, 1]] & /@ keywords}, SmoothHistogram[pos, Frame -> None, BaseStyle -> White, PlotLegends -> Placed[keywords, {{.93, .8}}], AspectRatio -> 1/3, ImageSize -> 700, PlotTheme -> "Marketing", PlotStyle -> {Automatic, Automatic, Dashed, Dashed, Dashed}, Filling -> {1 -> {2}}, FillingStyle -> Directive[White, Opacity[.8]]]] And now with a few computations `drama` reads the play and announces the verdict with just 3 images. Visually we see clearly what was important as the time went by. The 3rd image of interplay between "love", "hate", "life", and "death" speaks the most. drama[{"romeo", "juliet", "life", "death"}] drama[{"romeo", "juliet", "love", "hate"}] drama[{"love", "hate", "life", "death"}] ![enter image description here][8] To make a cloud app, we need to modify function a bit and use [CloudDeploy][9]. dramaFORM[keywords_String] := Rasterize@Module[ {pos, leg, keys = TextWords[ToLowerCase[keywords]]}, leg = {"romeo", "juliet"}~Join~keys; pos = StringPosition[romeojuliet, #][[All, 1]] & /@ leg; SmoothHistogram[DeleteCases[pos, {} | {_Integer}], Frame -> None, PlotLegends -> Placed[leg, Bottom], AspectRatio -> 1/3, ImageSize -> 700, PlotTheme -> "Marketing", PlotStyle -> {Automatic, Automatic}~Join~Table[Dashed, {Length[leg] - 2}], Filling -> {1 -> {2}}, FillingStyle -> Directive[White, Opacity[.8]]]] CloudDeploy[FormFunction[{ "x" -> <|"Label" -> "", "Interpreter"->"String", "Hint"->"hint: love, death", "Help"->Style["type Shakespeare's words separated by spaces or comma, be patient, wait, behold ;-)",Italic]|>}, dramaFORM[#x]&, AppearanceRules-><| "Title" -> Grid[{{"Evolution of topics through Romeo & Juliet"},{Spacer[{10,5}]},{img}},Alignment->Center], "Description" -> "DETAILS: http://wolfr.am/RomeoJuliet "|>, FormTheme -> "Black"], "RomeoAndJuliet", Permissions->"Public"] EXAMPLE: Wordcloud ------------------ It is also interesting to know how modern society sees Shakespeare. The code below for the word cloud runs over Encyclopedia Britannica article about Shakespeare. text=Import["http://www.britannica.com/print/article/537853"]; base[w_]:=With[{tmp=WordData[w,"BaseForm","List"]}, If[(Head[tmp]===Missing)||tmp==={},w,tmp[[1]]]]; SetAttributes[base,Listable]; tst=Quiet[base[TextWords[StringDelete[DeleteStopwords[ToLowerCase[text]],DigitCharacter..]]]]; blackLIST={"shakespeare","william","th","iii","iv","vi"}; WordCloud[DeleteCases[DeleteCases[tst,_First],Alternatives@@blackLIST], WordOrientation->{{-\[Pi]/4,\[Pi]/4}},AspectRatio->1/3, ScalingFunctions->(#^.01&),ImageSize->800] ![enter image description here][10] DATA & CODE SOURCES: -------------------- - [William Shakespeare Plays][11] - [Wolfram Demonstrations][12] - [The Complete Works of William Shakespeare, MIT][13] - [Open Source Shakespeare][14] [at0]: http://community.wolfram.com/web/jofreep [1]: https://www.wolframcloud.com/objects/vitaliyk/RomeoAndJuliet [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=2016-04-22_05-56-43.png&userId=11733 [3]: https://www.wolframcloud.com/objects/vitaliyk/RomeoAndJuliet [4]: http://blog.wolfram.com/2016/04/21/analyzing-shakespeares-texts-on-the-400th-anniversary-of-his-death/ [5]: https://www.wolframcloud.com/objects/jofree/Othello [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=2016-04-24_16-50-23.png&userId=11733 [7]: http://reference.wolfram.com/language/ref/SmoothKernelDistribution.html [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=sdfr45wtrhgfdasaf.png&userId=11733 [9]: http://reference.wolfram.com/language/ref/CloudDeploy.html [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=sadfsdgshdt5342.png&userId=11733 [11]: https://datahub.io/dataset/william-shakespeare-plays [12]: http://demonstrations.wolfram.com/search.html?query=shakespeare [13]: http://shakespeare.mit.edu/ [14]: http://www.opensourceshakespeare.org/ Vitaliy Kaurov 2016-04-20T17:19:46Z https://community.wolfram.com/groups/-/m/t/1257547 # Intro One of the most popular Reddit's channels **Data Is Beautiful** (with multi-million membership of subscribers) has just started **Battle Competitions** for data visualizations that will run monthly. This is a call to Wolfram Community members to collaborate on **JAN 2018 Battle**. ***Direct reference to the JAN 2018 Battle***: https://redd.it/7nm6ed ## Solutions - **Heatmap of inter- and intra- species comparison** *by Vitaliy Kaurov*: - http://community.wolfram.com/groups/-/m/t/1257577 - **Bubble chart for 4D data** *by Sander Huisman*: - http://community.wolfram.com/groups/-/m/t/1257885 - **Population - pyramid like visualization**: *by George Varnavides* - http://community.wolfram.com/groups/-/m/t/1258056 - **Growth Rate in "Intensity Space"** *by Henrik Schachner* - http://community.wolfram.com/groups/-/m/t/1258281 - **Intraspecies comparison using RadarChart** *by Diego Zviovich* - http://community.wolfram.com/groups/-/m/t/1260507 - **Interspecies comparison using HeatmapPlot** by *Anton Antonov* - http://community.wolfram.com/groups/-/m/t/1261444 - **Scatter plot slices of temperature dynamics** *by Vitaliy Kaurov* - http://community.wolfram.com/groups/-/m/t/1261948 - **RadarChart for each pecies** by *Anton Antonov* - http://community.wolfram.com/groups/-/m/t/1261438 ## Rules of this thread Reddi requires direct links to the images. Hence a separate post is necessary. Here are the steps: - Make a separate post solving the challenge with detailed title describing your specific method of visualization and starting with tag [Reddit-DiBB0118] (Data is Beautiful Battle 01/2018) - Make a comment in this thread simply stating the title and copying your post URL. See my example in the comments. I will collect the solutions in the "solutions" section above. This method enables you to post your own posts on Reddit if you want to keeping your authorship. ## Goal I simply suggest that Wolfram Community members brainstorm in the comments below about how the best to visualize the dataset. Feel free to submit **your own** solutions to Reddit if you want to as they require the original authors. The main goal though is simply to have fun here on Community. **Don't forget to vote up the posts you like**. ## Important Battles have simple rules explained clearly in the Reddit battle thread linked above. To not get disqualified it is advised to read rules carefully. You can ask Reddit admins any additional questions directly in the thread comments. I recommend reading other people comments as they clarify the nature of the dataset. ## Getting the data w/ Wolfram Language (WL) The dataset is located at a web page: http://aquatext.com/tables/algaegrwth.htm The nature of the data is clear from the website description. It is easy to get the raw data with the following WL command: raw = Import["http://aquatext.com/tables/algaegrwth.htm", "Data"] /."0..06" -> .06; You need `/."0..06" -> .06` because the data has a clerical error resulting in the import of a string instead of a number. One way of obtaining a simple rectangular array / table of data is: data=Cases[raw,{_String,__?NumberQ},Infinity]/. x_List/;First[x]=="Temperature":>{"Temperature",5,5,10,10,25,25,30,30}; which can be viewed as TableForm[data] ![enter image description here][3] [1]: https://www.reddit.com/r/dataisbeautiful/ [2]: http://community.wolfram.com/groups/-/m/t/1257577 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-01-02at6.20.30PM.png&userId=11733 Vitaliy Kaurov 2018-01-03T00:33:35Z https://community.wolfram.com/groups/-/m/t/326240 Triggered by the recent outbreak of Ebola India Bruckner, a pupil from Aberdeen's [St Margaret's School for Girls][1], and myself worked on a little model this summer to understand the basics of the spreading of diseases in populations and the relationship to transportation networks. The model is very basic, but shows some interesting features and is very straight forward to implement in Mathematica. When I was typing these lines I saw that Arnoud Buzing had posted something, reason enough to interrupt my typing and to check out what he had posted: [Visualizing the Ebola Outbreak][2]. I hope that my post is going to complement Arnoud's to some extent. So, my question is how the global air transport network might lead to a spreading of a disease. I will use a very standard SIR (susceptible-infected-recovered) model, which is certainly far from being ideal for Ebola; [but similar types of models are to too bad either][3]. It rather simulates an outbreak of some generic disease from which you recover. If we assumed that everyone died in an outbreak the SIR model might also be appropriate. I will introduce the equations below. I also need a list of all airports and all flight connections. On the website [Openflights.org][4] you will find all data we need. I saved the file "airports.dat" and the file "routes.dat". So that's the data. I first import the data: airports = Import["~/Desktop/airports.dat", "CSV"]; routes = Import["~/Desktop/routes.dat", "CSV"]; This is a plot of all airports in that database. GeoRegionValuePlot[Table[GeoPosition[airports[[i, {7, 8}]]] -> 1., {i, 1, Length[airports]}], PlotStyle -> PointSize[0.003], PlotRange -> 1, ImageSize -> Full] which gives ![enter image description here][5] Alright, now the routes. First, we create a list of rules for all airport IDs and their coordinates: codecoords = Table[airports[[i, 5]] -> GeoPosition[airports[[i, {7, 8}]]], {i, 1,Length[airports]}]; We then calculate the links: links = Monitor[Table[routes[[j, {3, 5}]] /. codecoords, {j, 1, Length[routes]}], ProgressIndicator[j, {1, Length[routes]}]]; and clean out missing data: linksclean = Select[links, Head[#[[1]]] == GeoPosition && Head[#[[2]]] == GeoPosition &]; Now comes a nice figure: With[{locations = RandomChoice[linksclean, 14000]}, GeoGraphics[{{Green, Opacity[0.3],AbsoluteThickness[0.0001], GeoPath[locations, "Geodesic"]}}, GeoRange -> "World", GeoProjection -> Automatic, GeoBackground -> GeoStyling["ReliefMap"], ImageSize -> {1200, 600}]] which gives ![enter image description here][6] Ok. Interestingly I can only plot 16000 max at a time. Somewhere between 16-17k the Kernel quits. That might be Integer related. Could be a limit in the programming of Geographics. Not sure. I have more than enough memory and can plot the remaining 2-3k airports in a second figure and use Show to display all. (It would be great if someone from WRI could comment.) Anyway, let's go to some modelling. The basic idea of an [SIR model][7] is that a population is modelled in three compartments Susceptibles (S), Infected (I) and Recovered (R). I will use a time-discrete model; there are continuous models around, too, and if anyone is interested I can provide the ODE model as well. Here are the three equations: sus[i_] := sus[i] = sus[i - 1] - [Rho] sus[i - 1] inf[i - 1]; inf[i_] := nf[i] = inf[i - 1] + [Rho] sus[i - 1] inf[i - 1] - [Lambda] inf[i - 1]; rec[i_] := rec[i] = rec[i - 1] + [Lambda] inf[i - 1]; The meaning of sus, inf and rec should be clear by now; they are given as percentages of the total population, their sum is 100%. The variable i represents time. $ ho$ is an infection rate and $lambda$ is a recovery rate. The infections increase with the product of susceptibles and infected. By adding the right hand sides it becomes clear that the population does not change over time. We come up with some values for the parameters and iterate: sus[1] = 0.95; inf[1] = 0.05; rec[1] = 0; [Rho] = 0.2; [Lambda] = 0.1; tcourse = Table[{sus[i], inf[i], rec[i]}, {i, 1, 100}]; The time course looks like this: ListPlot[Transpose[tcourse]] ![enter image description here][8] The monotonously decreasing function show susceptibles, the increasing function recovered, and the remaining curve the infected. We now need to do some cleaning up of the original airport data, before we proceed to a multi-airport/city model. (*Extract the names and GPS coordinates*) rawdata = Sort[Select[airports[[All, {5, 7, 8}]], #[[1]] != "" &]][[81 ;;]]; (*These are just the coordinates*) airportcoords = rawdata[[All, {2, 3}]]; (*These are just the names. *) names = rawdata[[All, 1]]; (*Here are the names to indices*) rules = MapThread[#1 -> #2 &, {names, Range[Length[names]]}]; (*The "population" is initially set to 1 for all airports, this allows us to take different airport sizes into consideration later.*) pop = Table[1., {j, 1, Length[names]}]; routesraw = Import["~/Desktop/routes.dat", "CSV"]; (*There are many links so this takes a while*) links = Select[routesraw[[All, {3, 5}]] /. rules, NumberQ[#[[1]]] && NumberQ[#[[2]]] &]; From that we now construct (a first guess at) the coupling or adjacency matrix: couplingdummy = Table[0, {i, 1, Length[names]}, {j, 1, Length[names]}]; For[k = 1, k <= Length[links], k++, couplingdummy[[links[[k, 1]], links[[k, 2]]]] = 1; couplingdummy[[links[[k, 2]], links[[k, 1]]]] = 1]; I do know about ConstantArray, but for some reason that does not work. The first line constructs a matrix full of zeros and the second adds ones where there are links. The problem is that apparently in that dataset some airports are not linked at all. We can sort them out by: indices = Select[Table[If[Total[couplingdummy[[i]]] > 0, i], {i, 1, Length[couplingdummy]}], NumberQ]; We now delete the columns and rows of the couplingdummy matrix intermed = couplingdummy[[#]] & /@ indices; transintermed = Transpose[intermed]; coupling = transintermed[[#]] & /@ indices; Again I had a much more elegant way of doing this, with the advantage that it did not work. To speed up the following calculations I use that the coupling matrix is sparse, but I like the original too much to throw it away just yet. coulinginterm = coupling; coupling = SparseArray[coulinginterm]; We adapt our "population/airport size" vector: pop = Table[1., {j, 1, Length[indices]}]; and set the following parameters: [Rho] = 0.2; [Lambda] = 0.1; Mairports = Length[indices]; [Mu] = 0.05; $\rho$ and $lambda$ are as before. Mairports is the number of airports that we model and $\mu$ is a "migration rate". It comes from the original model which we built for different cities were it describes the migration between different cities. Here is models the "propensity to fly". We now define an effective coupling matrix. It is the adjacency matrix times the population vector (i.e. people in the catchment area of the airport). In our case the vector has all ones, so it is just the adjacency matrix. It allows us later to model more general situations. meanNN = coupling.pop; When we want to model the outbreak as populations at the positions of all airports, each of which is described by an SIR model, we need to couple lots of populations, because there are lots of airports. The following uses the sparsity of the adjacency matrix to speed up the calculation. sumind = Table[Take[Flatten[ArrayRules[coupling[[k, All]]][[All, 1]]], Length[ArrayRules[coupling[[k, All]]]] - 1], {k, 1, Mairports}]; It generates a list of all airports that are coupled/linked to a given airport. Now we are ready to write down the central equations: sus[i_, j_] := sus[i, j] = (1 - [Mu]) (sus[i - 1, j] - [Rho] sus[i - 1, j] inf[i - 1, j]) + [Mu] Total[Table[sus[i - 1, sumind[[j, u]]]*pop[[sumind[[j, u]]]], {u, 1, Length[sumind[[j]]]} ] ]/meanNN[[j]]; inf[i_, j_] := inf[i, j] = (1 - [Mu]) (inf[i - 1, j] + [Rho] sus[i - 1, j] inf[i - 1, j] - [Lambda] inf[i - 1, j]) + [Mu] Total[Table[inf[i - 1, sumind[[j, u]]]*pop[[sumind[[j, u]]]], {u, 1, Length[sumind[[j]]]} ] ]/meanNN[[j]]; rec[i_, j_] := rec[i, j] = (1 - [Mu]) (rec[i - 1, j] + [Lambda] inf[i - 1, j]) + [Mu] Total[Table[rec[i - 1, sumind[[j, u]]]*pop[[sumind[[j, u]]]], {u, 1,Length[sumind[[j]]]} ] ]/meanNN[[j]]; The terms with the Total are "migration terms" that describe the travelling behaviour of the people in the catchment area of the airports. i is the time index and j labels the airports. Next come the initial conditions: For[i = 1, i <= Mairports, i++, sus[1, i] = 1.; inf[1, i] = 0.0; rec[1, i] = 0.;] sus[1, 1] = 0.95; inf[1, 1] = 0.05; rec[1, 1] = 0.0; In the catchment areas of all airports there are only susceptibles, apart from airport number 1, which will have 5% infected people. Now we can finally iterate the whole thing: tcourse = Monitor[Table[{sus[i, j], inf[i, j], rec[i, j]}, {i, 1, 500}, {j, 1,Mairports}], ProgressIndicator[i, {0, 500}]]; Great. Let's save that just in case your notebook tends to crash at this point, just l like mine did when I was playing with this. Export["~/Desktop/SIR-tcourse.csv", tcourse]; If you wish you can now plot the time course of some of the airport catchment areas: ListPlot[Flatten[Table[{tcourse[[All, j, 1]], tcourse[[All, j, 2]], tcourse[[All, j, 3]]}, {j, 1, 200}], 1], ImageSize -> Large] ![enter image description here][9] Now that is not very helpful yet. To generate nicer plots, i.e. to normalise, we first calculate the maximal number of sick people at any of the airports: maxsick = Max[Flatten[tcourse[[All, All, 2]]]]; We then generate movie frames, and go and get some coffee.... frames = Monitor[Table[GeoRegionValuePlot[Table[GeoPosition[airportcoords[[indices[[i]]]]] -> inf[k, i]/maxsick, {i, 1, Length[indices]}], PlotStyle -> PointSize[0.003], PlotRange -> 1, ImageSize -> Full,ColorFunction -> "Rainbow"], {k, 1, 300}], ProgressIndicator[k, {0, 300}]]; Actually, you might want to get another coffee when you want to export the frames: Export["~/Desktop/SIR-frames-World.gif", frames]; Alright. That gif is a bit large to embed it into this post, but you can download it from [here][10]. All I can do is show you some frames to get an idea of how this looks: ![enter image description here][11] Of course we can look at the network structure and try to understand the pattern of infections. This command is useful: CommunityGraphPlot[AdjacencyGraph[Normal[coupling]]] ![enter image description here][12] You clearly see the communities in North America, Europe and Asia. This one is also pretty: Show[TreePlot[Subgraph[grph, ConnectedComponents[grph][[1]]], Center, PlotStyle -> Directive[Gray, Opacity[0.02]]], TreePlot[Subgraph[grph, ConnectedComponents[grph][[1]]], Center, EdgeRenderingFunction -> None]] ![enter image description here][13] We have several enhancements of this. First we can easily look at different countries individually. What if an ebola patient arrives at some airport in the US? [See simulation here][14]. (Careful 50 MBs!) There is also something we can do if we want to go the the level of cities. The main problem is that the Wolfram database does not yet have data for all streets between cities. In one of the online conferences it was said that that will be introduced in some later version, which I cannot wait to play with. Until then we have to cheat. (or use some online database; I prefer cheating.) We developed a model of the spreading of a disease in Nigeria. So we could go about this like this: Clear["Global`*"] CountryData["Nigeria", "Population"] Graphics[CountryData["Nigeria", "Polygon"]] Then get city names, coords and population: names = CityData[{All, "Nigeria"}]; citypop = Table[CityData[names[[i]], "Population"], {i, 1, Length[names]}]; citycoords = Table[CityData[names[[i]], "Coordinates"], {i, 1, Length[names]}]; Here comes the cheat. Because we don't have the streets we use Delaunay triangulation: Needs["ComputationalGeometry`"] dtri = DelaunayTriangulation[citycoords]; list = {}; Table[ Do[AppendTo[list, {i, dtri[[All, 2]][[i, j]]}], {j, 1, Length[dtri[[All, 2]][[i, All]]]}], {i, 1, Length[dtri]}]; coupling = Table[0, {i, 1, Length[names]}, {j, 1, Length[names]}]; For[i = 1, i < Length[list] + 1, i++, coupling[[list[[i]][[1]], list[[i]][[2]]]] = 1;] coulinginterm = coupling; coupling = SparseArray[coulinginterm]; which gives the following network Graphics[Join[ Table[Circle[citycoords[[i]], 0.02], {i, 1, Length[names]}], DeleteCases[ Flatten[Table[ If[coupling[[i, j]] == 1, Line[{citycoords[[i]], citycoords[[j]]}]] , {i, 1, Length[names]}, {j, 1, i}], 1], Null]]] ![enter image description here][15] The point in the middle corresponds to the airport where it all starts; then come its neighbours and then their neighbours etc. You could now animate this and change the colours to see how the disease spreads through the different layers. It would be nice if someone could implement that. There are obviously some problems, i.e. "streets" leaving the country etc, but the general idea should work. The rest is quite the same as before: (*Paramters*) [Rho] = 0.2; [Lambda] = 0.1; Mcities = Length[names]; [Mu] = 0.05; (*Initiation*) For[i = 1, i <= Mcities, i++, sus[1, i] = 1.; inf[1, i] = 0.0; rec[1, i] = 0.;] (*Starting Outbrake at the following city*) sus[1, 1] = 0.95; inf[1, 1] = 0.05; rec[1, 1] = 0.0; meanNN = coupling.citypop; sumind = Table[ Take[Flatten[ArrayRules[coupling[[k, All]]][[All, 1]]], Length[ArrayRules[coupling[[k, All]]]] - 1], {k, 1, Mcities}]; sus[i_, j_] := sus[i, j] = (1 - [Mu]) (sus[i - 1, j] - [Rho] sus[i - 1, j] inf[i - 1, j]) + [Mu] Total[ Table[sus[i - 1, sumind[[j, u]]]*citypop[[sumind[[j, u]]]], {u, 1, Length[sumind[[j]]]} ] ]/meanNN[[j]]; inf[i_, j_] := inf[i, j] = (1 - [Mu]) (inf[i - 1, j] + [Rho] sus[i - 1, j] inf[i - 1, j] - [Lambda] inf[i - 1, j]) + [Mu] Total[ Table[inf[i - 1, sumind[[j, u]]]*citypop[[sumind[[j, u]]]], {u, 1, Length[sumind[[j]]]} ] ]/meanNN[[j]]; rec[i_, j_] := rec[i, j] = (1 - [Mu]) (rec[i - 1, j] + [Lambda] inf[i - 1, j]) + [Mu] Total[ Table[rec[i - 1, sumind[[j, u]]]*citypop[[sumind[[j, u]]]], {u, 1, Length[sumind[[j]]]} ] ]/meanNN[[j]]; This time we try to work in parallel: LaunchKernels[]; tcourse = ParallelTable[{sus[i, j], inf[i, j], rec[i, j]}, {i, 1, 500}, {j, 1, Mcities}]; // AbsoluteTiming (There is something strange here. This ran in MMA9 in 6.3 seconds- I still have data from the course I taught last year. MMA10 takes ages. After the installation of MMA10 also MMA9 seems to take longer 43 seconds. Is this a bug report?). Note that this time the population sizes of the cities are taken into account and are relevant. If you run Manipulate[ Graphics[{Line[Flatten[CountryData["Nigeria", "Coordinates"], 1]], Join[Table[{ RGBColor[tcourse[[t, i, 2]], tcourse[[t, i, 1]], tcourse[[t, i, 3]]], Disk[citycoords[[i]], 0.1]}, {i, 1, Length[names]}]]}], {t, 1, 500, 1}] or poly = Graphics[Polygon[Flatten[CountryData["Nigeria", "Coordinates"], 1]], ImagePadding -> None]; Animate[ImageSubtract[ Graphics[ListDensityPlot[ Join[{{4, 3.25, 0}, {4, 14, 0}, {14, 3.25, 0}, {14, 14, 0}}, Table[{citycoords[[k]][[1]], citycoords[[k]][[2]], 1. - tcourse[[t, k, 1]]}, {k, 1, Mcities}]], InterpolationOrder -> 3, ColorFunction -> "Rainbow", PlotRange -> All, Frame -> False, PlotRangePadding -> None]], poly], {t, 1, 500, 1}, DefaultDuration -> 20.] or better infmax = Max[tcourse[[All, All, 2]]]; frames = Table[ ImageSubtract[ Graphics[ ListDensityPlot[ Join[{{4, 3.25, 0}, {4, 14, 0}, {14, 3.25, 0}, {14, 14, 0}}, Table[{citycoords[[k]][[1]], citycoords[[k]][[2]], tcourse[[t, k, 2]]/infmax}, {k, 1, Mcities}]], InterpolationOrder -> 3, ColorFunction -> "Rainbow", PlotRange -> All, Frame -> False, PlotRangePadding -> None, ColorFunctionScaling -> False]], poly], {t, 1, 500, 6}]; you get nice animations like this one: ![enter image description here][16] I have noticed that Nigeria needs to be rotated, but I hope that the idea becomes clear. I am also aware that there are many flaws in this. SIR is certainly not the best way forward to model Ebola. Any population dynamicist and/or health expert can certainly come up with an endless list of problems. The network is not perfect. For more serious applications we actually use models for the cities, i.e. street connections among close cities and airport connections among countries etc. The problem is that if we simulate between 200-20000 cities per country plus the airports, a standard laptop runs into trouble. On the bright side, we have a cluster on which this kind of larger simulations work just fine. Hope that you like this anyway, Marco [1]: http://www.st-margaret.aberdeen.sch.uk [2]: http://community.wolfram.com/groups/-/m/t/325956 [3]: http://mtbi.asu.edu/files/Mathematical_Models_to_Study_the_Outbreaks_of_Ebola.pdf [4]: http://openflights.org/data.html [5]: /c/portal/getImageAttachment?filename=Airportsall.jpg&userId=48754 [6]: /c/portal/getImageAttachment?filename=Airports-world.jpg&userId=48754 [7]: http://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology [8]: /c/portal/getImageAttachment?filename=SingleSIR.jpg&userId=48754 [9]: /c/portal/getImageAttachment?filename=SIR-airportstcourse.jpg&userId=48754 [10]: https://www.dropbox.com/s/9y6d16z82qjw261/SIR-frames.gif?dl=0 [11]: /c/portal/getImageAttachment?filename=SIR-airports-frames.jpg&userId=48754 [12]: /c/portal/getImageAttachment?filename=Airports-CommNetwork.jpg&userId=48754 [13]: /c/portal/getImageAttachment?filename=SIR-networkrings.jpg&userId=48754 [14]: https://www.dropbox.com/s/0qwgmi7ks8dpsjh/SIR-USA-frames.gif?dl=0 [15]: /c/portal/getImageAttachment?filename=SIR-Nigerianetwork.jpg&userId=48754 [16]: /c/portal/getImageAttachment?filename=SIR-movie.gif&userId=48754 Marco Thiel 2014-08-22T20:18:58Z https://community.wolfram.com/groups/-/m/t/2007434 Our newest [Daily Study Group][1] offers a jump-start on earning Wolfram certifications and covers topics including Wolfram Notebooks, image processing and multiparadigm data science. Sign up at: https://wolfr.am/NpiuhRsg [1]: https://www.wolfram.com/wolfram-u/special-event/study-groups/ Jamie Peterson 2020-06-18T21:21:10Z https://community.wolfram.com/groups/-/m/t/2825207 A Wolfram U Daily Study Group on Introduction to Probability begins on **February 27th 2023**. Join me and a group of fellow learners to learn about the world of probability and statistics using the Wolfram Language. Our topics for the study group include the characterisation of randomness, random variable design and analysis, important random distributions and their applications, probability-based data science and advanced probability distributions. The idea behind this study group is to rapidly develop an intuitive understanding of probability for a college student, professional or interested hobbyist. A basic working knowledge of the Wolfram Language is recommended but not necessary. We are happy to help beginners get up to speed with Wolfram Language using resources already available on Wolfram U. Please feel free to use this thread to collaborate and share ideas, materials and links to other resources with fellow learners. > [**REGISTER HERE**][1] ![enter image description here][2] ![Wolfram U Banner][3] [1]: https://www.bigmarker.com/series/daily-study-group-probability-wsg36/series_details [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=NormalConvergence.gif&userId=11733 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=banner.jpg&userId=2823613 Marc Vicuna 2023-02-07T01:15:38Z https://community.wolfram.com/groups/-/m/t/258719 One of our members, Allison Newman, was wondering where the sign from recent XKCD comic - [url=http://xkcd.com/1375]Astronaut Vandalism[/url] could be really located. [img=width: 317px; height: 400px;]/c/portal/getImageAttachment?filename=astronaut_vandalism.png&userId=11733[/img] Could we locate where the sign is? At first it seems like a triangulation problem but because "Space" is probably not a point B) we have to live with two possible solutions from disk intersections. Check out first [url=http://reference.wolfram.com/language/ref/GeoGraphics.html]GeoGraphics[/url] and [url=http://reference.wolfram.com/language/ref/GeoDisk.html]GeoDisk[/url]. With just a little bit of code we can build: [mcode]GeoGraphics[{ Red, GeoDisk[ Entity["City", {"Memphis", "Tennessee", "UnitedStates"}], Quantity[98, "Miles"], {135, 225}], Blue, GeoDisk[ Entity["City", {"Jackson", "Mississippi", "UnitedStates"}], Quantity[115, "Miles"], {-45, 45}] }, GeoZoomLevel -> 9, ImageSize -> 500][/mcode] [img=width: 800px; height: 940px;]/c/portal/getImageAttachment?filename=sdaf233qterwgfda2343qgrefa.png&userId=11733[/img] So if you zoom a bit you'll se that eastern point could be somewhere around [url=http://wolfr.am/1tXg6se]Derma, Mississippi[/url]. Indeed for Memphis: [mcode]GeoDistance[ Entity["City", {"Derma", "Mississippi", "UnitedStates"}], Entity["City", {"Memphis", "Tennessee", "UnitedStates"}], UnitSystem -> "Imperial"] Out[] = 83.5647 mi[/mcode] and for Jackson [mcode]GeoDistance[ Entity["City", {"Derma", "Mississippi", "UnitedStates"}], Entity["City", {"Jackson", "Mississippi", "UnitedStates"}], UnitSystem -> "Imperial"] Out[] = 109.373 mi[/mcode] I am sure someone could do better than this - please do B)  [url=http://reference.wolfram.com/language/ref/GeoNearest.html]GeoNearest[/url] possibly? Vitaliy Kaurov 2014-05-30T18:14:33Z https://community.wolfram.com/groups/-/m/t/1167544 *NOTE: all Wolfram Language code and data are available in the attached notebook at the end of the post.* ---------- For my class this fall, I developed a little primer on Association and Dataset that I think might be useful for many people. So, I'm sharing the attached notebook. It's somewhat about the concepts embedded inside these features. It's intended for people at a beginner-intermediate level of Mathematica/Wolfram Language programming, but might be of value even to some more advanced users who have not poked about the Dataset functionality. The sections of the notebook are: 1. The world before Associations and Datasets 2. Datasets without Associations 3. Enter the Association 4. Creating a Dataset from a List of Associations 5. Nice queries with Dataset 6. Query 7. Some Recipes #The world before Associations and Datasets# Here' s an array of data. The data happens to represent the cabin class, age, gender, and survival of some of the passengers on the Titanic. t = {{"1st", 29, "female", True}, {"1st", 30, "male", False}, {"1st", 58, "female", True}, {"1st", 52, "female", True}, {"1st", 21, "female", True}, {"2nd", 54, "male", False}, {"2nd", 29, "female", False}, {"3rd", 42, "male", False}}; As it stands, our data is a List of Lists. Head[t] > List Head /@ t > {List, List, List, List, List, List, List, List} Suppose I wanted to get the second and fifth rows of the data. This is how I could do it. t[[{2, 5}]] > {{"1st", 30, "male", False}, {"1st", 21, "female", True}} Suppose we want to group the passengers by gender and then compute the mean age. We could do this with the following pretty confusing code. Use and enjoy. Constructive feedback appreciated. grouped = GatherBy[t, #[[3]] &]; justTheAges = grouped[[All, All, 2]]; Mean /@ justTheAges > {189/5, 42} Or I could write it as a one liner this way. Map[Mean, GatherBy[t, #[[3]] &][[All, All, 2]]] > {189/5, 42} But either way, realize that I have to remember that gender is the third column and that age is the second column. When there is a lot of data, this can get hard to remember. #Datasets without Associations# I could, if I wanted, convert this data into a Dataset. I do this below simply by wrapping Dataset about t. You see there is now some formatting about the data. But there are no column headers (because no one has told Dataset what to use). And there are no row headers, again because no one has told Dataset what to use. t2 = Dataset[t] ![enter image description here][1] The head of the expression has changed. Head[t2] > Dataset Now, I can now access the data in a different way. Query[{2, 5}][t2] ![enter image description here][2] Or, I can do this. Mathematica basically converts this expression into Query[{2,5}][t2]. The expression t2[{2,5}] is basically syntactic sugar. t2[{2, 5}] ![enter image description here][3] ##Digression : Using Query explicitly or using syntactic sugar## Why, by the way would anyone use the longer form if Mathematica does the work for you? Suppose you want to store a Dataset operation -- perhaps a complex series of Dataset operations -- but you want it to work not just on a particular Dataset but on any Dataset (that is compatible). Here's how you could do it. q = Query[{2, 5}] > Query[{2, 5}] q[t2] ![enter image description here][4] Now, let' s create a permutation of the t2 Dataset so that the rows are scrambled up. t2Scrambled = t2[{1, 4, 8, 3, 2, 7, 5}] ![enter image description here][5] We can now run the q operation on t2Scrambled. Notice that the output has changed even though the query has stayed the same. q[t2Scrambled] ![enter image description here][6] We can also generate Query objects with functions. Here's a trivial example. There are very few languages of which I am aware that have the ability to generate queries by using a function. The one other example is Julia. makeASimpleQuery[n_] := Query[n] makeASimpleQuery[{3, 4, 7}][t2] ![enter image description here][7] ##MapReduce operations on Dataset objects## Now, if I want to know the mean ages of the genders I can use this code. This kind of grouping of data and then performing some sort of aggregation operation on the groups is sometimes known as a MapReduce. (I'm not a fan of the name, but it is widely used). It's also sometimes known as a rollup or an aggregation. Query[GroupBy[#[[3]] &], Mean, #[[2]] &][t2] ![enter image description here][8] Or this shorthand form in which the Query is constructed. t2g = t2[GroupBy[#[[3]] &], Mean, #[[2]] &] ![enter image description here][9] I think this is a little cleaner. But we still have to remember the numbers of the columns, which can be challenging. By the way, just to emphasize how we can make this all functional, here's a function that creates a query that can run any operation (not just computing the mean) on the Dataset grouped by gender and then working on age. genderOp[f_] := Query[GroupBy[#[[3]] &], f, #[[2]] &] genderOp[Max][t2] ![enter image description here][10] To test your understanding, see if you can find the minimum age for each class of passenger on the Titanic in our Dataset **t2**. Query[GroupBy[#[[1]] &], Min, #[[2]] &][t2] ![enter image description here][11] #Enter the Association# ##Review of Association## If you feel comfortable with Associations, you can skip this section; otherwise read it carefully. Basically the key to understanding most Dataset operations is understanding Associations. ###Construction of Associations### Now let' s alter the data so that we don't have to remember those facts. To do this we will create an **Association**. Here's an example called **assoc1**. Notice that we do so by creating a sequence of rules and then wrapping it in an Association head. Notice that the standard output does not preserve the word "Association" as the head but, just as List is outputted as stuff inside curly braces, Association is outputted as stuff inside these funky "<|" and "|>" glyphs. assoc1 = Association["class" -> "1st", "age" -> 29, "gender" -> "female", "survived" -> True] > <|"class" -> "1st", "age" -> 29, "gender" -> "female", "survived" -> True|> I could equivalently have created a list of rules rather than a sequence. Mathematica would basically unwrap the **List** and create a sequence. assoc1L = Association[{"class" -> "1st", "age" -> 29, "gender" -> "female", "survived" -> True}] > <|"class" -> "1st", "age" -> 29, "gender" -> "female", "survived" -> True|> We can use **AssociationThread** to create Associations in a different way. The first argument is the list of things that go on the left hand side of the Rules -- the "keys" -- and the second argument is the list of things that go on the right hand side of the Rules -- the "values". assoc1T = AssociationThread[{"class", "age", "gender", "survived"}, {"1st", 29, "female", True}] > <|"class" -> "1st", "age" -> 29, "gender" -> "female", "survived" -> True|> Now let's use **AssociationThread** function to create a list of Associations similar to our original data. convertListToAssociation = list \[Function] AssociationThread[{"class", "age", "gender", "survived"}, list] > Function[list, AssociationThread[{"class", "age", "gender", "survived"}, list]] I start with t and Map the **convertListToAssociation** function over the rows of the data. I end up with a list of Associations. t3 = Map[convertListToAssociation, t] ![enter image description here][12] ###Keys and Values### Associations have keys and values. These data structures are used in other computer languages but known by different names: *Python* and *Julia* call them dictionaries. *Go* and *Scala* call them maps. *Perl* and *Ruby* call them hashes. *Java* calls it a *HashMap*. And *Javascript* calls it an object. But they all work pretty similarly. Anyway, the keys of an **Association** are the things on the left hand side of the Rules. Keys[assoc1] > {"class", "age", "gender", "survived"} And the values of an Association are the things on the right hand side of the Rules. Values[assoc1] > {"1st", 29, "female", True} That' s about all there is too it. Except for one thing. Take a look at the input and output that follows. assoc2 = Association["a" -> 3, "b" -> 4, "a" -> 5] > <|"a" -> 5, "b" -> 4|> You can' t have duplicate keys in an Association. So, when Mathematica confronts duplicate keys, it uses the last key it saw. You might think this is a minor point, but it is actually very important in coding. We will see why soon. ###Nested Associations### A funny thing happens if you nest an **Association** inside another **Association**. Association[assoc1, assoc2] > <|"class" -> "1st", "age" -> 29, "gender" -> "female", "survived" -> True, "a" -> 5, "b" -> 4|> You end up with a single un - nested (flat) association. That's a little unusual for Mathematica, but we can exploit this flattening as a way of adding elements to an Association. Association[Association["dances" -> False], assoc1] > <|"dances" -> False, "class" -> "1st", "age" -> 29, "gender" -> > "female", "survived" -> True|> Or, here' s a function that exploits the flattening to add elements to an **Association**. addstuff = Association[#, "dances" -> False, "sings" -> True] & > Association[#1, "dances" -> False, "sings" -> True] & addstuff[assoc1] > <|"class" -> "1st", "age" -> 29, "gender" -> "female", "survived" -> True, "dances" -> False, "sings" -> True|> ###Extracting Values from Associations### Just as the values contained in a **List** can be accessed by using the **Part** function, the values contained in an **Association** can likewise be accessed. Suppose, for example that I wanted to compute double the age of the person in **assoc1**. It turns out there are a lot of ways of doing this. The first is to treat the Association as a list except that the indices, instead of being integers, are the "keys" that are on the left hand side of the rules. 2*Part[assoc1, "age"] > 58 2*assoc1[["age"]] > 58 A second way is to use Query. We can wrap the "key" in the head **Key** just to make sure Mathematica understands that the thing is a Key. 2*Query[Key["age"]][assoc1] > 58 Usually we can omit the Key and everything works fine. 2*Query["age"][assoc1] > 58 A third way is to write a function that has an association as its argument. af = Function[Slot["age"]] > "#age &" Now look what we can do. 2*Query[af][assoc1] > 58 We can shorten this approach by using a simpler syntax for a function. 2*Query[#age &][assoc1] > 58 Note, though that this still will not work. Basically, Mathematica is confused. It thinks the function itself is the key. 2*assoc1[af] > 2 Missing["KeyAbsent", #age &] But here' s a simple workaround. For very simple functions, I can just use the name of the key. 2*assoc1["age"] > 58 ##A Note on Slot Arguments## And please pay attention to this : sometimes the Mathematica parser gets confused when it confronts a "slot argument" written as #something. If you see this happening, write it as Slot["something"]. Slot["iamaslot"] === #iamaslot > True Here' s another problem. What if the key in the association has spaces or non-standard characters in it. Any of these, for example, are perfectly fine keys: the string "I have a lot of spaces in me", the string "I_have_underscores", the symbol True, the integer 43. But if we try to denote those keys by putting a hash in front of them, it will lead to confusion and problems. problemAssociation = Association["I have a lot of spaces in me" -> 1, "I_have_underscores" -> 2, True -> 3, 43 -> 4] > <|"I have a lot of spaces in me" -> 1, "I_have_underscores" -> 2, True -> 3, 43 -> 4|> {Query[#I have a lot of spaces in me &][problemAssociation], Query[#I _have _underescores &][problemAssociation]} ![enter image description here][13] Here' s a solution. {Query[Slot["I have a lot of spaces in me"] &][problemAssociation], Query[Slot["I_have_underscores"] &][problemAssociation]} > {1, 2} Here' s how we solve the use of True and an integer as keys. We preface them with **Key**. {Query[#True &][problemAssociation], Query[#43 &][problemAssociation]} ![enter image description here][14] {Query[Key[True]][problemAssociation], Query[Key[43]][problemAssociation]} > {3, 4} ##Working with Associations and Lists of Associations## Here' s something we can do with the data in the form of an Association. I could ask for the gender of the person in the third row as follows. Notice I did not have to remember that "gender" was generally in the third position. t3[[3]][["gender"]] > "female" So, even if I scramble the rows, I can still use the same code. t3Scrambled = Map[convertListToAssociation, t[[All, {4, 1, 3, 2}]]] ![enter image description here][15] t3Scrambled[[3]][["gender"]] > female I could also group the people according to their cabin class. Here I use Query on a list of Associations. Query[GroupBy[#class &]][t3] ![enter image description here][16] Again, the following code, which does not explicitly use **Query**, won' t work. Basically, nothing has told Mathematica to translate t3[stuff___] \[RightArrow]Query[stuff][t3]. If t3 had a head of Dataset, Mathematica would know to make the translation. t3[GroupBy[#class &]] ![enter image description here][17] I can also get certain values for all the Associations in a list of Associations. Query[All, #age &][t3] > {29, 30, 58, 52, 21, 54, 29, 42} I can also map a function onto the result. I don't have to go outside the Query form to do so. Query[f, #age &][t3] > f[{29, 30, 58, 52, 21, 54, 29, 42}] Or, without exiting the Query form, I can map a function onto each element of the result. Query[Map[f], #age &][t3] > {f[29], f[30], f[58], f[52], f[21], f[54], f[29], f[42]} I could also do the same thing as follows. Query[All, #age &, f][t3] > {f[29], f[30], f[58], f[52], f[21], f[54], f[29], f[42]} #Creating a Dataset from a List of Associations# To get full use out of Query and to permit syntactic shorthands, we need for Mathematica to understand that the list of Associations is in fact a Dataset. Here' s all it takes. d3 = Dataset[t3] ![enter image description here][18] We can recover our original list of associations by use of the **Normal** command. t3 === Normal[d3] > True With the data inside a Dataset object we now have pretty formatting. But we have more. We can still do this. We get the same result but in a more attractive form. d3g = Query[GroupBy[#class &]][d3] ![enter image description here][19] But now this shorthand works too. d3g = d3[GroupBy[#class &]] ![enter image description here][20] And compare these two elements of code. When the data is in the form of a dataset, Mathematica understands that the stuff in the brackets is not intended as a key but rather is intended to be transformed into a Query. {Query[#age &][t3[[1]]], d3[[1]][#age &]} > {29, 29} ##A Dataset that is an Association of Associations## Let' s look under the hood of **d3g**. d3gn = Normal[d3g] ![enter image description here][21] Note : if you *really* want to look under the hood of a **Dataset** ask to see the **Dataset** in **FullForm**. You can also get more information by running the undocumented package Dataset`, but this is definitely NOT recommended for the non-advanced user. What we see is an Association in which each of the values is itself a list of Associations. We can map a function over d3gn. Map[f, d3gn] ![enter image description here][22] I can of course do the mapping within the **Query** construct. Query[All, f][d3gn] ![enter image description here][23] If I try synactic sugar, it doesn' t work because d3gn is not a Dataset. d3gn[All, f] > Missing["KeyAbsent", All] But, if I use the Dataset version, it does work. (The first line may be an ellipsis depending on your operating system and display, but if you look under the hood it looks just like the values for 2nd and 3rd. I have no idea why an ellipsis is being inserted. d3g[All, f] ![enter image description here][24] ##A Dataset that just has a single Association inside.## We can also have a Dataset that just has a single Association inside. Mathematica presents the information with the keys and values displayed vertically. Dataset[d3[[1]]] ![enter image description here][25] In theory, we could have a Dataset that just had a single number inside it. Dataset[6] ![enter image description here][26] #Nice queries with Dataset# Now I can construct a query that takes a dataset and groups it by the gender column. It then takes each grouping and applies the Mean function to at least part of it. What part? The "age" column part. Notice that I no longer have to remember that gender is the third column and age is the second column. qd = Query[GroupBy[#gender &], Mean, #age &] > Query[GroupBy[#gender &], Mean, #age &] Now I can run this query on t3. qd[d3] ![enter image description here][27] We can now learn a lot about Query. So long as our data is in the form of a Dataset we can write the query as either a formal Query or use syntactic sugar. #Query# A major part of working with data is to understand **Query**. Let's start with a completely abstract **Query**, that we will call **q1**. q1 = Query[f]; Now let' s run q1 on t3. q1[t3] ![enter image description here][28] We end up with a list of Associations that has f wrapped around it at the highest level. It's the same as if I wrote the following code. f[t3] === q1[t3] > True Now, let' s write a **Query** that applies the function g at the top level of the list of associations and the function **f** at the second level, i.e. to each of the rows. Why does it work at the second level? Because it's the second argument to **Query**. q2 = Query[g, f]; q2[t3] ![enter image description here][29] The result is the same as if I mapped **f** onto t3 at its first level and then wrapped **g** around it. g[Map[f, t3, {1}]] === q2[d3] Query[All, MapAt[StringTake[#, 1] &, #, {{"class"}, {"gender"}}] &][d3] Here' s a function **firstchar** that takes the first character in a string. firstchar = StringTake[#, 1] & > StringTake[#1, 1] & Now, let' s construct a query **cg1** that applies **firstchar** to the class and gender keys in each row. cg1 = Query[All, a \[Function] MapAt[firstchar, a, {{"class"}, {"gender"}}]] > Query[All, Function[a, MapAt[firstchar, a, {{"class"}, {"gender"}}]]] We apply **cg1** to our little dataset **d3**. cg1[d3] ![enter image description here][30] What if we want to apply the same function to every element of the Dataset. We just apply it at the lowest level. Here's one way. Query[Map[f, #, {-1}] &][d3] ![enter image description here][31] We can also combine it with column wise and entirety wise operations. For reasons that are not clear, Mathematica can't understand this as a Dataset and returns the Normal form. Query[(Map[f, #, {-1}] &) /* entiretywise, columnwise][d3] ![enter image description here][32] Here' s how we could actually a multilevel **Query**. Suppose we want to write a function that computes the fraction of the people in this little dataset that survived. The first step is simply going to be to extract the survival value and convert it to 1 if True and 0 otherwise. There's a built in function Boole that does this. {Boole[True], Boole[False]} > {1, 0} q3 = Query[something, assoc \[Function] assoc["survived"] /. {True -> 1, _ -> 0}] > Query[something, Function[assoc, assoc["survived"] /. {True -> 1, _ > -> 0}]] q3[t3] > something[{1, 0, 1, 1, 1, 0, 0, 0}] So, now we have something wrapping a list of 1 s and 0 s. By making **something** the **Mean** function, we can achieve our result. q4 = Query[Mean, Boole[#survived] &] > Query[Mean, Boole[#survived] &] q4[d3] > 1/2 We can also examine survival by gender. Notice that **Query** is a little like **Association**: it gets automatically flattened. Query[GroupBy[#gender &], q4][t3] > <|"female" -> 4/5, "male" -> 0|> If the data is held in a **Dataset**, we can also write the final step as follows. d3[GroupBy[#gender &], q4] ![enter image description here][33] Notice that even if we omit the "Query", this code works. Mathematica just figures out that you meant Query. The code immediately above is in the form we typically see and often use. #Some Recipes# titanic = ExampleData[{"Dataset", "Titanic"}] ![enter image description here][34] How to add a value to the Dataset based on values external to the existing columns. Here' s some additional data. Notice that the data is the same length as the titanic dataset. stuffToBeAdded = Table[Association["id" -> i, "weight" -> RandomInteger[{80, 200}]], {i, Length[titanic]}] ![enter image description here][35] We use **Join** at level 2. augmentedTitanic = Join[titanic, stuffToBeAdded, 2] ![enter image description here][36] ##How to add a column to a Dataset based on values in the existing columns and to do so row-wise## Notice that the query below does NOT change the value of the titanic dataset. To change the value of the titanic dataset, one would need to set titanic to the result of the computation. Remember, Mathematica generally does not have side effects or do modifications in place. Query[All, Association[#, "classsex" -> {#class, #sex}] &][titanic] ![enter image description here][37] We can add multiple columns this way. Query[All, Association[#, "classsex" -> {#class, #sex}, "agesqrt" -> Sqrt[#age]] &][titanic] ![enter image description here][38] ##How to change the value of an existing column : row - wise## Age everyone one year. Query[All, Association[#, "age" -> #age + 1] &][titanic] ![enter image description here][39] How to change the value of columns selectively. Query[All, Association[#, "age" -> If[#sex === "male", #age + 1, #age]] &][titanic] ![enter image description here][40] How to create a new column based on some aggregate operator applied to another column. With[{meanAge = Query[Mean, #age &][titanic]}, Query[All, Association[#, "ageDeviation" -> #age - meanAge] &]][titanic] ![enter image description here][41] Can you develop your own recipes? [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=17751.png&userId=20103 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=47222.png&userId=20103 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=60813.png&userId=20103 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=83664.png&userId=20103 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=44435.png&userId=20103 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=105416.png&userId=20103 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=49777.png&userId=20103 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=16898.png&userId=20103 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=28239.png&userId=20103 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=262710.png&userId=20103 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=749611.png&userId=20103 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=209912.png&userId=20103 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=925313.png&userId=20103 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=851614.png&userId=20103 [15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=627315.png&userId=20103 [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=679516.png&userId=20103 [17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=674717.png&userId=20103 [18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=707518.png&userId=20103 [19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=1053319.png&userId=20103 [20]: http://community.wolfram.com//c/portal/getImageAttachment?filename=579820.png&userId=20103 [21]: http://community.wolfram.com//c/portal/getImageAttachment?filename=980221.png&userId=20103 [22]: http://community.wolfram.com//c/portal/getImageAttachment?filename=450322.png&userId=20103 [23]: http://community.wolfram.com//c/portal/getImageAttachment?filename=113723.png&userId=20103 [24]: http://community.wolfram.com//c/portal/getImageAttachment?filename=932624.png&userId=20103 [25]: http://community.wolfram.com//c/portal/getImageAttachment?filename=754825.png&userId=20103 [26]: http://community.wolfram.com//c/portal/getImageAttachment?filename=836826.png&userId=20103 [27]: http://community.wolfram.com//c/portal/getImageAttachment?filename=393227.png&userId=20103 [28]: http://community.wolfram.com//c/portal/getImageAttachment?filename=577928.png&userId=20103 [29]: http://community.wolfram.com//c/portal/getImageAttachment?filename=158329.png&userId=20103 [30]: http://community.wolfram.com//c/portal/getImageAttachment?filename=984930.png&userId=20103 [31]: http://community.wolfram.com//c/portal/getImageAttachment?filename=664831.png&userId=20103 [32]: http://community.wolfram.com//c/portal/getImageAttachment?filename=1090332.png&userId=20103 [33]: http://community.wolfram.com//c/portal/getImageAttachment?filename=450733.png&userId=20103 [34]: http://community.wolfram.com//c/portal/getImageAttachment?filename=740234.png&userId=20103 [35]: http://community.wolfram.com//c/portal/getImageAttachment?filename=143835.png&userId=20103 [36]: http://community.wolfram.com//c/portal/getImageAttachment?filename=36.png&userId=20103 [37]: http://community.wolfram.com//c/portal/getImageAttachment?filename=37.png&userId=20103 [38]: http://community.wolfram.com//c/portal/getImageAttachment?filename=38.png&userId=20103 [39]: http://community.wolfram.com//c/portal/getImageAttachment?filename=39.png&userId=20103 [40]: http://community.wolfram.com//c/portal/getImageAttachment?filename=40.png&userId=20103 [41]: http://community.wolfram.com//c/portal/getImageAttachment?filename=41.png&userId=20103 Seth Chandler 2017-08-20T19:26:30Z https://community.wolfram.com/groups/-/m/t/463721 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=4902Hero.gif&userId=20103 [2]: https://www.wolframcloud.com/obj/5838919a-e64d-49fe-9193-c6ce123184f2 Marco Thiel 2015-03-21T01:18:08Z https://community.wolfram.com/groups/-/m/t/2051264 Probably most of you heard the sad news that there was a giant explosion in the port of Beirut today August 3rd 2020. Several videos were released on which we can do analysis. Note that the method I will use was also famously used by G.I. Taylor to find the energy of the Trinity nuclear bomb test, and he found the right amount to within 10%! We will not be so lucky as the video quality was relatively poor as compared to the high-speed imaging done back then. I extracted several frames from one of the videos: ![enter image description here][1] SetDirectory[NotebookDirectory[]]; v1 = Import["1.mp4"]; fra = VideoExtractFrames[v1, Interval[{11, 12}]] fra = ImageRotate[#, Right] & /@ fra; For each of the frames I identified the explosion by clicking 3 point on the circle: data={ {7,{{157.15625,365.20703125000006`},{233.83984375,379.76562500000006`},{272.015625,312.91015625000006`}}}, {8,{{318.16796874999994`,322.81640625000006`},{228.7890625,462.8515625},{103.61328125,393.38281250000006`}}}, {9,{{341.03515625000006`,311.34765625},{308.27734375,478.125},{93.86328125,420.34375}}}, {10,{{359.08984375,315.546875},{351.48828125,478.63671875000006`},{86.55078125,454.5078125}}}, {11,{{375.62109375,325.64453125},{330.05859375,535.3984375},{62.0390625,434.51171875}}}, {12,{{376.0390625,326.765625},{337.94140625,539.9257812499999},{46.4140625,462.55859375}}} }; The first is the index of the frames, the last elements are points of the circle: circs = CircleThrough /@ data[[;; 6, 2]]; r = circs[[All, 2]]; Here is the visualization: Table[HighlightImage[fra[[data[[i, 1]]]], circs[[i]], "Boundary"], {i, Length[data]}] ![enter image description here][2] Notice that I tracked the orange 'glow', not the shockwave or the smoke that was there partially before the main explosion (so on the conservative side and underestimating the energy release). From Google earth I estimated the size of the face of the building on the left (a grain elevator) and found that every pixel corresponds to 0.59 m roughly (~22 meters corresponding to ~37 pixels). cali = 0.5888486673789164`; realr = r cali The timestamps can be found from the video framerate. Information[Import["video.mp4"]]. And so the timestamps are created and the dataset is created: t = (Range[0, Length[realr] - 1]) 1/29.97; tr = Transpose[{t, realr}] Since the explosion started between two frames we include that in the fit (the t0): fit = FindFit[ tr, { a (x + t0)^0.4, 0 < t0 < 1/30}, {{a, 200}, {t0, 1/60}}, x] realfit = a (x + t0)^0.4 /. fit tzero = t0 /. fit realfitshifted = a (x)^0.4 /. fit prefactor = a /. fit The fit can be found [here][3] and is based on dimensional analysis with the variable E (energy), r (radius of the explosion), t (time), and ρ (density). This also explains the exponent 0.4 used for fitting. We plot the data and the fit: Show[{ListPlot[Transpose[{t + tzero, realr}]], Plot[realfitshifted, {x, 0, 0.2}]}, PlotRange -> {{0, 0.2}, {0, 120}}, Frame -> True, FrameLabel -> {"t", "r [m]"}] ![enter image description here][4] Which is a pretty good fit. We can now calculate the energy back from the explosion: ClearAll[r, e, t, \[Rho]] r == (e t^2/\[Rho])^(1/5) Refine[DivideSides[%, t^(2/5)], t > 0] %[[2]] == Quantity[prefactor, "Meters"/"Seconds"^(2/5)] % /. \[Rho] -> Quantity[1, "Kilograms"/"Meters"^3] energy = e /. Solve[%, e][[1]] Yielding: Quantity[4.2808721214488837`*^11, "Joules"] and we can convert it to kiloton of TNT: UnitConvert[energy, "KilotonsOfTNT"] yielding: Quantity[0.102315, "KilotonsOfTNT"] This number is comparable to the 2015 Tianjin explosion (0.3 kilo tonnes of TNT). [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2020-08-04at21.44.20.png&userId=73716 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2020-08-05at12.00.12.png&userId=73716 [3]: https://en.wikipedia.org/wiki/Nuclear_weapon_yield#Calculating_yields_and_controversy [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2020-08-04at21.53.18.png&userId=73716 Sander Huisman 2020-08-04T19:57:48Z https://community.wolfram.com/groups/-/m/t/1727272 Introduction ---------- A couple of days ago on 1 July [The Economist][1] tweeted [this][2]: > Since he was elected in 2016 Donald Trump has made 1,384 mentions of foreign countries on Twitter. Can you guess which one he named most often? [It claims][3] that in spite of the "special relationship" with the UK, it is only ranked 15th of the countries and territories tweeted about. It also says that Puerto Rico, Mexico and China are in fifth, fourth and third places respectively (countries and territories). According to The Economist North Korea is ranked in second place with 163 mentions. A couple of years ago I read the excellent book "A Mathematician Reads the Newspaper" by John Allen Paulos; and I wonder how much of the daily news coverage can we check using the Wolfram Language. *In a future post I will speak about another project that we are doing with several members of this community that goes in a similar direction. We call it "computational conversations". With a bit of luck you might hear about it at the [Wolfram Technology Conference][4] later this year.* Initial analysis ---------- It turns out that I have been monitoring @realDonaldTrump's tweets using IFTTT since early 2017. I attach excel files to this post. To have a look at the first tweet we first set the directory and load the raw data files: SetDirectory[NotebookDirectory[]] dataraw = Import /@ FileNames["Trump*.xlsx"]; As the first file (without a number) will be read in last (alphabetical order), this is the first tweet data: dataraw[[5, 1, 1]] // TableForm ![enter image description here][5] It is from January 26th, 2017, a couple of days after his inauguration. In oder to figure out which countries Mr Trump talks about we use the function TextCases, a recently updated function: tweettexts = Join[dataraw[[1, 1]], dataraw[[2, 1]], dataraw[[3, 1]], dataraw[[4, 1]], dataraw[[5, 1]]][[All, 2]]; locations = TextCases[StringJoin[tweettexts], "LocationEntity" -> "Interpretation", VerifyInterpretation -> True]; I find Length@locations 5768 locations; these will not only include direct mentions of countries but also locations within countries. These locations will be in Entity-form: locations[[1;;20]] ![enter image description here][6] Let's get that apart. First we make a list of all countries in the world: purecountries = # -> {#} & /@ EntityList[EntityClass["Country", "Countries"]]; If we select all direct mentions of countries we obtain: Select[locations, MemberQ[purecountries[[All, 1]], #] &] // Length 3624 mentions; if we exclude the 1349 mentions the US, we are left with 2275 country names. Despite our list starting with later tweets we obtain substantially more mentions of countries than The Economist (1,384). We can now generate a table of the mentions of all countries: TableForm[Flatten /@ Transpose[{Range[Length[#] - 1], Delete[#, 5]}] &@({#[[1]], #[[2]]} & /@ Normal[ReverseSort[Counts[CommonName@(Select[locations, MemberQ[purecountries[[All, 1]], #] &])]]])] ![enter image description here][7] (This is only the top of the list.) Note, that North Korea is missing, but will be very prominent in the next table.... Next we can check for "indirect" mentions of a country, i.e. Louvre would lead to a mention of France etc. We will find many more entities and will first generate a list of substitution rules: countriesrules = # -> Check[GeoIdentify["Country", #], {#}] & /@ (Complement[DeleteDuplicates[locations], EntityList[EntityClass["Country", "Countries"]]]); We will ignore the error messages for now. We can then generate a table that includes the "indirect" mentions, too: TableForm[Flatten /@ Transpose[{Range[Length[#] - 1], Delete[#, 5]}] &@({#[[1]], #[[2]]} & /@ Normal[ReverseSort[Counts[CommonName@(DeleteMissing[Flatten[locations /. countriesrules]])]]])] ![enter image description here][8] Note, that on rank 4 we find Media, which is not a country. It is easy to clean out, but I leave it in to show the performance of the code so far. We could now make typical representations such as GeoBubbleCharts: GeoBubbleChart[Counts[DeleteMissing[Flatten[locations /. countriesrules]]], GeoBackground -> "Satellite"] ![enter image description here][9] We can now make a BarChart (on a logarithmic scale) selecting "purecountries" like so: BarChart[ReverseSort@<| Select[Normal@ Counts[DeleteMissing[Flatten[locations /. countriesrules]]], MemberQ[purecountries[[All, 1]], #[[1]]] &]|>, ScalingFunctions -> "Log", ChartLabels -> (Rotate[#, Pi/2] & /@ CommonName[ ReverseSortBy[ Select[Normal@ Counts[DeleteMissing[Flatten[locations /. countriesrules]]], MemberQ[purecountries[[All, 1]], #[[1]]] &], Last][[All, 1]]]), PlotTheme -> "Marketing", LabelStyle -> Directive[Bold, 15]] ![enter image description here][10] We can also represent that on a world wide map: styling = {GeoBackground -> GeoStyling["StreetMapNoLabels", GeoStylingImageFunction -> (ImageAdjust@ColorNegate@ColorConvert[#1, "Grayscale"] &)], GeoScaleBar -> Placed[{"Metric", "Imperial"}, {Right, Bottom}], GeoRangePadding -> Full, ImageSize -> Large}; GeoRegionValuePlot[ Log@<|Select[Normal@Counts[DeleteMissing[Flatten[locations /. countriesrules]]], MemberQ[purecountries[[All, 1]], #[[1]]] &]|>, Join[styling, {ColorFunction -> "TemperatureMap"}]] ![enter image description here][11] Further analysis ---------- We can of course look at many other features of the tweets. One is a simple sentiment analysis. I am not at all convinced that the result of this attempt are useful or representing an actual pattern. But this is what we could do: emotion[text_] := "Positive" - "Negative" /. Classify["Sentiment", text, "Probabilities"] and then tweetssentiments = emotion /@ tweettexts; ListPlot[tweetssentiments, PlotRange -> All, LabelStyle -> Directive[Bold, 15], AxesLabel -> {"tweet number", "sentiment"}] ![enter image description here][12] Using a SmoothHistogram, we see a pattern of "extremes", negative, neutral, positive: SmoothHistogram[tweetssentiments, PlotTheme -> "Marketing", FrameLabel -> {"sentiment", "probablitiy"}, LabelStyle -> Directive[Bold, 16], ImageSize -> Large] ![enter image description here][13] We can also ask for less relevant information, such as the colours mentioned in the tweets: textcasesColor = TextCases[StringJoin[tweettexts], "Color" -> "Interpretation", VerifyInterpretation -> True] ![enter image description here][14] So there is a lot of white, some black, red and green: ReverseSort@Counts[textcasesColor] ![enter image description here][15] Let's blend these colours together: Graphics[{Blend[textcasesColor], Disk[]}] ![enter image description here][16] We can also look for "profanity" in tweets: textcasesProfanity = TextCases[StringJoin[tweettexts], "Profanity"]; and represent these tweets in a table: Column[textcasesProfanity, Frame -> All] ![enter image description here][17] It is not quite clear to my why some of the tweets are classified as containing profanity. For some tweets it is relatively obvious, I think. Twitter handles ---------- Another interesting analysis is to look at the twitter handles that @realDonaldTrump uses: textcasesTwitterHandle = TextCases[StringJoin[tweettexts], "TwitterHandle"]; Here are counts of the 50 most common handles: twitterhandles50 = Normal[(ReverseSort@Counts[ToLowerCase /@ textcasesTwitterHandle])[[1 ;; 50]]] ![enter image description here][18] Last but not least we can make a BarChart of that: BarChart[<|twitterhandles50|>, ChartLabels -> (Rotate[#, Pi/2] & /@ twitterhandles50[[All, 1]]), LabelStyle -> Directive[Bold, 14]] ![enter image description here][19] and to compare the same on a logarithmic scale: BarChart[<|twitterhandles50|>, ChartLabels -> (Rotate[#, Pi/2] & /@ twitterhandles50[[All, 1]]), LabelStyle -> Directive[Bold, 14], ScalingFunctions -> "Log"] ![enter image description here][20] A little word cloud ---------- Just to finish off we will generate a little word cloud like so: allwords = Flatten[TextWords /@ tweettexts]; WordCloud[ToLowerCase /@ DeleteCases[DeleteStopwords[ToString /@ allwords], "&"]] ![enter image description here][21] The cloud picks up on "witch hunt" and "collusion", "@foxandfrieds" and "Russia", "fake", "border" as well as other terms that indeed are relatively prominent in the media. Conclusion ---------- The main objective of this was to look try to reproduce at least qualitatively the results of the twitter analysis of @realDonaldTrump's tweets by The Economist using the Wolfram Language. We have been using a slightly different period of the tweets. We have been looking at direct mentions and "indirect" ones. I have not made any manual comparison of the results. I am not sure whether the recognition has worked and I only post it as a first cursory analysis. It was relatively easy to go beyond the analysis and look at other features of the tweets, too. [1]: https://www.economist.com [2]: https://twitter.com/TheEconomist/status/1145467208950329344 [3]: https://www.economist.com/graphic-detail/2019/06/04/the-world-according-to-donald-trump [4]: http://www.wolfram.com/events/technology-conference/2019/ [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1000.12.59.png&userId=48754 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1000.27.41.png&userId=48754 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1000.34.11.png&userId=48754 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1000.41.34.png&userId=48754 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1000.43.14.png&userId=48754 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1000.45.38.png&userId=48754 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1001.24.25.png&userId=48754 [12]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1000.53.33.png&userId=48754 [13]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1000.55.22.png&userId=48754 [14]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1001.05.39.png&userId=48754 [15]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1001.07.12.png&userId=48754 [16]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1001.08.07.png&userId=48754 [17]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1001.09.33.png&userId=48754 [18]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1001.12.15.png&userId=48754 [19]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1001.13.26.png&userId=48754 [20]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1001.14.50.png&userId=48754 [21]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2019-07-1001.33.04.png&userId=48754 Marco Thiel 2019-07-10T00:40:42Z https://community.wolfram.com/groups/-/m/t/942720 As many of you already know, the breaking news today was [Bob Dylan and his Nobel Prize in Literature][1]. You can also take a look at [New York Times announcement][2]. Somehow, maybe because Bob conquered a global ocean of hearts and changed the mind of modern society, this sounds very exciting and calls for a celebration. So what can we do to celebrate a computational way? I challenge you, ladies and gentlemen, to come up with an idea, piece of code, visualization and more to pay our tribute to one of the greatest voices of our generation. Here is an example. Bob Dylan Infinite Universe ------------------ Bob interacted with our culture so much his impact can hardly be measured. But we can take a look at Wikipedia data of his page connectivity to other pages. Even going just 2 levels deep and limiting 100 inter-page links per page we get the whole Universe of Bob Dylan. You can get lost by rolling over the nodes of the network. Infinitely. Well, almost. Watch animation and see if you can recognize some names. **Please also look through the comments way down to observe more of the visible Universe ;-)** linksTO=WikipediaData["Bob Dylan","BacklinksRules","MaxLevelItems"->100,"MaxLevel"->2]; linksFROM=WikipediaData["Bob Dylan","LinksRules","MaxLevelItems"->100,"MaxLevel"->2]; Graph[Union[Join[linksTO, linksFROM]], VertexLabels -> Placed["Name", Tooltip], VertexSize -> Scaled[.01], DirectedEdges -> False, Background -> Black, EdgeStyle -> Directive[White, Opacity[.5]], VertexStyle -> Directive[Red, Opacity[.8]]] ![enter image description here][3] [1]: https://www.nobelprize.org/nobel_prizes/literature/laureates/2016/ [2]: http://www.nytimes.com/2016/10/14/arts/music/bob-dylan-nobel-prize-literature.html [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=456rtrt56564564trhstea5467.gif&userId=11733 Vitaliy Kaurov 2016-10-13T17:11:04Z https://community.wolfram.com/groups/-/m/t/3062832 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=w5qgsdf.jpg&userId=11733 [2]: https://www.wolframcloud.com/obj/04458d24-aacf-4cd7-8a5d-46efde37e927 Marco Thiel 2023-11-09T22:51:38Z https://community.wolfram.com/groups/-/m/t/2380163 I recently noticed that Mathematica uses MxNet as the backend for neural networks. It seems to have been integrated in 2015. The blog post listing the rationale is [here](https://www.oreilly.com/content/apache-mxnet-in-the-wolfram-language/). MxNet seems to have not gained the momentum to become popular, you can see the [trends](https://paperswithcode.com/trends) from "papers with code". Lack of popularity means framework may be slow to develop. For instance, this [question](https://mathematica.stackexchange.com/questions/256323/neural-network-automatic-differentiation-autograd) about using neural networks to fit ODEs from Joshua Schrier. It requires underlying framework to support higher order gradients. There's an [issue](https://github.com/apache/incubator-mxnet/issues/10002) to add support in MxNET but progress has stalled. Meanwhile PyTorch/TensorFlow/JAX support this feature. Yaroslav Bulatov 2021-10-06T00:49:07Z https://community.wolfram.com/groups/-/m/t/289856 For me, it's strange the column redundant implementation of Dataset, that do not allow a memory efficient representation of a simpler tabular data format (as SQL data). As each column name is in each line, is't not memory efficient. See this test: compare[lines_,columns_]:=Module[{l1,l2}, l1=RandomInteger[1000000,{lines,columns}]; l2=AssociationThread["Columns"<>ToString@#&/@Range[columns]-> #]&/@l1; ByteCount[l2]/ByteCount@Dataset[l1]//N ] compare[100000,4] > 19.9959 I get approximately 20x more memory for it!.. I know that Dataset is a cool tool for Hierarchical data structure. But I believe that 95% of data problems are still rectangular (all SQL queries are), and the current format of Dataset is very memory inefficient for that. Maybe Dataset could have a option TabularData-> True, so: ds=Dataset[<|"col1" -> {1, 2, 3, 4}, "col2" -> {"a", "b", "c", "d"}|>, TabularData-> True] Here we can check that this structure do not need more memory: compare[lines_,columns_]:=Module[{l1,l2}, l1=RandomInteger[1000000,{lines,columns}]; l2=Thread["Columns"<>ToString@#&/@Range[columns]-> Transpose@l1]; ByteCount[l2]/ByteCount@Dataset[l1]//N ] compare[100000,4] > 1.0007 There are some Wolfram plans better handler Tabular Data in Dataset? Rodrigo Murta 2014-07-10T03:48:54Z https://community.wolfram.com/groups/-/m/t/1407711 Introduction ---------- On Tuesday, 14 August 2018, a large suspension bridge collapsed in Genoa, Italy, during a violent storm. GeoGraphics[GeoMarker@Entity["City", {"Genoa", "Liguria", "Italy"}]["Coordinates"], GeoRange -> "Country"] ![enter image description here][1] At least 39 people died. The disaster was extensively covered in the international press, see [here][2] and [here][3]. In the latter article by the BBC a graph is shown that indicates that Italy's infrastructure spending in roads has degrease from about 14 billion Euros in 2007 to slightly above 5 billion Euros in 2015. Is there a lack of investment into the road infrastructure? What is the state of our national bridges? Can a computational approach help citizens to better understand what the situation is and help them to make political decisions? In this post I will explore some very simple representations to visualise some datasets; I will use the example of Germany for the most part. Many representations are inspired by an article in "[Die Welt][4]". At the bottom of the article they also offer a link to a [google spreadsheet with data][5] that we will use later. We will start with the data from the "[Bundesstaat fuer Strassenwesen (BASt)][6]", which is available [here][7]. Data import and preparation ---------- The data files we will look at contain the locations of all bridges in Germany with a quality grade and some additional information. We will start by importing the data data = Import["/Users/thiel/Desktop/Zustandsnoten-excel.xlsx"]; The content of that table looks like so: ![enter image description here][8] It has 51592 rows. The important column for us are the last three. The last two contain location data of the bridge (in the UTM32N system). The third column to the end contains a grade indicating the quality of the bridge. We will store the quality rules in the variable states: states = {{1., 1.4, "very good"}, {1.5, 1.9, "good"}, {2.0, 2.4, "safe & sound"}, {2.5, 2.9, "suffient"}, {3.0, 3.4, "poor"}, {3.5,4.0, "very poor"}}; The first two elements of every entry indicate the range of values in which a certain descriptor (third entry) is achieved. The translations to English are a bit subjective. The last two categories (poor and very poor) are considered to be problematic. We start with a graphic of the locations of all bridges in Germany - in fact we have to filter out some for which the location data is not available. styling = {GeoBackground -> GeoStyling["StreetMapNoLabels", GeoStylingImageFunction -> (ImageAdjust@ColorNegate@ColorConvert[#1, "Grayscale"] &)], GeoScaleBar -> Placed[{"Metric", "Imperial"}, {Right, Bottom}], GeoRangePadding -> Full, ImageSize -> Large}; GeoGraphics[{GeoStyling[RGBColor[1, 0.85, 0], Opacity[1]], GeoDisk[#, Quantity[3, "Kilometers"]]} & /@ (GeoPosition[GeoGridPosition[#, GeoProjectionData["UTMZone32"]]] & /@ (Cases[ToExpression /@ data[[1, 2 ;;, -2 ;;]], {_Real, _Real}])), styling] ![enter image description here][9] Note, that we convert the UTM32 data to standard GeoPosition entries within the command. Also, it might be quite unexpected that I use GeoGraphics with GeoDisk to visualise the data - GeoListPlot appears to be the more natural choice. Using the latter, I did have problems to plot all points and to properly size the dots. So I went for GeoGraphics and GeoDisk. Cleaning the data some more ---------- In order to visualise the positions of the bridges that are in a poor or very poor state. For that we define a little helper function: fquality = (Which @@ Flatten[{#[[1]] <= dummy <= #[[2]], #[[3]]} & /@ states] /. dummy -> #) & Not elegant, but it does the job. With that we adjust the dataset cleanDataset = {GeoPosition[GeoGridPosition[#[[2 ;; 3]], GeoProjectionData["UTMZone32"]]], #[[1]], fquality[#[[1]]]} & /@ Cases[ToExpression /@ data[[1, 2 ;; ;; 1, -3 ;;]], {_, _Real, _Real}]; The entries look like this now: cleanDataset[[1]] ![enter image description here][10] We also want to compute in which province (Bundesland) the bridge is located. So first we get the list of provinces: provinces = Entity["Country", "Germany"][EntityProperty["Country", "AdministrativeDivisions"]]; The idea was to use GeoWithinQ or so to compute this. This for example whereQ = Which[GeoWithinQ[Entity["AdministrativeDivision", {"BadenWurttemberg", "Germany"}], #1], Entity["AdministrativeDivision", {"BadenWurttemberg", "Germany"}], GeoWithinQ[Entity["AdministrativeDivision", {"Bavaria", "Germany"}], #1], Entity["AdministrativeDivision", {"Bavaria", "Germany"}], GeoWithinQ[Entity["AdministrativeDivision", {"Berlin", "Germany"}], #1], Entity["AdministrativeDivision", {"Berlin", "Germany"}], GeoWithinQ[Entity["AdministrativeDivision", {"Brandenburg", "Germany"}], #1],Entity["AdministrativeDivision", {"Brandenburg", "Germany"}], GeoWithinQ[Entity["AdministrativeDivision", {"Bremen", "Germany"}], #1], Entity["AdministrativeDivision", {"Bremen", "Germany"}], GeoWithinQ[Entity["AdministrativeDivision", {"Hamburg", "Germany"}], #1], Entity["AdministrativeDivision", {"Hamburg", "Germany"}], GeoWithinQ[Entity["AdministrativeDivision", {"Hesse", "Germany"}], #1], Entity["AdministrativeDivision", {"Hesse", "Germany"}], GeoWithinQ[Entity["AdministrativeDivision", {"LowerSaxony", "Germany"}], #1], Entity["AdministrativeDivision", {"LowerSaxony", "Germany"}], GeoWithinQ[Entity["AdministrativeDivision", {"MecklenburgWesternPomerania", "Germany"}], #1], Entity["AdministrativeDivision", {"MecklenburgWesternPomerania", "Germany"}], GeoWithinQ[ Entity["AdministrativeDivision", {"NorthRhineWestphalia", "Germany"}], #1], Entity["AdministrativeDivision", {"NorthRhineWestphalia", "Germany"}], GeoWithinQ[Entity["AdministrativeDivision", {"RheinlandPalatinate", "Germany"}], #1], Entity["AdministrativeDivision", {"RheinlandPalatinate", "Germany"}], GeoWithinQ[Entity["AdministrativeDivision", {"Saarland", "Germany"}], #1], Entity["AdministrativeDivision", {"Saarland", "Germany"}], GeoWithinQ[Entity["AdministrativeDivision", {"Saxony", "Germany"}], #1], Entity["AdministrativeDivision", {"Saxony", "Germany"}], GeoWithinQ[ Entity["AdministrativeDivision", {"SaxonyAnhalt", "Germany"}], #1], Entity["AdministrativeDivision", {"SaxonyAnhalt", "Germany"}], GeoWithinQ[Entity["AdministrativeDivision", {"SchleswigHolstein", "Germany"}], #1], Entity["AdministrativeDivision", {"SchleswigHolstein", "Germany"}],GeoWithinQ[ Entity["AdministrativeDivision", {"Thuringia", "Germany"}], #1], Entity["AdministrativeDivision", {"Thuringia", "Germany"}]] & does the trick: whereQ[cleanDataset[[3, 1]]] (*Entity["AdministrativeDivision", {"SchleswigHolstein", "Germany"}]*) but is far too slow. So we will use the polygons of the provinces: polygons = (#["Polygon"] /. GeoPosition -> Identity) & /@ provinces; It turns out that this is not sufficient. Some provinces "have a hole", i.e. a city inside that is not part of the province but rather an administrative unit in its own right. This leads to two provinces having the head FilledCurve rather than Polygon. We can fix this like so: polygonsfixed = If[Head[#] === FilledCurve, Polygon[#[[1, 1, 1, 1]] /. Line -> Polygon], #] & /@ polygons; This basically ignores the hole. What we will do later is make sure that we first check whether a bridge is in the city (the hole inside) - if not we use the larger province surrounding it. The next command takes a while to run: Monitor[bridgeprovince = (provinces[[#]] & /@ Flatten /@ Table[Position[RegionMember[#, cleanDataset[[k, 1, 1]]] & /@ polygonsfixed, True], {k, 1, Length[cleanDataset]}]) /. {} -> {Missing}, k] It turns out that some bridges are not properly localised. The following code fixes that (it relies on the fact that the list is ordered by province anyway!). bridgeprovinceclean = bridgeprovince[[All, 1]] //. {a___, x_, Missing, b___} -> {a, x, x, b}; Let's build one list of lists containing all that data: totalData = Flatten /@ Transpose[{cleanDataset, bridgeprovinceclean /. Missing -> Entity["AdministrativeDivision", {"SchleswigHolstein", "Germany"}]}]; The data looks like this now: totalData[[1 ;; 5]] // TableForm ![enter image description here][11] Some statistics and visualisations of the data ---------- We can now count how many bridges in Germany are in a certain state: tallyData = Tally[totalData[[All, -2]]][[Ordering[Flatten[Position[states, #] & /@ Tally[totalData[[All, -2]]][[All, 1]], 1][[All, 1]]]]] which gives ![enter image description here][12] Let's make a bar chart of that: BarChart[tallyData[[All, 2]], ChartLabels -> Evaluate[Rotate[#, Pi/2] & /@ tallyData[[All, 1]]], LabelStyle -> Directive[Bold, 15], PlotTheme -> "Marketing"] ![enter image description here][13] Most bridges are in the "safe & sound" state; a non-negligible number is in the states "poor" or "very poor". Let's first select the bridges in the last two categories: badBridges = Select[totalData, (#[[3]] == "poor" || #[[3]] == "very poor") &]; There are 2413 bridges in that category now, and there are their locations: GeoGraphics[{GeoStyling[RGBColor[1, 0.0, 0], Opacity[1]], GeoDisk[#, Quantity[3, "Kilometers"]]} & /@ badBridges[[All, 1]], styling] ![enter image description here][14] So, if you live in Germany, there is probably one of those bad bridges close to you. A GeoSmoothHistogram helps us to visualise the distribution of those bridges: GeoSmoothHistogram[badBridges[[All, 1]], GeoRange -> Entity["Country", "Germany"], ColorFunction -> "TemperatureMap", styling] ![enter image description here][15] It is obvious that there are some areas where the problems are more acute than in other regions. We can represent good vs not-so-good bridges in Germany like so: nonbadBridges = Select[totalData, ! (#[[3]] == "poor" || #[[3]] == "very poor") &]; GeoGraphics[Join[{GeoStyling[RGBColor[0, 1., 0], Opacity[1]], GeoDisk[#, Quantity[3, "Kilometers"]]} & /@ nonbadBridges[[All, 1]], {GeoStyling[RGBColor[1, 0.0, 0], Opacity[1]], GeoDisk[#, Quantity[3, "Kilometers"]]} & /@ badBridges[[All, 1]]], styling] ![enter image description here][16] Satellite images ---------- The Wolfram Language allows us to take a deeper look at the individual bridges by using satellite data. Let's first get some images of bridges in a poor state: GeoImage[#, GeoRange -> Quantity[50, "Meters"]] & /@ RandomSample[badBridges][[1 ;; 25, 1]] ![enter image description here][17] These images have an excellent resolution and allow us to study the bridges in greater detail: ![enter image description here][18] Likewise we can look at the good bridges: goodBridges = Select[totalData, ! (#[[3]] == "good" || #[[3]] == "very good") &]; GeoImage[#, GeoRange -> Quantity[50, "Meters"]] & /@ RandomSample[goodBridges][[1 ;; 25, 1]] ![enter image description here][19] Again the resolution is very good for individual images: ![enter image description here][20] I was first thinking about developing a machine learning approach to try and estimate the state of a bridge from images, but that seems impossible at this level. How good are different administrative divisions addressing infrastructure ---------- Next, we look for regional variations. We gather the data by province: byprovince = GatherBy[totalData, Last]; and plot the ratio of poor bridges vs all bridges GeoRegionValuePlot[Rule @@@ SortBy[Table[{byprovince[[k, 1, -1]], N@Length[Select[byprovince[[k, All, 3]], (# == "poor" || # == "very poor") &]]/ Length[byprovince[[k]]]}, {k, 1, Length[byprovince]}], Last], GeoRange -> Entity["Country", "Germany"], styling, ColorFunction -> "TemperatureMap"] ![enter image description here][21] Relationship between age and state of the bridge ---------- "Die Welt" offers consolidated data in an [additional link][22] (which also contains the name of the province, so we could have saved the effort above). We import the data weltData = Import["/Users/thiel/Desktop/DIE WELT_ Zustand der Fernstraßenbrücken - Daten.tsv"]; It also contains the grades of the bridges for several consecutive test periods and for some bridges (in two of the provinces) the year it was built. Let's clean that weltDatawDates = Select[weltData[[2 ;;]], #[[-1]] =!= "" &] and collect the built year vs rating: builtdatevsmark = {#[[1]], ToExpression[StringReplace[ToString[#[[2]]], "," -> "."]]} & /@ weltDatawDates[[All, {-1, 4}]] A simple ListPlot of that looks rather boring: ListPlot[builtdatevsmark] ![enter image description here][23] and is difficult to interpret. But with a little bit of work we can make it much more useful BubbleChart[Flatten /@ Tally[builtdatevsmark], PlotRange -> {{1900, 2020}, All}, ColorFunction -> Function[{x, y, r}, Lighter[Green, r/41.]], AspectRatio -> 0.3, BubbleSizes -> {0.01, 0.05}, Background -> Black,LabelStyle -> Directive[Bold, 15, White], Epilog -> {Red, Line[{{1890, 2.95}, {2020, 2.95}}], Blue, Line[{{1960, 0.8}, {1960, 3.8}}], Line[{{1980, 0.8}, {1980, 3.8}}], Green, Line[{{1890, 1.95}, {2020, 1.95}}]}] ![enter image description here][24] The x-axis shows the year the bridge was built. The y-axis the quality of the bridge - small numbers are better marks. The size of the bubbles roughly indicates the number of bridges at that coordinate. The red line indicates the quality threshold. Everything above is rather problematic. The green line separates the bridges in excellent states (below it) from the ok-ish bridges (above). The blue lines indicate a period from 1960-1980, which has a high proportion of bridges with issues. Bridges build after 1980 appear to be in a much better state in general. Bridges built after 2000 are usually in quite good states. This broadly suggests that bridges that are up to 20 years old are usually quite good. Bridges that are between 20-40 years old get slowly worse, and from 40-60 years old they start needing more maintenance. Another historically interesting fact is that this suggests that there were many bridges built during the Third Reich, from say 1935 to about 1941 or so. In the 10 years after the war notably fewer bridges seem to have been built. To help illustrate that point one could also plot the regression line (orange below). We will use a linear and a quadratic model: nlm = LinearModelFit[builtdatevsmark, {1, x}, x] nlm2 = LinearModelFit[builtdatevsmark, {1, x, x^2}, x] and then plot it: Show[BubbleChart[Flatten /@ Tally[builtdatevsmark], PlotRange -> {{1900, 2020}, All}, ColorFunction -> Function[{x, y, r}, Lighter[Green, r/41.]], AspectRatio -> 0.3, BubbleSizes -> {0.01, 0.05}, Background -> Black, LabelStyle -> Directive[Bold, 15, White], Epilog -> {Red, Line[{{1890, 2.95}, {2020, 2.95}}], Blue, Line[{{1960, 0.8}, {1960, 3.8}}], Line[{{1980, 0.8}, {1980, 3.8}}], Green, Line[{{1890, 1.95}, {2020, 1.95}}]}], Plot[nlm[x], {x, 1900, 2020}, PlotStyle -> Orange]] ![enter image description here][25] or for the quadratic model (same code as above but nlm2 instead of nlm): ![enter image description here][26] Fitting a 5th order polynomial gives this: ![enter image description here][27] We can also plot bands for the single prediction error. nlm2["SinglePredictionBands"] For nlm2 that looks like so: Show[BubbleChart[Flatten /@ Tally[builtdatevsmark], PlotRange -> {{1900, 2020}, All}, ColorFunction -> Function[{x, y, r}, Lighter[Green, r/41.]], AspectRatio -> 0.3, BubbleSizes -> {0.01, 0.05}, Background -> Black, LabelStyle -> Directive[Bold, 15, White], Epilog -> {Red, Line[{{1890, 2.95}, {2020, 2.95}}], Blue, Line[{{1960, 0.8}, {1960, 3.8}}], Line[{{1980, 0.8}, {1980, 3.8}}], Green, Line[{{1890, 1.95}, {2020, 1.95}}]}], Plot[{-129.53690036856392` + 0.14262757468838871` x - 0.00003842565681513884` x^2 - 1.9603506747760122` \[Sqrt](99.64290933950231` - 0.20387038807958635` x + 0.00015683763427764343` x^2 - 5.3645803097817624`*^-8 x^3 + 6.883592653276127`*^-12 x^4), -129.53690036856392` + 0.14262757468838871` x - 0.00003842565681513884` x^2 + 1.9603506747760122` \[Sqrt](99.64290933950231` - 0.20387038807958635` x + 0.00015683763427764343` x^2 - 5.3645803097817624`*^-8 x^3 + 6.883592653276127`*^-12 x^4), nlm2[x]}, {x, 1900, 2020}, PlotStyle -> {Yellow, Yellow, Orange}, Filling -> {{1 -> {3}, 2 -> {3}}}]] ![enter image description here][28] Infrastructure spending ---------- The OECD offers great data on the infrastructure spending of different countries [here][29]. We import that data: spending = Import["/Users/thiel/Desktop/DP_LIVE_15082018232318196.csv"]; and select the infrastructure spending for roads for different countries over the years: germanySpending = Select[spending, #[[3]] == "ROAD" && #[[1]] == "DEU" &]; spainSpending = Select[spending, #[[3]] == "ROAD" && #[[1]] == "ESP" &]; italySpending = Select[spending, #[[3]] == "ROAD" && #[[1]] == "ITA" &]; franceSpending = Select[spending, #[[3]] == "ROAD" && #[[1]] == "FRA" &]; usaSpending = Select[spending, #[[3]] == "ROAD" && #[[1]] == "USA" &]; greeceSpending = Select[spending, #[[3]] == "ROAD" && #[[1]] == "GRC" &]; ukSpending = Select[spending, #[[3]] == "ROAD" && #[[1]] == "GBR" &]; russiaSpending = Select[spending, #[[3]] == "ROAD" && #[[1]] == "RUS" &]; luxemburgSpending = Select[spending, #[[3]] == "ROAD" && #[[1]] == "LUX" &]; russiaSpending = Select[spending, #[[3]] == "ROAD" && #[[1]] == "RUS" &]; If we plot this, we reproduce the plot of CNN, and get a better range than the BBC: DateListPlot[{{DateObject[ToString[#[[1]]]], #[[2]]} & /@ germanySpending[[All, -3 ;; -2]], {DateObject[ ToString[#[[1]]]], #[[2]]} & /@ spainSpending[[All, -3 ;; -2]], {DateObject[ ToString[#[[1]]]], #[[2]]} & /@ italySpending[[All, -3 ;; -2]], {DateObject[ToString[#[[1]]]], #[[2]]} & /@ ukSpending[[All, -3 ;; -2]], {DateObject[ ToString[#[[1]]]], #[[2]]} & /@ russiaSpending[[All, -3 ;; -2]]}, PlotTheme -> "Marketing", LabelStyle -> Directive[Bold, 15], PlotLegends -> {"Germany", "Spain", "Italy", "United Kingdom", "Russia"}, ImageSize -> Large, FrameLabel -> {"year", "spending"}] ![enter image description here][30] The BBC article only shows the range from 2007 to 2016, a phase in which the infrastructure spending in Italy (and Spain) has fallen substantially. This could be used to make the case that a lack of infrastructure spending by Italy might have contributed to the disaster. The full period from 1995, however, shows that Italy went through a phase of higher investments from 2004-2008 and then returned to values slightly lower than before 2004. Note also that this is the total spending and not normalised by population size, GDP or size of the road network. If for example we also plot the graph for the United States the diagram looks like this: DateListPlot[{{DateObject[ToString[#[[1]]]], #[[2]]} & /@ germanySpending[[All, -3 ;; -2]], {DateObject[ ToString[#[[1]]]], #[[2]]} & /@ spainSpending[[All, -3 ;; -2]], {DateObject[ ToString[#[[1]]]], #[[2]]} & /@ italySpending[[All, -3 ;; -2]], {DateObject[ToString[#[[1]]]], #[[2]]} & /@ ukSpending[[All, -3 ;; -2]], {DateObject[ ToString[#[[1]]]], #[[2]]} & /@ russiaSpending[[All, -3 ;; -2]], {DateObject[ ToString[#[[1]]]], #[[2]]} & /@ usaSpending[[All, -3 ;; -2]]}, PlotTheme -> "Marketing", LabelStyle -> Directive[Bold, 15], PlotLegends -> {"Germany", "Spain", "Italy", "United Kingdom", "Russia", "USA"}, ImageSize -> Large, FrameLabel -> {"year", "spending"}, PlotRange -> All] ![enter image description here][31] where the United States dwarf the spending by the (much smaller) European countries. There does, however, appear to be a trend of increasing infrastructure spending in the United States. (Preliminary) Conclusion ---------- We have developed a couple of graphs visualising some data about the state of bridges and produced some graphs that would appear in news coverage of the Genoa disaster. We have pulled in satellite date to inspect the bridges and seen how easy it is in the Wolfram Language to asks your own questions and get meaningful representations. This is, of course, only a shot computational essay, and by no means a comprehensive analysis. In fact, many questions can be asked next. Do different types of bridges age better (or worse)? Could additional data such as accelerometer data from mobile phones in cars that drive over bridges help to "measure" the state of the bridge? Could we use dash cam footage? There is data on cars using individual roads; could we build a model to suggest which bridges to fix first in order to minimise risk to people? It is should be easy to use TravelDirectionsData to get a warning about problematic bridges on the way. If citizens perform this or similar types of analysis it might help to prompt politicians or the administration to act when there is a hazard. The Wolfram Language is a great tool to enable us to do so. PS: By the way, if you want to do the same analysis for the United States, [this dataset][32] might be useful. [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1523.02.53.png&userId=48754 [2]: https://edition.cnn.com/2018/08/15/europe/italy-genoa-morandi-bridge-collapse-intl/index.html [3]: https://www.bbc.co.uk/news/world-europe-45193614 [4]: http://www.welt.de/politik/interaktiv/bruecken/deutschlands-bruecken-wettlauf-gegen-den-verfall.html [5]: https://docs.google.com/spreadsheets/d/1h_%5C%20NKP3lvTnoQBFjCRa9gJLLOi2rxtIMeNucvAUsPwtI/edit?pref=2&pli=1#gid=0 [6]: https://www.bast.de/BASt_2017/DE/Statistik/statistik-node.html [7]: https://www.bast.de/BASt_2017/DE/Statistik/Bruecken/Zustandsnoten-excel.html?nn=1819430 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1523.12.22.png&userId=48754 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1523.25.29.png&userId=48754 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1523.30.53.png&userId=48754 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1523.46.34.png&userId=48754 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1523.49.10.png&userId=48754 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1523.49.53.png&userId=48754 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1523.53.04.png&userId=48754 [15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1523.54.32.png&userId=48754 [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1523.56.13.png&userId=48754 [17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1523.59.58.png&userId=48754 [18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1600.01.18.png&userId=48754 [19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1600.02.39.png&userId=48754 [20]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1600.03.25.png&userId=48754 [21]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1600.07.28.png&userId=48754 [22]: https://docs.google.com/spreadsheets/d/1h_%5C%20NKP3lvTnoQBFjCRa9gJLLOi2rxtIMeNucvAUsPwtI/edit?pref=2&pli=1#gid=0 [23]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1600.12.48.png&userId=48754 [24]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1600.14.35.png&userId=48754 [25]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1601.20.48.png&userId=48754 [26]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1601.23.19.png&userId=48754 [27]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1601.32.50.png&userId=48754 [28]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1601.48.06.png&userId=48754 [29]: https://data.oecd.org/transport/infrastructure-investment.htm [30]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1600.26.42.png&userId=48754 [31]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-08-1600.31.31.png&userId=48754 [32]: https://www.fhwa.dot.gov/bridge/nbi/ascii2017.cfm Marco Thiel 2018-08-15T23:46:05Z https://community.wolfram.com/groups/-/m/t/567757 Here I will show a way to do some analysis of GPS data using new functionality present in the Wolfram Language. I just moved to Lyon (France), where they organised a mountainbike tour last week ([Lyon Free VTT][1]). I opted for the 'expert' trail of roughly 60 km with 1000+ meters ascend. I recorded my trip using the Garmin GPSmap 62s and I'm wearing a heart rate sensor from Garmin as well. We start by setting the current directory to the notebookdirectoy, and look for the GPX file of the day of the event: SetDirectory[NotebookDirectory[]]; fn = FileNames["*2015-09-13*.gpx"][[1]] "Track_2015-09-13 121636.gpx" So we found the file. The GPX file is an XML formatted file. So we can read the data easily: data = Import[fn, "XML"]; The data is stored in "track points" which are abbreviated as `trkpt`. We can find the parts of the track by using `Cases`. FirstCase[data, XMLElement["trkpt", ___], Missing[], \[Infinity]] which looks like: XMLElement[trkpt,{lat->45.7242092583,lon->4.8268832266}, {XMLElement[ele,{},{173.17}], XMLElement[time,{},{2015-09-13T06:09:03Z}], XMLElement[extensions,{},{ XMLElement[{http://www.garmin.com/xmlschemas/TrackPointExtension/v1,TrackPointExtension},{},{ XMLElement[{http://www.garmin.com/xmlschemas/TrackPointExtension/v1,hr},{},{108}]} ]} ]} ] Wow! Looks quite complicated, but the important thing is that it always starts with the Latitude and Longitude coordinates as `attributes -> values`, and then followed by some more data (in this case the time, elevation, and heart rate). So lets get all the track points: data = Cases[data, XMLElement["trkpt", {"lat" -> lat_, "lon" -> lon_}, other_] :> {ToExpression /@ {lat, lon}, other}, \[Infinity]] Note that I immediately detect the `latitude`, `longitude` and the `other` data and put them as a replacement rule right in `Cases`. I started recording when I left my house to go to the Start, and I also continued recording after I cycled back from the finish to my house. So we remove this data: data = Drop[Drop[data, 246], -128]; The trail is now stored as the first element of each element of data: trail = GeoPosition /@ data[[All, 1]]; We can now easily visualise this using the new GeoGraphics function (V10 and up): GeoGraphics[{Thick, Line[trail]}, ImageSize -> 400] Giving: ![enter image description here][2] That looks pretty good! Now let's retrieve the time, elevation and heart rate from the data. We overwrite data, because we already extracted the latitude and longitude from it: data = data[[All,2]]; First, let's try the time, we use the new `FirstCase` command (V10) as we only expect one time per track point entry. We make a pure function and map that over all the data: absolutetime = time = Map[ FirstCase[#,XMLElement["time",{},{time_}]:>AbsoluteTime[time]+$TimeZone*3600,Missing[],\[Infinity]]& , data ] time -= First[time] Note that I also add some seconds so that I have the time in my own timezone (the default is in Zulu (universal) time). In case the time is somehow not recorded it will returning `Missing[]`. I will also save a variable which is the time starting when I start called `time` (second line). We can do something similar for the elevation: elevation=Map[FirstCase[#,XMLElement["ele",{},{ele_}]:>ToExpression[ele],Missing[],\[Infinity]]&,data] and for the heart rate: hr=Map[FirstCase[#,XMLElement[{"http://www.garmin.com/xmlschemas/TrackPointExtension/v1","hr"},{},{hr_}]:>ToExpression[hr],Missing[],\[Infinity]]&,data]; Now that we have all the data, let's explore a little bit further. Let's first compute the distance travelled: distances=BlockMap[GeoDistance@@#&,trail,2,1]; distances=Prepend[QuantityMagnitude[distances,"Meters"],0.0]; distance=Accumulate[distances]; Here I used the new BlockMap function (V10.2) to compute the distance between each pair of points. (2 to denote pairs, and 1 to have overlapping pairs). I convert the results to just number (without units), otherwise the `Accumulate` is very slow (why is this slow Wolfram?). Lastly I keep track of the accumulated distance using the command `Accumulate`. Now we can plot this as a function of time: DateListPlot[{absolutetime,distance}\[Transpose],FrameLabel->"Meters",FrameStyle->12,ImageSize->400] giving: ![enter image description here][3] The total distance travelled is: Last[distance] 57106. This answer is in meters because I took the magnitude in "Meters". We can convert it to miles quite easily: UnitConvert[Quantity[%, "Meters"], "Miles"] 35.484mi About 35.5 miles. Let's calculate the speed: speed=Differences[distance]/Differences[time]; speed=Join[{speed[[1]]},MovingAverage[speed,2],{speed[[-1]]}]; The first order difference will give me the speed, I'm doing a moving average so that I have the average speed at an instant, I append and prepend the first and last speed, respectively, such as to have the length of `speed` and the length of `time` to be the same. Now we can plot the speed (which is in meter/second, so I multiply by 3.6 to get km/h): DateListPlot[Transpose[{absolutetime,3.6speed}],FrameLabel->"Kilometer/Hour",FrameStyle->12,ImageSize->1000,AspectRatio->1/5,PlotRange->All] ![enter image description here][4] My speed varies a lot as you can see. This is partly because this is an urban trail with climbs and descends, but also sharp corners, and up and down-going stairs. But also because of some GPS errors you have some fluctuations. This is apparent in the above plot, I'm quite fast, but not 80 km/h (50 miles/hour) fast!! So let's smooth the data over a few points: slen=3; smoothspeed=Join[Accumulate[speed[[;;slen]]]/Range[slen],MovingAverage[speed,2slen+1],Reverse[Accumulate[Reverse[speed[[-slen;;]]]]/Range[slen]]]; Which looks more realistic: DateListPlot[Transpose[{absolutetime,3.6smoothspeed}],FrameLabel->"Kilometer/Hour",FrameStyle->12,ImageSize->1000,AspectRatio->1/5,PlotRange->All] ![enter image description here][5] You can clearly see that I had a flat tire around 10:15 that took me 7-8 minutes to repair... Let's calculate the average speed (up to some time t): avgspeeds=Prepend[Rest[distance]/Rest[time],0]; And showing: DateListPlot[Transpose[{absolutetime,3.6avgspeeds}],FrameLabel->"Kilometer/Hour",FrameStyle->12,ImageSize->1000,AspectRatio->1/5,PlotRange->All] ![enter image description here][6] So I ended with a speed of `3.6 Last[avgspeeds] = 14.5 km/h` which is not too bad considering it includes also a flat tire. Plotting the elevation is also quite easy now: DateListPlot[Transpose[{absolutetime, elevation}], FrameLabel -> "Meters", FrameStyle -> 12, ImageSize -> 1000, AspectRatio -> 1/5, PlotRange -> All] ![enter image description here][7] There were some very steep hills in this track, and pieces where I had to change to my lowest gear to climb them. Also plotting the heart rate is quite easy: DateListPlot[Transpose[{absolutetime,hr}],FrameLabel->"BPM",FrameStyle->12,ImageSize->1000,AspectRatio->1/5,PlotRange->All] ![enter image description here][8] Let's compute the total number of heart beats and the average heart rate: hb=Interpolation[DeleteMissing[{time,hr/60.0}\[Transpose],1,2],InterpolationOrder->1]; hb=Round[Integrate[hb[t],{t,Min[time],Max[time]}]] avghr=60hb/(Max[time]-Min[time]) 38443 (* beats *) 162.8 (* average beats / minute *) That is quite high for my age (28 years)! Let's compute the elevation gain: elevationgain = Total[Select[Differences[elevation], Positive]] 1193.05 (* meters *) That explains partly the low speeds and the high heart rates! I've climbed almost 1200 meters. QuantityMagnitude[Quantity[elevationgain,"Meters"],"Stories"] or about 400-500 stories!! (depending on your metric of a storey) Let's make a visual plot where the climbs are by creating a map colored by the height: colordata = Rescale[elevation]; color = Blend[{{0, Darker@Green}, {0.5, Yellow}, {1, Red}}, #] & /@ colordata; GeoGraphics[{AbsoluteThickness[8], JoinForm["Round"], Line[trail, VertexColors -> color]}, ImageSize -> 400] ![enter image description here][9] Green means low, and red means high altitude. Using the same colors we can create also the accompanying plot: DateListPlot[Transpose[{absolutetime,elevation}],FrameLabel->"Meters",FrameStyle->12,ImageSize->1000,AspectRatio->1/5,PlotRange->All,ColorFunction->(Blend[{{0,Darker@Green},{0.5,Yellow},{1,Red}},#2]&),PlotStyle->Directive[AbsoluteThickness[5],JoinForm["Round"]]] ![enter image description here][10] Or the same but instead colored by the speed: colordata=Rescale[smoothspeed]; cf=Blend[{{0.2,Red},{0.45,Yellow},{0.8,Darker@Green}},#]&; color=cf/@colordata; g1=GeoGraphics[{AbsoluteThickness[8],JoinForm["Round"],Line[trail,VertexColors->color]},ImageSize->400] DateListPlot[Transpose[{absolutetime,3.6smoothspeed}],FrameLabel->"km/h",FrameStyle->12,ImageSize->1000,AspectRatio->1/5,PlotRange->All,ColorFunction->(cf[#2]&),PlotStyle->Directive[AbsoluteThickness[5],JoinForm["Round"]]] ![enter image description here][11] ![enter image description here][12] In the corners and in the climbs it is slow (red), while for the flat, down, and straight pieces it is fast (green). It looks like there is way more green, then red on the map, while for the bottom plot it is roughly equal, this is because for a low speed you travel only a little so just a bit of red on the map. While for high speed you travel more distance so it looks more green... This is part I, below I will post a second part of the analysis. With some nice general functionality. PS. Also have a look at the 2009 post at the wolfram blog when GeoGraphics (and other functions) were not introduced yet. [Wolfram Blog about GPS][13] [1]: http://www.lyonfreevtt.com [2]: /c/portal/getImageAttachment?filename=1_map.png&userId=73716 [3]: /c/portal/getImageAttachment?filename=2_distance.png&userId=73716 [4]: /c/portal/getImageAttachment?filename=3_speed.png&userId=73716 [5]: /c/portal/getImageAttachment?filename=3.5_smoothspeed.png&userId=73716 [6]: /c/portal/getImageAttachment?filename=4_avgspeed.png&userId=73716 [7]: /c/portal/getImageAttachment?filename=5_elevation.png&userId=73716 [8]: /c/portal/getImageAttachment?filename=5.5_hr.png&userId=73716 [9]: /c/portal/getImageAttachment?filename=6_elevationmap.png&userId=73716 [10]: /c/portal/getImageAttachment?filename=7_elevationcolor.png&userId=73716 [11]: /c/portal/getImageAttachment?filename=8_speedmap.png&userId=73716 [12]: /c/portal/getImageAttachment?filename=9_speedtime.png&userId=73716 [13]: http://blog.wolfram.com/2009/04/17/mapping-gps-data/ Sander Huisman 2015-09-19T17:56:26Z