https://community.wolfram.com RSS Feed for Wolfram Community showing ideas tagged with Audio, Sound and Music sorted by most replies. https://community.wolfram.com/groups/-/m/t/2774101 A Wolfram U daily study group covering the implementation of Wolfram Language for tasks ranging from basic programming to video analysis begins on January 17, 2023 and runs through February 3. This study group will run on weekdays from 11:00AM–12:00PM Central US time. This study group is an incredible way either to start learning Wolfram Language or to explore new functionality you haven't yet used. We will cover a very broad variety of topics, including but not limited to image and sound analysis, symbolics and numerics, function visualization and even cloud computation and deployment. We will even cover useful tips and tricks to help you work efficiently with notebooks! ![enter image description here][1] **No prior Wolfram Language experience is necessary.** As usual, we will have questions, study materials, quizzes along the way to help you master the subject matter. You can [**REGISTER HERE**][2]. I hope to see you there! ![enter image description here][3] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=meechstogram.png&userId=1711324 [2]: https://www.bigmarker.com/series/daily-study-group-wolfram-language-basics-wsg34/series_details?utm_bmcr_source=community [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=WolframUBanner%281%29%281%29.jpeg&userId=1711324 Arben Kalziqi 2023-01-11T04:29:36Z https://community.wolfram.com/groups/-/m/t/900782 Mathematica 11 is now out and ready to compute! Since the release of Version 10 two years ago, Mathematica has grown by leaps and bounds, with 500+ new functions. This version introduces both enhancements to computations across the board and completely new areas of functionality, continuing Mathematica's growth as the state-of-the-art technical platform—and we are very excited to share these developments with the world. Some of the most notable features include: **[3D Printing][1]** ![3D-Printed Triceratops][2] - Print your 3D models to local printers or online printing services with [Printout3D][3]. - Use [FindMeshDefects][4] and [RepairMesh][5] to fix your models before printing. - Hollow your models out with [ShellRegion][6] to lower the printing cost. **[Computational Audio][7]** ![Audio Image][8] - Version 11 includes a brand-new [Audio][9] object to represent audio created from importing or an array of data. - Edit your audio with [AudioPad][10], [AudioTrim][11], [AudioSplit][12], [AudioResample][13], etc. - Synthesize sounds using the new [AudioGenerator][14] function. - Apply filters, take [AudioMeasurements][15] and visualize your audio with [AudioPlot][16]. **[Neural Networks][17]** ![Neural Network][18] - Define network topologies with [NetGraph][19] or with a chain of layers with [NetChain][20]. - Train your neural nets with [NetTrain][21]. - Train networks on either CPUs or NVIDIA GPUs. - Are your training sets of images too large to hold in-memory? The neural network functions support out-of-core image datasets. **[Improved Machine Learning][22]** ![Identifying Notable Celebrities][23] - [ImageIdentify][24] now recognizes over 10,000 objects. - Extract features from images, text and other data types using [FeatureExtract][25]. - [Classify][26] works even better with images, including an option to customize what [FeatureExtractor][27] it uses. - Find formulas for time series data using [FindFormula][28]. - Find clusters in your data with enhanced options for [FindClusters][29], including new methods and setting a [CriterionFunction][30]. And these are only just a sample of the new features, including the [Wolfram Channel Framework][31], enhanced notebook processing, cloud-aware [WolframScript][32], improved [symbolic and numeric calculus][33] and so much more. To read more about what's new in Mathematica 11, read Stephen Wolfram's [release-day blog post][34] and check out the [New in 11 page][35]. [1]: http://www.wolfram.com/language/11/3d-printing/ [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=triceratops.png&userId=900759 [3]: http://reference.wolfram.com/language/ref/Printout3D.html [4]: http://reference.wolfram.com/language/ref/FindMeshDefects.html [5]: http://reference.wolfram.com/language/ref/RepairMesh.html [6]: http://reference.wolfram.com/language/ref/ShellRegion.html [7]: http://www.wolfram.com/language/11/computational-audio/ [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=audioplot.png&userId=900759 [9]: http://reference.wolfram.com/language/ref/Audio.html [10]: http://reference.wolfram.com/language/ref/AudioPad.html [11]: http://reference.wolfram.com/language/ref/AudioTrim.html [12]: http://reference.wolfram.com/language/ref/AudioSplit.html [13]: http://reference.wolfram.com/language/ref/AudioResample.html [14]: http://reference.wolfram.com/language/ref/AudioGenerator.html [15]: http://reference.wolfram.com/language/ref/AudioMeasurements.html [16]: http://reference.wolfram.com/language/ref/AudioPlot.html [17]: http://www.wolfram.com/language/11/neural-networks/ [18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=neuralnetworkdigits.png&userId=900759 [19]: http://reference.wolfram.com/language/ref/NetGraph.html [20]: http://reference.wolfram.com/language/ref/NetChain.html [21]: http://reference.wolfram.com/language/ref/NetTrain.html [22]: http://www.wolfram.com/language/11/improved-machine-learning/ [23]: http://community.wolfram.com//c/portal/getImageAttachment?filename=CumberbatchMachineLearning.png&userId=900759 [24]: http://reference.wolfram.com/language/ref/ImageIdentify.html [25]: http://reference.wolfram.com/language/ref/FeatureExtract.html [26]: http://reference.wolfram.com/language/ref/Classify.html [27]: http://reference.wolfram.com/language/ref/FeatureExtractor.html [28]: http://reference.wolfram.com/language/ref/FindFormula.html [29]: http://reference.wolfram.com/language/ref/FindClusters.html [30]: http://reference.wolfram.com/language/ref/CriterionFunction.html [31]: http://reference.wolfram.com/language/guide/Channel-BasedCommunication.html [32]: http://reference.wolfram.com/language/ref/program/wolframscript.html [33]: http://www.wolfram.com/language/11/symbolic-and-numeric-calculus/ [34]: http://blog.wolfram.com/2016/08/08/today-we-launch-version-11/ [35]: http://www.wolfram.com/mathematica/new-in-11/ Zachary Littrell 2016-08-08T16:43:43Z https://community.wolfram.com/groups/-/m/t/1967207 I first wrote this code in 1994. It was version three. I could not ever get it to run to completion. Puny desktops, and the early versions kept crashing. I got version 12 about a month ago. I finally get to hear what I write. It has been what? ...30 years. I'm not too bright. Could anyone help make the code more efficient? I am sorry if I am breaking the community rules...I am desperate...this is my message in a bottle. Best wishes. Patrick Michael Joseph Rose Patrick Rose 2020-05-07T11:24:59Z https://community.wolfram.com/groups/-/m/t/2153362 *Wolfram notebook is attached at the end of the post.* ---------- ![enter image description here][1] This post is about FEM simulation of violin vibration modes in 3D. As well known there are Helmholtz resonances of air inside the violin body with frequencies dependent on geometry of f-holes. This is the main reason why violin has so pronounced sound. To simulate these modes with Mathematica FEM we first define the body geometry (this is my design with given volume and area of f-holes, but it taken from the real violin): xy = {{3.805405405405406`,3.34954954954955`},{3.8252252252252257`,6.6990990990991`},{3.9441441441441443`,7.9081081081081095`}, {4.47927927927928`,8.601801801801802`},{5.014414414414414`,8.264864864864865`},{4.816216216216216`,7.8882882882882885`}, {4.895495495495496`,7.630630630630631`},{5.232432432432433`,7.432432432432433`},{5.47027027027027`,7.491891891891892`}, {5.648648648648649`,7.8882882882882885`},{5.668468468468468`,8.046846846846847`},{5.56936936936937`,8.403603603603605`}, {5.252252252252252`,8.681081081081082`},{4.855855855855856`,8.780180180180182`},{4.518918918918919`,8.8`}, {3.9639639639639643`,8.522522522522523`},{3.567567567567568`,7.967567567567568`},{3.3693693693693696`,7.372972972972973`}, {3.2306306306306305`,6.67927927927928`},{3.1513513513513516`,3.3693693693693696`},{3.1513513513513516`,2.655855855855856`}, {2.9729729729729732`,1.783783783783784`},{2.8738738738738743`,1.4666666666666668`},{2.100900900900901`,0.7927927927927928`}, {1.7243243243243245`,1.3081081081081083`},{2.021621621621622`,1.7639639639639642`},{2.0414414414414415`,2.0414414414414415`}, {1.9621621621621623`,2.23963963963964`},{1.6648648648648652`,2.4378378378378383`},{1.4666666666666668`,2.5171171171171176`}, {1.10990990990991`,2.338738738738739`},{0.891891891891892`,1.9423423423423425`},{0.9315315315315316`,1.4072072072072073`}, {1.5657657657657658`,0.7927927927927928`},{2.081081081081081`,0.6342342342342343`},{2.5963963963963965`,0.7927927927927928`}, {3.0918918918918923`,1.2090090090090093`},{3.5081081081081082`,1.902702702702703`},{3.706306306306306`,2.6954954954954955`}}; reg1 = RegionUnion[Disk[{0, 19.5/2}, 19.5/2], Disk[{0, 36 - 15.5/2}, 15.5/2], Rectangle[{-10, 15}, {10, 25}]]; reg2 = RegionDifference[reg1, RegionUnion[Disk[{-10, 20}, 9.5/2], Disk[{10, 20}, 9.5/2]]]; c0 = {0, 36 - 15.5/2}; c1 = {7.03562, 25}; f[x_] := c0[[2]] + x (c1[[2]] - c0[[2]])/(c1[[1]] - c0[[1]]); r1 = Norm[c1 - {10, f[10]}]; reg3 = RegionDifference[reg2, Disk[{10, f[10]}, r1]]; f1[x_] := c0[[2]] - x (c1[[2]] - c0[[2]])/(c1[[1]] - c0[[1]]); reg4 = RegionDifference[reg3, Disk[{-10, f1[-10]}, r1]]; c10 = {0, 19.5/2}; c11 = {8.215838362577491`, 15}; f11[x_] := c10[[2]] + x (c11[[2]] - c10[[2]])/(c11[[1]] - c10[[1]]); r2 = Norm[c11 - {10, f11[10]}]; reg5 = RegionDifference[reg4, Disk[{10, f11[10]}, r2]]; f12[x_] := c10[[2]] - x (c11[[2]] - c10[[2]])/(c11[[1]] - c10[[1]]); reg6 = RegionDifference[reg5, Disk[{-10, f12[-10]}, r2]]; p6 = RegionPlot[reg6, AspectRatio -> Automatic]; fh[xf_, yf_] := RegionUnion[ Polygon[Table[{xy[[i, 1]] - xf, xy[[i, 2]] + yf}, {i, Length[xy]}]], Polygon[Table[{-xy[[i, 1]] + xf, xy[[i, 2]] + yf}, {i, Length[xy]}]]]; General view of the violin body from the front and back side Show[p6, Graphics[fh[7, 12], AspectRatio -> Automatic]] dz = 3.79; reg8 = ImplicitRegion[Element[{x, y}, reg6] && 0 <= z <= dz, {x, y, z}]; mesh3d1 = DiscretizeRegion[reg8, {{-10, 10}, {0, 36}, {0, dz}}] ![Figure 1][2] Next step is the computation of air modes in the violin body with using ` NDEigensystem[]` as follows ca = 34321(*T=20C*); L = -Laplacian[u[x, y, z], {x, y, z}]; {vals, funs} = NDEigensystem[{L, DirichletCondition[u[x, y, z] == 0, Element[{x, y}, fh[7, 11.49]] && z == dz]}, u, Element[{x, y, z}, mesh3d1], 15]; Finally we visualize first 5 modes and the main mode in 3D {Table[DensityPlot[funs[[i]][x, y, dz/2], {x, -10, 10}, {y, 0, 36}, PlotRange -> All, PlotLabel -> ca Sqrt[vals[[i]]]/(2 Pi), ColorFunction -> "Rainbow", AspectRatio -> Automatic], {i, 1, 5}], DensityPlot3D[ funs[[1]][x, y, z], {x, -10, 10}, {y, 0, 36}, {z, 0, dz}, PlotRange -> All, PlotLabel -> ca Sqrt[vals[[1]]]/(2 Pi), ColorFunction -> "Rainbow", AspectRatio -> Automatic, PlotLegends -> Automatic, PlotPoints -> 100, BoxRatios -> Automatic, OpacityFunction -> None, Boxed -> False]} ![Figure 2][3] Therefore the first mode of 440.033 Hz is close to "A4" (440 Hz) tone. But we expecting "C4" (261.626 Hz), or "C#4" (277.183 Hz). The main reason of this discrepancies could be the wood plate vibration from the back side. Thus we define mesh, parameters of the wood plate and modes as follows dreg = DiscretizeRegion[reg6, {{-10, 10}, {0, 36}}, MaxCellMeasure -> .05] Y = 10.8*10^9; nu = 31/100; rho = 500; h = .003; d = 10^4 Sqrt[Y h^2/(12 rho (1 - nu^2))];Ld2 = {Laplacian[-d u[x, y], {x, y}] + v[x, y], -d Laplacian[v[x, y], {x, y}]}; {vals, funs} = NDEigensystem[{Ld2, DirichletCondition[u[x, y] == 0, True]}, {u, v}, Element[{x, y}, dreg], 5]; Table[DensityPlot[Re[funs[[i, 1]][x, y]], {x, y} \[Element] dreg, PlotRange -> All, PlotLabel -> vals[[i]]/(2 Pi), ColorFunction -> "Rainbow", AspectRatio -> Automatic], {i, 2, Length[vals]}] ![Figure 3][4] Hence for wood plate we have mode of 259.394 Hz and it is close to C4 tone. [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2021-01-04at1.47.31PM.jpg&userId=20103 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=bd9Wx.png&userId=1218692 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=w7dPm.png&userId=1218692 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=qb3HU.png&userId=1218692 Alexander Trounev 2021-01-03T23:13:58Z https://community.wolfram.com/groups/-/m/t/1732586 ![Feature Space Plot for speech data][1] ### *Figure Above: Feature Space Plot for speech data* ---------- ## Abstract In this project, I aim to design a system which is capable of detecting mood in human speech. Specifically, the system can be trained on a single user's voice given samples of emotional speech labeled as being angry, happy, or sad. The system is then able to classify future audio clips of the user's speech as having one of these three moods. I collected voice samples of myself and trained a Classifier Function based on this data. I experimented with many different available methods for the Classifier Function and for audio preprocessing to obtain the most accurate Classifier. ## Obtaining Data In this project, I focused on detecting mood for only one user (specifically, myself) since different people may express mood in different ways, leading to confusion for the Classifier. I initially recorded ten speech clips in each mood as training data, and ten different clips in each mood as testing data. I used [Audacity](https://www.audacityteam.org/) to record audio clips in 32-bit floating point resolution, exported as WAV files. If you're looking to replicate this project, you can use your own data, or use mine, available on [my GitHub repository](https://github.com/vedadehhc/MoodDetector). I later recorded additional clips, as discussed below. ## Feature Extraction In order to produce the most accurate Classifier, I extracted features which I thought were most useful to detecting mood. Specifically, I extracted the amplitudes, fundamental frequencies, and formant frequencies of each clip over the length of the clip using [AudioLocalMeasurements](https://reference.wolfram.com/language/ref/AudioLocalMeasurements.html). That is, I took multiple measurements of each of these values, over many partitions of the clip. I partitioned each clip into 75 parts, but feel free to experiment with different values. I also found the word rate, using Wolfram Language's experimental [SpeechRecognize](https://reference.wolfram.com/language/ref/SpeechRecognize.html) function, and the ratio of pausing time to total time of the clip using [AudioIntervals](https://reference.wolfram.com/language/ref/AudioIntervals.html). The final function takes the location of the audio file (could be local or on the web) and the number of partitions, and returns an association with the extracted features. It's important that the function return an association, since this makes things much easier when constructing the Classifier Function. The initial function for feature extraction is included below. extractFeatures[fileLocation_, parts_] := Module[{audio, assoc, pdur, amp, freq, form}, audio = Import[fileLocation]; pdur = AudioMeasurements[audio, "Duration"]/parts; amp = AudioLocalMeasurements[audio, "RMSAmplitude", PartitionGranularity -> pdur] // Normal; freq = AudioLocalMeasurements[audio, "FundamentalFrequency", MissingDataMethod -> {"Interpolation", InterpolationOrder -> 1}, PartitionGranularity -> pdur] // Normal; form = AudioLocalMeasurements[audio, "Formants", PartitionGranularity -> pdur] // Normal; assoc = <| Table["amplitude" <> ToString[i] -> amp[[i]][[2]], {i, Length[amp]}], Table["frequency" <> ToString[i] -> freq[[i]][[2]], {i, Length[freq]}], "wordrate" -> Length[TextWords[Quiet[SpeechRecognize[audio]]]]/ AudioMeasurements[audio, "Duration"], Table[Table["formant" <> ToString[i] <> "-" <> ToString[j] -> form[[i]][[j]], {j, Length[form[[i]]]}], {i, Length[form]}], "pauses" -> Total[Abs /@ Subtract @@@ AudioIntervals[audio, "Quiet"]]/ AudioMeasurements[audio, "Duration"] |>; Map[Normal, assoc, {1}] ] ### Get Training Data Using this function, I was able get the features for all of my training data. Here, I'll import my data from [my GitHub repository](https://github.com/vedadehhc/MoodDetector). angryTrainingFeats = Table[extractFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TrainingData/angry" <> ToString[i] <> ".wav", 75], {i, 10}]; happyTrainingFeats = Table[extractFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TrainingData/happy" <> ToString[i] <> ".wav", 75], {i, 10}]; sadTrainingFeats = Table[extractFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TrainingData/sad" <> ToString[i] <> ".wav", 75], {i, 10}]; ### Get Testing Data I also imported my testing data in the same way, so that it could be passed as an argument to the Classifier Function. I'll get this from GitHub as well. Note that this is the regular test data on GitHub. angryTestingFeats = Table[extractFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TestingData/Regular/a" <> ToString[i] <> ".wav", 75], {i, 10}]; happyTestingFeats = Table[extractFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TestingData/Regular/h" <> ToString[i] <> ".wav", 75], {i, 10}]; sadTestingFeats = Table[extractFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TestingData/Regular/s" <> ToString[i] <> ".wav", 75], {i, 10}]; ## Training a Classifier I constructed a Classifier Function using [Classify](https://reference.wolfram.com/language/ref/Classify.html). classifier = Classify[<|"angry" -> angryTrainingFeats, "happy" -> happyTrainingFeats, "sad" -> sadTrainingFeats|>, Method -> "LogisticRegression"] I ran the classifier on each set of training data. classifier[angryTrainingFeats] classifier[happyTrainingFeats] classifier[sadTrainingFeats] I experimented with all of the built-in options for [Method](https://reference.wolfram.com/language/ref/Method.html) to construct a Classifier Function with the best accuracy. I found that [LogisticRegression](https://reference.wolfram.com/language/ref/method/LogisticRegression.html) gave the best accuracy at 77% accuracy on the test data. However, I wanted to improve the accuracy further. ## Audio Pre-Processing One of the things that improved the Classifier's accuracy substantially was cleaning the audio before extracting features and classifying. Specifically, I trimmed the audio using [AudioTrim](https://reference.wolfram.com/language/ref/AudioTrim.html), and filtered each clip using [HighpassFilter](https://reference.wolfram.com/language/ref/HighpassFilter.html) before extracting features. The audio cleaning function is included below. cleanAudio[fileLocation_] := Module[{audio, trimmed, filtered}, audio = Import[fileLocation]; trimmed = AudioTrim[audio]; filtered = HighpassFilter[trimmed, Quantity[300, "Hertz"]] ] ### New Feature Extraction Function I updated the feature extraction function to include audio cleaning. extractCleanFeatures[fileLocation_, parts_] := Module[{audio, assoc, pdur, amp, freq, form}, audio = cleanAudio[fileLocation]; pdur = AudioMeasurements[audio, "Duration"]/parts; amp = AudioLocalMeasurements[audio, "RMSAmplitude", PartitionGranularity -> pdur] // Normal; freq = AudioLocalMeasurements[audio, "FundamentalFrequency", MissingDataMethod -> {"Interpolation", InterpolationOrder -> 1}, PartitionGranularity -> pdur] // Normal; form = AudioLocalMeasurements[audio, "Formants", PartitionGranularity -> pdur] // Normal; assoc = <| Table[ "amplitude" <> ToString[i] -> amp[[i]][[2]], {i, Length[amp]}], Table[ "frequency" <> ToString[i] -> freq[[i]][[2]], {i, Length[freq]}], "wordrate" -> Length[TextWords[Quiet[SpeechRecognize[audio]]]]/ AudioMeasurements[audio, "Duration"], Table[ Table["formant" <> ToString[i] <> "-" <> ToString[j] -> form[[i]][[j]], {j, Length[form[[i]]]}], {i, Length[form]}], "pauses" -> Total[Abs /@ Subtract @@@ AudioIntervals[audio, "Quiet"]]/ AudioMeasurements[audio, "Duration"] |>; Map[Normal, assoc, {1}] ] ### New Results Using audio cleaning, the accuracy of the classifier improved to 90%, much better than before. The results can be seen in the Confusion Matrix Plot below, generated using [ClassifierMeasurements](https://reference.wolfram.com/language/ref/ClassifierMeasurements.html). ![Confusion Matrix Plot for Regular Data][2] ## Testing with Neutral Statements Now, up until now, the statements used for recordings were all emotional in nature. For example, the clips recorded in an angry mood also had an angry statement being said. In order to control for this, I recorded neutral statements in each mood. That is, I recorded each of ten emotionally neutral statements in each of the three moods. If the Classifier still works on this data, that would show that it is not relying on the content of the speech, but on other audio features, which is the intended method. I imported these data using the clean feature extractor and, once again, I'll download them from my GitHub here. Note that this is the Neutral Statements data. nAngryCleanTestingFeats = Table[extractCleanFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TestingData/NeutralStatements/a" <> ToString[i] <> ".wav", 75], {i, 10}]; nHappyCleanTestingFeats = Table[extractCleanFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TestingData/NeutralStatements/h" <> ToString[i] <> ".wav", 75], {i, 10}]; nSadCleanTestingFeats = Table[extractCleanFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TestingData/NeutralStatements/s" <> ToString[i] <> ".wav", 75], {i, 10}]; ### Results for Neutral Statements I ran the clean Classifier on the new data, and the results were very positive. The classifier achieved an accuracy of 97% on the Neutral Statements data. The results are displayed in the Confusion Matrix Plot below, generated using [ClassifierMeasurements](https://reference.wolfram.com/language/ref/ClassifierMeasurements.html). ![Confusion Matrix Plot for Neutral Statements Data][3] ## Conclusions Overall, the Classifier was able to identify mood at 93% accuracy, even when the same statements were spoken in different moods. The composite Confusion Matrix Plot for all testing data can be seen below. ![Confusion Matrix Plot for all test data][4] ## Future Work In the future, I hope to improve the accuracy of the Classifier by providing additional training data, and to test it further with additional testing data. I also hope to expand the range of moods that the Classifier handles, including moods such as fear, calmness, and excitement. This, of course, would require the aforementioned additional data, and perhaps, more complex structures for the Classifier Function. In the future I would also like to experiment with multiple speakers, and determine whether classifiers for one speaker's moods can be used to determine those of another speaker. ## Acknowledgements I would like to thank my mentor Faizon Zaman for his guidance and assistance on this project. ## GitHub You can find my code for this project on [my GitHub repository](https://github.com/vedadehhc/MoodDetector), with all code available beginning July 12th, 2019. [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=featureSpacePlotNoLabel.png&userId=1724789 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=CMP1.png&userId=1724789 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=CMP2.png&userId=1724789 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=CMP3.png&userId=1724789 Dev Chheda 2019-07-12T01:16:42Z https://community.wolfram.com/groups/-/m/t/3062403 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=fg45qw.jpg&userId=11733 [2]: https://www.wolframcloud.com/obj/34ac42c4-10de-4201-a158-6af47eda2bd8 Marco Thiel 2023-11-08T21:48:25Z https://community.wolfram.com/groups/-/m/t/2166833 &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/cdf7d474-f4fb-4cbd-bbd5-f1fac8699f7a Siria Sadeddin 2021-01-18T22:46:11Z https://community.wolfram.com/groups/-/m/t/1862464 ###Introduction### Recently, I was learning about infinite space filling curves and their applications in real life. I became fascinated with [Hilbert Curves][1], one of many [plane-filling functions][2]. I wanted to see how these shapes could be used in image to video conversion. To begin with, I explored how pseudo Hilbert Curves could be used to convert a square image to audio so that each pixel color was associated with a specific sound. Theoretically, a person would be able to learn this association and would be able to reconstruct a mental image by listening to audio. This could help with people who are visually impaired, letting blind people "hear" pictures. The reverse is also true. People who are deaf will be able to "see" sound. I thought I could implement this using Mathematica, which I learned last summer at the Wolfram High School Camp. ###Why Hilbert Curves### ![Pseudo-Hilbert Curves from Order 1 to 10][3] As you can see above, as you increase the order, the limit of these curves start to fill an infinite amount of space. A true Hilbert Curve is actually $\lim_{n\to\infty} PseudoHilbertCurve_n$. Each one of these curves can be used on an image of dimensions 2 by 2, 4 by 4, 8 by 8, etc. The curve needed is accordingly: HilbertCurve[Log[2, ImageDimensions[image]]] Each line in the curve will go over one pixel in the image, starting at the bottom left and ending at the bottom right. ![Example of Order 3 Pseudo Hilbert Curve][4] But why would you need to use this specific pattern, as supposed to something more simple? Here: https://youtu.be/3s7h2MHQtxc?t=355. In short, if you were to increase the resolution of your image, you would now have to retrain your brain to re-associate the pixel value and the associated frequency at that point on the image. But this is not the case on a Hilbert Curve. As you increase the resolution of your image, and thus the order pseudo-Hilbert Curve, a point will just move closer and closer to its limit in the same space, which solves our problem. ###Converting Color to Sound### This was the main hurdle when writing the program. I needed to convert each pixel value to a unique sound frequency. My initial thought was to just create an association between each wavelength in the visible electromagnetic spectrum and a frequency, but I was quick to learn that doesn't work in the digital world. RGB colors could be a combination of different wavelengths. And I couldn't just add these wavelengths separately because there could be overlap. There was also brightness and darkness which isn't part of the light spectrum. So instead of the RGB format, I decided to use the HSB format (hue, saturation, and brightness). This is also more compatible for human learning, as the human eye uses these three characteristics to determine color, as supposed to the RGB values on a computer image. ![Graphics3D Visualizer of HSB][5] As you can see from the image, Hue could be used to determine the color. Now, instead of RGB with three values to one color, Hue gave me one number to a specific color. ![Graphics Hue Color Chart][6] But I still needed to show saturation and brightness in my sound waves. Color in a computer is expressed in three dimensions while sound has two: amplitude and frequency. I would be leaving one color property out. I could think of multiple ways to express all three as sound waves, but none of them made sure that similar colors sounded the same. For example, if I had a hue playing a certain frequency, I could define a range around this point that could express either saturation or brightness using a plus/minus system. To solve this, I decided to use [AudioChannel][7] functions. This way, one channel could express the hue through a specific frequency, while the other channel could express the saturation and brightness combined through volume and frequency respectively. This way, colors that were similar would sound similar too. ###Creating the Function### Obviously, I needed to get real images. I used *image1*, where the pixels were easily noticeable, and my baby picture for the school yearbook as *image2*, as a test image. The first one, being resized to a 8 by 8 image, would utilize an order 3 pseudo-Hilbert Curve ($\log _{2}8=3$), and the second one, being resized to a 32 by 32 image, would utilize an order 5 pseudo-Hilbert Curve ($\log _{2}32=5$). ![The images I used to test the program.][8] To get the HSB values, I used Wolfram's in-built [Hilbert Curve][9] and [Pixel Value][10] functions. I used Pixel Value to read the image's pixel values as bytes, which was converted to RGB and then to HSB. To get the points on the Hilbert Curve that was associated with their respective position on the image, I added {1,1} to each point so that it matched the pixel space (the origin on a Cartesian Plane is designated as (1,1) on an image). getHSB[image_] := ColorConvert[ RGBColor /@ Divide[PixelValue[ image, ({1, 1} + #) & /@ HilbertCurve[Log[2, ImageDimensions[image][[1]]]][[1]], "Byte"], 255.], "HSB"] In the Wolfram Language, HSB values range from 0 to 1 instead of the normal 0 to 360 which I guess gives the user more control of the specific color they want to implement. I had to scale these values to the frequency range I wanted for the hue and brightness. Through trial and error, I decided that 100-3900 Hz was a good range so that people of all ages could hear the lowest and highest frequencies. For the amplitude dictated by the saturation, I could just use the direct value as [SoundVolume][11]. I made it so that the sound for each pixel had a duration of 0.1 seconds for demonstrative purposes. In the real world, I would think that this should be a lot smaller so that images don't take too long to hear. getHSB[image_] := ColorConvert[ RGBColor /@ Divide[PixelValue[ image, ({1, 1} + #) & /@ HilbertCurve[Log[2, ImageDimensions[image][[1]]]][[1]], "Byte"], 255.], "HSB"] hueToFrequencyMatch[HSB_] := Rescale[HSB[[1]], {0, 1}, {100, 3900}] saturationToAmplitudeMatch[HSB_] := HSB[[2]] brightnessToFrequencyMatch[HSB_] := Rescale[HSB[[3]], {0, 1}, {100, 3900}] soundFrequency[frequency_] := Sound[Play[Sin[frequency*2 Pi t], {t, 0, 0.1}]] soundFrequencyVolume[frequency_, ampSaturation_] := Sound[soundFrequency[frequency], SoundVolume -> ampSaturation] After this, I just had to apply these functions over the HSB values of each pixel in the order dictated by the appropriate pseudo-Hilbert Curve, and then merge the hue sound and saturation+brightness sound as separate audio channels. convertHToSound[image_] := soundFrequency /@ hueToFrequencyMatch /@ getHSB[image] // AudioJoin convertBSToSound[picture_] := (soundList = {}; n = 1; While[n <= Length@getHSB[picture], AppendTo[soundList, soundFrequencyVolume[ getHSB[picture][[n]] // brightnessToFrequencyMatch, getHSB[picture][[n]] // saturationToAmplitudeMatch]]; n++]) convertHSBToSound[audio1_, audio2_] := AudioChannelCombine[{audio1, audio2}] Because hue is the most important color characteristic, I wanted to emphasize this AudioChannel more. Thus, editChannel2[audio_, factor_] := AudioPan[audio, -factor] To put everything together, I made a separate function to organize the results. seperateTable[picture_] := (convertBSToSound[picture]; Module[{a = convertHToSound[picture], b = soundList // AudioJoin}, c = editChannel2[convertHSBToSound[a, b], 0.05]; table = Grid[{{Image[picture, ImageSize -> 100], a, b, c}, {Blank[], AudioPlot@a, AudioPlot@b, AudioPlot@c}}]; ReplacePart[table, 1 -> Prepend[ First[table], {"Image", "Hue", "Saturation+Brightness", "HSB"}]]]) These were my results. I will attach the *hsb1* and *hsb2* sound files to this post, along with my notebook. ![Final image to sound representation.][12] Because they use two channels, you should use earbuds or headphones so that you can clearly differentiate the sounds coming through your left and right ears. ![Spectrogram of *hsb1* and *hsb2*][13] ###Visualizing the Process### animate[image_, audio_] := Module[{list = ({1, 1} + #) & /@ HilbertCurve[Log[2, ImageDimensions[image][[1]]]][[1]]}, (AudioPlay@audio; Animate[ ReplacePixelValue[Image[image, ImageSize -> 100], list[[1 ;; index]] -> Orange], {index, 1, list // Length, 1}, DefaultDuration -> QuantityMagnitude[ UnitConvert[Quantity[audio // Duration, "Seconds"]]], AnimationRepetitions -> 1])] I used this to create the following two animations. If you ran these programs, the HSB sound would play alongside the animation, showing exactly which pixel correlates to each sound. Unfortunately, I couldn't play the audio here on the post, but it is in the attached notebook if you would like to see. Here is what it looks like: ![*hsb1* Conversion][14] ![*hsb2* Conversion][15] ###Future Work### In the future, I hope to create a machine learning algorithm that can learn this association between audio and image in reverse (recreate the image from audio). If anyone has any ideas or pointers for me, please share in the comments below - I would really appreciate it! ##Updated Information:## Presented work at the [New Hampshire Science & Engineering Expo][16] and won the Office of Naval Research award. Youtube Link: [Video Presentation][17] [1]: http://mathworld.wolfram.com/HilbertCurve.html [2]: http://mathworld.wolfram.com/Plane-FillingFunction.html [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-01-19at12.07.19PM.png&userId=1725131 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=example.gif&userId=1725131 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=HSB.png&userId=1725131 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Hue.png&userId=1725131 [7]: https://reference.wolfram.com/language/ref/AudioChannels.html [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-01-19at1.39.00PM.png&userId=1725131 [9]: https://reference.wolfram.com/language/ref/HilbertCurve.html [10]: https://reference.wolfram.com/language/ref/PixelValue.html [11]: https://reference.wolfram.com/language/ref/SoundVolume.html [12]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-01-19at2.23.47PM.png&userId=1725131 [13]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-01-19at2.32.49PM.png&userId=1725131 [14]: https://community.wolfram.com//c/portal/getImageAttachment?filename=animation1.gif&userId=1725131 [15]: https://community.wolfram.com//c/portal/getImageAttachment?filename=animation2.gif&userId=1725131 [16]: https://nhsee.org/ [17]: https://youtu.be/gwpK2KIfVmA Srinath Rangan 2020-01-19T19:42:34Z https://community.wolfram.com/groups/-/m/t/1730213 # Overview Given an image of some sheet music, generate some internal representation of it that can then be played, reprinted or exported to some external musical notation software. In order to do this, first we split the sheet music into images of each staff, and remove the staff lines; we then find bounding boxes for each musical component using image segmentation, and classify them. This is all then passed to a parser that parses the final sheet music. # Finding the Staffs Staff lines are long, horizontal and thin; as such, they can be highlighted by using a bottom hat transform with a short vertical kernel, and the extracted with `ImageLines`. Then, by looking for close groups of five such lines, we can extract an image for each staff in the original score. We also find the median distance between the staff lines, both for padding the staff images and for reusing later. ``` detectStaffLineImages[image_] := Module[{ staffLines = Select[ImageLines[BottomHatTransform[image, BoxMatrix[{1, 0}]], Automatic, 0.0065][[All, 1]], (Abs@VectorAngle[Subtract @@ #, {-1, 0}] < 10 \[Degree]) &] }, \[CapitalDelta] = Median@Differences@Sort@Map[Mean, staffLines[[All, All, 2]]]; staffLineGroups = Gather[Sort[ staffLines, (Last@Mean@#1 > Last@Mean@#2) &], (Last@Mean@#1 - Last@Mean@#2 < 5 \[CapitalDelta]) &] // Echo; {Map[ ImageTrim[image, Flatten[#, 1], {0, 2.25 \[CapitalDelta]}] &, staffLineGroups ], \[CapitalDelta]} ] ``` ![Original page image][1] ![One of the split staff images][2] # Removing the Staff Lines Staff lines are dark, long, thin and horizontal. In order to remove them, we use a dilation with a short vertical kernel and the bottom hat transform of the image with a long horizontal kernel; by considering their closing together with taking the pixel-wise minimum and maximum of the resulting image with its morphological binarization, we get an image of the staff with no staff lines and not many artefacts. ``` removeStaffLines[image_, \[CapitalDelta]_] := With[ {withoutStaffLines = imageMin[Closing[image, BoxMatrix[{\[CapitalDelta]/6, 0}]], ColorNegate@ BottomHatTransform[image, BoxMatrix[{0, \[CapitalDelta]}]]]}, imageMax[withoutStaffLines, MorphologicalBinarize[withoutStaffLines, {0.66, 0.33}]] ] ``` ![Staffs removed][3] # Finding Bounding Boxes for the Musical Components ## Image Segmentation For the image segmentation, we use a SegNet [^segnet] trained on the DeepScores [^deepscores] [Segmentation Dataset](https://drive.google.com/drive/folders/1KFxqi0rO-bJrd03rLk87fF1iOmnjpaoG), with some preprocessing: Each pair of images (original and segmented), was separated into individual staffs and the symbol colors were negated (so that there would be more contrast with the black background). ![SegNet architecture][4] ![Segmented staff][5] ![`Binarize@*Blur` of the segmented staff][6] ## Finding the Bounding Boxes In order to find the bounding boxes from the segmented image, we look for the its morphological components using `ComponentMeasurements`. Before using `ComponentMeasurements`, we blur and binarize the segmented image, as it usually yields more accurate results; we also discard any bounding boxes with area less than 25 pixels, as that is very likely just noise. ![Bounding boxes][7] # Classifying the Musical Components For the component classification, we use a `ClassifierFunction` trained on the DeepScores [^deepscores] [Classification Dataset](https://drive.google.com/file/d/1bdBrX0dAX734I3MA_6-wH_-N2eqq_tf_/view), cropping the images in relation to a lookup table - this is so that the images contain only the actual symbol, and the classifier doesn't learn the context instead of the symbol. Unfortunately, the classifier trained during the Wolfram Summer Program misclassifies some images; however, it does still provide something that can be worked with. ![Classified symbols][8] # Parsing the Sheet Music Very limited parsing could be done during the Wolfram Summer Program: we sort the notes and pitch them, and add rests in as well. ## Finding the Pitches Pitching the notes depends on their distance to the bottom of the staff and a reference pitch, which is determined by the clef. Because of the way we trimmed the staff images, we know that the first staff line (from the bottom) has the Y coordinate $2.25 \Delta$, where $\Delta$ is the distance between the staff lines in the original image. We also know, from the way we trimmed the staffs from the original page, that the distance between the staff lines in the resized image is fixed - and is $\delta = 10$. `PitchNumber` calculates the distance from the note centroid to the bottom of the staff, normalizing by half of the distance between the staff lines. `PitchToLetter` finds the name for the given output of `PitchNumber` in relation to some reference pitch. ![Centroids of the correctly-classified notes][9] ``` PitchNumber[noteCentroid_, \[Delta]_, bottom_] := (noteCentroid[[2]] - bottom)/(0.5 \[Delta]) naturalNotes = Flatten[Table[# <> ToString[i] & /@ {"A", "B", "C", "D", "E", "F", "G"}, {i, 1, 7}], 1] PitchToLetter[pitchNumber_, referenceNumber_, referenceLetter_] := RotateLeft[naturalNotes, FirstPosition[naturalNotes, referenceLetter] - 1][[ Round[pitchNumber - referenceNumber] + 1]] ``` ## Basic Parsing In this case, the parsing is a simple sorting - there are no dynamics, articulation or even accidentals taken into consideration. ``` ParseSheetMusic[pitchedNotes_, rests_] := SortBy[Join[pitchedNotes, restsNotation], NotationCentroid[#][[1]] &] ``` In the end, this is our final result: ``` {SheetMusicNote[{66.5, 64.}, "B5"], SheetMusicRest[{97., 45.5}, "restHBar"], SheetMusicRest[{97.5, 64.5}, "restHalf"], SheetMusicNote[{121., 64.}, "B5"], SheetMusicNote[{159., 64.}, "B5"], SheetMusicNote[{196.5, 64.}, "B5"], SheetMusicRest[{287., 49.5}, "rest32nd"], SheetMusicNote[{327.5, 50.}, "F4"], SheetMusicNote[{366., 59.5}, "A5"], SheetMusicRest[{456., 75.}, "restQuarter"]} ``` [^segnet]: [https://arxiv.org/abs/1511.00561](https://arxiv.org/abs/1511.00561) [^deepscores]: [https://tuggeluk.github.io/papers/preprint_deepscores.pdf](https://tuggeluk.github.io/papers/preprint_deepscores.pdf) [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=original-sheet.png&userId=1726262 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=original-staff.png&userId=1726262 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=nostaffs.png&userId=1726262 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=segnet-architecture.png&userId=1726262 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=segmented.png&userId=1726262 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=binblur.png&userId=1726262 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=bboxes.png&userId=1726262 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=classified.png&userId=1726262 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=notes.png&userId=1726262 Daniel Csillag 2019-07-10T20:35:05Z https://community.wolfram.com/groups/-/m/t/3062832 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=w5qgsdf.jpg&userId=11733 [2]: https://www.wolframcloud.com/obj/04458d24-aacf-4cd7-8a5d-46efde37e927 Marco Thiel 2023-11-09T22:51:38Z https://community.wolfram.com/groups/-/m/t/1380021 ![Cover][1] ## Goal ## There are many ways to generate music and one of them is algorithmic, where music is generated with the help of a list of handcrafted rules. The approach in this project is different - I build a neural network that knows nothing about music but learns it from thousands of songs given in MIDI format. _Apart from just generating a meaningful sequence of notes I also wanted to add **dynamics** in loudness and humanlike mistakes in timing with **no restrictions for note durations**._ - **Why dynamics and timing?** There is no human who is able to play on a musical instrument with precisely the same loudness and strictly in time with a metronome(at least I can't). People do mistakes, but in the case of music, they are helping in creating what we call more alive music. It is a fact that dynamic music with slight time shifts sounds more interesting, so even when you write music in a program you supposed to add these "mistakes" by yourself. - **Why performances?** The dataset that I use for the project contains performances of [Yamaha e-piano competition][2] participants. This gives us a possibility to learn the dynamics and mistakes in timings. __Here's an [example][3] generated by the model.__ All the code, data and trained models can be found on [GitHub][32]. The examples will be attached to this post as files just in case. ---------- ## Inspiration ## This is not an original work and mostly it's an attempt to recreate the work of [Magenta][4] team from their blog [post][5]. Nevertheless, in this post, I will try to add more details to the **preprocessing** steps and how you can build a similar neural network model in Wolfram Language. ## Data ## I've used a [site][6] that has the Yamaha e-piano performances but also contains a set of classic and jazz compositions. In the original [work][7] Magenta team has used only the Yamaha dataset but with a heavy augmentation on top of that: Time-stretching (making each performance up to 5% faster or slower), Transposition (raising or lowering the pitch of each performance by up to a major third). Also, you can create your own list of MIDI files and build a dataset with the help of the code provided below in the post. Here are links to find a lot of free MIDI songs: [The Lakh MIDI Dataset][8](very well prepared a dataset for ML projects), [MidiWorld][9] and [FreeMidi][10] ## MIDI ## MIDI is short for Musical Instrument Digital Interface. It’s a language that allows computers, musical instruments, and other hardware to communicate. MIDI carries event messages that specify musical notation, pitch, velocity, vibrato, panning, and clock signals (which set tempo). _For the project, we need only events that denote where is every note starts/ends and with what are velocity and pitch._ ## Preprocessing The Data ## Even though MIDI is already a digital representation of music, we can't just take raw bytes of a file and feed it to an ML model as in the case of the models working with images. First of all, images and music are conceptually different tasks: the first is a single event(data) per item(an image), the second is a sequence of events per item(a song). Another reason is that raw MIDI representation and a single MIDI event itself contain a lot of irrelevant information to our task. Thus we need a special data representation, a MIDI-like stream of musical events. Specifically, I use the following set of events: - 88 **note-on** events, one for each of the 88 MIDI pitches of piano range. These events start a new note. - 88 **note-off** events, one for each of the 88 MIDI pitches of piano range. These events release a note. - 100 **time-shift** events in increments of 10 ms up to 1 second. These events move forward in time to the next note event. - 34 **velocity** events, corresponding to MIDI velocities quantized into 32 bins. These events change the velocity applied to subsequent notes. The neural network operates on a one-hot encoding over these 310 different events. This is the very same representation as in the original work but the number of note-on/note-off is fewer, I encode 88 notes in piano range instead of 127 notes in MIDI pitch range to reduce one-hot encoding vector size and make the process of learning easier. **For example**, if you want to encode 4 notes from C major with durations of a half second and with different velocities your sequence of events would be somewhat like this(for clarity I use only indices instead of the whole one-hot encoding): `{288, 60, 226, 148, 277, 62, 226, 150, 300, 64, 226, 152, 310, 67, 226, 155}` ![Preprocessing encoding C major example][11] In this particular example: - _60, 62, 64, 67_ are **note on** events(C5, D5, E5, G5). Values in a range from 1 to 88. - _148, 150, 152, 155_ are **note off** events. Values in a range from 89 to 176. - _226_ is a half second **time shift** event. Values in a range from 177 = 10 ms to 276 = 1 sec. - _288, 277, 300, 310_ are **velocity** events. Values in a range from 277 to 310. In this way, you can encode music that is expressive in dynamics and timing. Now, let's take a look on another example with a chord from the same notes but with different durations: `{300, 60, 62, 64, 67, 226, 152, 155, 226, 150, 226, 148}` ![C major chord][12] As you can see, if you want to play more than one note at once you just need to put them in a single bunch of note-on events(60, 62, 64, 67). Then you add time shift and note-off events as you needed. If you need a duration longer than 1 sec you can stack together more than one time-shift events({310, 310} = 2 sec time-shift). **WL and MIDI** Wolfram Language has a **built-in** support of MIDI files what is really simplifying initial work. To get data from MIDI file you need to import it with specific elements: ![WL MIDI Import Elements][13] In the code below I also extract and calculate needed information related to a tempo of a song. {raw, header} = Import[path, #]& /@ {"RawData", "Header"}; tempos = Cases[Flatten[raw], HoldPattern["SetTempo" -> tempo_] :> tempo]; microsecondsPerBeat = If[Length@tempos > 0, First[tempos], 500000]; (* If there is no explicit tempo we use default 120 bpm *) timeDivision = First@Cases[header, HoldPattern["TimeDivision" -> division_] :> division]; (* Convert timeDivision value to base of 2 *) timeDivisionBits = IntegerDigits[timeDivision, 2]; (* Pad zeros at the beginning if the value takes less then 16 bits *) timeDivisionBits = If[Length@timeDivisionBits < 16, PadLeft[timeDivisionBits, 16], timeDivisionBits]; (* The top bit responsible for the type of TimeDivision *) timeDivisionType = timeDivisionBits[[1]]; framesPerSecond = timeDivisionBits[[2 ;; 8]]; ticksPerFrame = timeDivisionBits[[9 ;; 16]]; ticksPerBeat = If[timeDivisionType == 0, timeDivision, 10^6 /(framesPerSecond * ticksPerFrame)]; secondsPerTick = (microsecondsPerBeat / ticksPerBeat) * 10^-6.; An example of raw data and header info from MIDI file in Wolfram Language: ![Raw MIDI output][14] **SetTempo** is a number of microseconds per beat(microseconds per quarter note). **Time Division** has two type of interpreting. If the top bit is 0 then the type is "ticks per beat" (or “pulses per quarter note”) otherwise, the type is "frames per second". We need those two values to calculate time per one **MIDI tick** that used in MIDI events as a time measurement. One MIDI event in WL representation looks like this `{56, {9, 0}, {46, 83}}` - 56 is a number of **MIDI ticks** that means the total amount of time that must pass from the previous MIDI event. It represents our **time-shift** event by simple multiplication of this number with **secondsPerTick**. - 9 is a status byte of MIDI events(9,8 are **note-on**, **note-off** respectively). - 0 is MIDI channel(irrelevant for us). - 46 indicates what is a pitch of this note(related to **note-on**/**note-off** events). - 83 is a number we encode in a **velocity** event. If you want to understand how a real raw MIDI data structured, this [blog][15] is specifically useful. Now, what we need is to parse a sequence of MIDI events and filter them only for events that are **note-on**, **note-off** and all the events that have the number of **MIDI ticks** greater than 0. Some of the meta-messages have irrelevant MIDI ticks thus we need to exclude them from final sequence - we just skip the events with value **F**(Meta message) in the MIDI status byte. After filtering MIDI data you get a sequence that is ready to be encoded to the final representation and will be fed to the model. ![Filtered MIDI events][16] To encode the sequence of MIDI events to the final representation I use the code below: EncodeMidi[track_, secondsPerTick_] := Block[{lastVelocity = 0}, ClearAll[list]; Flatten[ Map[ Block[{list = {}}, (* Add time shifts when needed *) If[TimeShiftByte[#, secondsPerTick] > 0, list = Join[list, EncodeTimeShift[TimeShiftByte[#, secondsPerTick]]]]; (* Proceed with logic only if it's a note event *) If[StatusByte[#] == NoteOnByte || StatusByte[#] == NoteOffByte, (* Add velocity if it's different from the last seen *) If[lastVelocity != QuantizedVelocity[VelocityByte[#]] && StatusByte[#] == NoteOnByte, lastVelocity = QuantizedVelocity[VelocityByte[#]]; list = Join[list, List[EncodeVelocity[VelocityByte[#]]]]; ]; (* Add note event *) list = Join[list, List[EncodeNote[NoteByte[#], StatusByte[#] == NoteOnByte]]]; ]; (* Return encoded list*) list ]&, track] , 1]]; This code has a lot of functions that I've written during the summer school but they are mostly utility short functions. You can check them and complete implementation on [GitHub][17]. When the code for the preprocessing is ready it's time to build a dataset. **Building Dataset** I've made a [notebook][18] that takes care of preprocessing of MIDI files and encode them into the final representation. (* Take all files names in Midi folder *) files = FileNames["*", NotebookDirectory[] <> "Midi"]; dataset = Flatten[EncodeTrack /@ files, 1]; During the encoding, each track is partitioning into smaller segments: encodings = Partition[EncodeMidi[GetMidiEvents[raw, secondsPerTick], secondsPerTick], 500]; In the original work, Magenta team split each song into 30-second segments to keep each example of manageable size. The problem is that partition by equal time doesn't give you the equal size of examples. Even though you can use varying input size in sequence models I wanted to use a static size of examples to speed up the training process. I was told that internally in WL(or maybe everywhere) it's more efficient to have the same size of every example for a model. However, I believe this kind of partition has a drawback, in a way that an equal number of encoded events could have a different duration in time thus adding inconsistency in the dataset. In my case, I've divided each song into segments of 500 encoded events. ![One Song Final Encoding][19] To reduce the size of the final dataset I use only indices for one-hot encodings. As the result, the final dimension of my dataset was **{99285, 500}** If you want to try partition by the time you need to edit `EncodeTrack` function in [`Midi.m`][20]. With this code, you will find positions of where to split a sequence on equal time segments: GetTimePositions[track_, seconds_, secondsPerTick_] := Block[{positions = {}, time = 0}, Do[ time = time + track[[i]][[1]] * secondsPerTick; If[time > seconds, positions = Append[positions, i]; time = 0;], {i, Length@track}]; positions ] Where parameter `track` is a sequence of MIDI events. Then you split the same `track` with the positions you've got from the function. segments = FoldPairList[TakeDrop, track, positions]; After that, you need to encode `segments` with the help of `EncodeMidi` function. If you do that there is one thing left - rework the model to accept varying input size but the next part will cover how to build a model with a static size of example. ---------- ## Building a Model ## Because music data is a sequence of events we need an architecture that knows how to remember, and predicts what is the next event based on all previous. This is exactly what Recurrent Neural Networks try to do - RNNs can use their internal state (memory) to process sequences of inputs. If you want to check more details I would recommend to watch this [introduction][21] video. On the abstract level, RNN learns the probabilities of events that follow after each other. Take for example this language model from Wolfram Neural Repository, it predicts the next character of a given sequence. NetModel["Wolfram English Character-Level Language Model V1"]["hello worl"] The output is **d**. You can get top 5 probabilities if you want. NetModel["Wolfram English Character-Level Language Model V1"]["hello worl", {"TopProbabilities", 5}] You will get: {"d" -> 0.980898, "e" -> 0.00808785, "h" -> 0.0045687, " " -> 0.00143807, "l" -> 0.000681855} In my work, I needed similar behavior but instead of characters, I wanted to predict encoded MIDI events. That is why the basis of the model I build is [Wolfram English Character-Level Language Model V1][22]. Also, after reading a [guide][23] about sequence learning with neural networks in WL I've decided to improve the training process with "teacher forcing" technique. **Teacher Forcing** In a simple language model, a model takes the last prediction from an input sequence and compute the class of it. But for "teacher forcing" we need to get classes of all predictions. ![Model comparison][24] Comparatively to the language model I've removed one `GatedReccurentLayer` and `Dropoutlayer` due to the not so big dataset(as precautions to avoid overfitting). Another benefit of using "teacher forcing" is that you don't need to separately create labels for every example. To compute the loss we make out of an input example two sequences: 1. Everything but the **last** element(Sequence**Most**Layer) 2. Everything but the **first** element(Sequence**Rest**Layer) ![Teacher Forcing Net][25] As you can notice the input is only one vector of indices with size 500 and labels for computing the loss are generating inside of a `NetGraph`. Here is a visualized example of the flow with simple input: ![Input flow explanation][26] You can find the code for creating the model in this [PerfrormanceRnnModel][27] notebook. After all the data is ready and the model is finalized we can start training. NetTrain[teacherForcingNet, <|"Input" -> dataTrain|>, All, TrainingProgressCheckpointing -> {"File", checkPointDir, "Interval" -> Quantity[5, "Minutes"]}, BatchSize -> 64, MaxTrainingRounds -> 10, TargetDevice -> "GPU", (* Use CPU if you don't have Nvidia GPU *) ValidationSet -> <|"Input" -> dataValidate|> ] A friendly advice - it's better to use **"Checkpoining"** during the training. This will keep your mental health safe and will work as assurance that all training progress is saved. I was training the model 30 rounds and it took around 4-5 hours on AWS' GPUs. First 10-15 rounds weren't showing any sight of problems but later training clearly started to overfit. ![Training loss][28] Unfortunately, I haven't had time to fix this problem because of the limited time but to overcome this problem I might reduce the size of GRUs from 512 to 256 and return Dropout layer. ## Generate Music ## To generate music we need a model that predicts the next event in a sequence as it was in the language model. To do that I take the trained model and extract out of it "PerformanceRNN Predict Model" part. predictNet = NetExtract[trainedNet, "predict"]; Next step is to convert this `predictNet` to a model that takes varying input size and return the class of the next event. generateModel = NetJoin[NetTake[predictNet, 3], { SequenceLastLayer[], NetExtract[predictNet, {4, "Net"}], SoftmaxLayer[]}, "Input" -> Automatic, "Output" -> NetDecoder[{"Class", Range[310]}] ] The resulting architecture is pretty the same as the language model from which I've started - it takes a sequence with varying size of encoded MIDI events `{177, 60, 90}` and predicts what could be next event `{177, 60, 90, ?}`. ![Model Comparison(Generation)][29] **Now, let's the fun begin!** generateDemo[net_, start_, len_] := Block[{obj = NetStateObject[net]}, Join@NestList[{obj[#, "RandomSample"]} &, start, len] ] This small function is all we need to generate a sequence of the desired length. `NetStateObject` helps to keep track of all sequences that were applied to the network, meaning every next prediction is the result of all previous events not only the recent one. `start` should be a sequence of encoded MIDI events. It also can be a single item sequence, say you want to start from a pause or a particular note. This is a possibility to some extent put the generation process in a particular direction. Okay, two lines of code left and you can hear play with generating of music: generatedSequence = Flatten[generateDemo[generateModel, {60, 216, 148, 62, 200, 150, 64, 236, 152, 67, 198, 155}, 500]]; ToSound[generatedSequence] These are other examples: [2][30], [3][31]. You can generate your own demos if download [repository][32] and open [PerformanceRNN][33] notebook. ## Further Work ## That was a very fun and challenging task for me. I can't say that I'm satisfied with the results but this a good start and I have a direction now. What I want to explore is Variational Autoencoder, especially [MusicVAE][34] that is made by the same Magenta team. However, I'll start with improving the existing model by changing the architecture and cleaning the dataset to have only performances from the Yamaha dataset. Thank you for reading the post, and feel free to ask any questions. ![Peace!][35] [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=2433article_cover.png&userId=1352227 [2]: http://www.piano-e-competition.com [3]: https://drive.google.com/open?id=1I7l6hrecWsuMxqvEdUiRWtg6N6NCW34R [4]: https://magenta.tensorflow.org/ [5]: https://magenta.tensorflow.org/performance-rnn [6]: http://www.kuhmann.com/Yamaha.htm [7]: https://github.com/tensorflow/magenta/tree/master/magenta/models/performance_rnn [8]: http://colinraffel.com/projects/lmd/ [9]: http://www.midiworld.com/ [10]: https://freemidi.org/ [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=5707Preprocessing_explanation.png&userId=1352227 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=6109Prep_ex_2.png&userId=1352227 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=MIDIimportelements.png&userId=1352227 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=7717raw_midi_output.png&userId=1352227 [15]: http://www.recordingblogs.com/wiki/musical-instrument-digital-interface-midi [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=filtered_midi_events.png&userId=1352227 [17]: https://github.com/Apisov/Performance-RNN-WL/blob/master/Project/Midi.m [18]: https://github.com/Apisov/Performance-RNN-WL/blob/master/BuildData.nb [19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=One_track_final_encoding.png&userId=1352227 [20]: https://github.com/Apisov/Performance-RNN-WL/blob/master/Project/Midi.m [21]: http://www.wolfram.com/wolfram-u/catalog/wl036/ [22]: https://resources.wolframcloud.com/NeuralNetRepository/resources/Wolfram-English-Character-Level-Language-Model-V1 [23]: http://reference.wolfram.com/language/tutorial/NeuralNetworksSequenceLearning.html#1013067167 [24]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Modelcomparison.png&userId=1352227 [25]: http://community.wolfram.com//c/portal/getImageAttachment?filename=3750Teacher_forcing.png&userId=1352227 [26]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Teacher_forcing_explanation.png&userId=1352227 [27]: https://github.com/Apisov/Performance-RNN-WL/blob/master/PerformanceRNNModel.nb [28]: http://community.wolfram.com//c/portal/getImageAttachment?filename=raw_midi.png&userId=1352227 [29]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Modelcomparison%28Generation%29.png&userId=1352227 [30]: https://drive.google.com/open?id=1GtlaOtTF_9rHiDVsrqmLnUaKCSAva1dP [31]: https://drive.google.com/open?id=1sEihbFJw4XbVZveYl8efoM8Ivq8781ar [32]: https://github.com/Apisov/Performance-RNN-WL [33]: https://github.com/Apisov/Performance-RNN-WL/blob/master/PerformanceRNN.nb [34]: https://magenta.tensorflow.org/music-vae [35]: http://community.wolfram.com//c/portal/getImageAttachment?filename=d2eedc8a1ea8fc6a62e23b151c7fb3675c8153cc.png&userId=1352227 Pavlo Apisov 2018-07-11T21:20:56Z https://community.wolfram.com/groups/-/m/t/1383518 ![A Representation of the emotion categorization system][1] ---------- #Abstract This project aims to develop a machine learning application to identify the sentiments in a music clip. The data set I used consists of one hundred 45-second clips from the Database for Emotional Analysis of Music and an additional 103 gathered by myself. I manually labeled all 203 clips and used them as training data for my program. This program works best with classical-style music, which is the main component of my data set, but also works with other genres to an reasonable extent. #Introduction One of the most important functions of music is to affect emotion, but the experience of emotion is ambiguous and subjective to individual. The same music may induce a diverse range of feelings in people as a result of different context, personality, or culture. Many musical features, however, usually lead to the same effect on the human brain. For example, louder music correlates more with excitement or anger, while softer music corresponds to tenderness. This consistency makes it possible to train a supervised machine learning program based on musical features. #Background This project is based on James Russell's circumplex model, in which a two-dimensional emotion space is constructed from the x-axis of valence level and y-axis of arousal level, as shown above in the picture. Specifically, valence is a measurement of an emotion's pleasantness, whereas arousal is a measurement of an emotion's intensity. Russell's model provides a metric on which different sentiments can be compared and contrasted, creating four main categories of emotion: Happy (high valence, high arousal), Stressed (low valence, high arousal), Sad (low valence, low arousal), and Calm (high valence, low arousal). Within these main categories there are various sub-categories, labeled on the graph above. Notably, "passionate" is a sub-category that does not belong to any main category due to its ambiguous valence value. ---------- #Program Structure The program contains a three-layer structure. The first layer is responsible for extracting musical features, the second for generating a list of numerical predictions based on different features, and the third for predicting and displaying the most probable emotion descriptors based on the second layer's output. ![enter image description here][2] ##First Layer The first layer consists of 23 feature extractors that generate numerical sequence based on different features: (*A list of feature extractors*) feMin[audio_] := Normal[AudioLocalMeasurements[audio, "Min", List]] feMax[audio_] := Normal[AudioLocalMeasurements[audio, "Max", List]] feMean[audio_] := Normal[AudioLocalMeasurements[audio, "Mean", List]] feMedian[audio_] := Normal[AudioLocalMeasurements[audio, "Median", List]] fePower[audio_] := Normal[AudioLocalMeasurements[audio, "Power", List]] feRMSA[audio_] := Normal[AudioLocalMeasurements[audio, "RMSAmplitude", List]] feLoud[audio_] := Normal[AudioLocalMeasurements[audio, "Loudness", List]] feCrest[audio_] := Normal[AudioLocalMeasurements[audio, "CrestFactor", List]] feEntropy[audio_] := Normal[AudioLocalMeasurements[audio, "Entropy", List]] fePeak[audio_] := Normal[AudioLocalMeasurements[audio, "PeakToAveragePowerRatio", List]] feTCent[audio_] := Normal[AudioLocalMeasurements[audio, "TemporalCentroid", List]] feZeroR[audio_] := Normal[AudioLocalMeasurements[audio, "ZeroCrossingRate", List]] feForm[audio_] := Normal[AudioLocalMeasurements[audio, "Formants", List]] feHighFC[audio_] := Normal[AudioLocalMeasurements[audio, "HighFrequencyContent", List]] feMFCC[audio_] := Normal[AudioLocalMeasurements[audio, "MFCC", List]] feSCent[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralCentroid", List]] feSCrest[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralCrest", List]] feSFlat[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralFlatness", List]] feSKurt[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralKurtosis", List]] feSRoll[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralRollOff", List]] feSSkew[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralSkewness", List]] feSSlope[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralSlope", List]] feSSpread[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralSpread", List]] feNovelty[audio_] := Normal[AudioLocalMeasurements[audio, "Novelty", List]] <br/> ##Second Layer Using data generated from the first layer, the valence and arousal predictors of the second layer provide 46 predictions for the audio input, based on its different features. (*RMSAmplitude*) (*Feature extractor*) feRMSA[audio_] := Normal[AudioLocalMeasurements[audio, "RMSAmplitude", List]] dataRMSA = Table[First[takeLast[feRMSA[First[Take[musicFiles, {n}]]]]], {n, Length[musicFiles]}]; (*Generating predictor*) pArousalRMSA = Predict[dataRMSA -> arousalValueC] ![Sample predictor function][3] <br/> ##Third Layer The two parts of the third layer, main category classifier and sub-category classifier, each utilize the tensors generated in the second layer to make a prediction within their realm of emotion. The output consists of two parts, a main category emotion and a sub-category emotion. (*Main*) emotionClassify1 = Classify[classifyMaterial -> emotionList1, PerformanceGoal -> "Quality"] (*Sub*) emotionClassify2 = Classify[classifyMaterial -> emotionList2, PerformanceGoal -> "Quality"] ![enter image description here][4] <br/> ##Output If the program receives an input that is longer than 45 second, it will automatically clip the audio file into 45 second segments and return the result for each. If the last segment is less than 45 seconds, the program would still work fine on it, though with reduced accuracy. The display for each clip includes a main-category and a sub-category descriptor, with each of their associated probability also printed. ###Sample testing: Debussy's Clair de Lune ![enter image description here][5] <br/> ---------- #Conclusion The program gives very reasonable result for most music in the classical style. However, the program have three shortcomings that I plan to fix in later versions of the this program. Firstly, the program may give contradictory result (ex. happy and depressed) if the sentiment dramatically changes in the middle of a 45 second segment, perhaps reflecting the music's changing emotional composition. The current 45 second clipping window is rather long and thus prone to capture contradicting emotions. In the next version of this program, the window will probably be shortened to 30 or 20 seconds to reduce prediction uncertainty. Secondly, the program's processing speed has a lot of room of improvement. It currently takes about one and half minutes to compute an one minute audio file. In future versions I will remove relative ineffective feature extractors to speed things up. Lastly, the data used in creating this application is solely from myself, and therefore it is prone to my human biases. I plan to expand the data set with more people's input and more genres of music. I have attached the application to this post so that everyone can try out the program. #Acknowledgement I sincerely thank my mentor, Professor Rob Morris, for providing invaluable guidance to help me carry out the project. I also want to thank Rick Hennigan for giving me crucial support with my code. [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=8714Emotion2DSpace.PNG&userId=1371765 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=2406DataStructure.PNG&userId=1371765 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=8990capture1.PNG&userId=1371765 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=5765capture2.PNG&userId=1371765 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=3607SampleTesting.PNG&userId=1371765 William Yicheng Zhu 2018-07-14T02:40:20Z https://community.wolfram.com/groups/-/m/t/2318363 ![Spectrogram of the THX sound recreation][1] &[Wolfram Notebook][2] [Original Notebook]: https://www.wolframcloud.com/obj/be1a176a-f620-4a53-8f37-6c99bff8ad9e [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2021-07-15124908.png&userId=2316922 [2]: https://www.wolframcloud.com/obj/6d89e81f-272f-4a00-8996-1ebf68e128ac Evelyn Di Genova 2021-07-15T21:14:09Z https://community.wolfram.com/groups/-/m/t/917048 The [new computational audio features][1] of Mathematica are really impressive. In no time you can cook up things as soon as you understand the basic algorithm. Even the simplest tricks can give quite surprising results. Before you read on, please do play this (file also uploaded to this post): ***[audio file][2]***. Have you noticed anything? I think that it sounds pretty much like the well known example ExampleData[{"Sound", "Apollo11SmallStep"}] As a matter of fact, it contains something a little special. If you import the file into Mathematica secsound = Import["/Users/thiel/Desktop/secretsound.wav"]; and calculate a spectrogram of it Spectrogram[secsound, ColorFunction -> "Rainbow", Frame -> None, ImageSize -> Full] you get this: ![enter image description here][3] How to inject images into spectrograms --------------------------- I will show now, how you can get this in a couple of lines of code. First we need to get a binary matrix (later we can do better!) of the message. This is quite straight forward: imgdata = Reverse@ImageData[ColorNegate@Image[Binarize[Rasterize[Text[" Wolfram "], RasterSize -> 100, ImageSize -> 100]]]]; There is nothing really tricky in here. The text is "Wolfram", I rasterise the image and resize it. Then I binarise it. The ColorNegate is needed to exchange ones and zeros in the matrix. Because I play with images and matrices, and they have a different coordinate system (the origin is at different corners) I need to Reverse the whole thing. I can plot the result like so: ArrayPlot[Reverse@imgdata] ![enter image description here][4] I now need to generate a sound that produces high amplitudes at the right places in the Spectrogram. At each and every place in the imagedata matrix that contains a one I need to proceed a little sine wave with the right frequency. The rows correspond to frequencies and the rows to time. So lets collect the different frequencies at each time: list = Flatten[Position[#, 1]] & /@ Transpose[imgdata]; Ok. Next we generate the corresponding sums of Sin-functions and generate a list of data from them. listcompete = 0.1*Flatten[Table[Table[N@Total[Sin[2 Pi 300 # t] &@(# & /@ list)[[k]]], {t, 0, 8.71/200., 1/16000}], {k, 1, 100}], 1]; We can generate the corresponding sound: Audio[Sound[SampledSoundList[listcompete, 8000]]] This gives you a little window like this ![enter image description here][5] which allows you to "listen to Wolfram in dolphin language". The file is attached to this post and sounds really cool. How to hide the images in sound files/music ------------------------------------------- In order to hide the string in the "A small step ..." recording, we first shift the frequencies a little bit and decrease the amplitude: listcompete2 = 0.02*Flatten[Table[Table[N@Total[Sin[2 Pi 1200 # t] &@(# & /@ list)[[k]]], {t, 0, 8.71/100., 1/64000}], {k, 1, 100}], 1]; It looks now like this: Spectrogram[Sound[SampledSoundList[listcompete2, 64000]], ColorFunction -> "Rainbow"] ![enter image description here][6] If you want to listen to it use: Audio[Sound[SampledSoundList[listcompete2, 64000]]] You should hear a sort of hissing sound. The new function AudioChannelCombine will now help us to merge the two sound objects: AudioChannelCombine[{Audio[ExampleData[{"Sound", "Apollo11SmallStep"}]], Audio[Sound[SampledSoundList[listcompete2, 64000]]]}] The resulting object is attached to this post. The spectrogram Spectrogram[AudioChannelCombine[{Audio[ExampleData[{"Sound", "Apollo11SmallStep"}]], Audio[Sound[SampledSoundList[listcompete2, 64000]]]}], ColorFunction -> "Rainbow", Frame -> None] ![enter image description here][7] clearly shows the secret message blended into the Apollo message. Like this you can export the sound object: secsound = Import["/Users/thiel/Desktop/secretsound.wav"]; Look for hidden messages ------------------------ This type of [procedure is quite well known][8]. I also saw it on [BBC's Click][9], but I cannot remember the episode. Now you can use this to look for hidden messages in the internet or in music. Here are two examples: [file1][10] and [file2][11], both from the website above. Let's look at file1 Spectrogram[snd, 150, Frame -> None] ![enter image description here][12] and file2 Spectrogram[Import["http://www.evansalazar.com/ohmpie/imageEncode/evan.mp3"], 150, Frame -> None, AspectRatio -> 1] ![enter image description here][13] You can clearly see that the second image is actually "grayscale". It is quite possible to achieve this by not Binarize-ing the image and using the grayscale values as amplitudes. Where to go from here? ---------------------- We could certainly try to increase the resolution and get everything into a nice little function to do all the steps for us. Another interesting thing would be to add multiple images/slices of a 3 D image into sound, and basically hide an entire 3D object. I'd love to see this being 3D printed. Can you detect the [message in this sound file][14]? Cheers, Marco [1]: http://www.wolfram.com/language/11/computational-audio/?product=language [2]: https://www.dropbox.com/s/83788i811rnjn39/secretsound.wav?dl=0 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2016-09-01at00.21.25.png&userId=48754 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2016-09-01at00.27.40.png&userId=48754 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2016-09-01at00.33.34.png&userId=48754 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2016-09-01at00.39.48.png&userId=48754 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2016-09-01at00.43.10.png&userId=48754 [8]: http://www.ohmpie.com/imageencode/ [9]: http://www.bbc.co.uk/programmes/b006m9ry [10]: http://www.evansalazar.com/ohmpie/imageEncode/ohmpie.mp3 [11]: http://www.evansalazar.com/ohmpie/imageEncode/evan.mp3 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2016-09-01at00.52.56.png&userId=48754 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2016-09-01at00.51.55.png&userId=48754 [14]: https://www.dropbox.com/s/g6cqzute897szrz/finalmessage.wav?dl=0 Marco Thiel 2016-09-01T00:10:48Z https://community.wolfram.com/groups/-/m/t/1247481 I'm a Mathematica novice, so i think there is a lot of space to improve the code. Here is [Youtube Link][1].Hopefully you like it. First I use the Bach's manuscript as the music score, but the output sound is chaos around the middle part.This score fit to play with the instrument, obviously not Mathematica. ![Bach's manuscript][2] After posting this awful version to my Weibo, my friend told me there is a [Canon played by Matlab][3], it's so amazing, the most important its score suits for playing with software, so i decided to translate this score from Matlab to Mathematica. ## Matlab code ## There are 3 parts in Matlab code. 1.First, tell the computer how to make certain sounds. There is a part of the code in Matlab (I'm a Chinese student, hopefully my poor English didn't confuse you. I will attach some concepts' wikilink, sorry about that) % 1/4 notes do0f = mod4.*cos(2*pi*ScaleTable(21)*f0*t4); re0f = mod4.*cos(2*pi*ScaleTable(22)*f0*t4); mi0f = mod4.*cos(2*pi*ScaleTable(23)*f0*t4); % 1/8 notes fa0e = mod8.*cos(2*pi*ScaleTable(1)*f0*t8); so0e = mod8.*cos(2*pi*ScaleTable(2)*f0*t8); la0e = mod8.*cos(2*pi*ScaleTable(3)*f0*t8); % 1/16 notes fa0s = mod16.*cos(2*pi*ScaleTable(1)*f0*t16); so0s = mod16.*cos(2*pi*ScaleTable(2)*f0*t16); la0s = mod16.*cos(2*pi*ScaleTable(3)*f0*t16); For clear to see, I just pick few notes information i.$\frac{1}{4}$,$\frac{1}{8}$,$\frac{1}{16}$means [the relative duration of a note][4](https://en.wikipedia.org/wiki/Note_value), the duration of Quarter note is twice that of Eighth note, the duration of 8th note is twice that of 16th note. In the Matlab code, the duration depends on $t4,t8, or\space t16$ ii.Obviously, "do0f" means the note's tone. If you plot the the curve of middle A, it will be like that ![Middle A][5] Middle A's frequency "f" is 440Hz. In the Matlab code, the frequency depends on "$ScaleTable(\space)\times f0$" iii.mod is a modification function, it makes notes sound more soft. the number behind "mod" means it's duration ![mod*Middle A][6] So the Matlab code looks more clear! It becomes$$notename=mod4\times cos(2\pi f t)$$ This the way how to make sounds in Matlab. 2.Second, tell the computer musical score This part is easy to understand, just input the score. % Base Melody cello = [do1f do1f so0f so0f la0f la0f mi0f mi0f... fa0f fa0f do0f do0f fa0f fa0f so0f so0f];...... % Long Melody violin = [mi2f mi2f re2f re2f do2f do2f ti1f ti1f... la1f la1f so1f so1f la1f la1f ti1f ti1f ...... 3.Third, arrange the different instruments' sequence Canon is an very interesting music style, you just need to put the same melody at 3 different time points, and done! (I know I didn't explain it clearly, you can see [Canon's wiki link][7]) It looks beautiful in code % violin1 v1 = [blkblock violin blkblock blkblock]; % violin2 v2 = [blkblock blkblock violin blkblock]; % violin3 v3 = [blkblock blkblock blkblock violin]; % Combination s = c1+v1+v2+v3; sound(s,fs); ## Translate to Mathematica ## We get all information about all notes and musical score, but it's Matlab code, not Mathematica, we need use Mathematica code translate it.(The Mathematica file at the bottom of the article, it's a little bit long, so I decided not to paste it here) 1. First, make the association between the note name and sound respectively in Mathematica. The output should be like this. I named this association "nametosoundass", remember this name. ![Note to sound association][8] 2.Next, we should get the list of sequence of every instrument should be play. i.Input the raw sequence in Matlab to Mathematica, as you can see, there are many characters we don't want to get, such as "...", "%%", return and whitespace cellonoteinfo = "do1f do1f so0f so0f la0f%% la0f mi0f mi0f...fa0f%% fa0f do0f do0f fa0f \n fa0f so0f so0f"; ii.Define a function to delete these unexpected characters infotonote[info_] := StringPartition[StringJoin@StringCases[info, _?LetterQ | _?DigitQ], 4]; iii.Get the sequence In[1]:= cellonote = infotonote[cellonoteinfo] Out[1]:= {"do1f", "do1f", "so0f", "so0f", "la0f", "la0f", "mi0f", "mi0f", \ "fa0f", "fa0f", "do0f", "do0f", "fa0f", "fa0f", "so0f", "so0f"} 3.Third, replace the note name in sequence by sound, then we will get the sound in sequence. We use the association "notetosoundass" cello = Sound[Flatten@Table[Replace[cellonote, notetosoundass, 1], 23], {0}]; vio1 = Sound[Replace[violinnote, notetosoundass, 1], {8}]; vio2 = Sound[Replace[violinnote, notetosoundass, 1], {16}]; vio3 = Sound[Replace[violinnote, notetosoundass, 1], {24}]; 0,8,16,24 at the end of code means the insert time of different instruments. 4.Finally, combine it ![Completed Canon][9] You can enjoy it [here][10] BTW, this code runs 2 minutes to generate final music, i think it can be better, if you have another idea, glad to see your comment. Both Matlab and Mathematica code are below. [1]: https://www.youtube.com/watch?v=cVKsSd6Ji1E&t=4s [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Pachelbel%27s_Canon_-_Mus.MS_16481-8_Page_1.jpg&userId=1247466 [3]: https://www.bilibili.com/video/av10699321/?from=search&seid=84894214186229529 [4]: https://en.wikipedia.org/wiki/Note_value [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A72017-12-17%E4%B8%8B%E5%8D%886.08.48.png&userId=1247466 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A72017-12-17%E4%B8%8B%E5%8D%886.16.13.png&userId=1247466 [7]: https://en.wikipedia.org/wiki/Canon_%28music%29 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A72017-12-17%E4%B8%8B%E5%8D%886.54.08.png&userId=1247466 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A72017-12-17%E4%B8%8B%E5%8D%887.31.58.png&userId=1247466 [10]: https://www.youtube.com/watch?v=cVKsSd6Ji1E&t=4s QiuHao Chen 2017-12-17T07:15:02Z https://community.wolfram.com/groups/-/m/t/2328597 > **GitHub Repository:** https://github.com/alecGraves/Howl &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/da6d3b76-7bbf-4a55-9fcd-058e7f1e17bf Alec Graves 2021-07-27T14:09:12Z https://community.wolfram.com/groups/-/m/t/2228704 Introduction ------------------ This is a light-hearted little piece of code I wrote to help deal with questions like "Are we nearly there yet?" of young children when driving a car. Here is the idea. My daughter is in second class and has learnt to count, but it takes time for children to appreciate numbers and magnitudes. This is an important part of "[Computational Thinking][1]"; there is a sort of progression when children learn how to count and work with numbers. Based on my observation (and that is based on a sample of 2 or so), counting starts with very concrete objects like pieces of chocolate or building bricks or so. Only later will children learn to appreciate numbers as more abstract entities. It is typical that little children struggle to understand the magnitude of a number - they might be quite poor at estimating ages of people for a long time and putting numbers in perspective. ("How come that 1000$ is a lot, but 1000 grains of sand is too little to play with?") If I answer the question "Are we nearly there yet?" with "30 minutes from now." It doesn't necessarily mean much. Saying "one episode of your favourite program" appears to be a more meaningful answer. Now, we came up with this concept, where I basically tell my daughter to count to a certain number: Clara: "How long until my food is ready?" Me:"Count to 150." Now I could do this very simply. There are all these calculations like: "To what number could I count if I were counting at a rate of one number per second and were to count all my life?" ![enter image description here][2] gives 72.6yr, which I can then UnitConvert to seconds: UnitConvert[Quantity[72.6, "Years"], "Seconds"] That gives about 2.29 * 10^ 9 seconds and that would be what a person could count to if they start counting when they are born, and never sleep or do anything else, and get to say one number per second. Some of these assumptions might not be realistic. Coding actual counting ------------------------------- I was interested in the actual counting time, i.e. ignoring sleep, breaks etc, and just wondering how many numbers I could really say in a given time, if I didn't do anything else. So I could ask Mathematica to make a sound file and speaking numbers for me. Counting up to 10 (at a reasonable speed) would look like this: SpeechSynthesize[StringRiffle[StringReplace[IntegerName[#], "\[Hyphen]" -> " "] & /@ Range[10], ", "]] Using Duration, I can find out how long it takes to say this: Duration[SpeechSynthesize[StringRiffle[StringReplace[IntegerName[#], "\[Hyphen]" -> " "] & /@ Range[10], ", "]]] I obtain 6.45732 s, which is too high a precision for my taste, but gives me an idea of how long it takes to count up to 10. So it takes an average of 0.645732 seconds to say one of the numbers. As numbers become larger, it will take longer to say them. Let's count up to 100: Duration[SpeechSynthesize[StringRiffle[StringReplace[IntegerName[#], "\[Hyphen]" -> " "] & /@ Range[100], ", "]]] which gives me 1min 44.4055 seconds, which is about 1.046 seconds per number. Counting up to 1000 is still an easy thing to do: Duration[SpeechSynthesize[StringRiffle[StringReplace[IntegerName[#], "\[Hyphen]" -> " "] & /@ Range[1000],", "]]] gives 27 min 50.0569 seconds, which is on average 1.67007 seconds per number. Now I could produce longer and longer sound files, but if I count up to larger numbers that is not the most practical thing to do. So lets do a random sampling approach. If we want to estimate how long it takes to count to one million, I could ask Mathematica to choose 100 numbers and then scale up to 1.000.000: 1000000/100*Duration[SpeechSynthesize[StringRiffle[StringReplace[IntegerName[#], "\[Hyphen]" -> " "] & /@ RandomSample[Range[1000000], 100], ", "]]] Every time you run this you get slightly different results. Here are the results from 10 "realisations": ![enter image description here][3] Of course, I can get better estimates by sampling more than 100 numbers. I would also like to use mixed units so give a better idea of the times. Here is a simple piece of code to do this: Monitor[ results = Table[{k, UnitConvert[k/1000*Duration[SpeechSynthesize[ StringRiffle[StringReplace[IntegerName[#], "\[Hyphen]" -> " "] & /@ RandomSample[Range[k], 1000], ", "]]], MixedUnit[{"Years", "Months", "Days", "Hours", "Minutes", "Seconds"}]]}, {k, NestList[10*# &, 1000, 6]}], k] Not the most elegant piece of code, but it gets the job done: ![enter image description here][4] Other languages ----------------------- Of course, we can try this in different languages: Monitor[resultsgerman = Table[{k, UnitConvert[k/1000*Duration[SpeechSynthesize[ StringRiffle[IntegerName[#, Language -> "German"] & /@ RandomSample[Range[k], 1000], ", "], Language -> "German"]], MixedUnit[{"Years", "Months", "Days", "Hours", "Minutes", "Seconds"}]]}, {k, NestList[10*# &, 1000, 6]}], k] which gives: ![enter image description here][5] So it appears that Germans count a bit more slowly than English native speakers. Let's have a look at Spanish: Monitor[resultsspanish = Table[{k, UnitConvert[k/1000*Duration[SpeechSynthesize[ StringRiffle[IntegerName[#, Language -> "Spanish"] & /@ RandomSample[Range[k], 1000], ", "], Language -> "Spanish"]], MixedUnit[{"Years", "Months", "Days", "Hours", "Minutes", "Seconds"}]]}, {k, NestList[10*# &, 1000, 6]}], k] which gives: ![enter image description here][6] I tried some other languages, too, but that doesn't contribute a lot to this parenting problem. Higher resolution ---------------------- For the next application, I want some higher resolution data for English. Monitor[resultsEnglishHighRes = Table[{k, UnitConvert[k/Min[1000, k]* Duration[SpeechSynthesize[StringRiffle[StringReplace[IntegerName[#], "\[Hyphen]" -> " "] & /@ RandomSample[Range[k], Min[1000, k]], ", "]]], MixedUnit[{"Years", "Months", "Days", "Hours", "Minutes", "Seconds"}]]}, {k, Floor /@ Table[10^k, {k, 1, 9, 1/10}]}], k] This gives me a long table; I'll only show every second entry though: ![enter image description here][7] That's some serious counting going on right here.... If we plot this, we get: ListLogLogPlot[{#[[1]], QuantityMagnitude[UnitConvert[#[[2]], "Seconds"]]} & /@ resultsEnglishHighRes, PlotRange -> All, LabelStyle -> Directive[Bold, Medium], AxesLabel -> {"Count to", "Time in seconds"}, PlotStyle -> Red] ![enter image description here][8] We can also calculate the average time per number if we count up: averagetime={#[[1]], QuantityMagnitude[UnitConvert[#[[2]], "Seconds"]]/#[[ 1]]} & /@ resultsEnglishHighRes; ListLogLogPlot[averagetime, LabelStyle -> Directive[Bold, Medium], AxesLabel -> {"Count up to", "<time> per number in secs"}] ![enter image description here][9] Making this into parenting approved functions --------------------------------------------------------- Of course, these graphs tell me how long it takes to count up to a certain number, but they do not allow me to answer my daughter's questions.... So I need to build something to make predictions about how far she has to count until we arrive. We could do some simple curve fitting, but if we are too lazy (and I sure am), we can use Mathematica's more advanced features to get this done - quick and dirty. Let's try two things: (i) Let's predict how long we need to count to a certain number. (ii) Predict the number we have to count to if we need to count for a given time. Let's predict how long we need to count to a certain number ----------------------------------------- Easy peasy. Here's a predictor ptime = Predict[ Rule @@@ ({#[[1]], UnitConvert[#[[2]], "Seconds"]} & /@ resultsEnglishHighRes), PerformanceGoal -> "Quality"] It's ok, but it doesn't give us units so let's do this: UnitConvert[ptime[1000000], MixedUnit[{"Years", "Months", "Days", "Hours", "Minutes", "Seconds"}]] So this predicts how long I have to count up to a million. I get: 1 mo 11 days 18h 21 min 26.8809s As before, this won't be as precise as it pretends to be (!) and also is a wee bit different from my direct (but sampled) counting approach, which gave me: 1 mo 6 days 15h 32 min 21.4512 s But then it is hopefully going to be more precise than the "one-number-per-second" assumption. Predict the number we have to count to if we need to count for a given time -------------------- Luckily there is not much brain-CPU-time required for this either. preversetime = Predict[Rule @@@ ({QuantityMagnitude[UnitConvert[#[[2]], "Seconds"]], #[[1]]} & /@ resultsEnglishHighRes), PerformanceGoal -> "Quality"] We can then add units and use units etc as above. Application ------------- Ok, now here's an ***hypothetical*** application - or an experiment which is of course not currently possible due to COVID restrictions. [planning to drive from Aberdeen to Edinburgh in Scotland] Clara: "Are we nearly there yet?" Me: "No, we are still in front of our house." Clara: "What do I have to count to?" Me: [takes out phone, opens Wolfram Cloud website (see below)] : "Count to 3573." Crisis averted. Happy child, happy parent. Here's how it works: Ceiling[preversetime[UnitConvert[TravelTime[Entity["City", {"Aberdeen", "AberdeenCity", "UnitedKingdom"}], Entity["City", {"Edinburgh", "Edinburgh", "UnitedKingdom"}]], "Seconds"]]] Then you deploy that to the Cloud: CloudDeploy[FormFunction[{"from" -> "Location", "to" -> "Location"}, Ceiling[preversetime[UnitConvert[TravelTime[#from, #to], "Seconds"]]] &], Permissions -> "Public"] and get a link to a website. Good times! Conclusion -------------- There are lots of things that can be improved and should be checked. All numbers were computed using a random subsample approach. So we would need to know error bars etc. That can be done, but is out of the scope of this post. We can run some easy tests though. For example: TravelTime[ Entity["City", {"Aberdeen", "AberdeenCity", "UnitedKingdom"}], Entity["City", {"Edinburgh", "Edinburgh", "UnitedKingdom"}]] Gives 2 h 11 mins. The predictor suggests that I should count to 3573. Well, let's quickly do that and see how long it takes: UnitConvert[Duration[SpeechSynthesize[StringRiffle[StringReplace[IntegerName[#], "\[Hyphen]" -> " "] & /@ Range[3573], ", "]]],MixedUnit[{ "Hours", "Minutes", "Seconds"}]] That gives 2 h 6 mins 24.2026s (with overwhelming precision!). Not too bad. The code has not been properly beta-tested and there might be bugs. I have not tried to count up to 1000000000 or so to check the results; this is basically due to my very limited life span. Also, I did not even try to make the code nice or efficient. It was basically just typing it in and getting it done - so there are lots of improvements to be made there. Of course, the cloud version can be adjusted to estimate the number to count to for any given period of time so that it becomes more useful for everyday parenting. All the best from Scotland, Marco [1]: https://www.computerbasedmath.org [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Averagelife.png&userId=48754 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2021-03-25at13.57.20.png&userId=48754 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2021-03-25at14.00.54.png&userId=48754 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2021-03-25at14.05.50.png&userId=48754 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2021-03-25at14.19.14.png&userId=48754 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2021-03-25at14.29.32.png&userId=48754 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2021-03-25at14.32.56.png&userId=48754 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2021-03-25at14.35.46.png&userId=48754 [10]: https://www.wolframcloud.com/obj/3c7431be-0c82-477f-8ede-f3ba1eb8d0c4 Marco Thiel 2021-03-25T15:11:00Z https://community.wolfram.com/groups/-/m/t/2386951 &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/a5b26a59-e94c-4335-a06d-be68868e3a88 [Original NB]: https://www.wolframcloud.com/obj/169c961e-e735-4ced-bafe-8b2f7440e60a James Lu 2021-10-18T02:02:33Z https://community.wolfram.com/groups/-/m/t/1383630 During this year's Wolfram Summer Camp, being mentored by Christian Pasquel, I developed a tool that identifies chord sequences in music (from MIDI files) and generates a corresponding graph. The graph represents all [unique] chords as vertices, and connects every pair of chronologically subsequent chords with a directed edge. Here is an example of a graph I generated: ![Graph genehrated from Bach's prelude no.1 of the Well Tempered Klavier (Book I)][1] Below is a detailed account on the development and current state of the project, plus some background on the corresponding musical theory notions. #Introduction       **GOAL** | The aim of this project is to develop a utility that identifies chords (e.g. C Major, A minor, G7, etc.) from MIDI files, in chronological order, and then generates a graph for visualizing that chord sequence. In the graph, each vertex would represent a unique chord, and each pair chronologically adjacent chords would be connected by a directed edge (i.e. an arrow). So, for example, if at some point in the music that is being analyzed there is a transition from a major G chord to a major C chord, there would be an arrow that goes from the G Major chord to the C Major chord. Therefore, the graph would describe a [Markov chain][2] for the chords. The purpose of the graph is to visualize frequent chord sequences and progressions within a certain piece of music.       **MOTIVATION** | While brainstorming for project ideas, I don't know why, I had a desire to do something with graphs. Then I asked myself, "What are graphs good at modelling?". I mentally browsed through my areas of interest, searching for any that matched that requirement. One of my main interests is music; I am somewhat of a musician myself. And, in fact, [musical] harmony *is* a good subject to be modelled by graphs. Harmony, one of the fundamental pillars of music (and perhaps the most important), not only involves the chords themselves, but, more significantly, the *transitions* between those, which is what gives character to music. And directed graphs, and, specifically, Markov models, are a perfect match for transitions between states. ---------- #Some background *Skip this if you aren't interested in the musical theory part or if you already have a background in music theory!* ##What is a chord? A chord is basically a group of notes played together (contemporarily). Chords are the quanta of musical "feeling"; the typical—but somewhat naïve—example is the sensation of major chords sounding "happy" and minor chords sounding "sad" or melancholic (more on types of chords later). Types of chords are defined by the [intervals][3] (distance in pitch) between the notes. The *root* of a chord is the "most important" or fundamental note of the cord, in the sense that it is the "base" from which the aforementioned intervals are measured. In other words, the archetype of chord defines the "feel" and the general harmonic properties of the chord, while the root defines the pitch of the chord. So a "C Major" chord is a chord with archetype "major triad" (more on that later) built on the note C; i.e., its root is C. The *sequence* of chords in a piece constitutes its **harmony**, and it can convey much more complex musical messages or feelings than a single chord, just as in language: a single word does have meaning, but a sentence can have a much more complex meaning than any single word. ##Patterns in chord sequences The main difference that between language and music is that language, in general, has a much stricter structure (i.e. the order of words, a.k.a. syntax) than music: the latter is an art, and there are no predetermined rules to follow. But humans \[have a tendency to\] like patterns, and music wouldn't be so universally beloved if it didn't contain any patterns. This also explains the unpopularity of [atonal music][4] (example [here][5]). But even atonal music has patterns: it may do its best to avoid harmonic patterns, but it still contains some level of rythmic, textural or other kinds of patterns. This is why using graphs to visualize chord sequences is interesting: it is a semidirect way of identifying the harmonic patterns that distinguish different genres, styles, forms, pieces or even fragments of music. In my project, I have mainly focused on the "western" conception of tonal music, an particularly in its "classical" version (what I mean by "classical" is, in lack of a better definition, a classification that encompasses all music where the composer is, culturally, the most important artist). That doesn't mean this tool isn't apt for other types of music; it just means it will analyze it from this specific standpoint. In tonal music, the harmonic patterns are all related to a certain notion of "center of gravity": the [*tonic*][6], which is, in some way the music's harmonic "home". Classical (as in pre-XX-century) tonal music usually ends (and often starts) with the tonic chord. In fact, we can further extend the analogy with gravity by saying that music consists in a game of tension, in which the closer you are to the center of gravity (the tonic), the greater the "pull". In an oversimplified manner, the musical equivalent of the [Schwarzschild radius][7] is the [dominant chord][8]: it tends towards the tonic. Well, not really, because you *can* turn back from it—and in fact a lot of interesting harmonical sequences consist in doing just that. ##Some types of chords In "classical" music (see definition above), there are mainly these kinds of chords (based on the amount of unique notes they contain): triad chords (i.e. three-note chords), seventh chords (i.e. four-note chords; we'll see why they're called *seventh* in a bit), and ninth chords (five-note chords). There is another main distinction: major and minor chords (i.e. the cliché "happy" vs "sad" distinction). ###Triad chords Probably the most simple and frequent chord is the triad chord (either major or minor). Here is a picture of a major and a minor triad C chord (left to right): ![Major and minor triad C chords (ltr)][9] ###Seventh chords [Seventh chords][10] are called so because they contain a seventh [interval][11]. Their main significance is in dominant chords, where they usually appear in the major-triad-minor-seventh (a.k.a ["dominant"][12]) form. Another important seventh chord form is the fully diminished seventh chord (these will be relevant for the code later), which also tends to resolve ("resolve" is music jargon for "transition to a chord with less tension") to tonic. ![Seventh chords][13] ###Ninth chords Although not extremely frequent, they do appear in classical music. The most "popular" is the dominant ninth chord (an extension of the dominant 7th). An alternative for this chord is the minor ninth dominant chord (built from the same dominant 7th chord, but with a minor ninth instead). <br> ---------- #Algorithms and Code In this section I'm going to walk through my code in order of execution. Four main parts can be distinguished in my project: importing and preprocessing, splitting the note sequence into "chunks" to be analyzed as chords, identifying the chord in each of those chunks, and visualizing the whole sequence as a graph. ##First phase: importing and preprocessing the MIDI file The first operation that needs to be done is importing the MIDI file and preprocessing it. This includes selecting which elements to import from the file, converting them to a given simplified form, and performing any sorting, deletion of superfluous elements, or other modification that needs to be done. For this purpose I defined the function `importMIDI`: importMIDI[filename_String] := MapAt[Flatten[#, 1] &, MapAt[flattenAndSortSoundNotes, Import[(dir <> filename <> ".mid"), {{"SoundNotes", "Metadata"}}], 1], 2] Here `dir` stands for the directory where I saved all my MIDIs (to avoid having to type in the whole directory every time). Notice that we're importing the music as SoundNotes *and* the file's metadata—we will need it for determining the boundaries of measures (see below). The function `flattenAndSortNotes` does what it sound like: it converts the list of `SoundNote`s that `Import` returned into a flattened list of notes (i.e. a single track), sorted by their starting time. It also gets rid of anything that isn't necessary for chord identification (i.e. rhythmic sounds or effects). Consult the attached notebook for the explicit definition. Here is the format the sequence of notes is returned in (i.e. `importMIDI[...][[1]]`): {{"C4", {0., 1.4625}}, {"E4", {0.18125, 1.4625}}, {"G4", {0.36875, 0.525}}, <<562>>, {"G2", {105., 107.963}}, {"G4", {105., 107.963}}} Each sub-list represents a note. Its first element is the actual pitch; the second is a list that represents the timespan (i.e. start and end time in seconds). <br> ##Second phase: splitting the note sequence into chunks The challenge in this part of the project is knowing how to determine which notes form a single chord; i.e., where to put the boundary between one chord and the next. The solution I came up with is not optimal, but, until now, nothing better has occurred to me (suggestions are welcome!). It involves determining where each measure start/end lies in time from the metadata and splitting each of those into a certain amount of sub-parts; then the notes are grouped by the specific sub-part of the specific measure they pertain to. The rationale behind this is that chords in classical music tend to be well-contained within measures or rational fractions of these. This procedure is contained in the function `chordSequenceAnalyzeUsingMeasures`. I'm going to go over it quickly: chordSequenceUsingMeasures[midiData_List /; Length@midiData == 2, measureSplit_: 2, analyzer_String: "Heuristic"] := Block[{noteSequence, metadata, chunkKeyframes, chunkedSequence, result}, (*Separate notes from metadata*) noteSequence = midiData[[1]]; metadata = midiData[[2]]; Until here it's pretty self evident. (*Get measure keyframes*) chunkKeyframes = divideByN[ measureKeyframesFromMetadata[ metadata, (Last@noteSequence)[[2, 2]]], measureSplit]; Here the function `measureKeyframesFromMetadata` is called. It fetches all of the `TimeSignature` and `SetTempo` tags in the metadata and identifies the position of each measure from them. `divideByN` subdivides each measure by `measureSplit` (an optional argument with default value `2`). (*Chunk sequence*) chunkedSequence = {}; Module[{i = 1}, Do[ With[{k0 = chunkKeyframes[[j]], k1 = chunkKeyframes[[j + 1]]}, Module[{chunk = {}}, While[ i <= Length@noteSequence && ( k0 <= noteSequence[[i, 2, 1]] < k1 || k0 < noteSequence[[i, 2, 2]] <= k1 ), AppendTo[chunk, noteSequence[[i]]] i++;]; AppendTo[chunkedSequence, chunk] ] ], {j, Length@chunkKeyframes - 1}]]; chunkedSequence = DeleteCases[chunkedSequence, l_List /; Length@l == 0]; Once the measures' timespan has been determined, a list of "chunks" (lists of notes grouped by measure part) is generated. (*Call analyzer*) Switch[analyzer, "Deterministic", result = chordChunkAnalyze /@ chunkedSequence, "Heuristic", result = heuristicChordAnalyze /@ justPitch /@ chunkedSequence ]; result = resolveDiminished7th[Split[result][[All, 1]]] ] Finally, each chunk is sent to the chord analyzer function `heuristicChordAnalyze`, which I'll talk about in the next section, along with the currently mysterious `resolveDiminished7th`. Since this algorithm for "chunking" a note sequence doesn't work for everything, I also developed an alternative, more naïve approach: chordSequenceNaïve[midiData_List /; Length@midiData == 2, analyzer_String: "Heuristic", n1_Integer: 6, n2_Integer: 1] := Module[{noteSequence, chunkedSequence, result}, (*Separate notes from metadata*) noteSequence = midiData[[1]]; (*Chunk sequence*) chunkedSequence = Partition[noteSequence, n1, n2]; (*Call analyzer*) result = heuristicChordAnalyze /@ justPitch@chunkedSequence; result = resolveDiminished7th[Split[result][[All, 1]]] ] <br> ##Phase 3: identifying the chord from a group of notes This has been the main conceptual challenge in the whole project. After some unsucsessful ideas, with some suggestions from Rob Morris (one of the mentors), whom I thank, I ended up developing the following algorithm. It iterates through each note and assigns it a score that represents the likeliness of that note being the root of the chord based on the presence of certain indicators (i.e. notes the presence of which define a chord, to some degree), each of which with a different weight: having a fifth, having a third, a minor seventh... Then the note with the highest chord is assumed to be the root of the chord. In code: heuristicChordAnalyze[notes_List] := Block[{chordNotes, scores, root}, (*Calls to helper functions*) chordNotes = octaveReduce /@ convertToSemitones /@ notes // DeleteDuplicates; (*Scoring*) scores = Table[Total@ Pick[ (*Score points*) {24, 16, 16, 8, 2, 3, 1, 1, 10, 15, 15, 18}, (*Conditions*) SubsetQ[chordNotes, #] & /@octaveReduce /@ {{nt + 7}, {nt + 4}, {nt + 3}, {nt + 10}, {nt + 11}, {nt + 2}, {nt + 5}, {nt + 9}, {nt + 4, nt + 10}, {nt + 3, nt + 6, nt + 10}, {nt + 3, nt + 6, nt + 9}, {nt + 1, nt + 4, nt + 10}} ] (*Substract outliers*) - 18*Length@Complement[chordNotes, octaveReduce /@ {nt, 7 + nt, 4 + nt, 3 + nt, 10 + nt, 11 + nt, 2 + nt, 5 + nt, 9 + nt, 6 + nt}], {nt, chordNotes}]; (*Return*) root = Part[chordNotes, Position[scores, Max @@ scores][[1, 1]]]; {root, Which[ SubsetQ[chordNotes, octaveReduce /@ {root + 10 , root + 2, root + 5, root + 9}], "13", SubsetQ[chordNotes, octaveReduce /@ {root + 10, root + 2, root + 5}], "11", SubsetQ[chordNotes, octaveReduce /@ {root + 4, root + 10, root + 2}], "Dom9", SubsetQ[chordNotes, octaveReduce /@ {root + 4, root + 10, root + 1}], "Dom9m", SubsetQ[chordNotes, octaveReduce /@ {root + 11, root + 7, root + 3}], "m7M", SubsetQ[chordNotes, {octaveReduce[root + 11], octaveReduce[root + 4]}], "7M", SubsetQ[chordNotes, {octaveReduce[root + 10], octaveReduce[root + 4]}], "Dom7", SubsetQ[chordNotes, {octaveReduce[root + 10], octaveReduce[root + 7]}], "Dom7", SubsetQ[chordNotes, {octaveReduce[root + 10], octaveReduce[root + 6]}], "d7", SubsetQ[ chordNotes, {octaveReduce[root + 9], octaveReduce[root + 6]}], "d7d", SubsetQ[chordNotes, {octaveReduce[root + 10], octaveReduce[root + 3]}], "m7", MemberQ[chordNotes, octaveReduce[root + 4]], "M", MemberQ[chordNotes, octaveReduce[root + 3]], "m", MemberQ[chordNotes, octaveReduce[root + 7]], "5", True, "undef"]} ] ###A note on notation In this project I use the following abbreviations for chord notation (they're not in the standard format). "X" represents the root of the chord. - *X-**5*** = undefined triad chord (just the root and the fifth) - *X-**M*** = Major - *X-**m*** = minor - *X-**m7*** = minor triad with minor (a.k.a dominant) seventh - *X-**d7d*** = fully diminished 7thchord - *X-**d7*** = half diminished 7thchord - *X-**Dom7*** = Dominant 7th chord - *X-**7M*** = Major triad with Major 7th - *X-**m7M*** = minor triad with Major 7th - *X-**Dom9*** = Dominant 9th chord - *X-**Dom9m*** = Dominant 7th chord with a minor 9th - *X-**11*** = 11th chord - *X-**13*** = 13th chord ###Dealing with diminished 7th chords Now, on to `resolveDiminished7th`. What is this function on about? Well, recall the fully diminished seventh chords I mentioned in the Background section. Here's the problem: they're completely symmetrical! What I mean by that is that the intervals between subsequent notes are identical, even if you [invert][14] the chord. In other words, the distance in semitones between notes is constant (it's 3) and is a factor of 12 (distance of 12 semitones = octave). So, given one of these chords, there is no way to determine which note is the root just by analyzing the chord itself. In the context of our algorithm, every note would have the same score! At this point I thought: "How do humans deal with this?". And I concluded that the only way to resolve this issue is to have some contextual vision (looking at the next chord, particularly), which is how humans do it. So what `resolveDiminished7th` does is it brushes through the chord sequence stored in `result`, looking for fully diminished chords (marked with the string "d7d"), and re-assigns each of those a root by looking at the next chord: resolveDiminished7th[chordSequence_List] := Module[{result}, result = Partition[chordSequence, 2, 1] /. {{nt_, "d7d"}, c2_List} :> Which[ MemberQ[octaveReduce /@ {nt, nt + 3, nt + 6, nt + 9}, octaveReduce[c2[[1]] - 1]], {{c2[[1]] - 1, "d7d"}, c2}, MemberQ[octaveReduce /@ {nt, nt + 3, nt + 6, nt + 9}, octaveReduce[c2[[1]] + 4]], {{c2[[1]] + 4, "d7d"}, c2}, MemberQ[octaveReduce /@ {nt, nt + 3, nt + 6, nt + 9}, octaveReduce[c2[[1]] + 6]], {{c2[[1]] + 6, "d7d"}, c2}, True, {{nt, "d7d"}, c2}]; result = Append[result[[All, 1]], Last[result][[2]]] ] ##Phase 4: Visualization Basically, my visualization function (`visualizeChordSequence`) is fundamentally a highly customized call of the `Graph` function; so I'll just paste the code below and then explain what some parameters do: visualizeChords[chordSequence_List, layoutSpec_String: "Unspecified", version_String: "Full", mVSize_: "Auto", simplicitySpec_Integer: 0, normalizationSpec_String: "Softmax"] := Module[{purgedChordSequence, chordList, transitionRules, weights, graphicalWeights, nOfCases, edgeStyle, vertexLabels, vertexSize, vertexStyle, vertexShapeFunction, clip}, (*Preprocess*) Switch[version, "Full", purgedChordSequence = StringJoin[toNoteName[#1], "-", #2] & @@@ chordSequence, "Basic", purgedChordSequence = Split[toNoteName /@ chordSequence[[All, 1]]][[All, 1]]]; (*Amount of each chord*) chordList = DeleteDuplicates[purgedChordSequence]; nOfCases = Table[{c, Count[purgedChordSequence, c]}, {c, chordList}]; (*Transition rules between chords*) Switch[version, "Full", transitionRules = Gather[Rule @@@ Partition[purgedChordSequence, 2, 1]], "Basic", transitionRules =(*DeleteCases[*) Gather[Rule @@@ Partition[purgedChordSequence, 2, 1]](*, t_/; Length@t\[LessEqual]2]*) ]; (*Get processed weight for each transition*) weights = Length /@ transitionRules; If[normalizationSpec == "Softmax", graphicalWeights = SoftmaxLayer[][weights]];; graphicalWeights = If[Min@graphicalWeights != Max@graphicalWeights, Rescale[graphicalWeights, MinMax@graphicalWeights, {0.003, 0.04}], graphicalWeights /. _?NumericQ :> 0.03 ]; (*Final transition list*) transitionRules = transitionRules[[All, 1]]; (*Graph display specs*) clip = RankedMax[weights, 4]; edgeStyle = Table[(transitionRules[[i]]) -> Directive[Thickness[graphicalWeights[[i]]], Arrowheads[2.5 graphicalWeights[[i]] + 0.015], Opacity[Which[ weights[[i]] <= Clip[simplicitySpec - 2, {0, clip - 2}], 0, weights[[i]] <= Clip[simplicitySpec, {0, clip}], 0.2, True, 0.6]], RandomColor[Hue[_, 0.75, 0.7]], Sequence @@ If[weights[[i]] <= Clip[simplicitySpec - 1, {0, clip - 1}], { Dotted}, {}] ], {i, Length@transitionRules}]; vertexLabels = Thread[nOfCases[[All, 1]] -> (Placed[#, Center] & /@ (Style[#[[1]], Bold, Rescale[#[[2]], MinMax[nOfCases[[All, 2]]], Switch[mVSize, "Auto", {6, 20}, _List, 10*mVSize[[1]]/0.3*{1, mVSize[[2]]/mVSize[[1]]}]]] & /@ nOfCases))]; vertexSize = Thread[nOfCases[[All, 1]] -> Rescale[nOfCases[[All, 2]], MinMax[nOfCases[[All, 2]]], Switch[mVSize, "Auto", (Floor[Length@chordList/10] + 1)*{0.1, 0.3}, _List, mVSize]]]; vertexStyle = Thread[nOfCases[[All, 1]] -> Directive[Hue[0.53, 0.27, 1, 0.6], EdgeForm[Blue]]]; vertexShapeFunction = Switch[version, "Full", Ellipsoid[#1, {3.5, 1} #3] &, "Basic", Ellipsoid[#1, {2, 1} #3] &]; Graph[transitionRules, GraphLayout -> Switch[layoutSpec, "Unspecified", Automatic, _, layoutSpec], EdgeStyle -> edgeStyle, EdgeWeight -> weights, VertexLabels -> vertexLabels, VertexSize -> vertexSize, VertexStyle -> vertexStyle, VertexShapeFunction -> vertexShapeFunction, PerformanceGoal -> "Quality"] ] There are five main things to focus on in the above definition: the graph layout (passed as the argument `layoutSpec`), the edge thickness (defined in `edgeStyle`), the vertex size (defined in `vertexSize`), the version (passed as argument `version`) and the simplicity specification (`simplicitySpec`). The graph layout is a `Graph` option that can be specified in the argument `layoutSpec`. If `"Unspecified"` is passed, an automatic layout will be used. I find that the best layouts tend to be, in order of preference, "BalloonEmbedding" and "RadialEmbedding"; nevertheless, neither are a perfect fit for every piece. In the future I would like to to implement custom (i.e. pre-defined) positioning, so that I can design it in a way that best fits this project. The edge thickness is a function of the amount of times a certain transition between two chords has occurred in the chord sequence. There is an option (namely the `normalizationSpec` argument) to enable or disable using a Softmax function for assigning thicknesses to edges. This is due to the fact that for simple/short chord sequences, Softmax is actually counterproductive because it suppresses secondary but still top-ranked transitions; i.e., it assigns a very high thickness to the most frequent transition and a low thickness to all other transitions (even those that come in second or third in frequency ranking). But for large or complex sequences it is actually useful, because it "gets rid of" a lot of the \[relatively\] insignificant instances, thus making the output actually understandable (and not just a [jumbled mess of thick lines][15]). The vertex size is proportional to the number of occurrences of each particular chord (that is, without taking into account the transitions). It can also be specified manually by passing `vSize` as a list `{a,b}` such that `a` is the minimum size an `b` is the maximum. The `version` can be either `"Full"` or `"Basic"`; the default is `"Full"`. The `"Basic"` version consists of a simplified chord set in which only the root note of the chord is taken into account, and not the archetype. For example, all C chords (M, Dom7, m...) would be represented by a single `"C"` vertex. Finally, the simplicity specification (`simplicitySpec`) is a number that can be thought of, in some way, as a "noise" threshold: as it gets larger, fewer edges "stand out"—that is, more of the lower-significance ones are rendered with reduced opacity or are shown as dotted lines. This is useful for large or complex sequences. <br> ---------- #Some examples Here I will show some specific examples generated with this tool. I tried to use different styles of music for comparison. - **Bach**'s [prelude no.1][16] from the Well Tempered Clavier: ![Visualization of Bach's prelude no.1 ][17] - **Debussy**'s [*Passepied*][18] from the *Suite Bergamasque*: ![Visualization of Debussy's *Passepied*][19] - A "template" blues progression: ![Blues template][20] - **Beethoven**'s second movement from the *Pathétique* sonata (no.8): ![Beethoven][21] - Any "reggaeton" song (e.g. Despacito): ![Reggaeton][22] #Microsite Check out the form page (a.k.a. microsite) of this project [here][23]: https://www.wolframcloud.com/objects/lammenspaolo/Chord%20sequence%20visualization [![enter image description here][24]][23] Briefly, here is what each option does (see the section **Algorithms and code** for a more detailed explanation): - **Chunkifier funciton**: choose between splitting notes by measures or by a constant amount of notes - **Measure split factor**: choose into how many pieces you want to divide measures (each piece will be analyzed as a separate chord) - **Graph layout**: choose the layout option for the `Graph` call - **Normalization function**: choose whether to apply a Softmax function to the weights of edges (to make results clearer in case of complex sequences). - **Version**: choose "Full" for complete chord info (e.g. "C-M", "D-Dom7", "C-7M"...) or "Basic" for just the root of the chord (e.g. "C", "D"...) - **Vertex size**: specify vertex size as a list `{a,b}` where `a` is the minimum and `b` is the maximum size - **Simplicity parameter**: visual simplification of the graph (a value of 0 means no simplification is applied) <br> #Conclusions I have developed a functional tool to visualize chord sequences as graphs. It is far from perfect, though. In the future, I would like improving the positioning of vertices, being able to eliminate insignificant transitions from the graph altogether, and making other visual adjustments. Furthermore, I plan to refine and optimize the chord analyzer, as right now it is just an experimental version that isn't too accurate. A better "chunkifier" function could be developed too. Finally, I'd like to thank my mentor Christian Pasquel and all of the other WSC staff for this amazing opportunity. I'd also like to thank my music theory teacher, Raimon Romaní, for making me, over the years, sufficiently less terrible at musical analysis to be able to undertake this project. [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Prelude.png&userId=1372342 [2]: https://en.wikipedia.org/wiki/Markov_chain "Wikipedia: Markov chain" [3]: https://en.wikipedia.org/wiki/Interval_(music) "Wikipedia: Interval" [4]: https://en.wikipedia.org/wiki/Atonality "Wikipedia: Atonality" [5]: https://youtu.be/L85XTLr5eBE "Schönberg's 4th string quartet on YouTube" [6]: https://en.wikipedia.org/wiki/Tonic_%28music%29 "Wikipedia: Tonic" [7]: http://astronomy.swin.edu.au/cosmos/S/Schwarzschild+Radius "Basic info on Schwartzschild radius" [8]: https://en.wikipedia.org/wiki/Dominant_(music) "Dominant chord" [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=2548Macro_analysis_chords_on_C.jpg&userId=1372342 [10]: https://en.wikipedia.org/wiki/Seventh_chord "Wikipedia: Seventh chord" [11]: https://en.wikipedia.org/wiki/Interval_(music) "Wikipedia: Interval" [12]: https://en.wikipedia.org/wiki/Dominant_seventh_chord "Wikipedia: Dominant seventh" [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=images.png&userId=1372342 [14]: https://en.wikipedia.org/wiki/Inversion_(music)#Chords "Wikipedia: Inversion#Chords" [15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Passepied.png&userId=1372342 "Jumbled mess!" [16]: https://www.youtube.com/watch?v=aengbLEFnM8 [17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Prelude.png&userId=1372342 [18]: https://www.youtube.com/watch?v=hDWbVP-5DSA "Passepied" [19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=deb_pass2.png&userId=1372342 [20]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Blues.png&userId=1372342 [21]: http://community.wolfram.com//c/portal/getImageAttachment?filename=pathetique.png&userId=1372342 [22]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Reggaeton.png&userId=1372342 [23]: https://www.wolframcloud.com/objects/lammenspaolo/Chord%20sequence%20visualization "Microsite" [24]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-07-19at1.53.02PM.png&userId=11733 Paolo Lammens 2018-07-14T05:10:03Z https://community.wolfram.com/groups/-/m/t/3299985 ![Signals, systems, and signal processing: a computational approach][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Main18102024.png&userId=20103 [2]: https://www.wolframcloud.com/obj/9fef1db9-ca5d-4080-9db5-fefdbb4d3085 Mariusz Jankowski 2024-10-17T21:55:31Z