https://community.wolfram.com RSS Feed for Wolfram Community showing any discussions tagged with Image Processing sorted by most likes. https://community.wolfram.com/groups/-/m/t/2593151 **Introducing our brand new YouTube channel, [Wolfram R&D][1]! Our channel features livestreams, behind-the-scenes creator presentations, insider videos, and more.** ---------- Join us for the unique Wolfram R&D livestreams on [Twitch][2] and [YouTube][3] led by our developers! You will see **LIVE** stream indicators on these channels on the dates listed below. The live streams provide tutorials and behind the scenes look at Mathematica and the Wolfram Language directly from developers. Join our livestreams every Wednesday at 11 AM CST and interact with developers who work on data science, machine learning, image processing, visualization, geometry, and other areas. ---------- ⭕ **UPCOMING** EVENTS - Jan 29 -- Reinforcement Learning Applied to Feedback Control with [Suba Thomas][61] ---------- ✅ **PAST** EVENTS - April 24 -- [FeynCalc][60] - April 3 -- [Explore the Total Solar Eclipse of April 2024][59] - Mar 22 -- [20 Years of xAct Tensor Computer Algebra][58] - Feb 28 -- [Zero Knowledge Authentication][57] - Jan 17 -- [Nutrients by the Numbers][56] - Dec 13 -- [Understanding Graphics][55] - Oct 18 -- [Overview of Number Theory][54] - Sep 27 -- [QMRITools: Processing Quantitative MRI Data][52] - Sep 13 -- [Make High Quality Graph Visualization][51] - Sep 6 -- [Insider's View of Graphs & Networks][53] - Aug 30 -- [Labeling Everywhere][49] - Aug 22 -- [Equation Generator for Equation-of-Motion Coupled Cluster Assisted by CAS][48] - Aug 16 -- [Foreign Function Interface][4] - July 26 -- [Modeling Fluid Circuits][6] - July 19 -- [Geocomputation][5] - July 5 -- [Protein Visualization][7] - Jun 14 -- [Chat Notebooks bring the power of Notebooks to LLMs][8] - May 31-- [Probability and Statistics: Random Sampling][9] - May 24 -- [Problem Solving][10] - May 17 -- [The state of Optimization][11] - May 10 -- [Building a video game with Wolfram notebooks][12] - April 26 -- [Control Systems: An Overview][13] - April 19 -- [MaXrd: A crystallography package developed for research support][14] - April 5th -- [Relational database in the Wolfram Language][15] - Mar 29th -- [Build your first game in the Wolfram Language with Unity game engine] [16] - Mar 22nd -- [Everything to know about Mellin-Barnes Integrals - Part II][17] - Mar 15th -- [Building your own Shakespearean GPT - a ChatGPT like GPT model][18] - Mar 8th -- [Understand Time, Date and Calendars][19] - Mar 1st -- [Introducing Astro Computation][20] - Feb 22nd -- [Latest features in System Modeler][21] - Feb 15th -- [Everything to know about Mellin-Barnes Integrals][22] - Feb 8th -- [Dive into Video Processing][23] - Feb 1st -- [PDE Modeling][24] - Jan. 25th -- [Ask Integration Questions to Oleg Marichev][25] - Jan. 18th -- [My Developer Tools][26] - Jan. 11th -- [Principles of Dynamic Interfaces][27] - Dec. 14th -- [Wolfram Resource System: Repositories & Archives][28] - Dec. 7th -- [Inner Workings of ImageStitch: Image Registration, Projection and Blending][29] - Nov. 30th -- [Q&A for Calculus and Algebra][30] - Nov. 23rd -- [xAct: Efficient Tensor Computer Algebra][31] - Nov. 16th -- [Latest in Machine Learning][32] - Nov. 9th -- [Computational Geology][33] - Nov. 2nd -- [Behind the Scenes at the Wolfram Technology Conference 2022][34] - Oct 26th -- [Group Theory Package (GTPack) and Symmetry Principles in Condensed Matter][35] - Oct 12th -- [Tree Representation for XML, JSON and Symbolic Expressions][36] - Oct. 5th -- [A Computational Exploration of Alcoholic Beverages][37] - Sept. 28th -- [Q&A with Visualization & Graphics Developers][38] - Sept. 14th -- [Paclet Development][39] - Sept. 7th -- [Overview of Chemistry][40] - Aug. 24th -- [Dive into Visualization][41] - Aug. 17th -- [Latest in Graphics & Shaders][42] - Aug. 10th -- [What's new in Calculus & Algebra][43] > **What are your interests? Leave a comment here on this post to share your favorite topic suggestions for our livestreams.** **Follow us on our live broadcasting channels [Twitch][44] and [YouTube][45] and for the up-to-date announcements on our social media: [Facebook][46] and [Twitter][47].** [1]: https://wolfr.am/1eatWLcDA [2]: https://www.twitch.tv/wolfram [3]: https://wolfr.am/1eatWLcDA [4]: https://www.youtube.com/watch?v=C82NHpy7D6k [5]: https://community.wolfram.com/groups/-/m/t/2985580 [6]: https://community.wolfram.com/groups/-/m/t/2982197 [7]: https://community.wolfram.com/groups/-/m/t/2982114 [8]: https://youtu.be/ZqawtrWwE0c [9]: https://community.wolfram.com/groups/-/m/t/2946101 [10]: https://community.wolfram.com/groups/-/m/t/2925156 [11]: https://community.wolfram.com/groups/-/m/t/2921756 [12]: https://community.wolfram.com/groups/-/m/t/2918746 [13]: https://community.wolfram.com/groups/-/m/t/2917597 [14]: https://community.wolfram.com/groups/-/m/t/2911327 [15]: https://community.wolfram.com/groups/-/m/t/2907390 [16]: https://community.wolfram.com/groups/-/m/t/2921593 [17]: https://community.wolfram.com/groups/-/m/t/2861119 [18]: https://community.wolfram.com/groups/-/m/t/2847286 [19]: https://community.wolfram.com/groups/-/m/t/2851575 [20]: https://community.wolfram.com/groups/-/m/t/2852934 [21]: https://community.wolfram.com/groups/-/m/t/2842136 [22]: https://community.wolfram.com/groups/-/m/t/2838335 [23]: https://community.wolfram.com/groups/-/m/t/2827166 [24]: https://community.wolfram.com/groups/-/m/t/2823264 [25]: https://community.wolfram.com/groups/-/m/t/2821053 [26]: https://youtu.be/istKGqpDUsw [27]: https://community.wolfram.com/groups/-/m/t/2777853 [28]: https://youtu.be/roCkXVkDuLA [29]: https://youtu.be/pYHAz-NatXI [30]: https://youtu.be/r7Hjdr_D7c4 [31]: https://community.wolfram.com/groups/-/m/t/2713818 [32]: https://community.wolfram.com/groups/-/m/t/2705779 [33]: https://community.wolfram.com/groups/-/m/t/2701172 [34]: https://youtu.be/UrM-OBu3H9o [35]: https://community.wolfram.com/groups/-/m/t/2678940 [36]: https://community.wolfram.com/groups/-/m/t/2649407 [37]: https://community.wolfram.com/groups/-/m/t/2635049 [38]: https://community.wolfram.com/groups/-/m/t/2618033 [39]: https://community.wolfram.com/groups/-/m/t/2616863 [40]: https://community.wolfram.com/groups/-/m/t/2613617 [41]: https://community.wolfram.com/groups/-/m/t/2605432 [42]: https://community.wolfram.com/groups/-/m/t/2600997 [43]: https://community.wolfram.com/groups/-/m/t/2596451 [44]: https://www.twitch.tv/wolfram [45]: https://wolfr.am/1eatWLcDA [46]: https://www.facebook.com/wolframresearch [47]: https://twitter.com/WolframResearch [48]: https://www.youtube.com/live/ElP55ZILxPw?si=nsAPOQ3u-RbvuGKX [49]: https://community.wolfram.com/groups/-/m/t/3007543 [50]: https://community.wolfram.com/web/charlesp [51]: https://community.wolfram.com/groups/-/m/t/3019288 [52]: https://www.youtube.com/live/KM1yWHRrF2k?si=g2R7rHB2IinVRpo6 [53]: https://community.wolfram.com/groups/-/m/t/3009184 [54]: https://community.wolfram.com/groups/-/m/t/3064700 [55]: https://community.wolfram.com/groups/-/m/t/3084291 [56]: https://community.wolfram.com/groups/-/m/t/3104670 [57]: https://community.wolfram.com/groups/-/m/t/3164204 [58]: https://youtube.com/playlist?list=PLdIcYTEZ4S8TSEk7YmJMvyECtF-KA1SQ2&si=paXZHs0ZzGdB7y1y [59]: https://youtube.com/playlist?list=PLdIcYTEZ4S8RyjEB7JSAsGerbYHl5xXeJ&si=xkNtkIDvKHFWHVmD [60]: https://youtu.be/KUWK19Gx2LE?si=qbKISbL8FtvweSWo [61]: https://community.wolfram.com/web/subat Charles Pooh 2022-08-05T21:37:19Z https://community.wolfram.com/groups/-/m/t/2498984 > *SHARE this contest*: https://wolfr.am/CompArt-22 # WINNERS Thank you to everyone who submitted entries into this contest! It was a blast to see all the amazing art you created. After deliberation by our judges, these are the winners: - **Honorable Mention**: ARTIST: [Daniel Hoffmann][1], EXHIBIT: "[The Memory of Persistence][2]" - **Staff Winner**: ARTIST: [Anton Antonov][3], EXHIBIT: "[Rorschach mask animations projected over 3D surfaces][4]" - **3rd Place**: ARTIST: [Jacqueline Doan][5], EXHIBIT: "[Kuramoto oscillators with phase lag][6]" - **2nd Place**: ARTIST: [Tom Verhoeff][7], EXHIBIT: "[Sculpture from 18 congruent pieces][8]" - **1st Place**: ARTIST: [Frederick Wu][9], EXHIBIT: "[Love heart jewelry IV: the giving tree][10]" ![enter image description here][11] ---------- # CONTEST Flex your creative and computational skill with Wolfram's Computational Art Contest that kicks off today, Monday, March 28th! Share your work with the community, and potentially win free Wolfram merchandise. Programmers and artists of all skill levels are encouraged to participate! This contest is inspired by Genuary, an annual project releasing generative art prompts during the month of January. We're elated to see the creative works of our users and engage with the community while exploring the scope of computational art within the Wolfram Language. ## Rules & Guidelines ## - Submission deadline is April 25th at 9am Central Time. Posts posted after will not be included in judging - Participants must fill out a detailed Community profile ( example: https://community.wolfram.com/web/claytonshonkwiler ) and create a Community post about their submission. The post must include the code used to create graphics and the final piece of art placed at the top of the post. An explanation of how their code works is required, moreover participants are encouraged to write more about their creative process. - Participants submit their entry by commenting on this post with an image of their art, along with a link to their Community post - Multiple submissions per participant are allowed, but please keep the number of submissions under three - Each participant can only win once. Participants' best-performing piece, as determined by the judges, will be used when determining winners - Both static images and animations can be submitted. Animations are preferred in a GIF format; if the animation is too large for a GIF, the post can point to a public YouTube video. - Submissions will be judged by a handful of Wolfram experts, with the following parameters: - Visual aesthetics - Wolfram Language code - Creativity - Explanation of process - Submissions from all areas of computational art are welcome - Submissions from former or current Wolfram employees are allowed, but will be judged as their own category with only one winner - Submitting previous work/posts is allowed, but must meet the requirements stated above ## Encouragements ## - Not sure where to start? We encourage you to look at other user's submission for inspiration, or look at some of the work in the visual arts group of Community: - Artists' group: https://wolfr.am/ART-examples - Artist (see Staff Picks section): https://community.wolfram.com/web/claytonshonkwiler - We encourage you to vote and comment on other people's submissions. - Please spread the word about this competition, among your friends and other social media! ## Prizes ## First, second, and third place winners will be featured on all of Wolfram's social media accounts, as well as receiving their choice of free Wolfram merchandise. We are able to ship merchandise to countries listed on the Wolfram Store: https://store.wolfram.com. If your country is not listed on the Wolfram Store, we strongly encourage you to still submit an entry, as we will feature winners submissions regardless of location. ### Important ### All contest rules have been explained above under Rules & Guidelines. It's encouraged for all participants to read the rules carefully to prevent disqualification. If you have any additional questions, ask directly in the thread comments or contact us by email at t-artcontest@wolfram.com . We recommend reading other people comments as they clarify the nature of the contest as well. Comments deemed by moderators as superfluous to the thread may be removed or transferred by moderators to keep competition professional. [1]: https://community.wolfram.com/web/danielsanderhoffmann [2]: https://community.wolfram.com/groups/-/m/t/2518220 [3]: https://community.wolfram.com/web/antononcube [4]: https://community.wolfram.com/groups/-/m/t/2518279 [5]: https://community.wolfram.com/web/jacquelinengocdoan [6]: https://community.wolfram.com/groups/-/m/t/2509110 [7]: https://community.wolfram.com/web/tverhoeff [8]: https://community.wolfram.com/groups/-/m/t/2513265 [9]: https://community.wolfram.com/web/wufei1978 [10]: https://community.wolfram.com/groups/-/m/t/2430827 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=news-congrads-kkluyshnik-02-04-19.jpg&userId=11733 Eryn Gillam 2022-03-28T17:35:43Z https://community.wolfram.com/groups/-/m/t/1659553 Dear all, inspired by another [great post][1] of [@Anton Antonov][at0] and in particular there by a remark of [@Vitaliy Kaurov][at1] pointing to [the art of knitting images][2] I could not resist trying with Mathematica. Clearly - this problem is crying out loudly for **Radon transform**! ![enter image description here][3] I start by choosing some example image, convert it to inverse grayscale and perform the Radon transform. ClearAll["Global`*"] img0 = RemoveBackground[ ImageTrim[ ExampleData[{"TestImage", "Girl3"}], {{80, 30}, {250, 240}}], {"Background", {"Uniform", .29}}]; img1 = ImageAdjust[ColorNegate@ColorConvert[RemoveAlphaChannel[img0], "Grayscale"]]; {xDim, yDim} = {180, 400}; (* i.e. angles between 1\[Degree] and 180\[Degree] *) rd0 = Radon[img1, {xDim, yDim}]; ImageCollage[{img0, ImageAdjust@rd0}, Method -> "Rows", Background -> None, ImagePadding -> 10] ![enter image description here][4] Every column of the Radon image represents a different angle of projection. So next I separate these columns into (here 180) single Radon images and do an inverse Radon transform on each: maskLine[a_] := Table[If[a == n, 1, 0], {n, 1, xDim}]; maskImg = Table[Image[ConstantArray[maskLine[c], yDim]], {c, 1, xDim}]; rdImgs = rd0 maskImg; ProgressIndicator[Dynamic[n], {1, xDim}] invRadImgs = Table[{ImageApply[If[# > 0, #, 0] &, InverseRadon[rdImgs[[n]]]], -(n - 91) \[Degree]}, {n, 1, xDim}]; These data already represent the angle dependent intensities for backpropagation. Now one just has *somehow* to translate these intensities into discretely spaced lines (because this is the actual task in analogy to the above mentioned knitting ). Here is my simple attempt, which e.g. for 69° gives the following result (I am not really happy with this - there is definitely room for improvement!): ![enter image description here][5] valsAngle[invRads_] := Module[{img, angle, data, l2}, angle = Last@invRads; data = Max /@ (Transpose@*ImageData@*ImageRotate @@ invRads); l2 = Round[Length[data]/2]; data = MapIndexed[{First[#2] - l2, #1} &, data]; {Select[ Times @@@ ({#1, If[#2 > .0003, 1, 0]} & @@@ ((Mean /@ # &)@*Transpose /@ Partition[data, 5])), # != 0 &], angle} (* limiting value of 0.0003 is just empirical! *) ]; va = valsAngle /@ invRadImgs; graphicsData[va_] := Module[{u, angle}, {u, angle} = va; InfiniteLine[# {Cos[angle], -Sin[angle]}, {Sin[angle], Cos[angle]}] & /@ u]; gd = graphicsData /@ va; Graphics[{Thickness[.0003], gd}, ImageSize -> 600, PlotRange -> {{-170, 170}, {-220, 220}}] ... and the result is a bunch of lines: ![enter image description here][6] [at0]: https://community.wolfram.com/web/antononcube [at1]: https://community.wolfram.com/web/vitaliyk [1]: https://community.wolfram.com/groups/-/m/t/1555648?p_p_auth=T7A50bYl [2]: http://artof01.com/vrellis/works/knit.html [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ImageOfLines.gif&userId=32203 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=img0rd0.jpg&userId=32203 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=linesample.png&userId=32203 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ImageOfLines.png&userId=32203 Henrik Schachner 2019-04-13T20:01:08Z https://community.wolfram.com/groups/-/m/t/2007434 Our newest [Daily Study Group][1] offers a jump-start on earning Wolfram certifications and covers topics including Wolfram Notebooks, image processing and multiparadigm data science. Sign up at: https://wolfr.am/NpiuhRsg [1]: https://www.wolfram.com/wolfram-u/special-event/study-groups/ Jamie Peterson 2020-06-18T21:21:10Z https://community.wolfram.com/groups/-/m/t/2774101 A Wolfram U daily study group covering the implementation of Wolfram Language for tasks ranging from basic programming to video analysis begins on January 17, 2023 and runs through February 3. This study group will run on weekdays from 11:00AM–12:00PM Central US time. This study group is an incredible way either to start learning Wolfram Language or to explore new functionality you haven't yet used. We will cover a very broad variety of topics, including but not limited to image and sound analysis, symbolics and numerics, function visualization and even cloud computation and deployment. We will even cover useful tips and tricks to help you work efficiently with notebooks! ![enter image description here][1] **No prior Wolfram Language experience is necessary.** As usual, we will have questions, study materials, quizzes along the way to help you master the subject matter. You can [**REGISTER HERE**][2]. I hope to see you there! ![enter image description here][3] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=meechstogram.png&userId=1711324 [2]: https://www.bigmarker.com/series/daily-study-group-wolfram-language-basics-wsg34/series_details?utm_bmcr_source=community [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=WolframUBanner%281%29%281%29.jpeg&userId=1711324 Arben Kalziqi 2023-01-11T04:29:36Z https://community.wolfram.com/groups/-/m/t/2135869 Introduction ------------ The goal of this post is to start from images like this example one : ![Girl3][1] and generate pictures like: ![Girl3Gray][2] or ![Girl3Color][3] Mathematica at least 12.1 will be required since Mixed Integer programming is used. Explanation ----------- Let's take the first image (black and white) as an example. Let's assume we have a collection of 16 tiles: ![GrayTiles][4] The problem to solve is how to place the tiles on the picture so that the gray content of a tile is close to the gray content of the picture below it and at the same time the topological constraints are satisfied. By topological constraints, I mean that the tiles must be compatible. This is allowed: ![Allowed][5] This is forbidden because the dark horizontal band is continuing as a white horizontal band. ![Forbidden][6] We are going to translate this problem into a set of equations on integer variables and with a linear cost function to optimize. The final problem will be solved with the LinearOptimization function from Mathematica 12.1. Each pixel of the image is encoded by a vector ![eq1][7] because there are 16 different tiles in this example. The components of the vector can be either 0 or 1. This is expressed as: VectorLessEqual[{0, v}], VectorLessEqual[{v, 1}], v \[Element] Vectors[nbTiles, Integers] For each pixel, only one tile can be used. We cannot put several tiles on a pixel but only choose one and only one. If we use the constraint: ![eq2][8] then we express that only one tile can be used. Indeed, since the component are integers and equal to 0 or 1, then the only way to satisfy this equation is that one of the components, and only one, is equal to one. Expressing the topological constraints is similar. We have equations like: ![eq3][9] This equation is describing a relationship between pixel (x,y) and pixel (x+1,y). The values on left and right side can either be 0 or 1 (at same time). When zero it means : none of the tiles is used. When one, it means one of the tile is used. So, the translation of the equation is: If the tile 0,3 or 5 are used at pixel (x,y) then the tiles 3,6,9 or 11 must be used at pixel (x+1,y). (I have not checked if it makes sense with the set of tiles I am using as example. The constraints for those tiles are probably different.). To describe the topological constraint of the tiles, we have functions like: rightSide[tileA[{a_,b_,c_}]] :={b,c}; This is giving a key describing the right side of the tile. Those keys are then used in associations to build the topological constraints. The key can be anything so if you want to add new tiles, you can just use the key you want to describe the sides of your tiles. The generic function rightSide must be extended with new cases when new tiles are added. Then, we need to express how good the tiles are approximating the original picture. For this, an error function is created. It is a sum of terms: ![eq4][10] It means that if the tile i is selected for pixel (x,y) then the approximation error is `Subscript[f, k]` The function averageColor must be extended with new tiles. It returns the color content of a tile : a RGBColor. The code is using a color distance to compute the errors. That's why the input picture is always converted to RGB and the alpha channel removed. So, finally we have translated our problem into a set of integer constraints and with a linear cost function to optimize. It is a mixed integer programming problem which can be solved with LinearOptimization. The tiles must be displayed. It is done by the function tileDraw and the tile is drawn in a square from (0,0) to (1,1) corners. The code is rasterizing those graphics into 50x50 pixel images. I have had lots of problems with those pictures due to rounding errors ... probably due to the very old GPU on my very old computer. So I tuned the vector code assuming the final tile image is 50x50 pixels. Now the pictures are well aligned, there is no more one row or one column of wrong pixels on the boundary of the tiles. But this may cause a problem on your configuration. So, if the tile pictures are not rendering correctly on your side, you'll need to tune my vector graphic code again. If the picture is too big, solving the full mixed integer programming problem may take too long. But we can solve a sub-optimal problem. We divide the picture into sub-pictures and solve the problem on each sub-picture then we recombine the solutions. For it to work : we must add new constraints to express compatibility between the pictures. For instance, the left side of a picture at (row,col) must be compatible with the right side of the picture at (row,col-1). So, the problem must be solved in a given order so that the constraints can be propagated from one sub-picture to the other. For some tiles, the sub-optimal solution can be very good from an artistic point of view (the dark tiles below are working well). For other tiles (the smith tiles in the notebook) either the sub-optimal problem cannot always be solved (because the constraints coming from the previous pictures can't be satisfied) or the sub-optimal problem will look bad from time to time. So this idea of using sub-picture is really dependent on the kind of tiles used. You need to experiment. But it is art after all. Example of use -------------- First, we get an example picture: srcImage = ImageCrop[ExampleData[{"TestImage", "Girl3"}], {190, 270}] The picture is resized, converted to RGB and any alpha channel removed. imgToAnalyze = ImageResize[ ImageAdjust[ RemoveAlphaChannel[ColorConvert[srcImage, "RGB"], Black]], {50, Automatic}] For the dark tiles (knots), we decide to only use gray levels. First color is the background of the tiles. Other colors are for the circles and vertical and horizontal bands. tileData = mkDarkTiles[RGBColor[ 0.5, 0.5, 0.5], {RGBColor[0., 0., 0.], RGBColor[1., 1., 1.]}]; It gives a total of 16 tiles. The more tiles, the more difficult it is to solve the problem. 16 is ok on my old computer. tileData["allTilesImg"] // Length The problem is solved on 15x15 sub pictures. solution = partitionSolve[tileData, imgToAnalyze, 15]; The final picture is generated from the tiles and the solution. img = createPict[tileData, solution]; And you'll get: ![Girl3Gray][2] Have fun ! I hope the vectorial code will not have to be tuned to generate the tile pictures (rounding errors). The notebook is attached to the post. [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Girl3.png&userId=89693 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Girl3Gray.png&userId=89693 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Girl3Color.png&userId=89693 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Tiles.png&userId=89693 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Allowed.png&userId=89693 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Forbidden.png&userId=89693 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=eq1.png&userId=89693 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=eq2.png&userId=89693 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=eq3.png&userId=89693 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=eq4.png&userId=89693 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=LenaGray.png&userId=89693 Christophe Favergeon 2020-12-11T16:08:18Z https://community.wolfram.com/groups/-/m/t/2435403 ![Designing Townscaper town on computable base-grid][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=frames2.gif&userId=20103 [2]: https://www.wolframcloud.com/obj/0d953f64-4678-4d55-9d6b-17997fa2f42a Silvia Hao 2022-01-02T11:23:53Z https://community.wolfram.com/groups/-/m/t/2342501 *MODERATOR NOTE: related resource function can be found here* https://resources.wolframcloud.com/FunctionRepository/resources/MandelbrotSetRemap ---- ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=frac_hero.jpg&userId=20103 [2]: https://www.wolframcloud.com/obj/17f4f067-3491-4d7a-bf84-5601782ad11e Mark Greenberg 2021-08-14T14:58:17Z https://community.wolfram.com/groups/-/m/t/960843 *MODERATOR NOTE:* a resource function based on this work is now available: https://resources.wolframcloud.com/FunctionRepository/resources/ContentAwareImageResize/ ---------- - **Update 1:** Now forward energy is also implemented. - **Update 2:** Now also extending an image has been implemented. ## Seam carving ## ![enter image description here][12] Hi All, As you know Wolfram Language can do a lot of image processing, but one thing it can't yet do is so-called liquid rescaling. Liquid rescaling is a way of cropping the image but as opposed to ImageCrop it is content-aware. Meaning it will crop first parts of the image that have less 'information'. I'll show you how we can implement such cropping in Wolfram Language. We start of with importing an image: img=Import["tower.png"]; ![enter image description here][1] I now define a so-called energy function that describes where the image is information-rich: EnergyFunction[img_Image] := GradientFilter[img, 1, Method -> "ShenCastan"] We can test this out: EnergyFunction[img] giving: ![enter image description here][2] Now the idea is to find vertical 'seams' such that the sum of the values of the energy function along a seam is as low as possible. This seam contains the least of 'energy' or 'information' and we will therefore remove it first. We have to use the following code to do that: MinFilter1[x_List] := Min /@ Partition[x, 3, 1, {2, 2}, 1.0 10^6] (* same as MinFilter[x,1] but 10x faster *) MinPosition[x_List] := First[Ordering[x, 1]] (* position of min element *) Neighbours[n_Integer?Positive, len_Integer?Positive] := Which[n == 1, If[len > 1, {1, 2}, {1}], n == len, If[len > 1, {len, len - 1}, {1}], True, {n - 1, n, n + 1} ] FindSeam[mat_List?MatrixQ] := Module[{dimx, dimy, seam, neighbours, values, newpos, ii}, {dimy, dimx} = Dimensions[mat]; seam = ConstantArray[-1, dimy]; seam[[-1]] = MinPosition[mat[[-1]]]; Do[ neighbours = Neighbours[seam[[ii + 1]], dimx]; values = mat[[ii, neighbours]]; newpos = neighbours[[MinPosition[values]]]; seam[[ii]] = newpos , {ii, dimy - 1, 1, -1} ]; seam ] ComputeEnergyField[img_Image] := FoldList[#2 + MinFilter1[#1] &, ImageData[EnergyFunction[img]]] ComputeEnergyField[mat_List] := ComputeEnergyField[Image[mat]] we can now test it out: seam = FindSeam[ComputeEnergyField[img]]; HighlightImage[img, Transpose[{seam, Range[Length[seam]]}]] seam will be a list of horizontal position of the pixel for each row of pixels. We can use HighlightImage to see the 'seam': ![enter image description here][3] We can remove (carve) that seam from the original image, after which we can repeat this process over and over and crop 1 pixel each time until we have the width that we need (note that we need to recalculate the energy function after every carve). Of course one could do the same with horizontal seams if one wants to crop in the vertical direction. Simply use the same code but rotate the image 90 degrees, remove some seems, and then rotate is 90 degrees back. To make it more interactive we can calculate all the seems and save these separately, after which we can very easily crop to a desired width: ClearAll[MinFilter1, MinPosition, Neighbours, FindSeam, EnergyFunction, ComputeEnergyField, Carve, FillNthPosition, CreateSeamcarveImageData, SeamCarve] MinFilter1[x_List] := Min /@ Partition[x, 3, 1, {2, 2}, 1.0 10^6] (* same as MinFilter[x,1] but 10x faster *) MinPosition[x_List] := First[Ordering[x, 1]] (* position of min element *) Neighbours[n_Integer?Positive, len_Integer?Positive] := Which[n == 1, If[len > 1, {1, 2}, {1}], n == len, If[len > 1, {len, len - 1}, {1}], True, {n - 1, n, n + 1} ] FindSeam[mat_List?MatrixQ] := Module[{dimx, dimy, seam, neighbours, values, newpos, ii}, {dimy, dimx} = Dimensions[mat]; seam = ConstantArray[-1, dimy]; seam[[-1]] = MinPosition[mat[[-1]]]; Do[ neighbours = Neighbours[seam[[ii + 1]], dimx]; values = mat[[ii, neighbours]]; newpos = neighbours[[MinPosition[values]]]; seam[[ii]] = newpos , {ii, dimy - 1, 1, -1} ]; seam ] EnergyFunction[img_Image] := GradientFilter[img, 1, Method -> "ShenCastan"] ComputeEnergyField[img_Image] := FoldList[#2 + MinFilter1[#1] &, ImageData[EnergyFunction[img]]] ComputeEnergyField[mat_List] := ComputeEnergyField[Image[mat]] Carve[mat_List?ArrayQ, seam_List] := MapThread[Delete, {mat, seam}, 1] FillNthPosition[x_List, n_Integer?Positive, fill_, empty_: 0] := Block[{pos, out}, out = x; pos = Position[out, empty, {1}, n, Heads -> False]; out[[pos[[n]]]] = fill; out ] CreateSeamcarveImageData[img_Image] := Block[{imagedata, dims, dimx, dimy, carveinfo, seam, energyinfo}, imagedata = ImageData[img]; dims = {dimy, dimx} = Dimensions[imagedata, 2]; carveinfo = ConstantArray[0, dims]; PrintTemporary[Dynamic[Row[{"Calculating: ", i, "/", dimx}]]]; Do[ energyinfo = ComputeEnergyField[imagedata]; seam = FindSeam[energyinfo]; carveinfo = MapThread[FillNthPosition[#1, #2, i, 0] &, {carveinfo, seam}]; imagedata = Carve[imagedata, seam]; , {i, dimx} ]; {img, carveinfo} ] SeamCarve[{img_Image, carveinfo_List}, n_Integer?NonNegative] := Block[{imgdata, pick, sel, m}, imgdata = ImageData[img]; If[Dimensions[imgdata, 2] == Dimensions[carveinfo], m = Clip[n, {0, Length[carveinfo[[1]]] - 1}]; sel = UnitStep[m - carveinfo]; Image[Pick[ImageData[img], sel, 0]] , Abort[]; ] ] Now we pre-calculate the positions of the seams: out = CreateSeamcarveImageData[img]; Manipulate[SeamCarve[out, n], {n, 0, 500, 1}] ![enter image description here][4] We can now interactively change the width without cropping essential features away. To give you a comparison, here I cut away 100 pixels (from 286 pixels to 186 pixels) using different methods: ![enter image description here][5] To give you a better idea of what the seams are I show you the seam information: Image[Rescale[out[[2]]]] Colorize[out[[2]]] ![enter image description here][6] Where (in the top figure) dark colors mean seams that will be removed first, while brighter seams are left for last. You can clearly see that the tower and the person are saved until the last. Using this code we can now crop the width of any picture. ## Object removal ## But we can extend this method: say that we want to remove an object from an image. We can use the same algorithm but now we use extra negative weight to the region we want to delete. Say we have the following image: ![enter image description here][7] and we would like to remove the guy on the left. We create a mask that includes his shadow: ![enter image description here][8] We update our code to include a negative mask: ClearAll[ComputeEnergyFieldNegativeMask, CreateSeamcarveImageDataNegativeMask] ComputeEnergyFieldNegativeMask[img_Image, mask_Image] := FoldList[#2 + MinFilter1[#1] &, ImageData[EnergyFunction[img]] - 10000 ImageData[mask]] ComputeEnergyFieldNegativeMask[mat_List, mask_List] := ComputeEnergyFieldNegativeMask[Image[mat], Image[mask]] CreateSeamcarveImageDataNegativeMask[img_Image, mask_Image] := Block[{maskdata, imagedata, dims, dimx, dimy, carveinfo, seam, energyinfo}, (* removal *) If[ImageDimensions[img] == ImageDimensions[mask], imagedata = ImageData[img]; maskdata = ImageData[ColorConvert[mask, "Grayscale"]]; dims = {dimy, dimx} = Dimensions[imagedata, 2]; carveinfo = ConstantArray[0, dims]; PrintTemporary[Dynamic[Row[{"Calculating: ", i, "/", dimx}]]]; Do[ energyinfo = ComputeEnergyFieldNegativeMask[imagedata, maskdata]; seam = FindSeam[energyinfo]; carveinfo = MapThread[FillNthPosition[#1, #2, i, 0] &, {carveinfo, seam}]; imagedata = Carve[imagedata, seam]; maskdata = Carve[maskdata, seam]; , {i, dimx} ]; {img, carveinfo} , Abort[]; ] ] Now trying out the code: img = Import["beachpeeps.jpg"]; mask = Import["beachpeepsmask.png"]; AbsoluteTiming[out = CreateSeamcarveImageDataNegativeMask[img, mask];] Manipulate[SeamCarve[out, n], {n, 0, 75, 1}] ![enter image description here][9] Or a static comparison: ![enter image description here][10] ## Object protection ## In the same spirit, we can protect certain areas in the image that are important to us: ClearAll[CreateSeamcarveImageDataPositiveMask, ComputeEnergyFieldPositiveMask] ComputeEnergyFieldPositiveMask[img_Image, mask_Image] := FoldList[#2 + MinFilter1[#1] &, ImageData[EnergyFunction[img]] + 10000 ImageData[mask]] ComputeEnergyFieldPositiveMask[mat_List, mask_List] := ComputeEnergyFieldPositiveMask[Image[mat], Image[mask]] CreateSeamcarveImageDataPositiveMask[img_Image, mask_Image] := Block[{maskdata, imagedata, dims, dimx, dimy, carveinfo, seam, energyinfo}, (* removal *) If[ImageDimensions[img] == ImageDimensions[mask], imagedata = ImageData[img]; maskdata = ImageData[ColorConvert[mask, "Grayscale"]]; dims = {dimy, dimx} = Dimensions[imagedata, 2]; carveinfo = ConstantArray[0, dims]; PrintTemporary[Dynamic[Row[{"Calculating: ", i, "/", dimx}]]]; Do[ energyinfo = ComputeEnergyFieldPositiveMask[imagedata, maskdata]; seam = FindSeam[energyinfo]; carveinfo = MapThread[FillNthPosition[#1, #2, i, 0] &, {carveinfo, seam}]; imagedata = Carve[imagedata, seam]; maskdata = Carve[maskdata, seam]; , {i, dimx} ]; {img, carveinfo} , Abort[]; ] ] Where now our mask includes all the 3 persons in the image, including there shadows: ![enter image description here][11] Again, we call our function: img = Import["beachpeeps.jpg"]; mask = Import["beachpeepsmask2.png"]; AbsoluteTiming[out = CreateSeamcarveImageDataPositiveMask[img, mask];] Manipulate[SeamCarve[out, n], {n, 0, 500, 1}] Giving: ![enter image description here][12] Or some static comparisons: ![enter image description here][13] ## Final thoughts ## For the energy-function we can use different functions, I used a simple (fast) GradientFilter, but other functions like ImageSaliencyFilter or EntropyFilter could also work. Seams now go from top to bottom, and the position of the seam changes at most 1 horizontal position per row of pixels, this can be changed without too much effort to be a bigger neighbourhood. Also this can be applied to videos but a 3-dimensional carve in space-time has to be calculated. Moreover, the energy function can be improved by including 'forward energy'. I can't wait for this functionality to be included in the Wolfram Language. **EDIT:**<br/> **See my reply below, on how to implement forward energy.** P.S. I'm still confused why the guy has a skateboard on the beach... [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=tower.png&userId=73716 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=energy.png&userId=73716 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=seam.png&userId=73716 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=tower.gif&userId=73716 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=comparison.png&userId=73716 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=seaminfo.png&userId=73716 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=beachpeeps.jpg&userId=73716 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=beachpeepsmask.png&userId=73716 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=guygone.gif&userId=73716 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=7920guygonecomparison.png&userId=73716 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=beachpeepsmask2.png&userId=73716 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=savepeople.gif&userId=73716 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=preservestack.jpg&userId=73716 Sander Huisman 2016-11-10T23:42:16Z https://community.wolfram.com/groups/-/m/t/2051264 Probably most of you heard the sad news that there was a giant explosion in the port of Beirut today August 3rd 2020. Several videos were released on which we can do analysis. Note that the method I will use was also famously used by G.I. Taylor to find the energy of the Trinity nuclear bomb test, and he found the right amount to within 10%! We will not be so lucky as the video quality was relatively poor as compared to the high-speed imaging done back then. I extracted several frames from one of the videos: ![enter image description here][1] SetDirectory[NotebookDirectory[]]; v1 = Import["1.mp4"]; fra = VideoExtractFrames[v1, Interval[{11, 12}]] fra = ImageRotate[#, Right] & /@ fra; For each of the frames I identified the explosion by clicking 3 point on the circle: data={ {7,{{157.15625,365.20703125000006`},{233.83984375,379.76562500000006`},{272.015625,312.91015625000006`}}}, {8,{{318.16796874999994`,322.81640625000006`},{228.7890625,462.8515625},{103.61328125,393.38281250000006`}}}, {9,{{341.03515625000006`,311.34765625},{308.27734375,478.125},{93.86328125,420.34375}}}, {10,{{359.08984375,315.546875},{351.48828125,478.63671875000006`},{86.55078125,454.5078125}}}, {11,{{375.62109375,325.64453125},{330.05859375,535.3984375},{62.0390625,434.51171875}}}, {12,{{376.0390625,326.765625},{337.94140625,539.9257812499999},{46.4140625,462.55859375}}} }; The first is the index of the frames, the last elements are points of the circle: circs = CircleThrough /@ data[[;; 6, 2]]; r = circs[[All, 2]]; Here is the visualization: Table[HighlightImage[fra[[data[[i, 1]]]], circs[[i]], "Boundary"], {i, Length[data]}] ![enter image description here][2] Notice that I tracked the orange 'glow', not the shockwave or the smoke that was there partially before the main explosion (so on the conservative side and underestimating the energy release). From Google earth I estimated the size of the face of the building on the left (a grain elevator) and found that every pixel corresponds to 0.59 m roughly (~22 meters corresponding to ~37 pixels). cali = 0.5888486673789164`; realr = r cali The timestamps can be found from the video framerate. Information[Import["video.mp4"]]. And so the timestamps are created and the dataset is created: t = (Range[0, Length[realr] - 1]) 1/29.97; tr = Transpose[{t, realr}] Since the explosion started between two frames we include that in the fit (the t0): fit = FindFit[ tr, { a (x + t0)^0.4, 0 < t0 < 1/30}, {{a, 200}, {t0, 1/60}}, x] realfit = a (x + t0)^0.4 /. fit tzero = t0 /. fit realfitshifted = a (x)^0.4 /. fit prefactor = a /. fit The fit can be found [here][3] and is based on dimensional analysis with the variable E (energy), r (radius of the explosion), t (time), and ρ (density). This also explains the exponent 0.4 used for fitting. We plot the data and the fit: Show[{ListPlot[Transpose[{t + tzero, realr}]], Plot[realfitshifted, {x, 0, 0.2}]}, PlotRange -> {{0, 0.2}, {0, 120}}, Frame -> True, FrameLabel -> {"t", "r [m]"}] ![enter image description here][4] Which is a pretty good fit. We can now calculate the energy back from the explosion: ClearAll[r, e, t, \[Rho]] r == (e t^2/\[Rho])^(1/5) Refine[DivideSides[%, t^(2/5)], t > 0] %[[2]] == Quantity[prefactor, "Meters"/"Seconds"^(2/5)] % /. \[Rho] -> Quantity[1, "Kilograms"/"Meters"^3] energy = e /. Solve[%, e][[1]] Yielding: Quantity[4.2808721214488837`*^11, "Joules"] and we can convert it to kiloton of TNT: UnitConvert[energy, "KilotonsOfTNT"] yielding: Quantity[0.102315, "KilotonsOfTNT"] This number is comparable to the 2015 Tianjin explosion (0.3 kilo tonnes of TNT). [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2020-08-04at21.44.20.png&userId=73716 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2020-08-05at12.00.12.png&userId=73716 [3]: https://en.wikipedia.org/wiki/Nuclear_weapon_yield#Calculating_yields_and_controversy [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2020-08-04at21.53.18.png&userId=73716 Sander Huisman 2020-08-04T19:57:48Z https://community.wolfram.com/groups/-/m/t/1025046 This is a photo of some tree branches that I took at the park today. [<img src="http://community.wolfram.com//c/portal/getImageAttachment?filename=20170304_121322.jpg&userId=38370" width="500">](http://community.wolfram.com//c/portal/getImageAttachment?filename=20170304_121322.jpg&userId=38370) It looks like a fractal, and a nice candidate for measuring the fractal dimension. Here's how we can do that in Mathematica. Let us start by binarizing the image. img = Import["~/Downloads/20170304_121322.jpg"]; bin = LocalAdaptiveBinarize[ImageAdjust[img, 0.3], 200, {0.9, 0, 0}]; [<img src="http://community.wolfram.com//c/portal/getImageAttachment?filename=bin.png&userId=38370" width="300">](http://community.wolfram.com//c/portal/getImageAttachment?filename=bin.png&userId=38370) For a better result, I increased the contrast slightly (`ImageAdjust`), then used local adaptive binarization. This method chooses a different binarization threshold for each pixel based on its neighbourhood. It helps preserve small branches while avoiding including any part of the clouds (the weather was not very good today). The easiest way to measure fractal dimension is *box counting*: overlay a square grid, and see how many of the grid cells contain some of the object in the image. What would we get if the object is a line? Table[ Rasterize[Graphics[{Antialiasing -> False, Line[{{0, 0}, {1, 1}}]}], ImageSize -> 256, RasterSize -> k], {k, 2^Range[2, 8]} ] // ListAnimate The box count would double (i.e. multiply by $2^1$) every time the box size is halved. What would happen if the object is a filled disk? Table[ Rasterize[Graphics[{Antialiasing -> False, Disk[]}], ImageSize -> 256, RasterSize -> k], {k, 2^Range[2, 8]} ] // ListAnimate The box count would quadruple (i.e. multiply by $2^2$) every time the box size is halved. In general, for a $d$-dimensional object the box count $n$ is proportional to power $-d$ of the box size $l$:  $n \sim l^{-d}$. For some objetcs, $d$ can be a fractional (non-integer) number. Such objects are called fractals. Let's try it on the tree photo! First, let us choose box sizes in pixels so that an integer number of boxes fit in the image. seq = Reverse[Intersection @@ Divisors /@ ImageDimensions[img]] (* {1008, 504, 336, 252, 168, 144, 126, 112, 84, 72, 63, 56, 48, 42, 36, 28, 24, 21, 18, 16, 14, 12, 9, 8, 7, 6, 4, 3, 2, 1} *) Luckily, the image width and height had many small prime factors in common, so we have many box sizes to work with. Now let us count boxes. {width, height} = ImageDimensions[img] (* {3024, 4032} *) result = { width/#, Last@First@ImageLevels@Image[ BlockMap[Min, ImageData[bin], {#, #}], "Bit"] } & /@ seq; // AbsoluteTiming (* {9.79212, Null} *) [BlockMap][1] will partition the image data into blocks (i.e. boxes) and take the minimum of each. Since black pixels are represented by 0 and white pixels by 1, we get 0 only for those boxes that contain at least one black pixel, i.e. they contain a bit of the tree branches. `BlockMap` was introduced in Mathematica 10.2. Those using older versions can substitute `BlockMap[f, matrix, {k,k}]` by `Map[f, Partition[matrix, {k,k}], {2}]`. [ImageLevels][2] counts how many times each possible pixel value appears in an image. To make sure that the only possible pixel values are 0 and 1, we create a `"Bit"` image. Is the result really a power function of the form $n = \text{(const.)} l^d$? The easy way to test this is to plot it on a log-log plot. Power functions appear as a line when plotted on a logarithmic scale. ListLogLogPlot[result] ![enter image description here][3] The slope of the line will give the exponent: $\ln n = \ln\text{(const.)} + d \ln l$. fm = LinearModelFit[Log[result], {1, x}, x] It is about $\approx 1.85$. Despite the thick tree trunks, the image of the branches behaves as a *lower-than-two dimensional* structure. Let us plot the power law fit together with the original data: plot = Show[ Plot[fm[x], {x, Log[result[[1, 1]]], Log[result[[-1, 1]]]}, PlotStyle -> Black], ListLogLogPlot[result, PlotStyle -> Red], AspectRatio -> 1, Axes -> False, Frame -> True ] ![enter image description here][4] You can see that the power law is a pretty good fit. The slope of the curve is more or less the same at all size scales. This indicates a certain kind of scale invariance in the image. Indeed, it is clear that if we magnified a small part of the image, it would look similar to the whole: small branches are miniature copies of larger ones. This is a characteristic feature of fractals. To visualize the box counting method, we can show the series of finer and finer box-grids: sz = 252; imgs = Image[BlockMap[Min, ImageData[bin], {#, #}], "Bit"] & /@ seq Scaling them to the same size and animating them show how they approach the original image better and better: imgs = If[First@ImageDimensions[#] > sz, ImageResize[Image[#, Real], sz], (* downscale as grayscale *) ImageResize[#, sz, Resampling -> "Nearest"]] & /@ imgs; (* upscale as bitmap *) frm = Table[ ImageCompose[ImageResize[img, 252], {im, 0.5}], {im, imgs}]; ListAnimate[frm] ![enter image description here][5] We can show the animation alongside the scaling plot: frames = Table[ Row[{ Show[plot, AspectRatio -> 1, ImageSize -> 336, Epilog -> {AbsolutePointSize[10], Red, Point@Log[result[[i]]]}], Image[frm[[i]], Magnification -> 1] }], {i, Length[imgs]}]; With some extra styling it looks like this: ![enter image description here][6] I hope you enjoyed this small demonstration! [1]: http://reference.wolfram.com/language/ref/BlockMap.html [2]: http://reference.wolfram.com/language/ref/ImageLevels.html [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=plot.png&userId=38370 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=plot2.png&userId=38370 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ezgif.com-gif-maker.gif&userId=38370 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=animation.gif&userId=38370 Szabolcs Horvát 2017-03-04T17:33:57Z https://community.wolfram.com/groups/-/m/t/2542490 ![enter image description here][1] The excellent [community contribution][2] by [@Greg Hurst][at0] and a Wikipedia [animation][3], Inspired me to look further into this "rolling shutter effect on rotating objects". When we capture a video of a rotating disk with a rolling shutter, we have two independent movements: the colored disk rotating at *rps revolutions per second* and the shutter line sweeping one frame at *fps frames per second*. The ratio rps/fps is the driver of the rolling shutter effect (or the disk rps alone if we normalize the shutter fps to 1 frame per second). In order to best demonstrate this effect, ratios of rps/fps are taken to be in the range 1.5-2.5 This is a colored disk of m pixels wide, rotated over an angle theta. colors = {RGBColor[1, 0, 1], RGBColor[0.988, 0.73, 0.0195], RGBColor[ 0.266, 0.516, 0.9576], RGBColor[0.207, 0.652, 0.324], RGBColor[ 0, 0, 1], RGBColor[1, 0, 0]}; colorDisk[theta_, m_, cols_] := ImageResize[ Image[Graphics[ MapThread[{#3, Disk[{0, 0}, 1, {#1, #2} + theta]} &, {Pi Range[0, 5, 1]/3, Pi Range[1, 6]/3, colors}]]], m] A video of the rotating disk consists of a series of frames. Each frame is captured during one passage of the shutter line. The function *angularPosition* links the angular progress of the disk to the frame number (frm) and the row number at the position of the shutter line: angularPosition[frm_, row_, rps_, m_] := -2 Pi (-1 + (-1 + frm) m + row) rps/m Th function *diskFrameImage* computes the result of the shutter line swiping a colored disk (of size m, rotating at rps revolutions per second) at frame number frm and up to row number toRow: diskFrameImage[frm_, toRow_, rps_, m_, cols_] := ImageAssemble[ Transpose@{ParallelTable[ ImageTake[ colorDisk[angularPosition[frm, r, rps, m], m, colors], {r}], {r, toRow}]}] This is the first frame of a video of a disk rotating at a speed ratio rps/fps of 1 : With[{m = 200, frm = 1, rps = 1}, diskFrameImage[frm, m, rps, m, colors]] ![enter image description here][4] This shows the influence of the disk rps/fps ratio on the appearance of the first frame captured: With[{m = 200, frm = 1}, Grid[{{"rps=0.512", "rps=1.512", "rps=2.512"}, diskFrameImage[frm, m, #, m, colors] & /@ {.512, 1.512, 2.512}}]] ![enter image description here][5] Below is a GIF showing the capture of the first frame of a disk rotating at rps/fps 1. The code is used to generate all of the following GIFs. With[{m = 200, frm = 1, rps = 1.}, Animate[ Grid[{{ ImageCompose[ colorDisk[angularPosition[frm, row, rps, m], m, colors], Graphics[Line[{{-m, 0}, {m, 0}}]], Scaled[{.5, (m - row)/m}]], ImageCompose[diskFrameImage[frm, row, rps, m, colors], Graphics[Line[{{-m, 0}, {m, 0}}]], Scaled[{.5, .01}]]}}, Alignment -> Top], {row, 1, m}]] ![enter image description here][6] Below are 4 examples showing the capture of the first frame of a video of a rotating disk with a roller shutter at 1fs. The disk rotates at 0.5 rps (top left), 1.0 rps (top right), 1.5 rps (bottom left) and 2.0 rps (bottom right). As the disk rotates faster relative to the shutter, the captured image becomes more complex. ![enter image description here][7] ![enter image description here][8] The subsequent frames are captured the same way. Below are the first 5 frames of a video of a disk rotating at 1.5123 rps: ![enter image description here][9] There ought to be a lot more images that result from the transformation of a rotation into a capture with a rolling shutter. I hope this contribution can inspire more community members. [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=combirotodisk15and2.gif&userId=20103 [2]: https://community.wolfram.com/groups/-/m/t/2489445 [3]: https://upload.wikimedia.org/wikipedia/commons/1/15/Rolling_shutter_effect_animation.gif [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=8345locusfulldiskrps1.png&userId=68637 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=9528colordiskrpscompare.png&userId=68637 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=newrotodiskfrm1rps05small.gif&userId=68637 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=combirotodisk05and10.gif&userId=68637 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=combirotodisk15and2.gif&userId=68637 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=allframesvideofinal.gif&userId=68637 [at0]: https://community.wolfram.com/web/ghurst Erik Mahieu 2022-06-02T12:22:57Z https://community.wolfram.com/groups/-/m/t/922544 Dear all, my data consist of a list of lists of points in 3D. After running the code ClearAll["Global`*"] SetDirectory[NotebookDirectory[]]; sliceData = << "SliceData.txt"; Graphics3D[Line /@ sliceData, Boxed -> True, Axes -> True] one gets: ![enter image description here][1] Clearly the data describe a 3D object (as "wire frame"). **QUESTION:** *How can those data be converted into a single 3D Mathematica object (graphics, mesh, ...)? Is there an already implemented way (a routine) I am missing?* ---------- NOTE on DATA: ------------- The data stem from our radiotherapy treatment planning system. There we are working with contours which are drawn on any single CT slice. Organs at risk and target volumes are defined by those contours. One special contour is the "BODY contour"; this is what is shown in my example data. Dose will be calculated only inside this BODY contour, therefore e.g. the volume around the ears and nose appears to be exaggerated (for being on the safe side). Those treatment plans can be exported as DICOM files and nicely imported in Mathematica. When high energy radiation is applied to a body it turns out that the dose next to the skin is highly diminished; this is due to the buildup effect of the dose. When full dose at the skin is wanted, one has to anticipate this buildup effect, and this can be realized by putting a "flab" onto the skin: A flab is a layer made of some tissue equivalent material. Optimal flabs can have quite irregular shapes. I recently learned at a conference that there is the possibility to 3D print those individual flabs - if one can provide the data ... So - probably to everybody's disappointment - I do not want to print any BODY contour, but I imagined a BODY contour might serve here as kind of a "honey pot". Best regards and many thanks! -- Henrik [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=sliceImg.png&userId=32203 Henrik Schachner 2016-09-11T20:40:44Z https://community.wolfram.com/groups/-/m/t/3062832 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=w5qgsdf.jpg&userId=11733 [2]: https://www.wolframcloud.com/obj/04458d24-aacf-4cd7-8a5d-46efde37e927 Marco Thiel 2023-11-09T22:51:38Z https://community.wolfram.com/groups/-/m/t/2153018 [![enter image description here][1]](https://easyzoom.com/image/238912) The entire text of The Great Gatsby rendered as colored pixels. Zoom interactively in here: https://easyzoom.com/image/238912 [![enter image description here][2]](https://easyzoom.com/image/238912) Hi all, making my first post here. I've been using Mathematica in my research for 10 years or so, but now that I'm an MFA student I'm finding it extremely useful in my art practice. This post is just a fun simple idea and I plan on posting my more complicated work soon. Let me know what you think! ---------- &[Wolfram Notebook][3] [Original]: https://www.wolframcloud.com/obj/76b4e5d2-aed6-4162-9543-39fa9139b920 [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ezgif-6-c1b05e197418.gif&userId=20103 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=booksAsPixels.jpg&userId=20103 [3]: https://www.wolframcloud.com/obj/2b820c1b-c4e8-495c-ae0f-194793676745 Jack Madden 2021-01-02T21:07:56Z https://community.wolfram.com/groups/-/m/t/884348 The aim of my project for the Wolfram Science Summer School was to build a neural network which could be able to colorize grayscale images in a realistic way. The network has been built following the article [1]. In this paper, the authors propose a fully automated approach for colorization of grayscale images, which uses a combination of global image features, which are extracted from the entire image, and local image features, which are computed from small image patches. Global priors provide information at an image level such as whether or not the image was taken indoors or outdoors, whether it is day or night, etc., while local features represent the local texture or object at a given location. By combining both features, it's possible to leverage the semantic information to color the images without requiring human interaction. The approach is based on Convolutional Neural Networks, which have a strong capacity for learning and is trained to predict the chrominance of a grayscale image using the CIE L*a*b* colorspace. Predicting colors has the nice property that training data is practically free: any color photo can be used as a training example. **Net Layers** The model consists of four main components: a low-level features network, a mid-level features network, a global features network, and a colorization network. First, a common set of shared low-level features are extracted from the image. Using these features, a set of global image features and mid-level image features are computed. Then, the mid-level and the global features are both fused by a "fusion layer" and used as the input to a colorization network that outputs the final chrominance map. Each layer has a ReLu transfer function except for the last convolution of the colorization network, where a sigmoid function is applied.The model is able to process images of any size, but it is most efficient when the input images are 224x224 pixels, as the shared low-level features layers can share outputs. Note that when the input image size is of a different resolution, while the low-level feature weights are shared, a rescaled image of size 224x224 must be used for the global features network.This requires processing both the original image and the rescaled image through the low-level features network, increasing both memory consumption and computation time. For this reason, we trained the model exclusively with images of size 224x224 pixels. *Low-Level Features Network* A 6-layer Convolutional Neural Network obtains low-level features directly from the input image. The convolution filter bank the network represents are shared to feed both the global features network and the mid-level features network. In order to reduce the size of the feature maps, we use convolution layers with increased strides instead of using max-pooling layers (as usual for similar kinds of networks). If padding is added to the layer, the output is effectively half the size of the input layer. We used 3x3 convolution kernels exclusively and a padding of 1x1 to ensure the output is the same size (or half if using a stride of 2) as the input. *Global Features Network* The global image features are obtained by further processing the low-level features with four convolutional layers followed by three fully-connected layers.This results in a 256-dimensional vector representation of the image. *Mid-Level Features Network* The mid-level features are obtained by processing the low-level features further with two convolutional layers. The output is bottlenecked from the original 512-channel low-level features to 256-channel mid-level features. Unlike the global image features, the low-level and mid-level features networks are fully convolutional networks, such that the output is a scaled version of the input. *Fusion Layer* In order to be able to combine the global image features, a 256-dimensional vector, with the (mid-level) local image features, a 28x28x256-dimensional tensor, the authors introduce a fusion layer. This can be thought of as concatenating the global features with the local features at each spatial location and processing them through a small one-layer network.This effectively combines the global feature and the local features to obtain a new feature map that is, as the mid-level features, a 3D volume. *Colorization Network* Once the features are fused, they are processed by a set of convolutions and upsampling layers, which use the nearest neighbour technique so that the output is twice as wide and twice as tall. These layers are alternated until the output is half the size of the original input. The output layer of the colorization network consists of a convolutional layer with a Sigmoid transfer function that outputs the chrominance of the input grayscale image. Finally, the computed chrominance is upsampled and combined with the input intensity/luminance image to produce the resulting color image. In order to train the network, we used the Mean Square Error (MSE) criterion. Given a color image for training, the input of the model is the grayscale image while the target output is the a*b* components of the CIE L*a*b* colorspace. The a*b* components are globally normalized so they lie in the [0,1] range of the Sigmoid transfer function. *Colorization with Classification* While training with only color images using the MSE criterion does give good performance, sometimes it could make obvious mistakes due to not properly learning the global context of the image, e.g., whether it is indoors or outdoors. As learning these networks is an non-convex problem, we facilitated the optimization by also training for classification jointly with the colorization. As we trained the model using a large-scale dataset for classification of N classes (Mathematica ImageIdentify dataset), we had classification labels available for training. These labels correspond to a global image tag and thus can be used to guide the training of the global image features. We did this by introducing another very small neural network that consists of two fully-connected layers: a hidden layer with 256 outputs and an output layer with as many outputs as the number of classes in the dataset. The input of this network is the second to last layer of the global features network with 512 outputs. We trained this network using the cross-entropy loss, jointly with the MSE loss for the colorization network. *Implementation* The aim of my project was to build the network described in the paper using the new NeuralNetworks framework of Mathematica 11. In order to achieve this, some adjustments were needed. First of all, we decided to train and evaluate the network only on images of 224x224 pixels size, in order to use (and train) only one low-level features network, instead of two with shared weights and different outputs. The final network has two inputs: the first one is the colored 224x224 px image, encoded by the "NetEncoder" function in LAB colorspace, the second one the class of the image. The two outputs (named "Loss" and "Output") represent the values of the two loss function used (one for the colorization, the other one for the classification), which are then summed together by the NetTrain function. The three color channels of the input image are split by the split layer: the L channel feeds the "low-level features" network, while the a,b channels are scaled and concatenated in order to obtain a target set for the mean squared loss function comparable with the output of the colorization network. The fusion layer has been replaced by a broadcast layer, which joins the rank 3- tensor, output of the mid-level network, with the vector from the global features network. However, the way they are combined is not exactly the same as the one described in the paper. To evaluate the trained network on a grayscale image it's necessary to drop some branches of the network, such as the classification network and the layers that process the a,b channels of the colored input image in order to produce the target set for the colorization loss function. ![Network described in the paper][1] ![Network implementation with Mathematica NeuralNetworks framework][2] **Results** ![enter image description here][3] **Conclusions** The network described in the paper has been trained on the Places scene dataset [Zhou et al. 2014], which consists of 2,448,872 training images and 20,500 validation images, with 205 classes corresponding to the types of the scene. They filtered the images by removing grayscale images and those that have little color variance with a small automated script. They trained using a batch size of 128 for 200,000 iterations corresponding to roughly 11 epochs. This takes roughly 3 weeks on one core of a NVIDIAR TeslaR K80 GPU. We needed to introduce some new layers in the existing framework and to fix some bugs, so we were able to train our network only for 14 hours on a dataset of 350000 images on one core of a GPU Titan machine. Furthermore, the images in our training set mainly represent specific items, so probably better results may be achieved introducing also images of different types of subjects (landscapes, human created images, indoors, etc). The results we obtained are showed in the section above and are quite good. We are confident that with a deeper and longer training our network would give considerably better results. **Open Problems / Future Developments** Due to the separation between the global and local features, it is possible to use global features computed on one image in combination with local features computed on another image, to change the style of the resulting colorization. One of the more interesting things the model can do is adapting the colorization of one image to the style of another. This is straight-forward to do with this model due to the decorrelation between the global features and the mid-level features. In order to colorize an image A using the style taken from an image B, it's necessary to compute the mid-level local features of image A and the global features from image B. Than it's possible to fuse these features and process them with the colorization network. Both the local and the global features are computed from grayscale images: it's not necessary to use any color information at all. The main limitation of the method lies in the fact that it is datadriven and thus will only be able to colorize images that share common properties with those in the training set. In order to evaluate on significantly different types of images, it would be necessary to train a the model for all type of images (indoor, outdoor, human-created...). In order to obtain good style transfer results, it is important for both images to have some semantic level of similarity between them. **References** [1] Satoshi Iizuka, Edgar Simo - Serra, and Hiroshi Ishikawa."Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification". [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=netwPicture.png&userId=884315 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=MyNetwork.png&userId=884315 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=result2.png&userId=884315 Sabrina Giollo 2016-07-07T19:53:19Z https://community.wolfram.com/groups/-/m/t/900782 Mathematica 11 is now out and ready to compute! Since the release of Version 10 two years ago, Mathematica has grown by leaps and bounds, with 500+ new functions. This version introduces both enhancements to computations across the board and completely new areas of functionality, continuing Mathematica's growth as the state-of-the-art technical platform—and we are very excited to share these developments with the world. Some of the most notable features include: **[3D Printing][1]** ![3D-Printed Triceratops][2] - Print your 3D models to local printers or online printing services with [Printout3D][3]. - Use [FindMeshDefects][4] and [RepairMesh][5] to fix your models before printing. - Hollow your models out with [ShellRegion][6] to lower the printing cost. **[Computational Audio][7]** ![Audio Image][8] - Version 11 includes a brand-new [Audio][9] object to represent audio created from importing or an array of data. - Edit your audio with [AudioPad][10], [AudioTrim][11], [AudioSplit][12], [AudioResample][13], etc. - Synthesize sounds using the new [AudioGenerator][14] function. - Apply filters, take [AudioMeasurements][15] and visualize your audio with [AudioPlot][16]. **[Neural Networks][17]** ![Neural Network][18] - Define network topologies with [NetGraph][19] or with a chain of layers with [NetChain][20]. - Train your neural nets with [NetTrain][21]. - Train networks on either CPUs or NVIDIA GPUs. - Are your training sets of images too large to hold in-memory? The neural network functions support out-of-core image datasets. **[Improved Machine Learning][22]** ![Identifying Notable Celebrities][23] - [ImageIdentify][24] now recognizes over 10,000 objects. - Extract features from images, text and other data types using [FeatureExtract][25]. - [Classify][26] works even better with images, including an option to customize what [FeatureExtractor][27] it uses. - Find formulas for time series data using [FindFormula][28]. - Find clusters in your data with enhanced options for [FindClusters][29], including new methods and setting a [CriterionFunction][30]. And these are only just a sample of the new features, including the [Wolfram Channel Framework][31], enhanced notebook processing, cloud-aware [WolframScript][32], improved [symbolic and numeric calculus][33] and so much more. To read more about what's new in Mathematica 11, read Stephen Wolfram's [release-day blog post][34] and check out the [New in 11 page][35]. [1]: http://www.wolfram.com/language/11/3d-printing/ [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=triceratops.png&userId=900759 [3]: http://reference.wolfram.com/language/ref/Printout3D.html [4]: http://reference.wolfram.com/language/ref/FindMeshDefects.html [5]: http://reference.wolfram.com/language/ref/RepairMesh.html [6]: http://reference.wolfram.com/language/ref/ShellRegion.html [7]: http://www.wolfram.com/language/11/computational-audio/ [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=audioplot.png&userId=900759 [9]: http://reference.wolfram.com/language/ref/Audio.html [10]: http://reference.wolfram.com/language/ref/AudioPad.html [11]: http://reference.wolfram.com/language/ref/AudioTrim.html [12]: http://reference.wolfram.com/language/ref/AudioSplit.html [13]: http://reference.wolfram.com/language/ref/AudioResample.html [14]: http://reference.wolfram.com/language/ref/AudioGenerator.html [15]: http://reference.wolfram.com/language/ref/AudioMeasurements.html [16]: http://reference.wolfram.com/language/ref/AudioPlot.html [17]: http://www.wolfram.com/language/11/neural-networks/ [18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=neuralnetworkdigits.png&userId=900759 [19]: http://reference.wolfram.com/language/ref/NetGraph.html [20]: http://reference.wolfram.com/language/ref/NetChain.html [21]: http://reference.wolfram.com/language/ref/NetTrain.html [22]: http://www.wolfram.com/language/11/improved-machine-learning/ [23]: http://community.wolfram.com//c/portal/getImageAttachment?filename=CumberbatchMachineLearning.png&userId=900759 [24]: http://reference.wolfram.com/language/ref/ImageIdentify.html [25]: http://reference.wolfram.com/language/ref/FeatureExtract.html [26]: http://reference.wolfram.com/language/ref/Classify.html [27]: http://reference.wolfram.com/language/ref/FeatureExtractor.html [28]: http://reference.wolfram.com/language/ref/FindFormula.html [29]: http://reference.wolfram.com/language/ref/FindClusters.html [30]: http://reference.wolfram.com/language/ref/CriterionFunction.html [31]: http://reference.wolfram.com/language/guide/Channel-BasedCommunication.html [32]: http://reference.wolfram.com/language/ref/program/wolframscript.html [33]: http://www.wolfram.com/language/11/symbolic-and-numeric-calculus/ [34]: http://blog.wolfram.com/2016/08/08/today-we-launch-version-11/ [35]: http://www.wolfram.com/mathematica/new-in-11/ Zachary Littrell 2016-08-08T16:43:43Z https://community.wolfram.com/groups/-/m/t/2152918 My question is based on the observation that when a new Mathematica version is released (this time being 12.2) some of the functions that don't even receive an update from the previous version often get broken. How is it so that the examples in the documentation are not checked for consistency of operation/results before a new Mathematica version is shipped? I believe that even with a team of 5-6 people in the quality control department (assuming there is any) it is possible to check the examples for all listed functions in Mathematica in about a week. Why is such a simple task and simple expectation so hard to meet? The issue at the end is that a buggy release will cause many of the mission critical functions for say an end user to break. It is a pity that I have to install not one version (that I can fully trust) but keep four versions of Mathematica (11.3, 12.0,12.1 and 12.2) simultaneously (grabbing over 50 GB). Reason: because some things work in one version and the others in another version. Please do enlighten me how hard are the user expectations to be met. We do not want more and more functions; we just want the functions to work reliably and correctly. There seems to be issues with the quality control here that needs to be sorted before releasing future versions. Take this simple example: ReplacePixelValue did not receive any update since 2014 (version 10). Then how come the second example in its documentation (in the "Applications" subsection) cease to function properly. Furthermore, I am attaching a notebook with another example for this particular function to demonstrate that something that works perfectly in 12.0 and 12.1 fails in the 12.2 release. &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/1cb0dbd1-e7cb-48cf-abd1-1324cf87e89e Ali Hashmi 2021-01-02T15:53:14Z https://community.wolfram.com/groups/-/m/t/2222977 &[Wolfram Notebook][1] [Original]: https://www.wolframcloud.com/obj/8a8fbd01-b0d8-4798-beec-166e0898b2b1 [1]: https://www.wolframcloud.com/obj/5d2efdc4-f66c-486b-9f5a-298d70381d5d Jack Madden 2021-03-18T17:28:29Z https://community.wolfram.com/groups/-/m/t/1787163 Tuseeta-san's [post][1] is how to convert a trained model of TensorFlow to Mathematica. Converting trained models from a language other than Mathematica to Mathematica is very beneficial to Mathematica users. So I'll show how to convert a trained model of PyTorch to Mathematica along with Tuseeta-san's post. ![enter image description here][2] ## Step 1: Figure out the architecture ## The [model][3] to be converted is Pose Estimation that detects the human skeleton (body parts and their connections) from an image. It's called OpenPose.The model consists of Feature map that extracts image features and six Stage maps. Feature map extracts image features from an input image(size:368*368). Each Stage map has two-branch, the first branch predicts confidence and the second predicts PAFs?Part Affinity Fields), along with the image feature .Two-branch are concatenated for next stage. ## Step 2: Coding it in Mathematica ## **Feature map** The Feature map consists of the first 23 layers of VGG-19, followed by 2 sets of Convolution and Ramp. Extract the first 23 layers of VGG-19. vgg19 = NetModel["VGG-19 Trained on ImageNet Competition Data"]; vgg19sub = Take[vgg19, {1, 23}]; Change Encoder. enc = NetExtract[vgg19, "Input"]; enc = NetReplacePart[ enc, {"ImageSize" -> {368, 368}, "VarianceImage" -> {0.229, 0.224, 0.225}, "MeanImage" -> {0.485, 0.456, 0.406}}]; featurefirst = NetReplacePart[vgg19sub, "Input" -> enc]; Add Convolution and Ramp. feature = NetAppend[ featurefirst, {"convadd1" -> ConvolutionLayer[256, 3, "Stride" -> 1, "PaddingSize" -> 1], "reluadd1" -> Ramp, "convadd2" -> ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], "reluadd2" -> Ramp}]; **Stage map** Each Stage map consists only of Convolutions and Ramps. Stage 1: The differences between two branches is that the last output channel number is 38 or 19. blk11 = NetChain[{ ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[512, 1, "Stride" -> 1, "PaddingSize" -> 0], Ramp, ConvolutionLayer[38, 1, "Stride" -> 1, "PaddingSize" -> 0]}]; blk12 = NetChain[{ ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[512, 1, "Stride" -> 1, "PaddingSize" -> 0], Ramp, ConvolutionLayer[19, 1, "Stride" -> 1, "PaddingSize" -> 0]}]; Stage 2?6: The difference between Stage1 and Stage2?6 is the kinds and the numbers of layers. blkx1 = NetChain[{ ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 1, "Stride" -> 1, "PaddingSize" -> 0], Ramp, ConvolutionLayer[38, 1, "Stride" -> 1, "PaddingSize" -> 0]}]; blkx2 = NetChain[{ ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 1, "Stride" -> 1, "PaddingSize" -> 0], Ramp, ConvolutionLayer[19, 1, "Stride" -> 1, "PaddingSize" -> 0]}]; Finally, create OpenPose. openpose = NetGraph[{ "feature" -> feature,(*feature*) "blk11" -> blk11, "blk12" -> blk12,(*stage 1*) "blk21" -> blkx1, "blk22" -> blkx2,(*stage 2*) "cat12" -> CatenateLayer[], "blk31" -> blkx1, "blk32" -> blkx2,(*stage 3*) "cat23" -> CatenateLayer[], "blk41" -> blkx1, "blk42" -> blkx2,(*stage 4*) "cat34" -> CatenateLayer[], "blk51" -> blkx1, "blk52" -> blkx2,(*stage 5*) "cat45" -> CatenateLayer[], "blk61" -> blkx1, "blk62" -> blkx2,(*stage 6*) "cat56" -> CatenateLayer[] }, {"feature" -> "blk11", "feature" -> "blk12",(*stage 1*) {"blk11", "blk12", "feature"} -> "cat12",(*stage 2*) "cat12" -> "blk21", "cat12" -> "blk22", {"blk21", "blk22", "feature"} -> "cat23",(*stage 3*) "cat23" -> "blk31", "cat23" -> "blk32", {"blk31", "blk32", "feature"} -> "cat34",(*stage 4*) "cat34" -> "blk41", "cat34" -> "blk42", {"blk41", "blk42", "feature"} -> "cat45",(*stage 5*) "cat45" -> "blk51", "cat45" -> "blk52", {"blk51", "blk52", "feature"} -> "cat56",(*stage 6*) "cat56" -> "blk61", "cat56" -> "blk62" }] ![enter image description here][4] ## Step 3: Importing the Weights and the Biases ## Download "[pose_model_scratch.pth][5]" as a trained model of PyTorch. Import the parameters, the weights and the biases. I referred "[How to import python pickle *.pkl?][6]" session = StartExternalSession["Python-NumPy"]; parameters = ExternalEvaluate[session, "import torch import numpy as np import pickle as pkl net_weights = torch.load( 'pose_model_scratch.pth', map_location={'cuda:0': 'cpu'}) keys = list(net_weights.keys()) parameters = {} for i in range(len(keys)): t = net_weights[keys[i]] x = t.numpy() parameters[keys[i]] = x.flatten() parameters"]; DeleteObject[session]; keys = Keys[parameters]; parameters = Values[parameters]; ## Step 4: Parsing the Weights and the Biases ## The parameters is 184 sets of one-dimensional lists. They consists of the Weights and the Biases of 92 Convolution layers in OpenPose. Get a list of layer names for OpenPose with depth level. Then, get a list of 92 names where convolution layer is used in it. layernames = GroupBy[Keys@NetInformation[openpose, "Layers"], First] // Values; convlayernames = (Position[ NetInformation[openpose, "Layers"], _ConvolutionLayer] // Flatten)[[All, 1]] ![enter image description here][7] As you can see in keys, you can see that the order of convolution layers of OpenPose is different from the order of convolution layers of "pose_model_scratch.pth" keys ![enter image description here][8] So, manually sort the order of convolution layers in OpenPose into the order of "pose_model_scratch.pth" convlayernamesGH = {{"feature", "conv1_1"}, {"feature", "conv1_2"}, {"feature", "conv2_1"}, {"feature", "conv2_2"}, {"feature", "conv3_1"}, {"feature", "conv3_2"}, {"feature", "conv3_3"}, {"feature", "conv3_4"}, {"feature", "conv4_1"}, {"feature", "conv4_2"}, {"feature", "convadd1"}, {"feature", "convadd2"}, {"blk11", 1}, {"blk11", 3}, {"blk11", 5}, {"blk11", 7}, {"blk11", 9}, {"blk21", 1}, {"blk21", 3}, {"blk21", 5}, {"blk21", 7}, {"blk21", 9}, {"blk21", 11}, {"blk21", 13}, {"blk31", 1}, {"blk31", 3}, {"blk31", 5}, {"blk31", 7}, {"blk31", 9}, {"blk31", 11}, {"blk31", 13}, {"blk41", 1}, {"blk41", 3}, {"blk41", 5}, {"blk41", 7}, {"blk41", 9}, {"blk41", 11}, {"blk41", 13}, {"blk51", 1}, {"blk51", 3}, {"blk51", 5}, {"blk51", 7}, {"blk51", 9}, {"blk51", 11}, {"blk51", 13}, {"blk61", 1}, {"blk61", 3}, {"blk61", 5}, {"blk61", 7}, {"blk61", 9}, {"blk61", 11}, {"blk61", 13}, {"blk12", 1}, {"blk12", 3}, {"blk12", 5}, {"blk12", 7}, {"blk12", 9}, {"blk22", 1}, {"blk22", 3}, {"blk22", 5}, {"blk22", 7}, {"blk22", 9}, {"blk22", 11}, {"blk22", 13}, {"blk32", 1}, {"blk32", 3}, {"blk32", 5}, {"blk32", 7}, {"blk32", 9}, {"blk32", 11}, {"blk32", 13}, {"blk42", 1}, {"blk42", 3}, {"blk42", 5}, {"blk42", 7}, {"blk42", 9}, {"blk42", 11}, {"blk42", 13}, {"blk52", 1}, {"blk52", 3}, {"blk52", 5}, {"blk52", 7}, {"blk52", 9}, {"blk52", 11}, {"blk52", 13}, {"blk62", 1}, {"blk62", 3}, {"blk62", 5}, {"blk62", 7}, {"blk62", 9}, {"blk62", 11}, {"blk62", 13} }; Get the position of each element of convlayernamesGH in OpenPose. convlayerpos = Flatten[Position[layernames, #] & /@ convlayernamesGH, 1] ![enter image description here][9] Reshape each one-dimensional list of parameters to the dimension of the corresponding weight or bias. getDimB[layer_] := Dimensions@NetExtract[layer, "Biases"] getDimW[layer_] := Dimensions@NetExtract[layer, "Weights"] convs = NetExtract[NetInitialize[openpose], #] & /@ convlayerpos; dimW = getDimW /@ convs; dimB = getDimB /@ convs; dim = Flatten[Transpose[{dimW, dimB}], 1]; parametersReshape = MapThread[ArrayReshape, {parameters, dim}]; ## Step 5: Linking the Weights and the Biases ## Replace the initial values of weights and biases in OpenPose with learned parameters, and finally get trained OpenPose. replacenames = Flatten[ Transpose[{Flatten@{#, "Weights"} & /@ convlayernamesGH, Flatten@{#, "Biases"} & /@ convlayernamesGH}], 1]; rule = Thread[replacenames -> parametersReshape]; trainedOpenPose = NetReplacePart[openpose, rule] ![enter image description here][10] ## Step 6: Making the tests ## For simplification, estimate pose for the image of single-person. The Output2 of OpenPose shows the confidence of 19 body parts in each part where the image is divided into 46 * 46. 1:Nose, 2:Neck, 3:RShoulder, 4:RElbow, 5:RWrist, 6:LShoulder, 7:LElbow, 8:LWrist, 9:RHip, 10:RKnee, 11:RAnkle, 12:LHip, 13:LKnee, 14:LAnkle, 15:REye, 16:LEye, 17:REar, 18:LEar, 19:Bkg Define the function to get the position of max of confidence of each body part. maxpts[img_, confidences_, idex_] := Module[{pos, pts, h}, pos = Reverse@First@Position[h = confidences[[idex]], Max@h]; pts = (pos/46)*ImageDimensions@img; pts = {pts[[1]], (ImageDimensions@img)[[2]] - pts[[2]]} ] Connect the detected body parts and show the result together on the original image. showpose[img_] := Module[{bodylist, size, out, confidences, pts, pose}, bodylist = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14}; size = {368, 368}; out = trainedOpenPose[img]; confidences = out[[2]]; pts = maxpts[img, confidences, #] & /@ bodylist; pose = Graphics[{ Yellow, Thickness[.0125], Line[pts[[#]] & /@ {1, 2}], Green, Line[pts[[#]] & /@ {2, 3, 4, 5}], Cyan, Line[pts[[#]] & /@ {2, 6, 7, 8}], Orange, Line[pts[[#]] & /@ {2, 9, 10, 11}], Magenta, Line[pts[[#]] & /@ {2, 12, 13, 14}], PointSize[Large], Red, Point[pts], White, Point[{{0, 0}, ImageDimensions@img}] }, ImagePadding -> All]; Show[img, pose] ] Let' try. img = Import["ichiro.jpg"]; showpose[img] ![enter image description here][11] ## Future work ## ? Estimate the pose of an image in which multi-person are by using PAFs of output1 of OpenPose. ? Convert a more accurate pose estimation model. [1]: https://community.wolfram.com/groups/-/m/t/1785523 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=00.jpg&userId=1013863 [3]: https://arxiv.org/pdf/1611.08050.pdf [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=752004.jpg&userId=1013863 [5]: https://www.dropbox.com/s/ae071mfm2qoyc8v/pose_model.pth?dl=0 [6]: https://mathematica.stackexchange.com/questions/181273/how-to-import-%5C%20python-pickle-pkl?rq=1 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=691905.jpg&userId=1013863 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=387908.jpg&userId=1013863 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=109406.jpg&userId=1013863 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=179107.jpg&userId=1013863 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=161401.jpg&userId=1013863 Kotaro Okazaki 2019-09-10T21:36:49Z