Message Boards Message Boards

0
|
3535 Views
|
3 Replies
|
0 Total Likes
View groups...
Share
Share this post:

Converting JSON to custom formatted CSV

Posted 2 years ago

I have a folder of about a million JSON files and I wrote this to convert them quickly to CSV, but the formatting is not correct for the software I'm feeding them into.

Here's the code I'm using:

Do[(Export["CSV_OUT_" <> FileBaseName@jsonlist[[i]] <> ".csv", 
Import[jsonlist[[i]]]]),{i, 1, Length[jsonlist]}];

The result looks like this:

"""from"" -> {""addresscity"" -> ""MORGANVILLE"", ""addresscountry"" -> ""UNITED STATES"", ""addressline1"" -> ""123 TEST ST"", ""addressline2"" -> ""STE 1"", ""addressstate"" -> ""NJ"", ""addresszip"" -> ""07751"", ""company"" -> """", ""name"" -> ""ELENA CROSS""}" """id"" -> ""sfmc4kjaugl7u8psvqfatp0""" """imbcode"" -> ""897714123456789""" """maildate"" -> """"" """mailtype"" -> ""uspsfirstclass""" etc.

The correct results would look like this in CSV format:
from addresscity,addresscountry,addressline1,addressline2,addressstate,addresszip,company,name,id,imbcode,maildate,mailtype,object,pressproof,size,targetdeliverydate,,to addresscity,addresscountry,addressline1,addressline2,addressstate,addresszip,company,name MORGANVILLE,UNITED STATES,123 TEST ST,STE 1,NJ,7751,,ELENA CROSS,sfmc4kjaugl7u8psvqfatp0,897714123456789,,uspsfirst_class,

How can I export correctly formatted CSV files like the one above quickly?

POSTED BY: Bob Hallam
3 Replies
Posted 2 years ago

Hi Bob,

CSV can only represent tabular data. The JSON sample contains hierarchical data, the "from" and "to" fields have nested keys and values. The usual way to deal with this is to unnest the nested structure by prefixing each field name in the inner structure with the field name of the outer structure.

unnest[prefix_String, assoc_Association] := 
 AssociationThread[prefix <> "." <> # & /@ Keys@assoc, Values@assoc];

toCSV[json_Association] := 
 json // 
   KeyValueMap[
    If[MatchQ[#2, _Association], 
      unnest[#1, #2], <|#1 -> #2|>] &] // {Flatten@Keys@#, Flatten@Values@#} &

json = Import["~/Downloads/sfm_c4kjaugl7u8psvqfatp0.json", "RawJSON"]
Export["sfm_c4kjaugl7u8psvqfatp0.json.csv", toCSV[json]]

This will work for one level of nesting, if there is more than one level then unnest has to be recursively called.

POSTED BY: Rohit Namjoshi
Posted 2 years ago

Can this be wrapped in a Do function to work on a folder of files?

POSTED BY: Updating Name
Posted 2 years ago

Sure, something like this

jsonFiles = FileNames["*.json", "path/to/folder"];

Export[# <> ".csv", toCSV@Import[#, "RawJSON"]] & /@ jsonFiles
POSTED BY: Rohit Namjoshi
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract