Community RSS Feed
https://community.wolfram.com
RSS Feed for Wolfram Community showing any discussions in tag Import and Export sorted by activeProblem doing a bulk import into SQL with GDELT data
https://community.wolfram.com/groups/-/m/t/3146986
Hi There,
This is a strange one (for me). I'm trying to import data from the GDELT project (http://gdeltproject.org/), one of the files is attached.
I have MS SQL installed on my PC. When I try to do bulk import from these files, I get errors about data truncating. On examining the files, it looks like there is some problems with commas in certain columns (37, 44, and 51).
If I open the files in Excel, Excel misinterprets these columns with a comma (even though this is supposed to be a tab delimited document).
When I try to do things like the following, it still fails to format the files in a way that SQL can read.
removeCommasInColumns[row_List, cols_List] := MapAt[StringReplace[#, "," -> ""] &, row, List /@ cols]
dataworldevents = Import["mypath\\20201114.export.CSV", "TSV"];
processedData = removeCommasInColumns[#, {37, 44, 51}] & /@ dataworldevents;
Export["mypath\\yourfile.csv", processedData, "TSV"];
This is the bulk upload query I'm using:
dbQuery = "BULK INSERT worldevents
FROM 'mypath\\20201114.export.CSV'
WITH
(
FIRSTROW = 1,
FIELDTERMINATOR = '\\t',
ROWTERMINATOR = '\\n',
);";
Any help would be appreciated. I'd like to import a lot of data so this seems like it would be the fastest way to do it.S G2024-03-25T13:21:35Z