Friday, August 14, 2009

How To Split A Huge CSV Excel Workbook Into Separate Files

Linked from Makeuseof.com

How To Split A Huge CSV Excel Workbook Into Separate Files:
csvHeadI recently started working with a company that does HUGE mailing lists. I deal with a lot of records everyday but this really is a lot of data.
They have an opt-in mailing list that they blast to once or twice a month. When I say huge I am talking about between 7 and 8 million email addresses.

Now if you have ever tried to open a file like this in excel you would see something that looks like this:


split excel workbook into seperate files
The reason you get this error is simple.
Excel can only handle 65,536 lines per sheet. It cannot display the rest of it and will truncate it. This is not good if you need to cut your list into smaller pieces. In my case I need to have less than 1 million records in my csv file to be able to import it into my mailing list program.
I tried using Base from Open Office with the same results. Now I could open this as a text file and chop it up manually or import it into an access database and try and manipulate it there as well. But hey let’s check out a free and EASY alternative to split an Excel workbook into seperate files. It is called FXFisherman’s CSV Splitter v1.1 by Sopheap LY.
We came across it here but I was able to trace it’s origins to the actual post on a forum it was created for and that is here. The original purpose of the app was to work with stock data in smaller chunks from a huge dump.
But hey I guess it will work on ANY csv file. Simply download the application and run it – no install required.
That will bring you to this screen:
split excel workbook into seperate files
Now this program has two simple options : you can split the csv file by the number of lines and specify the maximum number of files. Simply type in the number of lines you want to split each file into, and/or make the maximum pieces that you would want to see.
Next browse for your file by hitting the Browse file button and then hit the Split Now! Button.
When the program is complete you should see something that looks like this:
split excel workbook into seperate files
This tells me that my file of approximately 100,000 records was split successfully into a few even files. If they were more than 25 pieces with 50,000 lines in each it would have changed the algorithm to make each one slightly larger. Either which way this saved the day! I was able to import almost 4 million addresses at the end of the day into smaller chunks! (Thank god for interns!)
Do you know of a better, easier and / or faster way to accomplish the same thing?

Did you like the post? Please do share your thoughts in the comments section!
New on Twitter ? Now you can follow MakeUseOf on Twitter too.