0

I am wanting to add the ability to import massive CSV files into a table in the database. I've built an SSIS package that does this, but I just wanted to make sure that this was the correct way of doing it.

The text files are millions of rows with 50 columns. They don't open in notepad or notepad++ a lot of times. The SSIS package handles them with no problem and gets everything imported. Is SSIS the right way to deal with this? I just need to pass the file location parameter to the job and execute it right? Is there an easier way that I am overlooking?

Improve this question

1 Answer 1

Reset to default
2

The text files are millions of rows with 50 columns

Small. Why an SSIS package?

They don't open in notepad or notepad++

because both OPEN them - there is no need to open them and load them all into memory, a proper application can read them line by line.

Is SSIS the right way to deal with this?

No. SImply because the title is: Importing huge data from text file to sql using c# winforms

As Winforms can do that effectively - I am inserting around 100.000 rows into a database - in C# with quit eeasy coding (except some pages and a day to get sqlbulkcopy to work properly) and you say nothing about any transformations, SSIS just is another not needed technology AND makes things complicated (as in: more to install, or hav a packag on the server but then find a location for the file that the server can reach etc.

I am all for SSIS and if you have a larger SSIS infrastructure or do a lot of processing etc. and it makes architectural sense in a larger context - yes. But as the question stands, without additional reasons - absolutely not.

Heck, there is a good chance you can load the whole thing in one command into SQL server as SQL Server has some CSV processing capabilities:

http://blog.sqlauthority.com/2008/02/06/sql-server-import-csv-file-into-sql-server-using-bulk-insert-load-comma-delimited-file-into-sql-server/

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

I came back to mention that this solution did not work for me. Bulk insert is very fast and nice, but it is missing many features. For example, handling of quotation marks in CSV's. I have built the same solution in SSIS and it is just as fast and takes care of many of the issues that bulk insert cannot cover. SSIS is a much more powerful ETL technology.
I totally agree. I have used both in the past - it really depends on circumstances.
It's ok though, bulk insert might help other people's scenarios if they read this post in the future so thank you for your advice. I am sure I will even use it in the future in another project.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.