3

Situation: We receive statistical source data on monthly basis. It comes as a CD with an Access database which contains two tables. So we have 1 *.mdb file with 2 tables inside. Analysts should be able to import data, calculate and standardize it and check results. If results are fine data is populated. Analysts are not programmers and they don't have SQLServerManagementStudio.

My task: To create an access form as front-end for analysts with buttons: import, calculate, check, populate. Every button will fire the path trough query which run't stored procedure on sql server's side.

My problem: The bottleneck of my project is data import. It should be done automatically(NOT manually as analysts don't have management studio installed).

What I tried:

1) I linked SQL server's tables into access and tried to use regular queries to copy(INSERT INTO ** FROM *) but It takes too long. ( I have about 4 million rows in two tables)

2) As I fond out BULK INSERT is not working with importing from access file.

3) OPENROWSET approach:

SELECT *
FROM OPENROWSET( 'Microsoft.Jet.OLEDB.4.0','\\euro-dc\DOTS\2014\dots.mdb';'admin';'',TimeSeries)

Which returns me the error:

Msg 7308, Level 16, State 1, Line 1 OLE DB provider 'Microsoft.Jet.OLEDB.4.0' cannot be used for distributed queries because the provider is configured to run in single-threaded apartment mode.

Openrowset solution is not preferable. I would like to avoid it. Because it will will work on some computers and won't on other ones because of OLDB driver. After windows re-installation will need to check OLEDB driver ... it is not very comfortable as there are 15 analysts. Only if there is no other way to import data.

\euro-dc\dots folder is accessible for sql server - it is tested in my other project.

Will appreciate any fresh ideas. thanks

7
  • [stackoverflow.com/questions/22032222/… Commented Aug 11, 2014 at 14:23
  • @mohan111 maybe there is another way apart openrowset. Because than solution will work on some computers and won't on other ones. After windows re-installation will need to check OLEDB driver ... it is not very comfortable as there are 15 analysts. Commented Aug 11, 2014 at 14:40
  • Did you try copying the access files onto the local workstation before reading the records in to see if it speeds up the copy process? Sometimes I get a large performance decrease when trying to work with records on a CD or USB drive. Commented Aug 11, 2014 at 14:41
  • @RemedialBear yes my front-end solution is on my local machine. I tried to import 2 table from source DB into it and it was very quick. But when I try to put data into sql srv it takes ages Commented Aug 11, 2014 at 14:46
  • 1
    I would probably use one of the approaches that @Barry is suggesting in his answer (and related comments). Commented Aug 11, 2014 at 15:53

3 Answers 3

2

Use the import wizard and save the import specifications for each group of data you want to import. Then write some code/macro under a button that executes the import specifications. The tables you import into can be built in Access, then upsized to SQL Svr.

4 million recs is going to be slow, so you may want to put it in a separate MDB so that it doesn't tie up your analyst's working MDB.

Sign up to request clarification or add additional context in comments.

9 Comments

Do you mean I should save SSIS package? When I try it returns me "The attempted operation is not supported with this database version" (I have sql server 2012)
No. I'm not familiar with SSIS. So pardon me if I am missing your point. What I'm saying you should try is this: Fire up the Import Wizard and create your import ("File", "Get external Data", "Import" in Access 2003). Create your table in Access the first time you run this. Save this as an Import Specification. Now, do the import again, only this time specify the table you created the first time. Save this change.
(Took me longer than 5 min to edit)You can upsize the target table into SQL Server several ways, including using the Upsizing Wizard (Tools, Database utilities, Upsizing Wizard) or just manually typing the structure into SQL Server. Once you have it in SQL Server, attach it to your MDB, giving it the name used in your import spec.
@Almazini - Are you running SQL Server 2012 Express Edition? If so, then you may not be able to save SSIS packages with that version. (If I recall correctly that is one of the features that is disabled in the free Express Edition.)
@GordThompson I don't think Almazini is suggesting to use SSIS but rather the Import Wizard built into Access
|
2

Since you are dealing with a large number of records, you should consider the effect that indexes might be having on the performance of your data export/import process. A technique that some people use to speed things up in your situation is to first export the source data to a convenient intermediate file format (CSV or whatever), drop the indexes on the destination table, do a BULK INSERT to pull the intermediate file data into your destination table, then finally recreate the indexes on the destination table after the BULK INSERT completes.

This can help performance significantly, because recreating the indexes at the end (one time) is often much faster than the sum of all the index updates that occur when you insert rows one at a time.

I believe you should be able to programatically implement each of the steps above using VBA and T-SQL pass-through queries with Access.

Hope it helps. Good luck with your project.

2 Comments

that is exactly what I was trying to do yesterday. Idea was to convert tables to csv or txt and than do bulk insert. Problem is that access can not export tables in csv format and for txt format there is limitation of 65k rows.
A small VBA function that loops across the rows and columns of your table, writing the individual data items to a CSV file would do the trick. There is no limitation to 65K rows when you write your own export routine.
1

If you use something on the lines of the query below, it will run from MS Access. You also have the option of a pass-through query, or better yet, a little code to provide parameters.

INSERT INTO
[ODBC;DRIVER=SQL Server;Server=servername;Database=dbname;Integrated Security=SSPI].Table1
SELECT * FROM Table1

The easiest way to get the correct connection string is to link the table and then check the connect property.

You will also find connections strings at http://connectionstrings.com

Some examples, note that while names are used as a reference, it is really the order of the parameters that matters. For the most part, I check if I have a connection, to save time.

A query in MS Access

Set db = CurrentDb

sSQL = "INSERT INTO dbo_LocationLocationType (LocationID,LocationTypeID) VALUES (@LocationId,@LocationName)"
Set qdf = db.CreateQueryDef("", sSQL)

qdf.Parameters![@LocationId] = LocationID
qdf.Parameters![@LocationName] = LocationName

qdf.ReturnsRecords = False

qdf.Execute dbFailOnError

A stored procedure

 Dim cmd As New ADODB.Command
 Dim prm As ADODB.Parameter

    With cmd
        .ActiveConnection = "insert connection string"
        .CommandType = adCmdStoredProc
        .CommandText = "NameofStoredProcedure"

        Set prm = .CreateParameter("@LocationID", adInteger, adParamInput)
            prm.Value = frm.txtLocationID
            .Parameters.Append prm

        Set prm = .CreateParameter("@LocationName", adVarWChar, adParamInput, 255)
            prm.Value = frm.txtLocationDetailName
            .Parameters.Append prm

        .Execute

    End With

1 Comment

thanks for your answer. Connection between access and sql server is not a problem. I have linked tables and path-throw queries. but How you solution will change the situation with performance? As I know access sends one row in one transaction and that is why it takes so long to copy millions rows between servers.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.