3

I'm exporting data from SqlLite using TSQLConnection with the following procedure:

  Campos := TStringList.Create;
  SQLiteConnection.Execute('SELECT * FROM tabela', nil, results);
  if not results.IsEmpty then
  begin
    results.GetFieldNames(Campos);

    XLApp1 := createoleobject('excel.application');
    XLApp1.Workbooks.Add(xlWBatWorkSheet);
    Sheet := XLApp1.Workbooks[1].WorkSheets[1];
    linha := 1;
    begin
       pBar.Max := results.RecordCount;
       for coluna := 0 to results.FieldCount - 1 do
           Sheet.Cells[1, coluna + 1]  :=  Campos[coluna];

       while ( not results.Eof ) do
       begin
          linha := linha + 1;
          for coluna := 0 to results.FieldCount - 1 do
              Sheet.Cells[linha, coluna + 1]  :=  results.FieldByName(campos[coluna]).AsString;
          results.next;
          pBar.Position := pBar.Position + 1;
          Application.ProcessMessages;
       end;
   end;

);

It works ok. But it takes too much time to finish. 35 minutes to export a table with 40,000 records and 45 fields on i7 note with SSD.

So my question: is there a chance I could do a little faster?

13
  • Start by removing Application.ProcessMessages. It does not magically speed things up as most people think - it actually slows things down. It should never be used. Commented Sep 8, 2017 at 17:16
  • Why are you assigning Sheet twice in a row. Don't you get a compiler hint "Value assigned to 'Sheet' never used"? Commented Sep 8, 2017 at 17:17
  • @JerryDodge Even if i take it does not help too much and I cant give any feedback to user through my progressbar. In a routine that takes too much time that is not a good idea. Commented Sep 8, 2017 at 17:22
  • 1
    Assuming you will cleanup your code (from forcing message pumping for UI updating and moving this code into a thread to accesing field by index), you may try ODBC driver for MS Excel. Maybe it will go faster than OLE automation. Commented Sep 8, 2017 at 17:56
  • 2
    if you're using application.processmessage to move the progressbar, see this article zarko-gajic.iz.hr/delphi-tprogressbar-not-updating-fast-enough this probably won't solve your problem, as 40,000 records will take some time.. might be better off exporting to .csv, then import into Excel. Commented Sep 8, 2017 at 18:05

1 Answer 1

5

The easiest way to do it is to collect all of the data into a single variant array, and then pass that over to Excel in a single pass. The vast portion of the time is in writing to Excel, and reducing it to one write operation is much faster.

The code below is adapted from code to transfer data from a TStringGrid to an Excel worksheet. Results is from your original code, so consider that it's been set up exactly as you've done it above where indicated. (The below code is untested as modified, because clearly I don't have your data to test against. It works with the stringgrid, as I've mentioned - I just couldn't test the adaptations. If you run into issues, leave a comment and I'll try to address them.)

With 40K rows of 35 fields, you may need to break it up into blocks (even doing a few hundred rows at a time would be a major performance improvement over doing it one at a time).

var
  xls, wb, Range: OLEVariant;
  arrData: Variant;
  RowCount, ColCount, i, j: Integer;
begin
  // Set up your dataset as Result here, just as in your own code
  // ....

  //create variant array where we'll copy our data
  RowCount := Results.RecordCount;
  ColCount := StringGrid1.FieldCount;
  arrData := VarArrayCreate([1, RowCount, 1, ColCount], varVariant);

  //fill array
  j := 1;
  while not Results.Eof do
  begin
    for i := 1 to ColCount do
      arrData[i, j] := Results.Fields[i].AsString;
    Inc(j);
    Results.Next;
  end;

  //initialize an instance of Excel
  xls := CreateOLEObject('Excel.Application');

  //create workbook
  wb := xls.Workbooks.Add;

  //retrieve a range where data must be placed
  Range := wb.WorkSheets[1].Range[wb.WorkSheets[1].Cells[1, 1],
                                  wb.WorkSheets[1].Cells[RowCount, ColCount]];

  //copy data from allocated variant array
  Range.Value := arrData;

  //show Excel with our data
  xls.Visible := True;
end;
Sign up to request clarification or add additional context in comments.

5 Comments

Thanks. Actually when I asked this I had something like that in mind 'cause I use arrData to read from Excel and it's really fast but didn't know how to use this the other way around.
I've tested this routine and it worked well with minor adjustments. I changed arrData[i, j] := Results.Fields[i].AsString; to arrData[j, i] := Results.Fields[i-1].AsString; and its ok now. It's ok and really fast BUT when it comes to the line Range.Value := arrData;it takes about 25 minutes to copy all the data.
@ReginaldoRigo: Then don't try to do it all at once. Do it in smaller bundles.
It didn't help anything. I've tried breaking into chunks of 1000 items each and it takes around 28 seconds each. So, saving as CSV is the best option to me now.
@ReginaldoRigo: Then you're doing something wrong. I've used the code above to export large amounts of data from Delphi to Excel. Perhaps you shouldn't be trying to export 40K rows of 35 fields. Are you opening and closing Excel each time?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.