Is this Delphi dynamic array behaviour expected

Question

The question is - how dynamic arrays are managed internally by Delphi when they are set as a class member? Are they copied or passed by reference? Delphi 10.3.3 used.

The UpdateArray method deletes the first element from the array. But the array length stays 2. The UpdateArrayWithParam method also deletes the first element from the array. But the array length is correctly reduced to 1.

Here is a code sample:

interface

type
  TSomeRec = record
      Name: string;
  end;
  TSomeRecArray = array of TSomeRec;

  TSomeRecUpdate = class
    Arr: TSomeRecArray;
    procedure UpdateArray;
    procedure UpdateArrayWithParam(var ParamArray: TSomeRecArray);
  end;

implementation

procedure TSomeRecUpdate.UpdateArray;
begin
    Delete(Arr, 0, 1);
end;

procedure TSomeRecUpdate.UpdateArrayWithParam(var ParamArray: TSomeRecArray);
begin
    Delete(ParamArray, 0, 1);
end;

procedure Test;
var r: TSomeRec;
    lArr: TSomeRecArray;
    recUpdate: TSomeRecUpdate;
begin
    lArr := [];

    r.Name := 'abc';
    lArr := lArr + [r];
    r.Name := 'def';
    lArr := lArr + [r];

    recUpdate := TSomeRecUpdate.Create;
    recUpdate.Arr := lArr;
    recUpdate.UpdateArray;
    //(('def'), ('def')) <=== this is the result of copy watch value, WHY two values?

    lArr := [];

    r.Name := 'abc';
    lArr := lArr + [r];
    r.Name := 'def';
    lArr := lArr + [r];

    recUpdate.UpdateArrayWithParam(lArr);

    //(('def')) <=== this is the result of copy watch value - WORKS

    recUpdate.Free;
end;

Dynamic arrays are passed by reference. They are reference-counted and managed by the compiler. The documentation is pretty good. (And the details.) — Andreas Rejbrand
– Andreas Rejbrand, Commented Mar 12, 2020 at 18:17
Then why is the array length not updated after UpdateArray is called? — dwrbudr
– dwrbudr, Commented Mar 12, 2020 at 18:25
No it isn't. After recUpdate.UpdateArray is called, Length(lArr) is 2 — dwrbudr
– dwrbudr, Commented Mar 12, 2020 at 18:30
It's because of the Delete procedure. It has to reallocate the dynamic array, and so all pointers to it "must" move. But it only knows of one of these pointers, namely, the one you give to it. — Andreas Rejbrand
– Andreas Rejbrand, Commented Mar 12, 2020 at 18:48
I have analysed the problem and must admit I am unable to explain it. It feels like a bug. But maybe David or Remy or someone else knows more about this than I do. — Andreas Rejbrand
– Andreas Rejbrand, Commented Mar 12, 2020 at 19:27

Andreas Rejbrand · Accepted Answer · 2020-03-13 11:07:21Z

12

This is an interesting question!

Since Delete changes the length of the dynamic array -- just as SetLength does -- it has to reallocate the dynamic array. And it also changes the pointer given to it to this new location in memory. But obviously it cannot change any other pointers to the old dynamic array.

So it should decrease the reference count of the old dynamic array and create a new dynamic array with a reference count of 1. The pointer given to Delete will be set to this new dynamic array.

Hence, the old dynamic array should be untouched (except for its reduced reference count, of course). This is essentially documented for the similar SetLength function:

Following a call to SetLength, S is guaranteed to reference a unique string or array -- that is, a string or array with a reference count of one.

But surprisingly, this doesn't quite happen in this case.

Consider this minimal example:

procedure TForm1.FormCreate(Sender: TObject);
var
  a, b: array of Integer;
begin

  a := [$AAAAAAAA, $BBBBBBBB]; {1}
  b := a;                      {2}

  Delete(a, 0, 1);             {3}

end;

I have chosen the values so they are easy to spot in memory (Alt+Ctrl+E).

After (1), a points to $02A2C198 in my test run:

02A2C190  02 00 00 00 02 00 00 00
02A2C198  AA AA AA AA BB BB BB BB

Here the reference count is 2 and the array length is 2, as expected. (See the documentation for the internal data format for dynamic arrays.)

After (2), a = b, that is, Pointer(a) = Pointer(b). They both point to the same dynamic array, which now looks like this:

02A2C190  03 00 00 00 02 00 00 00
02A2C198  AA AA AA AA BB BB BB BB

As expected, the reference count is now 3.

Now, let's see what happens after (3). a now points to a new dynamic array at 2A30F88 in my test run:

02A30F80  01 00 00 00 01 00 00 00
02A30F88  BB BB BB BB 01 00 00 00

As expected, this new dynamic array has a reference count of 1 and only the "B element".

I would expect the old dynamic array, which b is still pointing to, to look as before but with a reduced reference count of 2. But it looks like this now:

02A2C190  02 00 00 00 02 00 00 00
02A2C198  BB BB BB BB BB BB BB BB

Although the reference count is indeed reduced to 2, the first element has been changed.

My conclusion is that

(1) It is part of the contract of the Delete procedure that it invalidates all other references to the initial dynamic array.

or

(2) It should behave as I outlined above, in which case this is a bug.

Unfortunately, the documentation for the Delete procedure doesn't mention this at all.

It feels like a bug.

Update: The RTL Code

I had a look at the source code of the Delete procedure, and this is rather interesting.

It might be helpful to compare the behaviour with that of SetLength (because that one works correctly):

If the reference count of the dynamic array is 1, SetLength tries simply to resize the heap object (and update the dynamic array's length field).
Otherwise, SetLength makes a new heap allocation for a new dynamic array with a reference count of 1. The reference count of the old array is decreased by 1.

This way, it is guaranteed that the final reference count is always 1 -- either it was that from the beginning or a new array has been created. (It is a good thing that you don't always make a new heap allocation. For instance, if you have a large array with a reference count of 1, simply truncating it is cheaper than copying it to a new location.)

Now, since Delete always makes the array smaller, it is tempting to attempt simply to reduce the size of the heap object where it is. And this is indeed what the RTL code attempts in System._DynArrayDelete. Hence, in your case, the BBBBBBBB is moved to the beginning of the array. All is well.

But then it calls System.DynArraySetLength, which is also used by SetLength. And this procedure contains the following comment,

// If the heap object isn't shared (ref count = 1), just resize it. Otherwise, we make a copy

before it detects that the object is indeed shared (in our case, ref count = 3), makes a new heap allocation for a new dynamic array, and copies the old (reduced) one to this new location. It reduces the ref count of the old array, and updates the ref count, length, and argument pointer of the new one.

So we ended up with a new dynamic array anyway. But the RTL programmers forgot that they had already messed up the original array, which now consists of the new array placed on top of the old one: BBBBBBBB BBBBBBBB.

edited Mar 13, 2020 at 11:07

answered Mar 12, 2020 at 19:19

Andreas Rejbrand

110k8 gold badges298 silver badges404 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

dwrbudr Over a year ago

Thanks Andreas! To be on the safe side, I'll just use pointers to dynamic arrays. I've tested how the same cases are handled with strings and it works as (2), e.g. new string is allocated (copy on write) and the original one (the local variable) is untouched.

Andreas Rejbrand Over a year ago

@dwrbudr: Personally, I'd avoid using Delete on a dynamic array. For one thing, it is not cheap on large arrays (since it necessarily needs to copy a lot of data). And this current issue makes me even more concerned, obviously. But let us also wait and see if the other Delphi members of the SO community agree with my analysis.

Andreas Rejbrand Over a year ago

@dwrbudr: Yes. Of course, dynamic arrays don't use copy-on-write semantics, but procedures like SetLength, Insert, and Delete obviously need to reallocate. Just changing an element (like b[2] := 4) will affect any other dynamic array variable pointing to the same dynamic array; there will be no copy.

J... Over a year ago

@AndreasRejbrand Indeed - there is absolutely no need to use Delete in new code in 2020. Let that dinosaur die.

GolezTrol Over a year ago

I can't seem to reproduce it in the slightly older Seattle.

|

Collectives™ on Stack Overflow

Is this Delphi dynamic array behaviour expected

1 Answer 1

Update: The RTL Code

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Update: The RTL Code

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related