how do you create a large array?

Question

Given

// r is a System.Data.IDataRecord
var blob = new byte[(r.GetBytes(0, 0, null, 0, int.MaxValue))];
r.GetBytes(0, 0, blob, 0, blob.Length);

and r.GetBytes(...) returns Int64 Since Array.zeroCreate and Array.init take Int32 how do I create an empty array that is potentially larger than Int32.MaxValue?

Are you sure you want to create an array that is potentially larger than 2GB? — Fyodor Soikin
– Fyodor Soikin, Commented Jun 6, 2016 at 17:34
Agree with @FyodorSoikin - it's very unlikely that your PC / server can handle 2.1 billion+ contiguous memory addresses adequately. What problem are you trying to solve? — Haney
– Haney, Commented Jun 6, 2016 at 17:35
I'm not sure if it is dense or sparse, I'm just trying to translate what C# is already doing into valid (maybe even idiomatic functional) F# where possible. — Maslow
– Maslow, Commented Jun 6, 2016 at 19:42
@GuyCoder F# uses System.Array everywhere. [| 3 |] is a System.Array. Microsoft.FSharp.Collections.Array is just a module containing functions that operate on standard .NET arrays; it is not a different array type. — Asik
– Asik, Commented Jun 6, 2016 at 20:57

Tomas Petricek · Accepted Answer · 2016-06-06 20:25:30Z

In .NET 4.5 and newer, the gcAllowVeryLargeObjects configuration element allows you to create individual objects that are larger than 2GB, but even that does not change the hard limit on number of elements in an array. To quote from the Remarks section:

The maximum number of elements in an array is UInt32.MaxValue.

The maximum index in any single dimension is 2,147,483,591 (0x7FFFFFC7) for byte arrays and arrays of single-byte structures, and 2,146,435,071 (0X7FEFFFFF) for other types.

I'm not entirely sure why UInt32 rather than Int32 (you can create array where the smallest index is Int32.MinValue, but I'm not sure you'll be able to do much with it :-)).

However, with the gcAllowVeryLargeObjects parameter, you can create very large arrays by creating an array of structs that contain multiple values - this allocates continuous memory block over 2GB:

[<Struct>]
type Quadruple<'T>(v1:'T, v2:'T, v3:'T, v4:'T) = 
  member x.V1 = v1
  member x.V2 = v2
  member x.V3 = v3
  member x.V4 = v4

let qa : Quadruple<byte>[] = Array.zeroCreate 10

I don't think this is particularly practical, but it is one possibility. In reality, it seems easier just to split your data into an array of arrays. One level of indirection will probably not cause too much overhead, when the individual arrays are around 2GB.

Asik · Accepted Answer · 2016-06-06 20:22:35Z

3

You can't. The maximum length of a single-dimensional array on .NET is System.Int32.MaxValue. Your C# code raises an OverflowException when the value is greater than this limit. The equivalent F# code is:

let blob =         
    let length = r.GetBytes(0, 0, null, 0, Int32.MaxValue)
    if length > int64(Int32.MaxValue) then
        raise (OverflowException())
    else
        Array.zeroCreate<byte>(int32(length))

edited Jun 6, 2016 at 20:22

answered Jun 6, 2016 at 20:17

Asik

22.3k8 gold badges76 silver badges141 bronze badges

1 Comment

Maslow Over a year ago

that's all the info I was looking for. (doing the exact same thing the C# code would do in F#) excellent.

Collectives™ on Stack Overflow

how do you create a large array?

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related