C# chunking two-dimensional array into batches

Question

I have a two-dimensional object[,] array which contains a matrix of rows and columns (object[nRows, nColumns]).

I would like to chunk this into a batch of rows - e.g. batches of 1,000 rows each which I can enumerate over.

In summary, I am looking for C# code that does the following but for two dimensional arrays (source):

private IEnumerable<T[]> SplitArray<T>(T[] sourceArray, int rangeLength)
{
    int startIndex = 0;

    do
    {
        T[] range = new T[Math.Min(rangeLength, sourceArray.Length - startIndex)];
        Array.Copy(sourceArray, startIndex, range, 0, range.Length);
        startIndex += rangeLength;
        yield return range;
    }
    while (startIndex < sourceArray.Length);            
}

This attempt at adapting the code for [,] arrays fails - rows/columns begin to get jumbled-up after the first iteration:

        private IEnumerable<T[,]> SplitArray<T>(T[,] sourceArray, int rangeLength)
        {
            int startIndex = 0;

            do
            {
                T[,] range = new T[Math.Min(rangeLength, sourceArray.GetLength(0) - startIndex), sourceArray.GetLength(1)];
                Array.Copy(sourceArray, startIndex, range, 0, range.Length);
                startIndex += rangeLength;
                yield return range;
            }
            while (startIndex < sourceArray.GetLength(0));
        }

What stops you from from just independently enumerating over different rows in your 2D array? There should be no need to create (partial) clones of the array just for that purpose. (On the other hand, if splitting and creating new arrays is what you want, what precisely is your problem in adapting the code in your question for 2D [,] arrays?) — user2819245
– user2819245, Commented Mar 29, 2019 at 13:57
Thanks! They need to be chunked as they are have to be processed in batches via COM interop into another application. Single rows cause too many calls, and doing all rows in one go leads to insufficient memory. I updated the question with my attempt at adapting the code. — JDR
– JDR, Commented Mar 29, 2019 at 14:02
Do I understand you correctly - you want to break up a 2 dimensional array in 'square' sub-arrays (blocks)? How do you want to deal with the edges/leftovers? — Bart van der Drift
– Bart van der Drift, Commented Mar 29, 2019 at 14:06
Note that Array.Copy treats a 2D array like a 1D array (where the elements of the 2D array appear in row-major order). Thus, the value of startIndex is not related to the length of the 1st rank (i.e. the length of the rows). Which means, there is little sense in combining sourceArray.GetLength(0) - startIndex as you did there. Also, startIndex += rangeLength doesn't make much sense, either, because you just copied range.Length elements, not rangeLength elements (as you seem to use rangeLength only as a range for one dimension in your 2D array) — user2819245
– user2819245, Commented Mar 29, 2019 at 14:09
@BartvanderDrift I think that's the issue I'm having. Each block should be a self-contained set of rows with all of its columns left intact - if I am making any sense. — JDR
– JDR, Commented Mar 29, 2019 at 14:10

Magnetron · Accepted Answer · 2019-03-29 14:34:59Z

2

This will solve your code issues. As Array.Copy threats the array as a single dimensional, you have to multiply by the number of columns to get the total amount of elements in some places:

private IEnumerable<T[,]> SplitArray<T>(T[,] sourceArray, int rangeLength)
{
    int startIndex = 0;
    do
    {
        T[,] range = new T[Math.Min(rangeLength, sourceArray.GetLength(0) - startIndex/sourceArray.GetLength(1)), sourceArray.GetLength(1)];
        Array.Copy(sourceArray, startIndex, range, 0, range.Length);
        startIndex += rangeLength*sourceArray.GetLength(1);
        yield return range;
    }
    while (startIndex < sourceArray.Length);
}

answered Mar 29, 2019 at 14:34

Magnetron

8,7732 gold badges31 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

David · Accepted Answer · 2019-03-29 14:32:40Z

By Using GetLength(int dimension), you can see how long a particular dimension is for an array, and then iterate through that. You'll also need to take the other dimensions as constants, and make sure the whole thing matches up to the Array.Rank value. From there, just look up the value via Array.GetValue(int[]). This may be a touch difficult since Array isn't generic:

public static IEnumerable<T> GetRow<T>(this Array source, int dimension, params int[] fixedDimensions)
{
    if(source == null) throw new ArgumentNullException(nameof(source));
    if(!typeof(T).IsAssignableFrom(source.GetType().GetElementType()) throw new OperationException($"Cannot return row of type {typeof(T)} from array of type {source.GetType().GetElementType()}");

    if(fixedDimensions == null) fixedDimensions = new T[0];
    if(source.Rank != fixedDimensions.Length + 1) throw new ArgumentException("Fixed dimensions must have exactly one fewer elements than dimensions in source", nameof(fixedDimensions));
    if(dimension > source.Rank) throw new ArgumentException($"Cannot take dimension {dimension} of an array with {source.Rank} dimensions!", nameof(dimension));
    if(dimension < 0) throw new ArgumentException("Cannot take a negative dimension", nameof(dimension));

    var coords = dimension == source.Rank
         ? fixedDimensions
            .Concat(new [] { 0 })
            .ToArray()
        : fixedDimensions
            .Take(dimension)
            .Concat(new [] { 0 })
            .Concat(fixedDimensions.Skip(dimension))
            .ToArray();

    var length = source.GetLength(dimension);
    for(; coords[dimension] < length; coords[dimension]++)
    {
        yield return (T)source.GetValue(coords);
    }
}

Bart van der Drift · Accepted Answer · 2019-03-29 14:40:34Z

1

I think you're looking for something like this:

private static List<T[]> SplitArray<T>(T[,] sourceArray)
{
    List<T[]> result = new List<T[]>();
    int rowCount = sourceArray.GetLength(0);
    for (int i = 0; i < rowCount; i++)
    {
        result.Add(GetRow(sourceArray, i));
    }

    return result;
}

private static T[] GetRow<T>(T[,] sourceArray, int rownumber)
{
    int columnCount = sourceArray.GetLength(1);
    var row = new T[columnCount];
    for (int i = 0; i < columnCount; i++)
    {
        row[i] = sourceArray[rownumber, i];
    }
    return row;
}

answered Mar 29, 2019 at 14:40

Bart van der Drift

1,33715 silver badges31 bronze badges

Collectives™ on Stack Overflow

C# chunking two-dimensional array into batches

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related