Take specific number of array first then process

Question

I have this code below that:

InstanceCollection instances = this.MyService(typeID, referencesIDs);

My problem here is when the referencesIDs.Count() is greater than a specific count, it throws an error which is related to SQL.

Suggested to me is to call the this.MyService multiple times so it won't process many referencesIDs.

What is the way to do that? I am thinking of using a while loop like this:

while (referencesIDs.Count() != maxCount)
{
    newReferencesIDs = referencesIDs.Take(500).ToArray();
    instances = this.MyService(typeID, newReferencesIDs);
    maxCount += newReferencesIDs.Count();
}

The problem that I can see here is that how can I remove the first 500 referencesIDs on the newReferencesIDs? Because if I won't remove the first 500 after the first loop, it will continue to add the referencesIDs.

Use Skip(), so newReferenceIDs = referenceIDs.Skip(500).Take(500).ToArray() — DavidG
– DavidG, Commented Jul 21, 2014 at 14:59

David · Accepted Answer · 2014-07-21 15:00:20Z

2

Are you just looking to update the referencesIDs value? Something like this?:

referencesIDs = referencesIDs.Skip(500);

Then the next time you call .Take(500) on referencesIDs it'll get the next 500 values.

Conversely, without updating the referencesIDs variable, you can include the Skip in your loop. Something like this:

var pageSize = 500;
var skipCount = 0;
while(...)
{
    newReferencesIDs = referencesIDs.Skip(skipCount).Take(pageSize).ToArray();
    skipCount += pageSize;
    ...
}

answered Jul 21, 2014 at 15:00

David

221k42 gold badges245 silver badges337 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

DavidG Over a year ago

Was just about to suggest a pageSize variable!

Sergey Kalinichenko · Accepted Answer · 2014-07-21 15:10:06Z

2

My first choice would be to fix the service, if you have access to it. A SQL-specific error could be a result of an incomplete database configuration, or a poorly written SQL query on the server. For example, Oracle limits IN lists in SQL queries to about 1000 items by default, but your Oracle DBA should be able to re-configure this limit for you. Alternatively, server side programmers could rewrite their query to avoid hitting this limit in the first place.

If this does not work, you could split your list into blocks of max size that does not trigger the error, make multiple calls to the server, and combine the instances on your end, like this:

InstanceCollection instances = referencesIDs
    .Select((id, index) => new {Id = id, Index = index})
    .GroupBy(p => p.Index / 500) // 500 is the max number of IDs
    .SelectMany(g => this.MyService(typeID, g.Select(item => item.Id).ToArray()))
    .ToList();

answered Jul 21, 2014 at 15:10

Sergey Kalinichenko

729k85 gold badges1.2k silver badges1.6k bronze badges

2 Comments

vgru Over a year ago

+1, one difference might be that this will change the order of IDs in case they weren't sorted (which may or may not matter to OP).

Gerald Over a year ago

Thank you for this. I know modifying the SQL is much better but I have SLA to meet. Thanks again!

vgru · Accepted Answer · 2014-07-21 15:24:51Z

If you want a general way of splitting lists into chunks, you can use something like:

/// <summary>
/// Split a source IEnumerable into smaller (more manageable) lists.
/// </summary>
public static IEnumerable<IList<TSource>> 
    SplitIntoChunks<TSource>(this IEnumerable<TSource> source, int chunkSize)
{
    long i = 1;
    var list = new List<TSource>();
    foreach (var t in source)
    {
        list.Add(t);
        if (i++ % chunkSize == 0)
        {
            yield return list;
            list = new List<TSource>();
        }
    }
    if (list.Count > 0)
        yield return list;
}

And then you can use SelectMany to flatten results:

InstanceCollection instances = referencesIDs
    .SplitIntoChunks(500)
    .SelectMany(chunk => MyService(typeID, chunk))
    .ToList();

Collectives™ on Stack Overflow

Take specific number of array first then process

3 Answers 3

1 Comment

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related