How to divide array of file paths into several smaller arrays?

Question

I've read several different questions from SO about this already but still haven't been able to get it to work. I'm using public static string[] files = Directory.GetFiles(CurrentDirectory, "*.wav", SearchOption.AllDirectories); to get an array of file paths that will then be passed of to a filestream. The operations the filestream does were taking too long with just one thread handling all of the files. So I decided I'd split the array and pass those smaller arrays to different threads.

The code I was using for that I got from another SO question, and used it to pass the split array, but it only worked with just one file in the first array, but I know what the problem was:

var thing = from index in Enumerable.Range(0, files.Length) 
          group files[index] by index/600;
foreach(var set in thing)
    string.Join(";", set.ToArray());

(This isn't exactly how I used it, I've messed with it so much I can't remember.) The problem with this is that everything was treated as just one massive file path, I have a foreach loop that gets each file from the smaller array, but it treated every file in it as just one, throwing the filepathtoolong exception when there was more than one file returned from the search. My function takes an array and then uses foreach (string file in smallerArray) to write to each one. What I need to do is break up the files array into 4 smaller arrays and start the new threads like new Thread(() => { DoWork(newArray); }).Start(); but nothing I've tried has worked.

Have you considered using Parallel LINQ instead of writing this yourself? — Tim B
– Tim B, Commented Aug 1, 2012 at 18:32
Probably better to use Directory.EnumerateFiles() as the source. — Henk Holterman
– Henk Holterman, Commented Aug 1, 2012 at 19:54

Jon Skeet · Accepted Answer · 2012-08-01 18:33:41Z

5

So I decided I'd split the array and pass those smaller arrays to different threads.

Sounds like you're doing it the hard way :) Let the framework handle it for you with Parallel.ForEach:

Parallel.ForEach(files, file => 
{
    // Do stuff with one file
});

answered Aug 1, 2012 at 18:33

Jon Skeet

1.5m893 gold badges9.3k silver badges9.3k bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

0_______0 Over a year ago

I'd forgotten about this. I have several thousand files, this would spawn a thread per file, right? Is there going to be a performance impact on running that many threads at the same time? Or will the framework limit the number of concurrently running threads when using Parallel.ForEach?

Tim B Over a year ago

See stackoverflow.com/questions/5512312/…

Jon Skeet Over a year ago

@0_______0: Sorry for not replying before. No, it doesn't spawn a thread per file. It creates a task per file, which isn't the same thing. It's like adding tasks into the thread pool. You can explicitly limit the degree of parallelism though - in particular, if these are files on the same disk, it's unclear that any parallelism will actually help. (It will depend on the disk type too.)

GSerjo · Accepted Answer · 2012-08-01 18:35:54Z

Here is example

    public static IEnumerable<IEnumerable<T>> Split<T>(this IEnumerable<T> source, int blockSize)
    {
        if (source == null)
            throw new ArgumentNullException("source");
        if (blockSize <= 0)
            throw new ArgumentException("blockSize = {0}".FormatWith(blockSize), "blockSize");
        var result = new List<IEnumerable<T>>();
        for (int blockStartIndex = 0; blockStartIndex < source.Count(); blockStartIndex += blockSize)
        {
            int blockStart = blockStartIndex;
            int blockEnd = blockStartIndex + blockSize - 1;
            IEnumerable<T> block = source.Where((x, i) => i >= blockStart && i <= blockEnd);
            result.Add(block);
        }
        return result;
    }

here is test

    [Test]
    public void TestSplit()
    {
        var list = new[] {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
        IEnumerable<IEnumerable<int>> splitted = list.Split(10);
        Assert.That(splitted.Count(), Is.EqualTo(1));
        Assert.That(splitted.First().Count(), Is.EqualTo(10));

        splitted = list.Split(11);
        Assert.That(splitted.Count(), Is.EqualTo(1));
        Assert.That(splitted.First().Count(), Is.EqualTo(10));

        splitted = list.Split(9);
        Assert.That(splitted.Count(), Is.EqualTo(2));
        Assert.That(splitted.First().Count(), Is.EqualTo(9));
        Assert.That(splitted.ElementAt(1).Count(), Is.EqualTo(1));

        splitted = list.Split(3);
        Assert.That(splitted.Count(), Is.EqualTo(4));
        Assert.That(splitted.First().Count(), Is.EqualTo(3));
        Assert.That(splitted.ElementAt(1).Count(), Is.EqualTo(3));
        Assert.That(splitted.ElementAt(2).Count(), Is.EqualTo(3));
        Assert.That(splitted.ElementAt(3).Count(), Is.EqualTo(1));
    }

Collectives™ on Stack Overflow

How to divide array of file paths into several smaller arrays?

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related