.NET Core high memory usage base64 encoding messages from database

Question

I have a large database of messages I need to send to other systems. The messages average 5-10mb in size but some are larger (50-70mb). I need to base64 encode each message to disk. My application is consuming a large amount of memory encoding the messages to disk and I cannot figure out why.

What I have tried: I have read multiple posts regarding dispose method. I have implemented the dispose function as much as I can (via using) to release resources but I am still observing high memory usage (700mb+) for my Console application.

To keep things simple, I selected the 10 largest files from my database that range from 55-75mb. Under normal circumstances, the application will be processing messages 24/7.

Here is what my memory usage looks like when executing the below code. The flat line indicates the end of processing the large files. Why are these resources not being released?

Here is a capture of my heap view:

Here's my code:

const string sql = "SELECT TOP 10 * FROM [dbo].[OutboundMessages] ORDER BY [Length] DESC";
using (SqlConnection connection = new SqlConnection(ConnectionString))
{
    connection.Open();
    using (SqlCommand command = new SqlCommand(sql, connection))
    using (SqlDataReader reader = command.ExecuteReader(CommandBehavior.SequentialAccess))
    {
        if (reader.HasRows)
        {
            while (reader.Read())
            {
                using (MemoryStream stream = new MemoryStream())
                {
                    int ordinal = reader.GetOrdinal("FileData");
                    long bytesRead, offset = 0;
                    byte[] buffer = new byte[4096];
                    while ((bytesRead = reader.GetBytes(ordinal, offset, buffer, 0, buffer.Length)) > 0)
                    {
                        offset += bytesRead;
                        stream.Write(buffer, 0, (int)bytesRead);
                    }
                    string file = Guid.NewGuid().ToString().Replace("-", string.Empty);
                    using (StreamWriter writer = new StreamWriter(file))
                    {
                        writer.Write(Convert.ToBase64String(stream.ToArray()));
                    }
                }
            }
        }
    }
}

.Net doesn't aggressively give back memory to the OS, and the GC will wait until it feels like it to collect and compact memory. There is not enough information in this post to say whether you have a problem or not.. I suggest doing more memory profiling, and not just relying on the graph — TheGeneral
– TheGeneral, Commented May 27, 2019 at 1:27
Thank you for your response. What can I add to my post to help you help me? — mr.coffee
– mr.coffee, Commented May 27, 2019 at 1:28
Maybe take a look at this learn.microsoft.com/en-us/visualstudio/profiling/… — TheGeneral
– TheGeneral, Commented May 27, 2019 at 1:31

mr.coffee · Accepted Answer · 2019-05-27 02:17:40Z

I solved the issue by writing the bytes directly to file in multiples of 3 bytes (3*1024=3072). Now my process memory stays at a stable 20mb. Writing 3 bytes (or multiples of) at a time ensures base64 does not lose it's binary to text formatting.

const string sql = "SELECT TOP 10 * FROM [dbo].[OutboundMessages] ORDER BY [Length] DESC";
using (SqlConnection connection = new SqlConnection(ConnectionString))
{
    connection.Open();
    using (SqlCommand command = new SqlCommand(sql, connection))
    using (SqlDataReader reader = command.ExecuteReader(CommandBehavior.SequentialAccess))
    {
        if (reader.HasRows)
        {
            while (reader.Read())
            {
                string file = Guid.NewGuid().ToString().Replace("-", string.Empty);
                using (StreamWriter writer = new StreamWriter(file))
                {
                    int ordinal = reader.GetOrdinal("FileData");
                    long bytesRead, offset = 0;
                    byte[] buffer = new byte[3072];
                    while ((bytesRead = reader.GetBytes(ordinal, offset, buffer, 0, buffer.Length)) > 0)
                    {
                        offset += bytesRead;
                        writer.Write(Convert.ToBase64String(buffer));
                    }
                }
            }
        }
    }
}

Collectives™ on Stack Overflow

.NET Core high memory usage base64 encoding messages from database

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related