0
$\begingroup$

Our test log files are stored in .asc format. Each file contains around 5000 rows of these Logging Strings as shown in the image. I would like to reduce the size of these files.They are around 4-5 MBs each bringing the total folder size for one test run around 700-900 MBs. The files are output from the system in only .asc format so i might have to opt for post-processing in this case. What would be the best way to reduce the size of the file, if i need reduction by a factor of 5 to 10?

  • Will a conversion to binary file format be suitable? If yes, is it going to be in .bin format or something else ?
  • If i opt for lossless compression technique for size reduction what would be a better option? Is there a ready tool that would run the technique for me like 7zip or should i be writing a program by myself? [I couldn't find much information leading to this point hence had to ask here]

Zip doesn't yield satisfactory result

One more question i had is why is it called .asc file ? It looks to me more like a .csv file or a .txt file. Is it that i see it in this format because of the file viewers i use ( notepad/notepad++/excel). Based on the formatting seeing in the image, isn't this more appropriate to be called a .csv file? This is just to enhance my knowledge about the difference between a csv, txt and .asc file.

enter image description here

Improve this question
$\endgroup$
6
  • $\begingroup$ asc stands for ASCII (readable text). $\endgroup$ Commented Oct 20, 2022 at 10:33
  • $\begingroup$ en.wikipedia.org/wiki/File_format $\endgroup$ Commented Oct 20, 2022 at 10:40
  • $\begingroup$ But do you really need to store all this data ? Do you exploit it ? $\endgroup$ Commented Oct 20, 2022 at 10:49
  • $\begingroup$ @YvesDaoust every information of each row string is needed. There are times when there are rows of strings are redundant . I could probably eliminate that but each data in a row is required $\endgroup$ Commented Oct 20, 2022 at 11:14
  • 1
    $\begingroup$ Lots of special-purpose compression you could do here. Observation 1: The date/time parts seem redundant to me the "2:38 AM" part is recoverable from 2:38:57.0, and the day of the week is recoverable from the date. Observation 2: The timestamps and date/times are ascending, so you only need to store deltas. Observation 3: There is a correlation between the final fields (the 0's and 1's), so "vertical coding" (i.e. think of transposing a matrix) is likely to pay off. $\endgroup$ Commented Oct 20, 2022 at 11:43

3 Answers 3

Reset to default
0
$\begingroup$

Unfortunately, conversion to a binary format will barely yield compression by a factor 2. Because your decimal numbers currently occupy 7 bytes, while a float takes 4 bytes. You can discard all repeating strings, which are useless, and pack the 17 final 0/1 values to 3 bytes. The date/times can be encoded as integers on 4 bytes (number of tenths of a second since a starting date).

Improve this answer
$\endgroup$
3
  • $\begingroup$ thanks for your response. I didn't get what you mean by repeating strings. Do you mean to say multiple rows of same data string? Or you mean repeating values in each string? As far as i understood, each row in the image is one single big string or did i interpret it wrong ? $\endgroup$ Commented Oct 20, 2022 at 11:23
  • $\begingroup$ hey @Yves i was going through your response again, could you maybe shed some more light on the decimal numbers occupying 7 bytes and float taking 4 bytes case. Where exactly where you pointing to in my database with this ? Thank you in advance :) $\endgroup$ Commented Oct 27, 2022 at 9:45
  • $\begingroup$ @SajeevPillai 0.8865, $\endgroup$ Commented Oct 27, 2022 at 10:07
0
$\begingroup$

The first column (1591231.911710) seems to be the number of seconds since some specific time (about 18 days earlier).

If you can preserve that time, " LoggingString := … 2021,02:38:57.0," becomes unnecessary, as it can be recalculated.

The times in the first column could be reduced by subtracting the previous line's value from each one.

So, the second line would be reduced to:

2.004776 70.66,13.4,…

Does the time need to have microsecond precision? If milliseconds are sufficient, some trailing digits could be removed:

2.005 70.66,13.4,…

Depending upon what they actually are, some further savings could be made by changing the values at the right the same way the time stamps were shortened, by recording them as the difference from the previous value. The "70.67"s would all become "0", and "13"s would become "0" or ".01" or "-.01".

And if those values all have at most 2 digits after the decimal point, that "." character could be removed.

And instances of "…,0,…" could even be reduced to "…,,…".

And since the whole thing now uses fewer than 17 distinct characters, each one could be packed into half a byte.

That should give an overall reduction in the lower end of the 5–10 range, and zipping the file might provide a little more saving.

Is it really worth the effort though, compared with the cost of additional storage?

Improve this answer
$\endgroup$
6
  • $\begingroup$ How do you conclude 5-10 range ? IMO, worse than 3. $\endgroup$ Commented Oct 20, 2022 at 15:30
  • $\begingroup$ @YvesDaoust, recording differences rather than values cuts it down by at least a factor of 2, and packing bytes gives another factor of 2. The other changes could easily push it past the 5 mark. $\endgroup$ Commented Oct 20, 2022 at 15:35
  • $\begingroup$ I am not convinced. And zipping an already "precompressed" file does not reduce it. $\endgroup$ Commented Oct 20, 2022 at 16:04
  • $\begingroup$ This does sound a lot complex than i imagined @RayButterworth.This scenario seems to be very case specific isn't it ? Is there any other method i could adopt not worrying about the contents of the file. It could happen that the test results come out in future in other structure and then this would be rendered useless $\endgroup$ Commented Oct 24, 2022 at 8:46
  • $\begingroup$ Also, 5-10 range means 5-10% or a factor of 5 - 10 ? @RayButterworth $\endgroup$ Commented Oct 24, 2022 at 8:47
0
$\begingroup$

The data consists of fixed text, and numbers that are multiples of 0.000001 or 0.1, converted to numbers or dates as text. So step 1 is that you multiply these numbers by 1,000,000 or 10 with an integer result.

Step 2 is recording not the numbers, but differences between consecutive number which are much smaller. And for the time information, you record the second differences. Then look up how h.264 store sequences of small integers with self-adjusting arithmetic coding with a different encoder for each number.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.