0

I am encountering an issue where memory is not being released back to the OS, even after clearing large in-memory tables and running .Q.gc[]. Despite multiple attempts, the process continues to consume a large amount of RAM.

The process handles real-time data ingestion and maintains large in-memory tables throughout the day. At the end of the day, data is partitioned on disk, and all in-memory tables are cleared using:

`tableName set 0#tableName  / Empty the table

Garbage collection is then triggered: .Q.gc[]

However, heap memory remains significantly larger than used memory.

Memory Stats After Cleanup (.Q.w[] Output):

(`used`heap`peak`wmax`mmap`mphy`syms`symw)!
(550678528j, 6174015488j, 219915747328j, 0j, 0j, 1622702747648j, 390966606j, 126656286272j)

Observations

used = 550MB but heap = 5.75GB, indicating significant fragmentation.

Peak memory usage reached 204GB at some point.

symw = 118GB, symbol memory is consuming a large portion of RAM.

.Q.gc[] runs successfully but does not free memory back to the OS.

Symbol memory (symw) is high, but symbols cannot be garbage collected without a restart. Are there any techniques I can try to reduce memory fragmentation? Is there a way to force KDB+ to release memory back to the OS without restarting?

Thank you in advance for your response.

2
  • 1
    Those are huge values for syms and symw and the creation of so many unique symbol values should be investigated. 126656286272%390966606=323 suggesting long symbols are being generated. On disk what is the size/count of the sym file? Does it have a similar issue? Commented Mar 20 at 10:27
  • I agree with Rian ... you're asking about how to shave off ~5gb in unreleased heap memory when there's a 120gb-sized elephant in the corner of the room. Even if you could get savings from the heap it's just a drop in the ocean. Your symbol problem needs to be solved first Commented Mar 25 at 13:42

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.