Is it possible to scale large data set in a local file and not overload the HEAP?

Question

I have a vue project where I've made an aggregate data grid similar to Excel. I have a dataset, here's just one object in the array of objects as a small sample

export default [     < --- jsonDataSmall.js
  {
    id: 1,
    names: "Alice Anderson",
    language: "English",
    country: "USA",
    game: "Chess",
    bought: "true",
    balance: "$5,000",
    rating: "3",
    winnings: "$61,800",
    jan: "$65,243",
    feb: "$64,235",
    mar: "$60,845",
    apr: "$61,245",
    may: "$63,542",
    jun: "$66,452",
    jul: "$60,124",
    aug: "$58,142",
    sep: "$60,312",
    oct: "$59,842",
    nov: "$60,125",
    dec: "$61,520",
  },
...
]

//GridTemplate.Vue
export default {
  data() {
    return {
      tableData: jsonDataSmall,

Now for a project I'm doing, I'm supposed to hardcode the JSON, not call it from an API. I need to add 500,000 objects to this data set and scale it locally, so that means like 3 million lines of code in this jsonDataSmall.js file. Yes I know, why not use a database for a large scale data set? I have no idea but in this project I'm required to do that. Now when I add pagination, I can get up to about 50,000 JSON. But after that I get the error that the HEAP is overloaded.

So my question is - is this even possible without overloading the HEAP? How can I render the data and make the table optimized while not overloading the HEAP?

I've heard of things like Streaming from ChatGPT but don't fully understand it, other than that I am stumped

The available heap (FYI, heap is not capitalized) depends on whether a program is compiled for 32-bit (2-4 GB maximum address space) or 64-bit (no limit), and how much physical RAM and disk-based virtual RAM is available. Another limit is the data structures in the code, they might not have been written to track millions of objects or multiple GBs of data, But to answer just the question title, yes it is possible on a PC or server with enough RAM and the right code to process 60-120+ GB of data in memory. — Dave S
– Dave S, Commented Apr 10, 2023 at 17:17
@DaveS Man, I have no idea - tbh I am a junior developer and made this table and he want to see it rendered to scale. But doesnt want me to use a database. I am the person who set up this data structure. I guess another question is "is this something youd expect a junior developer to do?" lol - I just have no idea why he wants it done this way and I'm not sure that with the amount of RAM that I have that it is possible, specifically because the development server wont even build and says the heap is overloading — timventura
– timventura, Commented Apr 10, 2023 at 21:08

Dapo Michaels · Accepted Answer · 2023-04-11 08:52:34Z

0

Checkout Virtual scroll list package, I used it to render 20,000 records on a webpage at once. It uses intersection observer to achieve it functionality

answered Apr 11, 2023 at 8:52

Dapo Michaels

3922 gold badges4 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Is it possible to scale large data set in a local file and not overload the HEAP?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related