How to Speed Up This Python Loop

Question

    downloadStart = datetime.now()
while (True):
    requestURL = transactionAPI.format(page = tempPage,limit = 5000)
    response = requests.get(requestURL,headers=headers)
    json_data = json.loads(response.content)
    tempMomosTransactionHistory.extend(json_data["list"])  
    if(datetime.fromtimestamp(json_data["list"][-1]["crtime"]) <  datetime(datetime.today().year,datetime.today().month,datetime.today().day - dateRange)):          
        break                       
    tempPage += 1
downloadEnd = datetime.now()

Any suggestions please threading or something like that ?

Outputs here

downloadtime 0:00:02.056010

downloadtime 0:00:05.680806

downloadtime 0:00:05.447945

Murali D · Accepted Answer · 2021-12-20 03:54:58Z

1

You need to improve it in two ways.

Optimise code within loop
Parallelize code execution

#1 By looking at your code I can see one improvement ie. create datetime.today object instead of doing 3 times. Check other methods like transactionAPI optimise further.

#2: If you multi core CPU machine then you take advantage of machine by spanning thread per page. Refer to modified code of above.

import threading

def processRequest(tempPage):
    requestURL = transactionAPI.format(page = tempPage,limit = 5000)
    response = requests.get(requestURL,headers=headers)
    json_data = json.loads(response.content)
    tempMomosTransactionHistory.extend(json_data["list"])
    
downloadStart = datetime.now()
while (True):
     #create thread per page
     t1 = threading.Thread(target=processRequest, args=(tempPage, ))
     t1.start()
     #Fetch datetime today object once instaed 3 times
     datetimetoday = datetime()
    if(datetime.fromtimestamp(json_data["list"][-1]["crtime"]) <  datetime(datetimetoday.year,datetimetoday.month,datetimetoday.day - dateRange)):          
        break                       
    tempPage += 1
downloadEnd = datetime.now()

answered Dec 20, 2021 at 3:54

Murali D

1161 silver badge8 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

hsnzkg Over a year ago

Thanks a lot. At the same time, I changed the download of data per request hourly. I download and update it hourly now and check it out. And now only sort time is important it has a small value like 0.3 seconds :)

Collectives™ on Stack Overflow

How to Speed Up This Python Loop

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related