python code on Heroku errors where it works locally

Question

I am getting too many errors on Heroku, so any print statements are not showing up in logs. This code works fine locally (python on windows 10) but wont work on Heroku, where it tells me there is nothing to concat...

path = r'.\store\tx' # use path for csv files
all_files = glob.glob(os.path.join(path , "tastyworks_tx*.csv")) # 

li = []
print(all_files)    

for filename in all_files:
    df = pd.read_csv(filename, index_col=None, header=0)
    print('filename: ', filename)
    input('pause here..')
    li.append(df)

frame = pd.concat(li, axis=0, ignore_index=True)   #append all the files to a single dataframe called "frame"

I have checked the \store\tx folder on Heroku and the tastyworks_tx*.csv files are present and I have other functions that can access files inside of the store folder but they use forward slanting (e.g. hist_data = pd.read_csv('./store/hist5m.csv', index_col=0, header=[0, 1]) works fine ).

On Heroku I get the following error.

2022-08-18T05:35:28.470644+00:00 app[web.1]: File "/app/./main.py", line 291, in tastytx 2022-08-18T05:35:28.470644+00:00 app[web.1]: frame = pd.concat(li, axis=0, ignore_index=True) #append all the files to a single dataframe called "frame"

...

raise ValueError("No objects to concatenate") 2022-08-18T05:35:28.470645+00:00 app[web.1]: ValueError: No objects to concatenate

my python dependencies used are same version as on the local setup, though the local setup has more installed. Here is my "requirements.txt" file for Heroku that installs fine:

pandas==1.4.2, pandas-datareader==0.10.0, DateTime==4.4, yfinance==0.1.72, fastapi==0.79.0, uvicorn==0.18.2, beautifulsoup4==4.11.1

I dont want to give Heroku my credit card info so cannot install Papertrail or better logging addons and cant figure out how to find out more of what is going wrong here due to limits on the logs.

"I have other functions that can access files inside of the store folder but they use forward slanting" -> Have you tried changing this code to use forward slashes? Backslashes are not valid directory separators on linux. — FiddleStix
– FiddleStix, Commented Aug 18, 2022 at 8:14
@FiddleStix the all_files variable ends up looking like this with the working code ['.\\store\\tx\\tastyworks_tx_2020.csv', '.\\store\\tx\\tastyworks_tx_2021.csv', '.\\store\\tx\\tastyworks_tx_2022.csv'] in windows, but I am so far unable to achieve the equivalent that works between platforms. I tried using pathlib Path but also not figured it yet. I think you might be right about this being the crux of the issue though. os.path.join is supposed to select the correct slashes for the OS but cant test in heroku without logs. — mdkb
– mdkb, Commented Aug 18, 2022 at 11:10
@Fiddlestix. you were right, if you want to add it as an answer I will close it. Basically, I got rid of path = r'.\store\tx' and changed the next line to all_files = glob.glob('./store/tx/tastyworks_tx*.csv', recursive = True) and it now works in Windows locally and in Heroku — mdkb
– mdkb, Commented Aug 18, 2022 at 11:49

FiddleStix · Accepted Answer · 2022-08-18 15:27:22Z

1

This is likely because you are using \ instead of / in the path.

On Linux, which is probably the OS for your Heroku machine, directories are separated with '/'. Your code will be looking for a file called precisely .\\store\\tx\\tastyworks_tx_2020.csv rather than a file called tastyworks_tx_2020.csv in .store/tx/.

Another (IMO more rubust) solution to replacing \ with / would be to use pathlib, which should work on all operating systems.

from pathlib import Path

the_path = Path("store") / Path("tx")
all_files = the_path.glob("tastyworks_tx*.csv") 
...

answered Aug 18, 2022 at 15:27

FiddleStix

3,90123 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

mdkb Over a year ago

when I checked what the new method creates in windows it shows me this ['./store/tx\\tastyworks_tx_2020.csv', './store/tx\\tastyworks_tx_2021.csv', './store/tx\\tastyworks_tx_2022.csv'] which has back slashes, but works both on windows python and on Heroku. When it was not working on Heroku I was getting ['.\\store\\tx\\tastyworks_tx_2020.csv', '.\\store\\tx\\tastyworks_tx_2021.csv', '.\\store\\tx\\tastyworks_tx_2022.csv'] with my original code. all attempts that failed related to ./store aspect being unable to pass the ./ so it might be the issue was that too.

Collectives™ on Stack Overflow

python code on Heroku errors where it works locally

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related