1

I have a quite large db of 750MB which I need to migrate from sqlite to postgres. As, the db is quite large I am facing some issues that some of the previous questions on the same topic did not had.

Few days ago, I have migrated one sqlite db of size 30MB without any issues with loaddata and dumpdata commands. But for this large db, one of my app throws Database image is malformed error when running command dumpdata. Another of my app dumps successfully but does not load. I have seen the with -v 3 verbose flag that the objects are not even processed. To be precise, while running the loaddata command data from json file is processed first to check duplicate primary key and other model constraints then those data are used to create model objects. But for this app, data are not processed in the first place.

Apart from this two commands there are some other methods that does the migration. But, the schema is completely changed in those way which I don't desire to do. Moreover I have faced issue DurationField becomes a string after migration and I couldn't typecast those fields. Bigint becomes int and varchar becomes text etc are some of the issues. I cannot afford to have this kind of issues.

Improve this question
6
  • Can you post the command you are using for loaddata? Commented Jul 12, 2021 at 14:11
  • ./manage.py loaddata -v 3 data.json Commented Jul 12, 2021 at 16:28
  • Are you defining more than one database? have you tried adding --app to the command, python manage.py loaddata -v3 --app YOURAPP data.json? Commented Jul 12, 2021 at 19:01
  • Yes I did similar thing. Dumped specific app and then tried to load the data. One of my app has the issue where it does not processing the objects in the json file. Commented Jul 13, 2021 at 14:07
  • 1
    Yes. I was. The issue is resolved now. I had a recursive model schema and one of the entry was corrupted which resulted in this issue Commented Jul 14, 2021 at 6:01

2 Answers 2

Reset to default
2

I found the issues for these problems.

  1. Database image is malformed occurred because I was copying entire db file directly. Just as, angardi suggested, I compressed the db file using gzip package and then copied it. Which solved this issue.

  2. There were two issues for which loaddata was not loading a certain app. That app contained a model which had a foreign key to itself. And one of the entry of the model had the foreign key set to the entry. This resulted in an infinite loop and the app wasn't loading. Even after fixing the entry I was facing the same issue. This happened because I dumped data of a specific app. One of the model of the app had a ManyToMany field set to a model of another app. Though, loaddata throws error when ForeignKey field reference is not found, I think it kept searching for the ManyToMany fields without throwing an error which turned into an infinite loop type of situation. All, I had to do here is, dump the entire database or the related app on the same file.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

Read this for the image malformed part.

When migrating from one database to another, you should first dump all data, then run django migrations on the new database and the load data. That should retain all column types.

When running loaddata, the tables that have no dependencies should be loaded first and then the ones that depend on those already loaded.

Improve this answer

3 Comments

As for the malformed image issue, I think the reason might be downloading the db. As the db is in remote server, I am actually using scp command to copy the db locally. Somehow, the db may have been corrupted when copying. For the migration, I am actually doing the same thing. But the problem is one of my app is not loaded when running loaddata. This is the main issue. As for the dependency, tables of this app has no dependency
Have you tried compressing the db file and then copying it?
No. I will try that.. Any idea for the loaddata issue?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.