Django using dumpdata to create database backups

Question

I have a production server running, and a local development one (this one uses the simple runserver, etc.).

I was planning on using Django's built in "dumpdata" command in order to create backups of the database every so often. Sadly, I can't seem to get "loaddata" to take in what "dumpdata" provides it.

On the production server:

python manage.py dumpdata > db_backup.json

This gives me a ~6MB file, which I then transfer over to my local machine. On my local machine:

bash-3.2$ python manage.py flush --settings=config.settings_dev
/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/models/fields/subclassing.py:80: DeprecationWarning: A Field class whose get_db_prep_save method hasn't been updated to take a `connection` argument.
  new_class = super(SubfieldBase, cls).__new__(cls, name, bases, attrs)
You have requested a flush of the database.
This will IRREVERSIBLY DESTROY all data currently in the 'dcif_db' database,
and return each table to the state it was in after syncdb.
Are you sure you want to do this?

    Type 'yes' to continue, or 'no' to cancel: yes

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): no
Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_manager(settings)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/core/management/base.py", line 351, in handle
    return self.handle_noargs(**options)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/core/management/commands/flush.py", line 74, in handle_noargs
    emit_post_sync_signal(set(all_models), verbosity, interactive, db)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/core/management/sql.py", line 190, in emit_post_sync_signal
    interactive=interactive, db=db)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/dispatch/dispatcher.py", line 172, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/contrib/auth/management/__init__.py", line 51, in create_permissions
    content_type=ctype
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/models/manager.py", line 138, in create
    return self.get_query_set().create(**kwargs)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/models/query.py", line 360, in create
    obj.save(force_insert=True, using=self.db)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/models/base.py", line 460, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/models/base.py", line 553, in save_base
    result = manager._insert(values, return_id=update_pk, using=using)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/models/manager.py", line 195, in _insert
    return insert_query(self.model, values, **kwargs)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/models/query.py", line 1436, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 791, in execute_sql
    cursor = super(SQLInsertCompiler, self).execute_sql(None)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 735, in execute_sql
    cursor.execute(sql, params)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/backends/util.py", line 34, in execute
    return self.cursor.execute(sql, params)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
    return self.cursor.execute(query, args)
django.db.utils.DatabaseError: can't adapt type '__proxy__'

I've had this problem a few weeks ago, and just running whatever command generated the error a second time made the error go away. Not the ideal solution, but I'm not sure how else to fix it.

Then:

bash-3.2$ python manage.py loaddata etc/db_backups/12_9_11.json --settings=config.settings_dev
/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/models/fields/subclassing.py:80: DeprecationWarning: A Field class whose get_db_prep_save method hasn't been updated to take a `connection` argument.
  new_class = super(SubfieldBase, cls).__new__(cls, name, bases, attrs)
Problem installing fixture 'etc/db_backups/12_9_11.json': Traceback (most recent call last):
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/core/management/commands/loaddata.py", line 174, in handle
    obj.save(using=using)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/core/serializers/base.py", line 165, in save
    models.Model.save_base(self.object, using=using, raw=True)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/models/base.py", line 570, in save_base
    created=(not record_exists), raw=raw, using=using)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/dispatch/dispatcher.py", line 172, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/Applications/djangostack-1.3-0/python/lib/python2.6/site-packages/djangobb_for_dcif/signals.py", line 18, in post_saved
    profile = post.user.forum_profile
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/models/fields/related.py", line 315, in __get__
    rel_obj = QuerySet(self.field.rel.to).using(db).get(**params)
  File "/Applications/djangostack-1.3-0/apps/django/lib/python2.6/site-packages/django/db/models/query.py", line 349, in get
    % self.model._meta.object_name)
DoesNotExist: User matching query does not exist.

Shouldn't the .json file provided by dumpdata be consistent, and not have missing objects as this error seems to claim?

On the production server, I'm using MySQL, and I'm using PostgreSQL locally, so I'm not sure if that's the cause of the problem.

Is there an easier way to do this?

Why wouldn't you develop using the same db server your application will be running on?? as you can see MySQL and Postgres are different enough so that it does matter. — dm03514
– dm03514, Commented Dec 9, 2011 at 15:49
Try use dumpscript from django-extensions It generates a standalone Python script that will repopulate the database using objects. So it should be db independent — Alexey Savanovich
– Alexey Savanovich, Commented Dec 9, 2011 at 18:23
@dm03514: The reason was because I started developing before the production one was online, and it was set up by someone else. Now that I've destroyed my local database anyway, I switched over to MySQL and everything seems to be working now. — numegil
– numegil, Commented Dec 10, 2011 at 2:18

soField · Accepted Answer · 2011-12-09 19:43:40Z

1

I think best way to do this export tables as simple sql inserts statements then you can re rerun insert sql scripts in target db

I'am using this method. I also tried dumpdata etc but insert statement way is easy and not complex.

answered Dec 9, 2011 at 19:43

soField

2,7369 gold badges37 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

wobbily_col Over a year ago

datadump will make it database independent though. I am looking to do something similar to convert from MySQL to Postgres

RUGVED · Accepted Answer · 2021-03-04 08:29:41Z

I wrote this blog where i explain how to convert all your database to csvs with django standalone script.
https://www.gyaanibuddy.com/blog/django-standalone-script-to-take-database-backup/

PROJECT_NAME = '--- ENTER YOUR PROJECT NAME HERE ---'

def main():
    from django.apps import apps
    import csv

    model_list = apps.get_models()
    model_name_list = [x.__name__ for x in model_list]
    
    for model in model_list:
        all_fields = model._meta.get_fields()
        columns = [x.name for x in all_fields]
        
        if not os.path.exists('csvs'):
            os.makedirs('csvs')

        with open(f'csvs/{model.__name__}.csv', mode='w') as csv_file:
            writer = csv.writer(csv_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
            
            # Writing column names
            writer.writerow(columns)

            objects = model.objects.all()
            
            for obj in objects:
                row = [str(getattr(obj, field_name,"NA")) for field_name in columns]
                writer.writerow(row)
    
if __name__ == '__main__':
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', '%s.settings' % PROJECT_NAME)
    import django
    django.setup()
    main()

Collectives™ on Stack Overflow

Django using dumpdata to create database backups

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related