Review Board Ticket #4920: Crash while running 'rb-site manage <path> dumpdb'

[Matthew Blythe]

To reply, visit https://hellosplat.com/s/beanbag/tickets/4920/

New ticket #4920 by mblythe For Beanbag, Inc. ▸ Review Board Crash while running 'rb-site manage <path> dumpdb' What version are you running? 4.0.0-rc1 What's the URL of the page containing the problem? N/A What steps will reproduce the problem? 'rb-site manage <path> dumpdb' What is the expected output? What do you see instead? I expect the command to progress to 100% and exit successfully. Instead, it errors out at about 43%. Output (filepaths changed because reasons): Traceback (most recent call last): File "/foobar/python3_user_packages/bin/rb-site", line 8, in <module> sys.exit(main()) File "/foobar/python3_user_packages/lib/python3.9/site-packages/reviewboard/cmdline/rbsite.py", line 2706, in main command.run(site, options) File "/foobar/python3_user_packages/lib/python3.9/site-packages/reviewboard/cmdline/rbsite.py", line 2476, in run site.run_manage_command(manage_command, manage_args) File "/foobar/python3_user_packages/lib/python3.9/site-packages/reviewboard/cmdline/rbsite.py", line 1047, in run_manage_command manage_util.execute() File "/foobar/python3_user_packages/lib/python3.9/site-packages/django/core/management/init.py", line 356, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/foobar/python3_user_packages/lib/python3.9/site-packages/django/core/management/base.py", line 283, in run_from_argv self.execute(args, cmd_options) File "/foobar/python3_user_packages/lib/python3.9/site-packages/django/core/management/base.py", line 330, in execute output = self.handle(*args, options) File "/foobar/python3_user_packages/lib/python3.9/site-packages/reviewboard/admin/management/commands/dumpdb.py", line 52, in handle value = serializer.serialize([obj]) File "/foobar/python3_user_packages/lib/python3.9/site-packages/django/core/serializers/base.py", line 97, in serialize self.end_object(obj) File "/foobar/python3_user_packages/lib/python3.9/site-packages/django/core/serializers/json.py", line 64, in end_object json.dump(self.get_dump_object(obj), self.stream, *self.json_kwargs) File "/foobar/lib/python3.9/json/init.py", line 179, in dump for chunk in iterable: File "/foobar/lib/python3.9/json/encoder.py", line 431, in _iterencode yield from _iterencode_dict(o, _current_indent_level) File "/foobar/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict yield from chunks File "/foobar/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict yield from chunks File "/foobar/lib/python3.9/json/encoder.py", line 438, in _iterencode o = _default(o) File "/foobar/python3_user_packages/lib/python3.9/site-packages/django/core/serializers/json.py", line 124, in default return super(DjangoJSONEncoder, self).default(o) File "/foobar/lib/python3.9/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.class.name} ' TypeError: Object of type bytes is not JSON serializable What operating system are you using? What browser? Red Hat Enterprise Linux 7 Please provide any additional information below. Using a postgresql 11.5 DB backend. Using python 3.9.1. Attempted the dump immediately after updating from ReviewBoard 3.0.20 New Priority:Medium Type:Defect

[Christian Hammond]

To reply, visit https://hellosplat.com/s/beanbag/tickets/4920/

New update by chipx86 For Beanbag, Inc. ▸ Review Board ▸ Ticket #4920 Hmm, that's strange. Looks more like an issue in Django's code. I'll see if I can reproduce. As mentioned on issue #4921, we're probably going to have to phase out these tools at some point, due to issues like this in Django. Native database dump/restore tools are more reliable. We do have a successor to dumpdb/loaddb that's in the works that avoids all the Django compatibility issues that these tools have (as they're based on Django's own loaddata/dumpdata, and aren't necessarily cross-version-compatible). That'll be coming as a free component to Power Pack in the future. Milestones: + Release-4.0 Tags: + Component:RB-Site Assigned to: + chipx86

[Christian Hammond]

To reply, visit https://hellosplat.com/s/beanbag/tickets/4920/

New update by chipx86 For Beanbag, Inc. ▸ Review Board ▸ Ticket #4920

After much analysis, and making a real attempt and keeping things compatible, the unfortunate conclusion is that we're removing these tools for 4.0. As mentioned, dumpdb/loaddb are fairly thin wrappers around Django's dumpdata/loaddata, and those are absolutely not forwards/backwards-compatible between the versions of Django used for Review Board 3.0/4.0. These were never really fully built for production use anyway, more for development (in fact, loading requires a development setup). There are plenty of issues deep within Django for compatible serialization, and we'd have to rewrite much of that stack, which isn't worth the time, given the work going into Power Pack. We'll be adding a deprecation notice and warning when running these tools in 3.0.21, and we'll be stubbing them out entirely in 4.0, showing just an error message with guidance on alternatives.

[Matthew Blythe]

To reply, visit https://hellosplat.com/s/beanbag/tickets/4920/

New update by mblythe

For Beanbag, Inc. ▸ Review Board ▸ Ticket #4920

absolutely not forwards/backwards-compatible between the versions of Django used for Review Board 3.0/4.0.

Yeah, it was pretty clear to me (from forum posts or documentation, I don't remember) that dumbdb/loaddb were only suitable for changing DB backends within the same version of Django and Review Board. And that was my use case - I wanted to migrate from postgresql to MySQL. guidance on alternatives At least for postgresql->MySQL migration, I had good luck with this process: 1) use pg_dump to snapshot the existing DB (data only, not table structures) 2) use https://github.com/ChrisLundquist/pg2mysql to convert it into a MySQL-compatible format (this required a bit of manual tweaking as well, mostly to the table names in the INSERT statements, and the syntax for binary values (i.e. '\x0123456789abcdef' to X'0123456789abcdef') ) 3) use 'rb-site install' to set up the new MySQL database 4) truncate the MySQL tables & load the data from the MySQL-compatible snapshot (need to disable foreign key checks during this step) I can understand you might be reluctant to 'bless' this flow by adding it to the documentation, but it would be nice if this were documented somewhere.