[GrandComicsDatabase/gcd-django] Add deterministic development data and dump setup (PR #748)
Adam Hernandez
Summary
I added the PR3 data layer for the new development environment:
- Added an idempotent
seed_development_datacommand with local accounts, catalog relationships, variants, reprints, stories, credits, features, universes, characters, groups, covers, images, and approved change history. - Added deterministic
[GCD DEV]records and small dummy image files so contributors can exercise representative workflows without a production database. - Added the one-command
./bin/dev setup --dump ARCHIVEworkflow for importing the public catalog dump into a clean migrated development database. - Updated the core setup guide and README to document the workflow and local credentials.
PR4 (Elasticsearch 7 search integration) will follow separately after this is merged.
Verification
- Focused seed test passes:
apps/gcd/tests/test_seed_development_data.py - Docker Compose rebuild and health-gated startup pass.
./bin/dev doctorpasses with Python 3.13.15, Django 5.2.17, and MySQL 8.0.46.- Local homepage returns HTTP 200.
- The seed command was run twice against the existing local catalog database without duplicating records.
- The public catalog dump import path was previously validated against the downloaded archive.
You can view, comment on, or merge this pull request online at:
https://github.com/GrandComicsDatabase/gcd-django/pull/748
Commit Summary
File Changes
(8 files)
- M README.md (9)
- A apps/gcd/management/__init__.py (1)
- A apps/gcd/management/commands/__init__.py (1)
- A apps/gcd/management/commands/seed_development_data.py (343)
- M apps/gcd/tests/test_dev_environment_core.py (12)
- A apps/gcd/tests/test_seed_development_data.py (57)
- M bin/dev (196)
- M docs/development/CORE_SETUP.md (63)
Patch Links:
- https://github.com/GrandComicsDatabase/gcd-django/pull/748.patch
- https://github.com/GrandComicsDatabase/gcd-django/pull/748.diff
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
gemini-code-assist[bot]
@gemini-code-assist[bot] commented on this pull request.
Code Review
This pull request introduces a new ./bin/dev setup command to easily bootstrap a local development environment with a small, deterministic dataset or an optional full catalog dump. It adds the seed_development_data management command, updates documentation, and includes corresponding tests. Feedback on the changes highlights two critical issues in the shell script: the need to validate the dump archive's existence and format upfront to prevent accidental database deletion, and the requirement to increase MySQL's group_concat_max_len session variable to avoid query truncation on large tables.
In bin/dev:
> + local archive="$1" replace="$2" yes="$3" + validate_setup_database_name + compose up -d --wait db + confirm_setup_replacement "$replace" "$yes" + fresh_docker_database
Validate the existence and file extension of the dump archive upfront before prompting the user or running fresh_docker_database. Currently, if a user provides a non-existent file or an unsupported file type, the existing local database is destroyed by fresh_docker_database before the script validates the archive and exits, leading to accidental data loss.
local archive="$1" replace="$2" yes="$3"
validate_setup_database_name
if [[ -n "$archive" ]]; then
if [[ ! -f "$archive" ]]; then
echo "Dump archive not found: $archive" >&2
exit 2
fi
if [[ "$archive" != *.zip && "$archive" != *.sql ]]; then
echo "The dump must be a .zip archive or .sql file." >&2
exit 2
fi
fi
compose up -d --wait db
confirm_setup_replacement "$replace" "$yes"
fresh_docker_database
In bin/dev:
> + if [[ -n "$missing_columns" ]]; then
+ echo "The dump has columns that are not present in this source checkout:" >&2
+ echo "$missing_columns" >&2
+ exit 1
+ fi
+ incompatible_columns="$(mysql_root_query "SELECT CONCAT(source_columns.table_name, '.', source_columns.column_name, ' (dump ', source_columns.column_type, ', target ', target_columns.column_type, ')') FROM information_schema.columns AS source_columns JOIN information_schema.columns AS target_columns ON target_columns.table_schema = '${MYSQL_DATABASE}' AND target_columns.table_name = source_columns.table_name AND target_columns.column_name = source_columns.column_name WHERE source_columns.table_schema = 'gcd_dev_import_stage' AND source_columns.table_name REGEXP '^(gcd|stddata)_' AND (source_columns.is_nullable <> target_columns.is_nullable OR (source_columns.column_type <> target_columns.column_type AND NOT (source_columns.column_type = 'datetime' AND target_columns.column_type = 'datetime(6)') AND NOT (source_columns.column_type = 'tinyint(1)' AND target_columns.column_type = 'int'))) ORDER BY source_columns.table_name, source_columns.ordinal_position;")"
+ if [[ -n "$incompatible_columns" ]]; then
+ echo "The dump schema is incompatible with this source checkout:" >&2
+ echo "$incompatible_columns" >&2
+ exit 1
+ fi
+}
+
+copy_dump_catalog() {
+ local catalog_copy_sql auto_increment_sql
+ catalog_copy_sql="$(mysql_root_query "SELECT CONCAT('DELETE FROM ', CHAR(96), '${MYSQL_DATABASE}', CHAR(96), '.', CHAR(96), table_name, CHAR(96), '; INSERT INTO ', CHAR(96), '${MYSQL_DATABASE}', CHAR(96), '.', CHAR(96), table_name, CHAR(96), ' (', GROUP_CONCAT(CONCAT(CHAR(96), column_name, CHAR(96)) ORDER BY ordinal_position SEPARATOR ','), ') SELECT ', GROUP_CONCAT(CONCAT(CHAR(96), column_name, CHAR(96)) ORDER BY ordinal_position SEPARATOR ','), ' FROM ', CHAR(96), 'gcd_dev_import_stage', CHAR(96), '.', CHAR(96), table_name, CHAR(96), ';') FROM information_schema.columns WHERE table_schema = 'gcd_dev_import_stage' AND table_name REGEXP '^(gcd|stddata)_' GROUP BY table_name ORDER BY table_name;")"
MySQL's default group_concat_max_len is 1024 characters. For tables with a large number of columns (such as gcd_issue or gcd_story), the concatenated column list generated by GROUP_CONCAT will be truncated, leading to syntax errors and import failures. Prepend SET SESSION group_concat_max_len = 1000000; to the query to prevent truncation.
catalog_copy_sql="$(mysql_root_query "SET SESSION group_concat_max_len = 1000000; SELECT CONCAT('DELETE FROM ', CHAR(96), '${MYSQL_DATABASE}', CHAR(96), '.', CHAR(96), table_name, CHAR(96), '; INSERT INTO ', CHAR(96), '${MYSQL_DATABASE}', CHAR(96), '.', CHAR(96), table_name, CHAR(96), ' (', GROUP_CONCAT(CONCAT(CHAR(96), column_name, CHAR(96)) ORDER BY ordinal_position SEPARATOR ','), ') SELECT ', GROUP_CONCAT(CONCAT(CHAR(96), column_name, CHAR(96)) ORDER BY ordinal_position SEPARATOR ','), ' FROM ', CHAR(96), 'gcd_dev_import_stage', CHAR(96), '.', CHAR(96), table_name, CHAR(96), ';') FROM information_schema.columns WHERE table_schema = 'gcd_dev_import_stage' AND table_name REGEXP '^(gcd|stddata)_' GROUP BY table_name ORDER BY table_name;")"
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
Adam Hernandez
@DeusExTaco pushed 1 commit.
- ce562b7 Validate development dump setup inputs
—
View it on GitHub or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
JochenGCD
@jochengcd commented on this pull request.
We could also consider to have management calls to create changesets for specific data objects.
Those without dependent objects are simple (see the docker-dev), but these are getting less.
At least for issues there is setup_issue, which is hand-written, but still should work. Since that needs updating once further related objects are added to an issue/sequence, it might be possible to automate something using routines / models from the OI. Reprints are not included, would need to be added. And with that in mind a check if a revision already exists, e.g. reprints are on two sequences and the backfilled revision might exist.
Maybe I should add an issue for this. Not part of this PR.
In apps/gcd/management/commands/seed_development_data.py:
> + if not image.image_file:
+ image.image_file.save(
+ f'gcd-dev-cover-{covered_issue.sort_code}.png',
+ ContentFile(SAMPLE_PNG),
+ save=True,
+ )
+
+ series.set_first_last_issues()
+ series.issue_count = series.active_issues().count()
+ series.save(update_fields=['issue_count'])
+ publisher.issue_count = Issue.objects.filter(series__publisher=publisher,
+ deleted=False).count()
+ publisher.series_count = publisher.active_series().count()
+ publisher.save(update_fields=['issue_count', 'series_count'])
+
+ def _seed_change_history(self):
This is an empty changeset. While fine for some aspects of test and development, it would pose problems for other aspects.
We could use setup_object from the gcd-django-docker-setup or generate changesets for (some of) the generated dev-data objects, or just the created series if one is for these purposes enough ?
https://github.com/GrandComicsDatabase/gcd-django-docker/blob/main/setup_initial_changesets.py
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.