I have been working on exactly this problem in recent days (pyramid and alembic in docker). The main issue is that the PostgreSQL connection information needs to be taken from the environment, not an INI file.
I will be posting my work in a Github repo soonish (in a week or two), but until then, here is my suggestion:
run_migrations_offline pulls the url out of the INI directly; you can instead construct it from the information in the environment (or skip using run_migrations_offline entirely)
run_migrations_online uses engine_from_config with the INI config. You could use the config.set_main_option method to set the url into the config (after constructing it from the environment variables), or use a different way of getting a sqlalchemy engine.