[dbashford-NOAA]

Resolves #389

Added a function to load bio data from a postgres db as an alternative to loading from a CSV

Cleaned version of PR #415

[dbashford-NOAA]

@leewujung
I opened this new PR for this work because it was easier to push a new branch instead of trying to fix all the upstream conflicts in the other one (#415) which can probably just be closed now.

Let me know if there is anything here I should change.

[leewujung]

@dbashford-NOAA : Agreed! I just closed #415.

[leewujung]

Thanks @dbashford-NOAA ! I think the code/tests look good on my side. Just requested @aliciabillings-noaa to review. :)

[brandynlucca]

@dbashford-NOAA I suggested adding requires-python/python-version fields in pr.yaml and project.toml. That should allow the tests to pass for 3.14; effecting those changes allowed my local tox tests to pass.

There was one exception: I was not able to run the acutal PostgreSQL database tests since I can't run Docker locally. Could we make the database tests runnable locally without Docker? Currently, the fixture skips all three when Docker or testcontainers is unavailable, so a passing local run can leave database ingestion untested. That makes it harder for contributors to catch regressions before submitting changes.

I suggest using an automatically managed local PostgreSQL instance while retaining Docker coverage in CI. Both paths could reuse the existing tests and seed data, allowing contributors to verify the same behavior without manually configuring a database. Setup failures should also produce errors rather than skips, so broken infrastructure cannot silently reduce coverage.

I tested this approach locally and the three previously skipped database tests passed.

[brandynlucca]

Suggested change

	python-version: ["3.12", "3.13"]
	python-version: ["3.12", "3.13", "3.14"]

[brandynlucca]

Suggested change

	requires-python = ">=3.12, <3.15"

[brandynlucca]

Suggested change

	run: |
	# Fixtures also exercise Docker automatically on Ubuntu runners.
	run: |

[brandynlucca]

Suggested change

	- name: Setup PostgreSQL Binaries
	# GitHub action to set up postgreSQL for all 3 platforms
	uses: ikalnytskyi/action-setup-postgres@v8
	with:
	username: test_user
	password: postgres
	database: test
	port: 5432
	postgres-version: '14'
	id: postgres

See my overall comment concerning this.

[brandynlucca]

Suggested change

	"isort",
	"isort",
	"pgembed==0.2.0",

[brandynlucca]

Suggested change

	import os
	import os
	import sys
	from importlib.resources import files

[brandynlucca]

Suggested change

	import pytest
	import pytest
	from pytest_postgresql import factories

[brandynlucca]

Suggested change

	# Bundled binaries let plain pytest run locally without Docker or server setup.
	native_postgres_process = factories.postgresql_proc(
	executable=str(
	files("pgembed") / "pginstall" / "bin" / ("pg_ctl.exe" if os.name == "nt" else "pg_ctl")
	),
	host="127.0.0.1",
	port=None,
	user="test_user",
	password="postgres",
	dbname="test",
	)
	native_postgres = factories.postgresql("native_postgres_process")

[brandynlucca]

Suggested change

	"""
	Session-scoped fixture to get database connection.
	- In GitHub Actions: Uses the postgres service from the workflow
	- Locally: Uses Testcontainers if Docker is available, skips if not
	"""
	is_github_action = os.environ.get("GITHUB_ACTIONS")
	if is_github_action:
	# In GitHub Actions use the postgres service from workflow
	yield type(
	"obj",
	(object,),
	{
	"get_connection_url": lambda self: "postgresql+psycopg://test_user:postgres@localhost:5432/test",
	"get_container_host_ip": lambda self: "localhost",
	"get_exposed_port": lambda self, port: 5432,
	},
	)()
	else:
	# Local development
	try:
	from testcontainers.postgres import PostgresContainer
	container = PostgresContainer(
	image="postgres:16", username="test_user", password="postgres", dbname="test"
	)
	container.start()
	yield container
	container.stop()
	except Exception as e:
	# Docker not available - skip integration tests
	pytest.skip(f"Docker must be running for Testcontainers: {e}")
	"""Exercise the existing PostgreSQL container on Linux CI."""
	from testcontainers.community.postgres import PostgresContainer
	with PostgresContainer(
	image="postgres:16",
	username="test_user",
	password="postgres",
	dbname="test",
	driver="psycopg",
	) as container:
	yield container

[brandynlucca]

Suggested change

	@pytest.fixture(scope="session")
	def database_credentials(postgres_container):
	"""
	Session-scoped fixture to:
	1. Connect to the PostgreSQL database (CI or local).
	2. Load 'test_bio_data.sql' into it.
	3. Yield the credentials dictionary in the format expected by load_biodata_db_views.
	Returns dict with keys: host, port, dbname, user, password, schema
	"""
	host = postgres_container.get_container_host_ip()
	port = postgres_container.get_exposed_port(5432)
	creds = {
	"host": host,
	"port": port,
	"dbname": "test",
	"user": "test_user",
	"password": "postgres",
	"schema": "public",
	}
	db_url = (
	f"postgresql+psycopg://"
	f"{creds['user']}:{creds['password']}@"
	f"{creds['host']}:{creds['port']}/"
	f"{creds['dbname']}"
	)
	try:
	engine = create_engine(db_url)
	with engine.begin() as connection:
	with open(TEST_SQL_FILE) as f:
	sql_script = f.read()
	connection.execute(text(sql_script))
	except Exception as e:
	pytest.fail(f"Failed to load {TEST_SQL_FILE}: {e}")
	yield creds
	@pytest.fixture(
	params=(
	["native", "docker"]
	if os.environ.get("GITHUB_ACTIONS") == "true" and sys.platform == "linux"
	else ["native"]
	)
	)
	def database_credentials(request):
	"""
	Fixture to:
	1. Start native PostgreSQL locally and on CI; also test Docker on Linux CI.
	2. Load 'test_bio_data.sql' into it.
	3. Return the credentials dictionary expected by load_biodata_db_views.
	Returns dict with keys: host, port, dbname, user, password, schema
	"""
	backend = request.param
	if backend == "native":
	connection = request.getfixturevalue("native_postgres")
	host, port = connection.info.host, connection.info.port
	dbname = connection.info.dbname
	else:
	container = request.getfixturevalue("postgres_container")
	host, port = container.get_container_host_ip(), container.get_exposed_port(5432)
	dbname = "test"
	creds = {
	"host": host,
	"port": port,
	"dbname": dbname,
	"user": "test_user",
	"password": "postgres",
	"schema": "public",
	}
	db_url = (
	f"postgresql+psycopg://"
	f"{creds['user']}:{creds['password']}@"
	f"{creds['host']}:{creds['port']}/"
	f"{creds['dbname']}"
	)
	engine = create_engine(db_url)
	try:
	with engine.begin() as connection:
	with open(TEST_SQL_FILE, encoding="utf-8") as f:
	sql_script = f.read()
	connection.execute(text(sql_script))
	except Exception as e:
	pytest.fail(f"Failed to load {TEST_SQL_FILE}: {e}")
	finally:
	engine.dispose()
	return creds