Hello,
In my company we are building an internal platform that runs many ETL processes over Spark 4.1 with PySpark.
We are using an AWS RDS Postgres DB as well as custom Postgres deployments for usage with AGE etc., all read/write via Spark.
We have an issue with connection retries during RDS rollovers or other intermittent issues.
For writers this is not a problem as we can add our own delay with backoff and then retry the writer. However, For readers, this is a problem because at the point of materialization the reader is already baked into the pipeline and it does not have and delays or backoffs during retries and we cannot replace or recreate the reader once the the full ETL plan is already built by Spark.
I'm wondering if it would be possible to add configurations for delays and backoffs?
We need to be able to handle connection refusals of up to 20 seconds at a time (once every few days).
Thanks!
Hello,
In my company we are building an internal platform that runs many ETL processes over Spark 4.1 with PySpark.
We are using an AWS RDS Postgres DB as well as custom Postgres deployments for usage with AGE etc., all read/write via Spark.
We have an issue with connection retries during RDS rollovers or other intermittent issues.
For writers this is not a problem as we can add our own delay with backoff and then retry the writer. However, For readers, this is a problem because at the point of materialization the reader is already baked into the pipeline and it does not have and delays or backoffs during retries and we cannot replace or recreate the reader once the the full ETL plan is already built by Spark.
I'm wondering if it would be possible to add configurations for delays and backoffs?
We need to be able to handle connection refusals of up to 20 seconds at a time (once every few days).
Thanks!