Gracefully handle missing availability service by lkeyson-earthscope · Pull Request #13 · EarthScope/piqqa

lkeyson-earthscope · 2026-07-27T22:56:35Z

The FDSN fdsnws-availability service and MUSTANG's ts_* time-series metrics (e.g. ts_percent_availability_total, ts_num_gaps) are currently down for an undetermined period. Though their return will likely occur at the same time, these two services could go down and come back independently of each other. This PR lets PIQQA reports keep generating while either or both are unavailable, and switch back onto them automatically (no manual flag/config change) once they recover.

What changed
Independent health checks, one per service, run once per report:

doAvailability() probes fdsnws-availability directly (PIQQA.py) and only treats it as down on an explicit HTTP 410 (or another HTTP error / connection failure) - no longer relies on a getcode() == 410 check that could never actually be reached (urlopen raises HTTPError for 4xx/5xx rather than returning a response object).
reportUtils.checkTsMetricsExist() (new) independently probes MUSTANG for ts_percent_availability_total against a stable reference station (IU.ANMO), so we don't conflate "availability service is down" with "ts_ metrics are down" - they're detected and handled separately.

Fallback paths when a service is down:

reportUtils.get_extents_from_mustang() (new) computes data start/end extents from MUSTANG's percent_availability measurements when fdsnws-availability is unreachable, replacing the availability-service lookup used to determine data start/end for PDF and spectrogram plots. If it cannot retrieve percent_availability metrics, it will use the report dates.
reportUtils.getDataStartEnd() (new) wraps the "get start/end for a target" logic (previously duplicated 4x across doPDFs/doSpectrograms) and picks between getAvailability() and get_extents_from_mustang() based on the availability-service flag.
doAvailability() skips straight to the percent_availability fallback (instead of attempting ts_percent_availability_total and only falling back after a failed retry-with-sleep) when checkTsMetricsExist() has already determined ts_ metrics are down — avoids ~5-10s of wasted retries per channel group.
doBoxPlots() drops ts_num_gaps_total and substitutes num_gaps for ts_num_gaps in the requested metric list when ts_ metrics are down.

Updated urls:

service.iris.edu --> service.earthscope.org

This MR also bumps python to 3.12, off of end-of-life 3.9.

Example report that was run on the latest code:
UW_2026.zip

…se merging for gaps

gillian-earthscope

Mostly just comments about updating IRIS references to EarthScope

gillian-earthscope · 2026-07-29T21:28:41Z

+    testURL = (
+        f"https://service.earthscope.org/mustang/measurements/1/query?"
+        f"metric=ts_percent_availability_total&net=IU&sta=ANMO&loc=00&chan=BHZ&"
+        f"format=text&timewindow={startDate},{endDate}&nodata=404"


Suggest using the metrics web service for a query that is not dependent on a network or time window:
https://service.earthscope.org/mustang/metrics/1/query?metric=ts_percent_availability_total&output=html&nodata=404

gillian-earthscope · 2026-07-29T21:32:13Z

+        except urllib.error.HTTPError as e:
+            if e.code == 410:
+                print(
+                    f"WARNING: {serviceKey}-availability service is down, falling back to MUSTANG percent_availability for data extents"


HTTP code 410 is specific for the reason "Gone". Suggest this warning instead of "down":

"WARNING: EarthScope {serviceKey}-availability service is gone, falling back to MUSTANG percent_availability for data extents"

gillian-earthscope · 2026-07-29T21:34:39Z

@@ -1728,8 +1792,8 @@
        f"This report is intended as a quick, broad overview of the quality of the data archived for the network "
        f"and time period specified above. IRIS' goal in generating these reports is to give PIs for temporary "


EarthScope's goal

gillian-earthscope · 2026-07-29T21:36:34Z

        f"and time period specified above. IRIS' goal in generating these reports is to give PIs for temporary "
        f"experiments better insight into the quality of their data archived at the DMC, as well as to demonstrate "
-        f'the utility provided by the IRIS DMC quality assurance system <a href="http://service.iris.edu/mustang/?" target="_blank">MUSTANG</a> '
+        f'the utility provided by the IRIS DMC quality assurance system <a href="http://service.earthscope.org/mustang/?" target="_blank">MUSTANG</a> '


EarthScope quality assurance system

gillian-earthscope · 2026-07-29T21:37:05Z

@@ -1768,7 +1839,7 @@
        f"are from the IRIS DMC  {availabilityServiceLink}. "


IRIS DMC -> EarthScope

gillian-earthscope · 2026-07-29T21:39:58Z

        f"<br/><br/>"
        f"The report will include selected channels if specified, or all available channels if not. To see the complete metadata holdings for this experiment, visit this link:"
        f'<p><a href="http://ds.iris.edu/mda/{network}/?starttime={startDate}T00:00:00&endtime={endDate}T23:59:59" target="_blank"> '
        f"IRIS Metadata Aggregator (MDA) for {network}: {startDate} to {endDate}</a></p>\n"


EarthScope Metadata Aggregator

gillian-earthscope · 2026-07-29T21:44:58Z

        f"and the many metrics and products it generates. "
        f"For PIs, we hope that these reports will "
        f"inform the range of data availability, continuity, and noise levels existing across the network and potentially "
        f"which stations are problematic and might be avoided for ongoing data analysis. For PASSCAL, these reports may provide lessons "


PASSCAL -> the EarthScope Primary Instrument Center (EPIC)

gillian-earthscope · 2026-07-29T21:45:54Z

@@ -1728,8 +1792,8 @@
        f"This report is intended as a quick, broad overview of the quality of the data archived for the network "
        f"and time period specified above. IRIS' goal in generating these reports is to give PIs for temporary "
        f"experiments better insight into the quality of their data archived at the DMC, as well as to demonstrate "


archived at EarthScope

gillian-earthscope · 2026-07-29T21:48:23Z

        f"<li> Available data extents in blue </li>  "
        f"<li> Gaps in black  </li>"
        f"</ol>"
        f"The metadata extents are retrieved from the IRIS DMC {stationServiceLink}, "


IRIS DMC -> EarthScope

gillian-earthscope · 2026-07-29T21:51:00Z

+    if not services:
+        availabilityIntroText = (
+            f"{availabilityIntroText} <br/><br/>"
+            f"<b>Note:</b> the availability service was unavailable when this report was generated, "


The EarthScope availability service ...

lkeyson-earthscope added 5 commits July 27, 2026 15:30

gracefully handle missing availability service

4323a90

if availability and percent_avail are missing, use report dates

62cf4d9

fix missing spectrograms

009e97e

add user agent to service calls

70a0aba

remove extra print

f53aa71

lkeyson-earthscope marked this pull request as draft July 28, 2026 14:21

lkeyson-earthscope added 6 commits July 28, 2026 07:46

update service url --> earthscope.org

10adfa0

add / to availability url

d2fa622

update url in readme

d68d0a9

fix warning message

bc3b998

fix links in report

96d8eda

modernize append->concat for dfs

9463207

lkeyson-earthscope marked this pull request as ready for review July 28, 2026 16:37

lkeyson-earthscope requested a review from gillian-earthscope July 28, 2026 16:37

lkeyson-earthscope self-assigned this Jul 28, 2026

lkeyson-earthscope added 10 commits July 28, 2026 09:38

bump version

46551cf

add handling for when station service fails

4367cef

move onto python 3.12

a9b7aac

update IRIS->EarthScope

31c60c1

add check for earthscope OR iris in fedcat return

9821ad6

remove temporary logging

6682b26

check availability service for both ph5 and fdsn services

b1f3b9a

fix typo lowest->highest

1e9f59a

remove future-deprecated params from availability call, create in-hou…

dd8caf7

…se merging for gaps

bypass ts_percent_availability_total if tsmetrics are down

fed2d93

gillian-earthscope reviewed Jul 29, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Gracefully handle missing availability service - #13

Gracefully handle missing availability service#13
lkeyson-earthscope wants to merge 21 commits into
mainfrom
handle-availability

lkeyson-earthscope commented Jul 27, 2026 •

edited

Loading

Uh oh!

gillian-earthscope left a comment

Uh oh!

gillian-earthscope Jul 29, 2026

Uh oh!

gillian-earthscope Jul 29, 2026

Uh oh!

gillian-earthscope Jul 29, 2026

Uh oh!

gillian-earthscope Jul 29, 2026

Uh oh!

gillian-earthscope Jul 29, 2026

Uh oh!

gillian-earthscope Jul 29, 2026

Uh oh!

gillian-earthscope Jul 29, 2026

Uh oh!

gillian-earthscope Jul 29, 2026

Uh oh!

gillian-earthscope Jul 29, 2026

Uh oh!

gillian-earthscope Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -1728,8 +1792,8 @@
		f"This report is intended as a quick, broad overview of the quality of the data archived for the network "
		f"and time period specified above. IRIS' goal in generating these reports is to give PIs for temporary "

		@@ -1768,7 +1839,7 @@
		f"are from the IRIS DMC {availabilityServiceLink}. "

Uh oh!

Conversation

lkeyson-earthscope commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gillian-earthscope left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lkeyson-earthscope commented Jul 27, 2026 •

edited

Loading