Skip to content

AP-687 Only download files if local file is older#13

Open
jason-raitz wants to merge 1 commit into
mainfrom
AP-687_only-redownload-changed-images
Open

AP-687 Only download files if local file is older#13
jason-raitz wants to merge 1 commit into
mainfrom
AP-687_only-redownload-changed-images

Conversation

@jason-raitz
Copy link
Copy Markdown
Contributor

  • given a UTC modified datetime string for the requested file, we now check if the file exists locally and has a newer mtime

 - given a UTC modified datetime string for the requested file,
   we now check if the file exists locally and has a newer mtime
@jason-raitz jason-raitz self-assigned this May 22, 2026
@jason-raitz jason-raitz marked this pull request as ready for review May 22, 2026 15:38
Copy link
Copy Markdown
Member

@awilfox awilfox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

r+wc; minor documentation fixes.

Comment thread CHANGELOG.md
Comment on lines +11 to +12
- Optional parameter to `fetch_file()` with a modified time of the remote file pulled from the TIND API
- `fetch_file()` uses this to avoid unnecessary downloads if a file already exists at the target
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't this be a single entry in the changelog?

Comment thread tind_client/client.py
Comment on lines 9 to +13
from io import StringIO
from pathlib import Path
from typing import Any, Iterator
import xml.etree.ElementTree as E
from datetime import datetime, timezone
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not only for alphabetisation, but because etree is sometimes considered external.

Suggested change
from io import StringIO
from pathlib import Path
from typing import Any, Iterator
import xml.etree.ElementTree as E
from datetime import datetime, timezone
from datetime import datetime, timezone
from io import StringIO
from pathlib import Path
from typing import Any, Iterator
import xml.etree.ElementTree as E

Comment thread tind_client/client.py
Comment on lines +76 to +78
"""Download a file from TIND and save it locally. If the file already exists in the output
directory and has a local modified timestamp that is newer than supplied ``modified``
timestamp, the file will not be re-downloaded.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"""Download a file from TIND and save it locally. If the file already exists in the output
directory and has a local modified timestamp that is newer than supplied ``modified``
timestamp, the file will not be re-downloaded.
"""Download a file from TIND and save it locally.
If the file already exists in the output directory and was modified at or after a supplied ``modified``
timestamp, the file will not be re-downloaded.

Comment thread tind_client/client.py
:param str file_url: The TIND file download URL.
:param str output_dir: Directory in which to save the file.
Falls back to ``default_storage_dir`` when empty.
:param str modified: Optional modified timestamp from the file metadata returned by TIND
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
:param str modified: Optional modified timestamp from the file metadata returned by TIND
:param str modified: Optional modified timestamp from the file metadata returned by TIND.
If not specified, the file will always be downloaded.

@awilfox
Copy link
Copy Markdown
Member

awilfox commented May 22, 2026

It would be nice to have a test. We should be able to mock stat to return specific dates instead of relying on fixture dates / filesystem stuff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants