Automatic yt-dlp updates

[Amdrel]

Description

This patch adds shell scripts that update yt-dlp and its optional dependencies everyday. The Toolforge yt-dlp is updated at 7:33 UTC (see jobs.yaml) and the encoders do it at 6:32 UTC. I staggered them to run after other jobs should have completed just to be safe.

The scripts restart the web service and celery processes respectively if the version changed so that the changes actually take effect. However, the Toolforge update script has to get a bit clever as it can't directly utilize the toolforge CLI.

In Toolforge I set it up to restart uwsgi by using the touch-reload option, which allows the job to touch a special file at www/python/reload and eventually uwsgi will detect that and restart the service. I tested this on video2commons-test already and it notices after a couple of minutes or so.

I also tested on encoder01 by manually copying the script over and running it.

Changes

• Add jobs.yaml file with the new update-yt-dlp job, along with already existing jobs.
  • Note: I backed up existing job definitions and didn't include the worker1 job on purpose as that job never worked and attempts to setup a Celery worker. I have the old definitions backed up though just in case.
• Add encoder update job to the puppet configuration.
• Add utils/update-yt-dlp-encoder.sh and utils/update-yt-dlp-toolforge.sh shell scripts.
• Add special www/python/reload file that uwsgi will check to perform restarts.
• Add uwsgi config at www/python/uwsgi.ini.
  • One already existed but wasn't committed to the repository. I carried over the configuration changes it made (log-maxsize = 10485760 specifically) and added the touch-reload key.

Deployment

Encoders

These can be deployed like usual with the GitHub Action. There is nothing special to do here.

Toolforge

We shouldn't use the action to deploy changes to Toolforge just this one time. The reason for this is I need to manually shuffle some untracked files around as the git pull that gets executed by the GitHub Action will fail as new files added in this commit exist in an untracked state in Toolforge, and git won't clobber them.

I also need to manually load the new jobs.yaml definition, which the GitHub Action doesn't do.

[Amdrel]

Here is context on why I made this patch: https://commons.wikimedia.org/wiki/Commons_talk:Video2commons#c-Amdrel-20260804154600-Jeff_G.-20260804145400

Ignore my blurb about the encoders not being affected. They only update if we push new code, which is not ideal.

[don-vip]

Wow it's really nice, thank you!

We will need to update those python 3.9 / 3.11 jobs to a supported version though.

[Amdrel]

I agree, and I do want to do that soon. Just didn't want to lump too many changes together all at once to minimize the chances of something going wrong.