Skip to content
New issue

Have a question about this project?Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to ourterms of serviceand privacy statement.We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix DAT models download #16302

Open
wants to merge 5 commits into
base: dev
Choose a base branch
from
Open

Fix DAT models download #16302

wants to merge 5 commits into from
+97 −28

Conversation

w-e-w
Copy link
Collaborator

@w-e-w w-e-w commented Jul 31, 2024 •
edited
Loading

Description

Fix DAT models download

the thedownload linkswe used was uploaded by n0kovo because theorignaldue to the orignal was hosted on Google Drive / Baidu Disk, the issue is n0kovo used LFS,
LFS on GitHub quota is exceededfor obvious reasonsand so the Auto download doesn't work

Worse it currently downloads the LFS pointer file so webui thinks that thinks is downloaded and errors

  • The fix I uploaded the the models tohttps://huggingface.co/w-e-w/DAT/tree/main/experiments/pretrained_models
  • in addition I also added hash check when downloading DAT models
  • Re-download ifmodel filesize < 200 bytes(likely an LFS pointer), (the three expected DAT models is almost 150MB)
  • add new keyword only argre_downloadtoload_file_from_url()default false, when set toTrue,re-download and override existing file

TODO

not sure if it will be in this PR or a separate one

  • loading local upscaler files seems to be a bit disorganized
    it seems like every architecture just seem to implement its own file loader (not model loader file loader)
    this seems to result in various bugs
    for DAT you will see duplicate entries if local files exist
    some other upscallers don't seem to list custom user models

Checklist:

@w-e-w w-e-w marked this pull request as ready for review July 31, 2024 10:11
@w-e-w w-e-w changed the title Fix dat models download Fix DAT models download Jul 31, 2024
@n0kovo
Copy link
Contributor

n0kovo commented Aug 10, 2024 •
edited
Loading

Well that explains a lot. Didn't realize there was a quota, but of course there is.

@w-e-w
Copy link
Collaborator Author

w-e-w commented Aug 10, 2024 •
edited
Loading

I guess that your Git LFS Data on GitHub Billing page has exploded

what sort of building plan are you on, did you get charged for the LFS data?

@w-e-w
Copy link
Collaborator Author

w-e-w commented Oct 17, 2024 •
edited
Loading

new commitrewrite load_file_from_url with requests, move to util

been getting this error when usingtorch.hub import download_url_to_file.download_url_to_fileto download files from hugging face

urllib.error.URLError:<urlopenerror[SSL:CERTIFICATE_VERIFY_FAILED]certificateverifyfailed:unabletogetlocalissuercertificate(_ssl.c:997)>

but request seems to work
so I rewritten load_file_from_url to use request

I also move itmodules.modelloader.load_file_from_urltomodules.utils.load_file_from_url
the reason is that it's a general useful function that someone may wish to use elsewhere

@w-e-w w-e-w mentioned this pull request Oct 17, 2024
Open
4 tasks
@w-e-w w-e-w changed the title Fix DAT models download [Fix]: DAT models download Oct 19, 2024
@w-e-w w-e-w changed the title [Fix]: DAT models download Fix DAT models download Oct 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AUTOMATIC1111 AUTOMATIC1111 Awaiting requested review from AUTOMATIC1111 AUTOMATIC1111 is a code owner

At least 0 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@w-e-w @n0kovo