Skip to content

[Feature] Add scan_cache_dir function#1348

Merged
Yunnglin merged 5 commits into
masterfrom
add/scan_cache
May 28, 2025
Merged

[Feature] Add scan_cache_dir function#1348
Yunnglin merged 5 commits into
masterfrom
add/scan_cache

Conversation

@Yunnglin

@Yunnglin Yunnglin commented May 26, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Scan snapshot downloaded datasets and models in ModelScope cache dir. #1326

Usage:

>>> from modelscope.hub.cache_manager import scan_cache_dir

>>> ms_cache_info = scan_cache_dir()
ModelScopeCacheInfo(...)

>>> print(ms_cache_info.export_as_table())

REPO ID             REPO TYPE REVISION SIZE ON DISK NB FILES LAST_ACCESSED LAST_MODIFIED LOCAL PATH                                               
------------------- --------- -------- ------------ -------- ------------- ------------- -------------------------------------------------------- 
AI-ModelScope/IQuiz dataset   master        5.23 MB        8 4 hours ago   4 hours ago   /root/.cache/modelscope/hub/datasets/AI-ModelScope/IQuiz 
Qwen/Qwen3-0.6B     model     master        1.41 GB        9 4 hours ago   4 hours ago   /root/.cache/modelscope/hub/models/Qwen/Qwen3-0___6B

CLI:

Scan default cache dir:

>>> modelscope scan-cache

REPO ID             REPO TYPE REVISION SIZE ON DISK NB FILES LAST_ACCESSED LAST_MODIFIED LOCAL PATH                                               
------------------- --------- -------- ------------ -------- ------------- ------------- -------------------------------------------------------- 
AI-ModelScope/IQuiz dataset   master        5.23 MB        8 4 hours ago   4 hours ago   /root/.cache/modelscope/hub/datasets/AI-ModelScope/IQuiz 
Qwen/Qwen3-0.6B     model     master        1.41 GB        9 4 hours ago   4 hours ago   /root/.cache/modelscope/hub/models/Qwen/Qwen3-0___6B     

Done in 0.0s. Scanned 2 repo(s) for a total of 1.42 GB.

or specify --dir:

>>> modelscope scan-cache --dir /mnt/workspace/.cache/modelscope
...
Done in 43.1s. Scanned 349 repo(s) for a total of 4.45 TB.

yingdachen
yingdachen reviewed May 27, 2025
View reviewed changes
Comment thread tests/hub/test_hub_scan_cache.py
"""Set up for tests."""
self.fake_cache_dir = '/fake/cache/path'

def test_scan_default_dir(self):

@yingdachen yingdachen May 27, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should ensure that at least one model exists, by downloading a known model first (it can be one with only readme file) in the test

@Yunnglin Yunnglin May 27, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@Yunnglin Yunnglin merged commit b034267 into master May 28, 2025
3 checks passed
@yingdachen yingdachen mentioned this pull request May 31, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yingdachen yingdachen yingdachen approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Yunnglin @yingdachen