[proxy] web.archive.org← back | site home | direct (HTTPS) ↗ | proxy home | ◑ dark◐ light
/ cpython Public
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-33695 shutil.copytree() + os.scandir() cache #7874

Merged
merged 19 commits into from Nov 12, 2018
Merged

bpo-33695 shutil.copytree() + os.scandir() cache #7874

merged 19 commits into from Nov 12, 2018

Conversation

Copy link
Contributor

giampaolo commented Jun 23, 2018

Copy link
Contributor Author

giampaolo commented Jun 23, 2018

Note: I explicitly avoided using a context manager around os.scandir() for now so that patch it's easier to review (will add it before pushing).

giampaolo changed the title bpo-33695 shutil os.scandir() cache bpo-33695 shutil.copytree() + os.scandir() cache Jun 23, 2018
giampaolo requested a review from vstinner Jun 26, 2018
Lib/shutil.py Outdated
@@ -53,6 +53,10 @@
COPY_BUFSIZE = 1024 * 1024 if _WINDOWS else 16 * 1024
_HAS_SENDFILE = posix and hasattr(os, "sendfile")
_HAS_FCOPYFILE = posix and hasattr(posix, "_fcopyfile") # macOS
_HAS_SAMESTAT = hasattr(os.path, 'samestat')
_HAS_SAMEFILE = hasattr(os.path, 'samefile')
Copy link
Member

serhiy-storchaka Jul 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is not it always true?

Copy link
Contributor Author

giampaolo Jul 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure. This was inherited by existing code. Doc says Availability: Unix, Windows. which suggests there may be cases where it's not available?

Lib/shutil.py Outdated Show resolved Hide resolved
Lib/shutil.py Outdated Show resolved Hide resolved
Lib/shutil.py Outdated Show resolved Hide resolved
Lib/shutil.py Outdated
if hasattr(os.path, 'samefile'):
if isinstance(src, os.DirEntry) and _HAS_SAMESTAT:
try:
return os.path.samestat(src.stat(), os.stat(dst))
Copy link
Member

serhiy-storchaka Jul 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See https://bugs.python.org/issue30480 .

Not always samefile() can be replaced by samestat() on Windows.

Copy link
Contributor Author

giampaolo Jul 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I wasn't aware of that. I'm confused though as it appears the problem should be fixed in samestat() (assuming it's possible), not in here. Also, as per https://bugs.python.org/issue30480 the problem already affects shutil regardless of this PR.

Copy link
Contributor Author

giampaolo Jul 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code updated. I think this is better left as-is as it just reflects the previous problematic behavior which IMO should be fixed separately.

Lib/shutil.py Show resolved Hide resolved
Lib/shutil.py Outdated Show resolved Hide resolved
Copy link
Contributor Author

giampaolo left a comment

@serhiy-storchaka , @vstinner,
thanks for joining the code review; replies to your comments are inline.

Lib/shutil.py Outdated
@@ -53,6 +53,10 @@
COPY_BUFSIZE = 1024 * 1024 if _WINDOWS else 16 * 1024
_HAS_SENDFILE = posix and hasattr(os, "sendfile")
_HAS_FCOPYFILE = posix and hasattr(posix, "_fcopyfile") # macOS
_HAS_SAMESTAT = hasattr(os.path, 'samestat')
_HAS_SAMEFILE = hasattr(os.path, 'samefile')
Copy link
Contributor Author

giampaolo Jul 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure. This was inherited by existing code. Doc says Availability: Unix, Windows. which suggests there may be cases where it's not available?

Lib/shutil.py Outdated Show resolved Hide resolved
Lib/shutil.py Outdated Show resolved Hide resolved
Lib/shutil.py Outdated
if hasattr(os.path, 'samefile'):
if isinstance(src, os.DirEntry) and _HAS_SAMESTAT:
try:
return os.path.samestat(src.stat(), os.stat(dst))
Copy link
Contributor Author

giampaolo Jul 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I wasn't aware of that. I'm confused though as it appears the problem should be fixed in samestat() (assuming it's possible), not in here. Also, as per https://bugs.python.org/issue30480 the problem already affects shutil regardless of this PR.

Lib/shutil.py Show resolved Hide resolved
Lib/shutil.py Outdated Show resolved Hide resolved
Lib/shutil.py Outdated Show resolved Hide resolved
Copy link
Contributor Author

giampaolo commented Aug 1, 2018

Is there anything else I can do regarding this PR? Are we happy?

giampaolo merged commit 19c46a4 into python:master Nov 12, 2018
Copy link

bedevere-bot commented Nov 12, 2018

@giampaolo: Please replace # with GH- in the commit message next time. Thanks!

giampaolo deleted the 33695-shutil-scandir-cache branch Nov 12, 2018
Copy link

ghost commented Feb 23, 2019

@giampaolo Maybe you can also look after my PR #11425 (bpo-35652) because it fixes the behaviour of this PR that the use_srcentry is only set if the original copy function is used. The problem is that a custom edited copy function can't use the srcentry directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants