Issue24538
Created on 2015-06-30 12:07 by Gerrit.Holl, last changed 2022-04-11 14:58 by admin. This issue is now closed.
| Pull Requests | |||
|---|---|---|---|
| URL | Status | Linked | Edit |
| PR 13212 | merged | python-dev, 2019-05-09 00:18 | |
| PR 13234 | merged | miss-islington, 2019-05-10 03:22 | |
| Messages (10) | |||
|---|---|---|---|
| msg245985 - (view) | Author: Gerrit Holl (Gerrit.Holl) * | Date: 2015-06-30 12:07 | |
`shutil.copystat` fails on [panfs](https://en.wikipedia.org/wiki/Panasas#PanFS) if the source file lacks u+w, because setting extended attributes results in a `PermissionError`. This leads to higher end functions such as `shutil.copytree` to fail. More seriously, it leads to `pip install .` to fail, as I [reported here](https://github.com/pypa/pip/issues/2941): In [55]: shutil.copystat("/src/on/panfs", "/tmp/fubar") --------------------------------------------------------------------------- PermissionError Traceback (most recent call last) <ipython-input-55-139c9fc77184> in <module>() ----> 1 shutil.copystat("/home/users/gholl/checkouts/pyatmlab/.git/objects/pack/pack-c1449559ec4287b3830efe2913608dddf2f21391.pack", "/tmp/fubar") /home/users/gholl/lib/python3.4/shutil.py in copystat(src, dst, follow_symlinks) 211 else: 212 raise --> 213 _copyxattr(src, dst, follow_symlinks=follow) 214 215 def copy(src, dst, *, follow_symlinks=True): /home/users/gholl/lib/python3.4/shutil.py in _copyxattr(src, dst, follow_symlinks) 151 try: 152 value = os.getxattr(src, name, follow_symlinks=follow_symlinks) --> 153 os.setxattr(dst, name, value, follow_symlinks=follow_symlinks) 154 except OSError as e: 155 if e.errno not in (errno.EPERM, errno.ENOTSUP, errno.ENODATA): PermissionError: [Errno 13] Permission denied: '/tmp/fubar' This occurs for any source file where the user write bit is not set. Now, I'm not sure if this should be addressed in `pip`, `shutil.copytree`, `shutil.copystat`, `setxattr`, `panfs`, or in none of the above. I'm reporting it in different places; please close this issue if this is the wrong place. |
|||
| msg245987 - (view) | Author: R. David Murray (r.david.murray) * | Date: 2015-06-30 13:08 | |
There are a couple of related open issues. I think there is an stdlib problem here, but I"m not sure what the solution is. |
|||
| msg245989 - (view) | Author: Gerrit Holl (Gerrit.Holl) * | Date: 2015-06-30 13:26 | |
Perhaps the solution would be some kind of flag, at least for copytree and possibly others, on what to do when attributes cannot be completely copied — a bit like numpys options to raise, warn, or ignore? |
|||
| msg301493 - (view) | Author: Bart Oldeman (Bart Oldeman) | Date: 2017-09-06 17:50 | |
The issue can be avoided by calling _copyxattr *before* instead of after os.chmod in shutil.copystat. That way the file is still writable. The same issue happens on the Lustre parallel file system, with the lustre.lov extended attribute. |
|||
| msg342022 - (view) | Author: Olexa Bilaniuk (obilaniu) * | Date: 2019-05-10 00:49 | |
This old bug now has a PR on Github authored by myself that may be reviewed. |
|||
| msg342051 - (view) | Author: Giampaolo Rodola' (giampaolo.rodola) * | Date: 2019-05-10 03:20 | |
Patch LGTM. I'd like to point out one thing for posterity. Current shutil code catches EPERM but not EACCES. While reviewing the patch I wondered whether simply catching (and ignoring) both error codes instead, but it turns out they are supposed to have 2 different meanings: https://stackoverflow.com/a/35879961/376587 Especially after _copyxattr is moved before chmod() EACCES (permission denied) should propagate instead of being silenced, and EPERM should be left in place because in this case EPERM acts more like an ENOTSUP (operation not supported) than EACCES (permission denied) and as such should rightly be ignored. |
|||
| msg342052 - (view) | Author: Giampaolo Rodola' (giampaolo.rodola) * | Date: 2019-05-10 03:22 | |
New changeset 79efbb719383386051c72f2ee932eeca8e033e6b by Giampaolo Rodola (Olexa Bilaniuk) in branch 'master': bpo-24538: Fix bug in shutil involving the copying of xattrs to read-only files. (PR-13212) https://github.com/python/cpython/commit/79efbb719383386051c72f2ee932eeca8e033e6b |
|||
| msg342433 - (view) | Author: Giampaolo Rodola' (giampaolo.rodola) * | Date: 2019-05-14 05:30 | |
New changeset 0a5b88e7f23b671d63896619b13148b0e4e2b5dd by Giampaolo Rodola (Miss Islington (bot)) in branch '3.7': bpo-24538: Fix bug in shutil involving the copying of xattrs to read-only files. (PR-13212) (#13234) https://github.com/python/cpython/commit/0a5b88e7f23b671d63896619b13148b0e4e2b5dd |
|||
| msg378292 - (view) | Author: Irit Katriel (iritkatriel) * | Date: 2020-10-08 23:14 | |
This seems complete, can it be closed? |
|||
| msg378300 - (view) | Author: Olexa Bilaniuk (obilaniu) * | Date: 2020-10-09 03:51 | |
Yes, it may be closed. The fix was merged in time to make it into Python 3.7.4 and 3.8.0, and solved the problems we were facing with fancy filesystem xattrs on read-only files. |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:58:18 | admin | set | github: 68726 |
| 2020-10-20 10:57:24 | iritkatriel | set | status: open -> closed resolution: fixed stage: patch review -> resolved |
| 2020-10-09 03:51:28 | obilaniu | set | messages: + msg378300 |
| 2020-10-08 23:14:59 | iritkatriel | set | nosy:
+ iritkatriel messages: + msg378292 |
| 2019-05-14 05:30:28 | giampaolo.rodola | set | messages: + msg342433 |
| 2019-05-10 03:26:24 | giampaolo.rodola | set | versions: + Python 3.7, Python 3.8, - Python 3.2, Python 3.3, Python 3.4 |
| 2019-05-10 03:22:47 | miss-islington | set | pull_requests: + pull_request13144 |
| 2019-05-10 03:22:24 | giampaolo.rodola | set | messages: + msg342052 |
| 2019-05-10 03:20:02 | giampaolo.rodola | set | messages: + msg342051 |
| 2019-05-10 00:49:13 | obilaniu | set | nosy:
+ giampaolo.rodola, tarek, obilaniu messages: + msg342022 |
| 2019-05-09 00:18:50 | python-dev | set | keywords:
+ patch stage: patch review pull_requests: + pull_request13123 |
| 2017-09-06 17:54:38 | Maxime Boissonneault | set | nosy:
+ Maxime Boissonneault |
| 2017-09-06 17:50:12 | Bart Oldeman | set | nosy:
+ Bart Oldeman messages: + msg301493 |
| 2015-06-30 13:26:51 | Gerrit.Holl | set | messages: + msg245989 |
| 2015-06-30 13:08:49 | r.david.murray | set | nosy:
+ r.david.murray messages: + msg245987 |
| 2015-06-30 12:07:29 | Gerrit.Holl | set | type: crash |
| 2015-06-30 12:07:21 | Gerrit.Holl | create | |