| Messages (21) |
|---|
|
msg361058 - (view) |
Author: EGuesnet (EGuesnet) |
Date: 2020-01-30 13:51 |
Hi,
I have an error during regression tests with Python3.8.1 on AIX 6.1 compiled with GCC 8.3.
It occurs only on 64 bit. Test passes on 32 bit.
```
======================================================================
ERROR: test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/test/test_zipfile.py", line 606, in test_add_file_after_2107
self.assertRaises(struct.error, zipfp.write, TESTFN)
File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/unittest/case.py", line 816, in assertRaises
return context.handle('assertRaises', args, kwargs)
File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/unittest/case.py", line 202, in handle
callable_obj(*args, **kwargs)
File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/zipfile.py", line 1739, in write
zinfo = ZipInfo.from_file(filename, arcname,
File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/zipfile.py", line 523, in from_file
mtime = time.localtime(st.st_mtime)
OverflowError: localtime argument out of range
```
The PR associated to the new behavior is: https://github.com/python/cpython/pull/12726 (new on Python 3.8). Code is AIX specific. Is the code 32 bit only, or maybe the test was not updated?
-----
I can reproduce the behavior as follow:
```
$ python3.8_32
Python 3.8.1 (default, Jan 27 2020, 11:34:59)
[GCC 8.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t
$ python3.8_64
Python 3.8.1 (default, Jan 27 2020, 11:30:15)
[GCC 8.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: localtime argument out of range
$ python3.7_32
Python 3.7.4 (default, Jan 15 2020, 15:50:53)
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t
$ python3.7_64
Python 3.7.4 (default, Jan 15 2020, 15:46:22)
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
time.struct_time(tm_year=2107, tm_mon=1, tm_mday=27, tm_hour=10, tm_min=7, tm_sec=32, tm_wday=3, tm_yday=27, tm_isdst=0)
```
|
|
msg361059 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2020-01-30 13:55 |
I suggest to skip the test if time.localtime() fails with OverflowError.
Can you please try attached patch on AIX, on the master branch if possible? (Or on Python 3.8 at least.)
|
|
msg361061 - (view) |
Author: EGuesnet (EGuesnet) |
Date: 2020-01-30 14:03 |
Tested on Python 3.8.1 on both 32 and 64 bits.
Test is now skipped.
test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... skipped 'time.localtime(4386268800) raises OverflowError'
|
|
msg361062 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2020-01-30 14:08 |
> Tested on Python 3.8.1 on both 32 and 64 bits. Test is now skipped.
Good. I created PR 18282 from my patch.
|
|
msg361063 - (view) |
Author: Michael Felt (Michael.Felt) * |
Date: 2020-01-30 14:20 |
Probably this broke the 64-bit usage.
diff --git a/Python/pytime.c b/Python/pytime.c
index 54ddfc952b..6f13e62490 100644
--- a/Python/pytime.c
+++ b/Python/pytime.c
@@ -1059,7 +1059,7 @@ _PyTime_localtime(time_t t, struct tm *tm)
return 0;
#else /* !MS_WINDOWS */
-#ifdef _AIX
+#ifdef defined(_AIX) and (SIZEOF_LONG ==4)
/* bpo-34373: AIX does not return NULL if t is too small or too large */
if (t < -2145916800 /* 1902-01-01 */
|| t > 2145916800 /* 2038-01-01 */) {
Testing...
|
|
msg361064 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2020-01-30 14:36 |
> Probably this broke the 64-bit usage.
I'm not sure of the meaning of your patch. Are you saying that localtime() supports timestamp after the year 2038 on 64-bit AIX? Did you test that time.localtime(2**32) actually works as expected?
In my timezone, Linux, I get:
$ python3
Python 3.7.6 (default, Dec 19 2019, 22:52:49)
>>> import time; time.localtime(2**32)
time.struct_time(tm_year=2106, tm_mon=2, tm_mday=7, tm_hour=7, tm_min=28, tm_sec=16, tm_wday=6, tm_yday=38, tm_isdst=0)
|
|
msg361065 - (view) |
Author: Michael Felt (Michael.Felt) * |
Date: 2020-01-30 14:40 |
From memory I recall the 64-bit version worked with values above the threshold value that broke the 32-bit library.
And the additional test was needed because the AIX library (iirc did not return NULL on error) - so had to test range before the call to get an overflow.
Working up a test of 32 and 64-bit enviornments.
|
|
msg361066 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2020-01-30 14:47 |
New changeset c232c9110cfefa0935cbf158e35e91746a8a9361 by Victor Stinner in branch 'master':
bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX (GH-18282)
https://github.com/python/cpython/commit/c232c9110cfefa0935cbf158e35e91746a8a9361
|
|
msg361067 - (view) |
Author: EGuesnet (EGuesnet) |
Date: 2020-01-30 14:52 |
> I'm not sure of the meaning of your patch. Are you saying that localtime() supports timestamp after the year 2038 on 64-bit AIX? Did you test that time.localtime(2**32) actually works as expected?
I think it worked as expected before 3.8 on 64 bit.
On AIX 64bit, with Python3.7.4,
$ python3_64
Python 3.7.4 (default, Jan 15 2020, 15:46:22)
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time; time.localtime(2**32)
time.struct_time(tm_year=2106, tm_mon=2, tm_mday=7, tm_hour=7, tm_min=28, tm_sec=16, tm_wday=6, tm_yday=38, tm_isdst=0)
and on 32bit
$ python3_32
Python 3.7.4 (default, Jan 15 2020, 15:50:53)
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time; time.localtime(2**32)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t
With the new Python 3.8.1, on both 32 and 64bits,
Python 3.8.1 (default, Jan 30 2020, 11:23:14)
[GCC 8.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time; time.localtime(2**32)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: localtime argument out of range
|
|
msg361068 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2020-01-30 14:57 |
> bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX (GH-18282)
Even if localtime() is updated to support larger timestamp, this change is still needed for 32-bit AIX.
|
|
msg361069 - (view) |
Author: miss-islington (miss-islington) |
Date: 2020-01-30 15:05 |
New changeset b841633cc2d7619cf4a7db108d91b14926450a6e by Miss Islington (bot) in branch '3.8':
bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX (GH-18282)
https://github.com/python/cpython/commit/b841633cc2d7619cf4a7db108d91b14926450a6e
|
|
msg361070 - (view) |
Author: Michael Felt (Michael.Felt) * |
Date: 2020-01-30 15:06 |
Seems to be working on 64-bit, starting 32-bit test (with overflow expected). Once finished will post a PR.
root@x065:[/data/prj/python/python3-3.9]./python
Python 3.9.0a3+ (heads/bpo-39502-dirty:8d49f7ceb4, Jan 30 2020, 14:47:52) [C] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
time.struct_time(tm_year=2107, tm_mon=1, tm_mday=27, tm_hour=9, tm_min=7, tm_sec=32, tm_wday=3, tm_yday=27, tm_isdst=0)
>>> time.gmtime(4325562452)
time.struct_time(tm_year=2107, tm_mon=1, tm_mday=27, tm_hour=9, tm_min=7, tm_sec=32, tm_wday=3, tm_yday=27, tm_isdst=0)
>>>
|
|
msg361071 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2020-01-30 15:08 |
> Seems to be working on 64-bit, starting 32-bit test (with overflow expected). Once finished will post a PR.
test_time already has tests for timestamps after year 2038, no?
|
|
msg361116 - (view) |
Author: Michael Felt (Michael.Felt) * |
Date: 2020-01-31 15:02 |
OK. There have been some comments/questions re: PR18282.
a) my PR is missing that PR, seems we just missed each other.
b) when using my patch I took a suggestion from issue39460 to test again:
root@x065:[/home/python/python3-3.9]./python -m test -v test_zipfile -m test_add_file_after_2107 --tempdir /tmp
== CPython 3.9.0a3+ (heads/bpo-39502-dirty:c54fe56362, Jan 30 2020, 15:52:37) [C]
== AIX-3-00C291F54C00-powerpc-64bit big-endian
== cwd: /tmp/test_python_303120
== CPU count: 4
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_zipfile
test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.257s
OK
== Tests result: SUCCESS ==
1 test OK.
Total duration: 653 ms
Tests result: SUCCESS
Without the --tempdir setting it "skips" to PASS:
root@x065:[/home/python/python3-3.9]./python -m test -v test_zipfile -m test_add_file_after_2107
== CPython 3.9.0a3+ (heads/bpo-39502-dirty:c54fe56362, Jan 30 2020, 15:52:37) [C]
== AIX-3-00C291F54C00-powerpc-64bit big-endian
== cwd: /data/prj/python/python3-3.9/build/test_python_364784
== CPU count: 4
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_zipfile
test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... skipped 'Linux VFS/XFS kernel bug detected: mtime_ns=91301504000000000'
----------------------------------------------------------------------
Ran 1 test in 0.010s
OK (skipped=1)
== Tests result: SUCCESS ==
1 test OK.
Total duration: 252 ms
Tests result: SUCCESS
FYI
|
|
msg361118 - (view) |
Author: Michael Felt (Michael.Felt) * |
Date: 2020-01-31 15:08 |
p.s., I manually added #18282 to the test, and the results are the same as without - using 64-bit. Will rebuild the 32-bit and try test again.
|
|
msg361272 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2020-02-03 08:40 |
> test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... skipped 'Linux VFS/XFS kernel bug detected: mtime_ns=91301504000000000'
AIX kernel seems to have a bug. You should report it to AIX.
>>> time.localtime(91301504)
time.struct_time(tm_year=1972, tm_mon=11, tm_mday=22, tm_hour=18, tm_min=31, tm_sec=44, tm_wday=2, tm_yday=327, tm_isdst=0)
|
|
msg361303 - (view) |
Author: Michael Felt (Michael.Felt) * |
Date: 2020-02-03 14:45 |
Not sure I understand what bug I am supposed to report. I apologize if
my message https://bugs.python.org/issue39502#msg361116.
I assume your comment re: time.localtime(91301504) comes from this bit
of the test message (mtime_ns=91301504000000000).
Assuming our TZ used are different I am verifying AIX works correctly
for the constant 91301504
root@x065:[/data/prj/python/python3-3.9]./python
Python 3.9.0a3+ (heads/master-dirty:78c7183f47, Feb 3 2020, 14:10:40) [C] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.gmtime(91301504)
time.struct_time(tm_year=1972, tm_mon=11, tm_mday=22, tm_hour=17, tm_min=31, tm_sec=44, tm_wday=2, tm_yday=327, tm_isdst=0)
>>> time.localtime(91301504)
time.struct_time(tm_year=1972, tm_mon=11, tm_mday=22, tm_hour=17, tm_min=31, tm_sec=44, tm_wday=2, tm_yday=327, tm_isdst=0)
root@x065:[/data/prj/python/python3-3.9]set | grep TZ
TZ=UTC0
Again, my apologies for the confusion: the message
"skipped 'Linux VFS/XFS kernel bug detected: mtime_ns=91301504000000000" only occurs when the test is run on a NFS
mount hosted by a SAN based on a Linux kernel.
When I specify --tempdir=/tmp (i.e., using jfs2 as filesystem) the test passes normally.
On 03/02/2020 09:40, STINNER Victor wrote:
> STINNER Victor <vstinner@python.org> added the comment:
>
>> test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... skipped 'Linux VFS/XFS kernel bug detected: mtime_ns=91301504000000000'
> AIX kernel seems to have a bug. You should report it to AIX.
>
>>>> time.localtime(91301504)
> time.struct_time(tm_year=1972, tm_mon=11, tm_mday=22, tm_hour=18, tm_min=31, tm_sec=44, tm_wday=2, tm_yday=327, tm_isdst=0)
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue39502>
> _______________________________________
>
|
|
msg361323 - (view) |
Author: David Edelsohn (David.Edelsohn) * |
Date: 2020-02-03 19:30 |
I think that Victor means AIX kernel and filesystems are not prepared for Y2038.
|
|
msg361603 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2020-02-07 17:56 |
New changeset de6f38db4859f7b8fe4da4556f06c52675fff24a by Michael Felt in branch 'master':
bpo-39502: Fix 64-bit Python PyTime_localtime() on AIX (GH-18285)
https://github.com/python/cpython/commit/de6f38db4859f7b8fe4da4556f06c52675fff24a
|
|
msg361604 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2020-02-07 17:57 |
Does time.gmtime() accept year after 2038 on 64-bit AIX? Example on Linux:
>>> time.gmtime(4386268800)
time.struct_time(tm_year=2108, tm_mon=12, tm_mday=30, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=365, tm_isdst=0)
|
|
msg365056 - (view) |
Author: Michael Felt (Michael.Felt) * |
Date: 2020-03-26 09:42 |
My apologies for the late reply -
Here is 3.6.10:
Python 3.6.10 (default, Mar 24 2020, 14:12:31) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.gmtime(4386268800)
time.struct_time(tm_year=2108, tm_mon=12, tm_mday=30, tm_hour=0,
tm_min=0, tm_sec=0, tm_wday=6, tm_yday=365, tm_isdst=0)
And, a patched 3.9:
root@x064:[/data/prj/python/python-3.9]./python
Python 3.9.0a4+ (default, Mar 13 2020, 08:03:36) [C] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.gmtime(4386268800)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t
So, it should be working - but is not after the patches. I'll work on an
update asap.
On 07/02/2020 18:57, STINNER Victor wrote:
> STINNER Victor <vstinner@python.org> added the comment:
>
> Does time.gmtime() accept year after 2038 on 64-bit AIX? Example on Linux:
>
>>>> time.gmtime(4386268800)
> time.struct_time(tm_year=2108, tm_mon=12, tm_mday=30, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=365, tm_isdst=0)
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue39502>
> _______________________________________
>
|