Issue 33901: test_dbm_gnu.test_reorganize() failed on x86-64 High Sierra 3.x

http://buildbot.python.org/all/#/builders/145/builds/81

x86-64 High Sierra 3.x:

======================================================================
FAIL: test_reorganize (test.test_dbm_gnu.TestGdbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Lib/test/test_dbm_gnu.py", line 77, in test_reorganize
    self.assertTrue(size0 < size1)
AssertionError: False is not true


The only change of this build is the commit cb970730e3ca2522e9b1700dcaf0a06b7e898db6: bpo-33630. But I hardly believe that it's related.

New changeset 22525de737679ace1488e63b7ed289bdb253ffc7 by Serhiy Storchaka in branch 'master':
Use more specific asserts in dbm tests. (GH-7786)
https://github.com/python/cpython/commit/22525de737679ace1488e63b7ed289bdb253ffc7

http://buildbot.python.org/all/#/builders/145/builds/84

======================================================================
FAIL: test_reorganize (test.test_dbm_gnu.TestGdbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Lib/test/test_dbm_gnu.py", line 77, in test_reorganize
    self.assertGreater(size1, size0)
AssertionError: 16777216 not greater than 16777216
----------------------------------------------------------------------

16777216 = 2**24 = 16 MiB

Looks too large for the size of an empty database.

@Matt Billenstein: Hi! Would you mind to have a look at this issue? It seems like test_dbm_gnu started to fail today. Did you upgrade the buildbot worker recently?

For comparison, on Linux size0 = 12288 = 12 KiB. 3 decimal orders smaller.

size0=16777216 (16 MiB). On my Fedora 28, size0=12288 (12 KiB).

Even more strange: another test starts by removing the filename, just in case!?

    def test_error_conditions(self):
        # Try to open a non-existent database.
        unlink(filename)
        ...

Maybe the setUp() method must fail if filename already exists?

The buildbot runs tests in subprocesses:
"Run tests in parallel using 2 child processes"

regrtest spawns child procsses which creates their own temporary empty current directory and TESTFN is related to the current directory. If the filename/database existed before test_reorganize(), it's likely a bug in test_dbm_gnu: but there is a single TestCase class with a tearDown() method which calls unlink(filename).

--

Maybe x86-64 High Sierra 3.x has a different dbm library version which always creates an empty database with 16 MiB preallocated on disk!?

Ok, I reproduced the issue on macOS High-Sierra and gdbm 1.15. Creating a DB creates a file of 16 MiB.

macbook:master haypo$ brew info gdbm
gdbm: stable 1.15 (bottled)
GNU database manager
https://www.gnu.org/software/gdbm/
/usr/local/Cellar/gdbm/1.15 (19 files, 569.8KB)
  Poured from bottle on 2018-06-19 at 13:48:35
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/gdbm.rb
==> Options
--with-libgdbm-compat
	Build libgdbm_compat, a compatibility layer which provides UNIX-like dbm and ndbm interfaces.

and

macbook:master haypo$ ./python.exe
Python 3.8.0a0 (heads/master:22525de, Jun 19 2018, 13:56:57) 
[Clang 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dbm.gnu
>>> import os
>>> dbm.gnu.open("x", "c").close()
>>> os.path.getsize("x")
16777216

PR 7791 makes the test less useful.

I suggest to replace 10000 with size0 or max(size0, 10000).

New changeset 1261bfa83db30b1cf86c1fb816cc167db77874cd by Victor Stinner in branch 'master':
bpo-33901: Fix test_dbm_gnu for gdbm 1.15 (GH-7791)
https://github.com/python/cpython/commit/1261bfa83db30b1cf86c1fb816cc167db77874cd

Considering adding gdbm version to pythoninfo? It is possible to expose version info from gnu dbm.

Did not increasing the size of the value fixed the test?

Xiang: "Considering adding gdbm version to pythoninfo? It is possible to expose version info from gnu dbm."

I wrote PR 7794. What do you think of this?

Serhiy Storchaka: "Did not increasing the size of the value fixed the test?"

Honestly, I don't understand the purpose of the test. Why does *Python* check such low level implementation detail, the exact file size used by gdbm? We are supposed to only test the API, no? If someone wants to change something else, I would suggest to remove test_reorganize().

How can we test an API if calling it doesn't have any effect? reorganize() can be a no-op method, and still pass the test.

New changeset c44d8e5db6fb9d3847c49e9c9718f2b4cf71f506 by Victor Stinner in branch 'master':
bpo-33901: Better test_dbm_gnu.test_reorganize() fix (GH-7795)
https://github.com/python/cpython/commit/c44d8e5db6fb9d3847c49e9c9718f2b4cf71f506

New changeset 13c79c677f9ec9437c82eda72fa1c2d288d8fceb by Victor Stinner in branch '3.7':
bpo-33901: Fix test_dbm_gnu for gdbm 1.15 (GH-7798)
https://github.com/python/cpython/commit/13c79c677f9ec9437c82eda72fa1c2d288d8fceb

New changeset fe8122d7b7c747fc344dde8467b09de6b5888d01 by Miss Islington (bot) in branch '3.6':
bpo-33901: Fix test_dbm_gnu for gdbm 1.15 (GH-7798)
https://github.com/python/cpython/commit/fe8122d7b7c747fc344dde8467b09de6b5888d01

The buildbot is green again!

New changeset 00f9edb98dd64e14daf5c44f303deca5cbc3cdeb by Victor Stinner in branch 'master':
bpo-33901: Add _gdbm._GDBM_VERSION (GH-7794)
https://github.com/python/cpython/commit/00f9edb98dd64e14daf5c44f303deca5cbc3cdeb

Ok, the test have been fixed in 3.6, 3.7 and master. I added a private version number in master.

If someone wants to add a public version number, please go ahead but open a new issue.

Thanks Serhiy and Xiang for the reviews and to help to debug this bug.

Oh, the test also fails on 2.7:

x86-64 High Sierra 2.7
http://buildbot.python.org/all/#builders/140/builds/25

I backported the fix to 2.7: PR 7818.

New changeset f2918881b7b2e13ed1091dad482aec2382358bcb by Victor Stinner in branch '2.7':
bpo-33901: Fix test_gdbm for gdbm 1.15 (GH-7798) (GH-7818)
https://github.com/python/cpython/commit/f2918881b7b2e13ed1091dad482aec2382358bcb

stage: patch review -> resolved