Issue 21538: plistlib unable to load iOS7 Safari History.plist

plistlib has key error when attempting to load the iOS7  
var/mobile/Applications/com.apple.mobilesafari/Library/Safari/History.plist.  No detected issues with other iOS7 plists.

In [8]: with open('cases/vitale_test_iphone/unback/AppDomain-com.apple.mobilesafari/Library/Safari/History.plist', 'rb') as f:
   ...:     plist = plistlib.lo
plistlib.load   plistlib.loads  
   ...:     plist = plistlib.load(f)
   ...:     
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-8-7824ac03848c> in <module>()
      1 with open('cases/vitale_test_iphone/unback/AppDomain-com.apple.mobilesafari/Library/Safari/History.plist', 'rb') as f:
----> 2     plist = plistlib.load(f)
      3 

/usr/lib/python3.4/plistlib.py in load(fp, fmt, use_builtin_types, dict_type)
    993         p = _FORMATS[fmt]['parser'](use_builtin_types=use_builtin_types)
    994 
--> 995     return p.parse(fp)
    996 
    997 

/usr/lib/python3.4/plistlib.py in parse(self, fp)
    620             ) = struct.unpack('>6xBBQQQ', trailer)
    621             self._fp.seek(offset_table_offset)
--> 622             offset_format = '>' + _BINARY_FORMAT[offset_size] * num_objects
    623             self._ref_format = _BINARY_FORMAT[self._ref_size]
    624             self._object_offsets = struct.unpack(

KeyError: 3

Can you attach a zipped version of the failing plist or at least identify what data in it is causing the exception?  Presumably that plist is user-specific, i.e. contains browser history.

I have shared the file for which I have permission, but neither of two
History.plists in my possession will load, both with the same traceback.

Thank you.

On Mon, May 19, 2014 at 4:18 PM, Ned Deily <report@bugs.python.org> wrote:

>
> Ned Deily added the comment:
>
> Can you attach a zipped version of the failing plist or at least identify
> what data in it is causing the exception?  Presumably that plist is
> user-specific, i.e. contains browser history.
>
> ----------
> nosy: +ned.deily
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue21538>
> _______________________________________
>

John, a quick look at the file shows that it contains browsing history information that might be personally identifiable.  I don't think that is appropriate to store on a public web site like this one so I've deleted the file.  Is there some way to provide a simpler file with non-personal history?

Ned, I understand.  I did get permission to post that history, and as I
feared, the sanitized History.plist I'm attaching now doesn't have the
loading issue that the first did.  Possibly a URL character is throwing the
exception?

Can you suggest where I can put a print statement in the library or some
other way to post to get you meaningful debug information?  I don't have
any experience with pdb, and a verbose python execution is pretty lengthy.
 As I said before, I have two more extensive History.plist files from two
different devices that won't load.

On Mon, May 19, 2014 at 5:51 PM, Ned Deily <report@bugs.python.org> wrote:

>
> Ned Deily added the comment:
>
> John, a quick look at the file shows that it contains browsing history
> information that might be personally identifiable.  I don't think that is
> appropriate to store on a public web site like this one so I've deleted the
> file.  Is there some way to provide a simpler file with non-personal
> history?
>
> ----------
> nosy: +ned.deily
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue21538>
> _______________________________________
>

I think this patch should fix the issue. But tests needed.

And here is a patch with test.

Serhiy, I applied the patch and can confirm this is a fix (three previous
failing loads of History.plists from three different devices now load
successfully).  Thank you, I would not likely have identified the issue
myself.

On Mon, May 19, 2014 at 11:08 PM, Serhiy Storchaka
<report@bugs.python.org>wrote:

>
> Changes by Serhiy Storchaka <storchaka@gmail.com>:
>
>
> Removed file: http://bugs.python.org/file35299/plistlib_read_refs.patch
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue21538>
> _______________________________________
>

Patch plistlib_read_refs_2.patch corrects load error for iOS Safari History.plist.

Issue shouldn't be closed as resolved until patch is committed.

Ronald, is it good to you?

The patch looks good.

New changeset f0452bc62cc3 by Serhiy Storchaka in branch '3.4':
Issue #21538: The plistlib module now supports loading of binary plist files
http://hg.python.org/cpython/rev/f0452bc62cc3

New changeset b2c5d0cba5fd by Serhiy Storchaka in branch 'default':
Issue #21538: The plistlib module now supports loading of binary plist files
http://hg.python.org/cpython/rev/b2c5d0cba5fd

Thank you John for your report.

messages: + msg218838
stage: test needed -> patch review

keywords: + patch
type: behavior
stage: test needed

messages: + msg218834

type: crash -> (no value)
components: + Library (Lib)
versions: + Python 3.5

messages: + msg218825