Issue 29412: IndexError thrown on email.message.Message.get

Test case:

  import email
  import email.policy
 
  txt = '''From: juckelman@strozfriedberg.co.uk
  To: (Recipient list suppressed)
  Date: Thu, 22 Aug 2013 04:13:02 +0000
  Subject: ADSF-1082
 
  Hey!
  '''
 
  msg = email.message_from_string(txt)
  msg.get('to')
  msg = email.message_from_string(txt, policy=email.policy.default)
  msg.get('to')

The second msg.get() throws an IndexError:

  Traceback (most recent call last):
    File "test.py", line 16, in <module>
      print(msg.get('to'))    # throws IndexError
    File "/usr/lib64/python3.5/email/message.py", line 472, in get
      return self.policy.header_fetch_parse(k, v)
    File "/usr/lib64/python3.5/email/policy.py", line 153, in header_fetch_parse
      return self.header_factory(name, ''.join(value.splitlines()))
    File "/usr/lib64/python3.5/email/headerregistry.py", line 586, in __call__
      return self[name](name, value)
    File "/usr/lib64/python3.5/email/headerregistry.py", line 197, in __new__
      cls.parse(value, kwds)
    File "/usr/lib64/python3.5/email/headerregistry.py", line 337, in parse
      kwds['parse_tree'] = address_list = cls.value_parser(value)
    File "/usr/lib64/python3.5/email/headerregistry.py", line 328, in value_parser
      address_list, value = parser.get_address_list(value)
    File "/usr/lib64/python3.5/email/_header_value_parser.py", line 2336, in get_address_list
      token, value = get_address(value)
    File "/usr/lib64/python3.5/email/_header_value_parser.py", line 2313, in get_address
      token, value = get_group(value)
    File "/usr/lib64/python3.5/email/_header_value_parser.py", line 2269, in get_group
      token, value = get_display_name(value)
    File "/usr/lib64/python3.5/email/_header_value_parser.py", line 2095, in get_display_name
      token, value = get_phrase(value)
    File "/usr/lib64/python3.5/email/_header_value_parser.py", line 1770, in get_phrase
      token, value = get_word(value)
    File "/usr/lib64/python3.5/email/_header_value_parser.py", line 1745, in get_word
      if value[0]=='"':
  IndexError: string index out of range

The docs say that email.policy.default has raise_on_defect set to False, hence parse errors ought to be reported via EmailMessage.defects, not by throwing an exception.

Does the patch from issue 27931 fix your problem as well?  I haven't looked closely enough to see if I think it should, I'm just hoping :)

No dice. I get the same exception with issue27931_v2.patch. I briefly looked at the other two, and don't expect those will help, either.

This seems not related to #27931.

The problem is that the receiver's content is only CFWS. It's just like it's empty and for the default policy, it checks `value[0] == '"'` in `get_word`.

I'm really short on time to even review patches these days, but I'll see if I can pry any loose if someone wants to propose a patch.

I'm working on a patch now.

Here's a patch, complete with tests.

If the value is all CFWS, then get_cfws(value)[1], which is what's left after the CFWS is extracted, is the empty string---which is why value[0] throws in this case.

I can't reproduce this problem with the latest master branch, it was perhaps fixed with some other PR. 

This is also a dupe of bpo-31445.

@barry, @david: I think this issue can be closed.

For the record, this is how I tested using the master branch:

>>> msg = email.message_from_string('  To: (Recipient list suppressed)')
>>> msg['To']
>>> import email.policy
>>> msg = email.message_from_string('  To: (Recipient list suppressed)', policy=email.policy.default)
>>> msg
<email.message.EmailMessage object at 0x7f377512b370>
>>> msg['To']
>>> msg.get('to')

Nevermind, I was wrong, I was able to reproduce it:

>>> msg = email.message_from_string('To: (Recipient list suppressed)', policy=email.policy.default))
  File "<stdin>", line 1
SyntaxError: unmatched ')'
>>> msg = email.message_from_string('To: (Recipient list suppressed)', policy=email.policy.default)
>>> msg.get('to')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/maxking/Documents/cpython/Lib/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/home/maxking/Documents/cpython/Lib/email/policy.py", line 163, in header_fetch_parse
    return self.header_factory(name, value)
  File "/home/maxking/Documents/cpython/Lib/email/headerregistry.py", line 589, in __call__
    return self[name](name, value)
  File "/home/maxking/Documents/cpython/Lib/email/headerregistry.py", line 197, in __new__
    cls.parse(value, kwds)
  File "/home/maxking/Documents/cpython/Lib/email/headerregistry.py", line 340, in parse
    kwds['parse_tree'] = address_list = cls.value_parser(value)
  File "/home/maxking/Documents/cpython/Lib/email/headerregistry.py", line 331, in value_parser
    address_list, value = parser.get_address_list(value)
  File "/home/maxking/Documents/cpython/Lib/email/_header_value_parser.py", line 1951, in get_address_list
    token, value = get_address(value)
  File "/home/maxking/Documents/cpython/Lib/email/_header_value_parser.py", line 1928, in get_address
    token, value = get_group(value)
  File "/home/maxking/Documents/cpython/Lib/email/_header_value_parser.py", line 1884, in get_group
    token, value = get_display_name(value)
  File "/home/maxking/Documents/cpython/Lib/email/_header_value_parser.py", line 1710, in get_display_name
    token, value = get_phrase(value)
  File "/home/maxking/Documents/cpython/Lib/email/_header_value_parser.py", line 1385, in get_phrase
    token, value = get_word(value)
  File "/home/maxking/Documents/cpython/Lib/email/_header_value_parser.py", line 1360, in get_word
    if value[0]=='"':
IndexError: string index out of range

New changeset 7213df7bbfd85378c6e42e1ac63144d5974bdcf6 by Barry Warsaw (Abhilash Raj) in branch 'master':
bpo-29412: Fix indexError when parsing a header value ending unexpectedly (GH-14387)
https://github.com/python/cpython/commit/7213df7bbfd85378c6e42e1ac63144d5974bdcf6

New changeset b950cdb4beabeb093fa3ccc35f53d51cc0193aba by Barry Warsaw (Miss Islington (bot)) in branch '3.7':
bpo-29412: Fix indexError when parsing a header value ending unexpectedly (GH-14387) (GH-14412)
https://github.com/python/cpython/commit/b950cdb4beabeb093fa3ccc35f53d51cc0193aba

New changeset 82654a037211a3466a294d53952926fc87f8403d by Barry Warsaw (Miss Islington (bot)) in branch '3.8':
bpo-29412: Fix indexError when parsing a header value ending unexpectedly (GH-14387) (GH-14411)
https://github.com/python/cpython/commit/82654a037211a3466a294d53952926fc87f8403d