Issue 35922: robotparser crawl_delay and request_rate do not work with no matching entry

RobotFileParser.crawl_delay and RobotFileParser.request_rate raise AttributeError for a robots.txt with no matching entry for the given user-agent, including no default entry, rather than returning None which would be correct according to the documentation.  E.g.:

>>> from urllib.robotparser import RobotFileParser
>>> parser = RobotFileParser()
>>> parser.parse([])
>>> parser.crawl_delay('example')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/urllib/robotparser.py", line 182, in crawl_delay
    return self.default_entry.delay
AttributeError: 'NoneType' object has no attribute 'delay'

Thanks for your report Joseph, I opened a new PR to fix this.

The PR is looking good, I'll likely merge it soon.

I'm quite sure this should go into 3.8, but should it be backported to 3.7?  This is certainly a bugfix, but still a slight change of behavior, so perhaps we should avoid changing this in 3.7?

Yes, this looks like a bugfix. Who wants an AttributeError? :-)

New changeset 8047e0e1c620f69cc21f9ca48b24bf2cdd5c3668 by Tal Einat (Rémi Lapeyre) in branch 'master':
bpo-35922: Fix RobotFileParser when robots.txt has no relevant crawl delay or request rate (GH-11791)
https://github.com/python/cpython/commit/8047e0e1c620f69cc21f9ca48b24bf2cdd5c3668

New changeset 58a1a76baefc92d9e2392a5dbf65e39e44fb8f55 by Miss Islington (bot) in branch '3.8':
bpo-35922: Fix RobotFileParser when robots.txt has no relevant crawl delay or request rate (GH-11791)
https://github.com/python/cpython/commit/58a1a76baefc92d9e2392a5dbf65e39e44fb8f55

New changeset 45d6547acfb9ae1639adbe03dd14f38cd0642ca2 by Miss Islington (bot) in branch '3.7':
bpo-35922: Fix RobotFileParser when robots.txt has no relevant crawl delay or request rate (GH-11791)
https://github.com/python/cpython/commit/45d6547acfb9ae1639adbe03dd14f38cd0642ca2

Rémi, thanks for the great work writing the PR and quickly going through several iterations of reviews and revisions!

stage: patch review -> resolved