Issue 10796: Improve doc for readline.set_completer_delims()

Python's readline library generates out of the choices provided by a custom completion function the wrong terminal input. Say, the completion function suggests 'foobar' and 'foobaz' as matching completion strings, readline should produce the characters 'fooba' to prompt and await further interaction by the user. 
This is the intended behaviour of readline and this works in Python as well. However this does nod work anymore as soon, as the suggestion 
list contains dashes `-' as input argument. 

A working as supposed example:

>>> import readline
>>> 
>>> def complete(text, state):
...   if (state == 0):
...     return "abc"
...   if (state == 1):
...     return "ade"
...   else:
...     return None
... 
>>> 
>>> readline.parse_and_bind("Tab: complete")
>>> readline.set_completer(complete)
>>> 
>>> raw_input()
a
abc  ade  
a
'a'
>>> 

remark: I entered a and hit tab. readline produces abc/ade as valid choices, stopping at the first ambiguous character. Now consider the following example:

>>> import readline
>>> 
>>> def complete(text, state):
...   if (state == 0):
...     return "a-bc"
...   if (state == 1):
...     return "a-de"
...   else:
...     return None
... 
>>> 
>>> readline.parse_and_bind("Tab: complete")
>>> readline.set_completer(complete)
>>> 
>>> raw_input()
a-a-a-
'a-a-a-'
>>> 

The intended behaviour is the very same as for the first example. Readline should produce 'a-' and offer a-bc and a-de as valid choices. Instead it produces an additional a- for every time I hit tab. 

Same for Python3.1:

$ python3.1
Python 3.1.2 (release31-maint, Sep 26 2010, 13:51:01) 
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
>>> 
>>> def complete(text, state):
...   if (state == 0):
...     return "a-bc"
...   if (state == 1):
...     return "a-de"
...   else:
...     return None
... 
>>> 
>>> readline.parse_and_bind("Tab: complete")
>>> readline.set_completer(complete)
>>> 
>>> input()
a-a-a-a-a-
'a-a-a-a-a-'
>>> 


Other programming languages falling back to the GNU C-readline library don't have this problem. Consider the roughly equivalent example in Perl which works as expected:

use Term::ReadLine;

sub complete
{
        my ($text, $state) = @_;
        if ($state == 0)
        {
                return "a-bc";
        }
        elsif ($state == 1)
        {
                return "a-cd";
        }
        else
        {
                return undef;
        }
}

my $term = new Term::ReadLine 'sample';
my $attribs = $term->Attribs;

$term->parse_and_bind("Tab: complete");
$attribs->{completion_entry_function} = \&complete;

$term->readline();



Running it:

$ perl rl 
a-
a-bc  a-cd  
a-

I can’t reproduce with 2.7 or 3.2 with a readline module linked to GNU readline.

I think this is the default behavior of readline module.

the default word delimiters for completion contains dash(-). so completion breaks at dash.

  >>> import readline
  >>> readline.get_completer_delims()
  ' \t\n`~!@#$%^&*()-=+[{]}\\|;:\'",<>/?'

In contrast, the default word delimitors of GNU readline is "\t\n\"\\'`@$><=;|&{(" and perl binding does not change it.

when I remove dash from delims like below, it works.

   >>> readline.set_completer_delims(readline.get_completer_delims().replace('-', ''))
   >>> input()
   a-
   a-bc  a-de
   a-

I think I get it: Python sets custom delimiters that include '-' because for the Python REPL, it’s not possible to have one identifier with a dash, so it’s considered a delimiter and when you press tab, a new completion is started.  For a custom REPL however, you may want to have dashes as legal parts of your completed words, so you want to call set_completer_delims.  Reclassifying as a doc patch.

I propose to address this with the general documentation bug, Issue 6953

New changeset 6137c46cb8df by Martin Panter in branch '2.7':
Issue #6953: Rearrange and expand Readline module documentation
https://hg.python.org/cpython/rev/6137c46cb8df

New changeset b1acd6cf15b6 by Martin Panter in branch '3.5':
Issue #6953: Rearrange and expand Readline module documentation
https://hg.python.org/cpython/rev/b1acd6cf15b6

My update includes a new section called Completion, with the following text:

By default, Readline is set up to be used by “rlcompleter” to complete Python identifiers for the interactive interpreter. If the “readline” module is to be used with a custom completer, a different set of word delimiters should be set.

I hope that is enough to call this fixed :)

resolution: fixed
stage: needs patch -> resolved

messages: + msg255316

dependencies: + readline documentation needs work

messages: + msg174686
stage: needs patch

messages: + msg156739
versions: - Python 2.6, Python 3.1