Issue 29456: bugs in unicodedata.normalize: u1176, u11a7 and u11c3

unicodedata can't normalize(NFC) hangul strings which contain \u1176(HANGUL JUNGSEONG A-O).

>>> from unicodedata import normalize
>>> normalize("NFC", "\u1100\u1176\u11a8")
'깍'

=> should be "\u1100\u1176\u11a8" not '깍' (\uae4d)

I attached a patch for this issue. (Fixing boundary of modern medial vowels)

How about the third character's range? The code seems assuming it's [11a7..11c3] while the spec is [11a8..11c2]?

>>> unicodedata.normalize("NFC", "\u1100\u1175\u11a7")
'기'

while it should be '기ᆧ'?

I think you are right. The modern final consonants is [11a8..11c2].
I attached another patch for this issue.

Is there anything need more?

We have moved our code hosting to GitHub, would you mind turn your patch into a GitHub PR first Wonsup?

Ok, I'll do it.

Any updates? I need this fix for my project.

I added some test cases for this issue. Please, someone check this.

I think it can be merged. Is there anything I need to do?

Hi Wonsup, sorry for the delay. I get really busy with my work these days. If no one get involved I'd try to find time reviewing your patch this week.

This patch fixes changes in Unicode 4.1.0.
I think it well reviewed and it is time to merge.
Who can commit this patch? 

@animalize says:
Let me give a supplement:

Before Unicode 4.1.0 (draft), here is: TBase <= code <= TBase+TCount
see: http://www.unicode.org/reports/tr15/tr15-24.html#hangul_composition

After Unicode 4.1.0, here is TBase < code < TBase+TCount, which in line with the latest version (Unicode 10.0)
see: http://www.unicode.org/reports/tr15/tr15-25.html#hangul_composition

This change happened in 2005.

Hello?

ping, this was forgotten.

Hello!

Sorry for the absence and late response. I just reviewed it and think it's ready. I think the change in the unicode standard is more like a bug in the implementation than an intentional change. It's mentioned in Unicode 3.0 the third character is out of bounds when TIndex <= 0 or TIndex >= TCount. We have a ucd_3_2_0 in unicodedata.

I'll merge it after resolve the CI bot.

New changeset d134809cd3764c6a634eab7bb8995e3e2eff14d5 by Xiang Zhang (Wonsup Yoon) in branch 'master':
bpo-29456: Fix bugs in unicodedata.normalize: u1176, u11a7 and u11c3 (GH-1958)
https://github.com/python/cpython/commit/d134809cd3764c6a634eab7bb8995e3e2eff14d5

New changeset 0e2b76ea4e48d0fc1ca34ae4ffbb2fd6c19664bb by Miss Islington (bot) in branch '3.7':
bpo-29456: Fix bugs in unicodedata.normalize: u1176, u11a7 and u11c3 (GH-1958)
https://github.com/python/cpython/commit/0e2b76ea4e48d0fc1ca34ae4ffbb2fd6c19664bb

New changeset e2e7ff0d0378ba44f10a1aae10e4bee957fb44d2 by Miss Islington (bot) in branch '3.6':
bpo-29456: Fix bugs in unicodedata.normalize: u1176, u11a7 and u11c3 (GH-1958)
https://github.com/python/cpython/commit/e2e7ff0d0378ba44f10a1aae10e4bee957fb44d2

New changeset 1889c4cbd62e200fa4cde3d6219e0aadf9bd8149 by Xiang Zhang in branch '2.7':
bpo-29456: Fix bugs in unicodedata.normalize: u1176, u11a7 and u11c3 (GH-1958) (GH-7704)
https://github.com/python/cpython/commit/1889c4cbd62e200fa4cde3d6219e0aadf9bd8149

> We have a ucd_3_2_0 in unicodedata.

Probably this 3.2 unicodedata is used for IDNA2003.
In IDNA2003 there is a step: normalize the domain_name string to Unicode Normalization Form C.

Now we changed the Composition code of Hangul to Unicode Standard 4.1+, and fixed the bug even in Unicode Standard 4.1-.
Should this (Unicode Standard 4.1+ behavior) cause a security vulnerability for someone who is using IDNA2003 via ucd_3_2_0?

As I said, I checked Unicode 3.0 for the hangul composition algorithm. It looks consistent with Unicode 4.1+. 3.0 only gets description but no sample implementation. So I think the changed code also applies to Unicode 3.0+.

You are right.

I found a Normalization Test Suite for Unicode 3.2
http://www.unicode.org/Public/3.2-Update/NormalizationTest-3.2.0.txt

\u1176 is not in the range of the second character.
\u11a7, \u11c3 are not in the range of the third character.

Thanks for your confirmation, Ma Lin. Also thanks for Wonsup!

versions: + Python 3.5, Python 3.7

messages: + msg287079