Text normalization in list_phones.py
#272
Comments
|
Does |
|
I tested with several languages and only found one problem with NKFD normalization: <ç> is a valid IPA symbol, but the cedilla is not, so |
|
That could work.
|
|
I found two differences between the NFD and NFKD normalizations. First, if we use the NFKD normalization, then While I was working on this, I discovered that there are a number of instances where a character with a diacritic is included as a key in and then checking if a phone passes |
|
That seems like an improvement to me.
Given how much trouble this is, though, and how recently `ipapy` was added
to this approach, maybe we should just drop it and revert @agutkin's change
there.
…On Mon, Nov 23, 2020 at 6:44 PM ajmalanoski ***@***.***> wrote:
I found two differences between the NFD and NFKD normalizations. First, if
we use the NFKD normalization, then ipapy.is_valid_ipa will return False
for anything with an alveolar click symbol <ǃ> in it, because that symbol
gets converted to an exclamation point. Second, if we use NFKD
normalization, then ipapy.is_valid_ipa will accept any superscript
consonant or vowel symbols, including those that are not part of the
standard IPA repertoire. On the other hand, the NFD normalization leads
some valid IPA (e.g. /dⁿ/) to be erroneously rejected. On the whole,
however, I think that the NFKD normalization leads bad superscripts to be
accepted more often than NFD leads good superscripts to be rejected.
While I was working on this, I discovered that there are a number of
instances where a character with a diacritic is included as a key in
ipapy.UNICODE_TO_IPA, but the diacritic itself is not. For example, <t̼>
is in ipapy.UNICODE_TO_IPA, but the linguolabial diacritic itself is not.
Because of ipapy.is_valid_ipa works by checking to see if the individual
characters in a string are in ipapy.UNICODE_TO_IPA, any string that
contains the linguolabial marker will be rejected, even if it shouldn't be.
While this isn't exactly the same as the case with <ç> (<◌̧> alone is not
an IPA symbol), we can get around both these problems by defining the
following set
OTHER_VALID_IPA = frozenset(
phone
for phone in ipapy.UNICODE_TO_IPA.keys()
if not ipapy.is_valid_ipa(unicodedata.normalize('NFD', phone))
)
and then checking if a phone passes ipapy.is_valid_ipa OR is in
OTHER_VALID_IPA. It's not a perfect solution, because, say, <t̼ː> would
still be erroneously rejected, I think.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#272 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABG4OKADXEEUY6DUXYYQ5DSRLXW5ANCNFSM4T5QTWXA>
.
|
|
It's up to you. I've already made the change on my local copy, though, so I could just push the changes and file a pull request. |
|
Let's do the change for now.
…On Mon, Nov 23, 2020 at 6:54 PM ajmalanoski ***@***.***> wrote:
It's up to you. I've already made the change on my local copy, though, so
I could just push the changes and file a pull request.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#272 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABG4OO76Z4ODC4366KT5LLSRLY3BANCNFSM4T5QTWXA>
.
|
|
Yeah, do it. But |
|
Honestly I think I would just maintain a `frozenset` myself for this, if
that's the case.
…On Mon, Nov 23, 2020 at 6:57 PM Alexander Gutkin ***@***.***> wrote:
Yeah, do it.
But surely there must be some better packages out there?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#272 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABG4OKAFGY26YX2VE6RQVTSRLZHJANCNFSM4T5QTWXA>
.
|
|
Should this be closed @ajmalanoski or are there outstanding issues here? |
|
I wasn't sure if you wanted me to look for a better package, like @agutkin was suggesting |
|
Okay, not a crazy idea, if you want to. Let's keep it open then.
…On Wed, Nov 25, 2020 at 1:59 PM ajmalanoski ***@***.***> wrote:
I wasn't sure if you wanted me to look for a better package, like @agutkin
<https://github.com/agutkin> was suggesting
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#272 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABG4OMVRIL6KZ4ULWYFDATSRVHYVANCNFSM4T5QTWXA>
.
|
In the Slovenian data, some of the vowels with tone marking (e.g. /é/) are transcribed using the precomposed characters (so here, U+00E9 instead of the sequence U+0065 U+0301). The module
ipapydoes not recognize the precomposed characters as valid IPA, solist_phones.pyincorrectly rejects them. So, I thought it might make sense to normalize the phones as part oflist_phones.py.The text was updated successfully, but these errors were encountered: