Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Text normalization in list_phones.py #272

Open
ajmalanoski opened this issue Nov 21, 2020 · 12 comments
Open

Text normalization in list_phones.py #272

ajmalanoski opened this issue Nov 21, 2020 · 12 comments

Comments

@ajmalanoski
Copy link
Collaborator

@ajmalanoski ajmalanoski commented Nov 21, 2020

In the Slovenian data, some of the vowels with tone marking (e.g. /é/) are transcribed using the precomposed characters (so here, U+00E9 instead of the sequence U+0065 U+0301). The module ipapy does not recognize the precomposed characters as valid IPA, so list_phones.py incorrectly rejects them. So, I thought it might make sense to normalize the phones as part of list_phones.py.

@kylebgorman
Copy link
Collaborator

@kylebgorman kylebgorman commented Nov 22, 2020

Does ipapy accept them in NKFD normalization? if so you could just use unicodedata.normalize to convert to NFKD, check against ipapy, and then convert back.

@ajmalanoski
Copy link
Collaborator Author

@ajmalanoski ajmalanoski commented Nov 23, 2020

I tested with several languages and only found one problem with NKFD normalization: <ç> is a valid IPA symbol, but the cedilla is not, so ipapy.is_valid_ipa(unicodedata.normalize('NFKD', 'ç')) returns False. To solve this, I think we could just check the NFKC normalization in addition to the NFKD normalization. I don't think this will create any problems here, since we're just looking at individual phones.

@kylebgorman
Copy link
Collaborator

@kylebgorman kylebgorman commented Nov 23, 2020

That could work.

  1. What happens if we use NFD instead of NFKD?
  2. What if we check is_valid_ipa(nfkd_phone) or phone == "ç"? Do you think that's simpler than trying both normalizations?

@ajmalanoski
Copy link
Collaborator Author

@ajmalanoski ajmalanoski commented Nov 23, 2020

I found two differences between the NFD and NFKD normalizations. First, if we use the NFKD normalization, then ipapy.is_valid_ipa will return False for anything with an alveolar click symbol <ǃ> in it, because that symbol gets converted to an exclamation point. Second, if we use NFKD normalization, then ipapy.is_valid_ipa will accept any superscript consonant or vowel symbols, including those that are not part of the standard IPA repertoire. On the other hand, the NFD normalization leads some valid IPA (e.g. /dⁿ/) to be erroneously rejected. On the whole, however, I think that the NFKD normalization leads bad superscripts to be accepted more often than NFD leads good superscripts to be rejected.

While I was working on this, I discovered that there are a number of instances where a character with a diacritic is included as a key in ipapy.UNICODE_TO_IPA, but the diacritic itself is not. For example, <t̼> is in ipapy.UNICODE_TO_IPA, but the linguolabial diacritic itself is not. Because ipapy.is_valid_ipa works by checking to see if the individual characters in a string are in ipapy.UNICODE_TO_IPA, any string that contains the linguolabial marker will be rejected, even if it shouldn't be. While this isn't exactly the same as the case with <ç> (<◌̧> alone is not an IPA symbol), we can get around both these problems by defining the following set

OTHER_VALID_IPA = frozenset(
        phone
        for phone in ipapy.UNICODE_TO_IPA.keys()
        if not ipapy.is_valid_ipa(unicodedata.normalize('NFD', phone))
)

and then checking if a phone passes ipapy.is_valid_ipa OR is in OTHER_VALID_IPA. It's not a perfect solution, however
– for example, <t̼ː> would still erroneously be rejected, I think.

@kylebgorman
Copy link
Collaborator

@kylebgorman kylebgorman commented Nov 23, 2020

@ajmalanoski
Copy link
Collaborator Author

@ajmalanoski ajmalanoski commented Nov 23, 2020

It's up to you. I've already made the change on my local copy, though, so I could just push the changes and file a pull request.

@kylebgorman
Copy link
Collaborator

@kylebgorman kylebgorman commented Nov 23, 2020

@agutkin
Copy link
Contributor

@agutkin agutkin commented Nov 23, 2020

Yeah, do it.

But surely there must be some better packages out there?

@kylebgorman
Copy link
Collaborator

@kylebgorman kylebgorman commented Nov 24, 2020

@kylebgorman
Copy link
Collaborator

@kylebgorman kylebgorman commented Nov 25, 2020

Should this be closed @ajmalanoski or are there outstanding issues here?

@ajmalanoski
Copy link
Collaborator Author

@ajmalanoski ajmalanoski commented Nov 25, 2020

I wasn't sure if you wanted me to look for a better package, like @agutkin was suggesting

@kylebgorman
Copy link
Collaborator

@kylebgorman kylebgorman commented Nov 25, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants