Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug with emoji encoding? #396

Open
tkossak opened this issue May 4, 2020 · 1 comment
Open

Bug with emoji encoding? #396

tkossak opened this issue May 4, 2020 · 1 comment
Labels

Comments

@tkossak
Copy link

@tkossak tkossak commented May 4, 2020

Hi, when I get text from telegram message that contains emoji, I can't index the message string properly. Here's sample code:

    import pyrogram
    import emoji

    chat_id = <INSERT CHAT ID HERE>
    msg_id = <INSERT MESSAGE ID HERE>
    app = pyrogram.Client(<SETUP APP HERE>)
    app.start()

    em = emoji.EMOJI_ALIAS_UNICODE[':headphone:']
    new_text = f'a {em} a'

    try:
        app.edit_message_text(
            chat_id=chat_id,
            message_id=msg_id,
            text=new_text,
        )
    except pyrogram.errors.BadRequest as e:
        None  # bad request 400 - can't update when new text is the same as old

    msg = app.get_messages(
        chat_id=chat_id,
        message_ids=msg_id,
    )
    old_text = msg.text

    print([c for c in new_text])
    print(new_text[2])
    print([c for c in old_text])
    print(old_text[2])  # this line throws error
    app.stop()

Output is:

Pyrogram v0.17.1, Copyright (C) 2017-2020 Dan <https://github.com/delivrance>
Licensed under the terms of the GNU Lesser General Public License v3 or later (LGPLv3+)

['a', ' ', '🎧', ' ', 'a']
🎧
['a', ' ', '🎧', ' ', 'a']
Traceback (most recent call last):
  File "telegram_actions.py", line 345, in <module>
    test2()
  File "telegram_actions.py", line 49, in test2
    print(old_text[2])
  File "/home/kossak/.cache/pypoetry/virtualenvs/telegram-send-audiobook-H_xae_Ep-py3.8/lib/python3.8/site-packages/pyrogram/client/types/messages_and_media/message.py", line 58, in __getitem__
    return parser_utils.remove_surrogates(parser_utils.add_surrogates(self)[item])
  File "/home/kossak/.cache/pypoetry/virtualenvs/telegram-send-audiobook-H_xae_Ep-py3.8/lib/python3.8/site-packages/pyrogram/client/parser/utils.py", line 37, in remove_surrogates
    return text.encode("utf-16", "surrogatepass").decode("utf-16")
UnicodeDecodeError: 'utf-16-le' codec can't decode bytes in position 2-3: unexpected end of data

I use emoji package for generating emojis, but the same error is thrown when pyrogram gets message sent directly from telegram app.

Is it a bug in the encoding of the message text? Am I doing something wrong?
I need to index the characters because I'm inserting the Message.entities as markdown links, so I can send the original links along with the updated message.

I know I can manually convert it using old_text = str(msg.text) (and then indexing works) but then msg.entities offsets are wrong.

Thank you in advance for any help.

I use:

  • python 3.8.2
  • pyrogram 0.17.1 (the same error is for 0.16)
  • Linux Manjaro
@tkossak
Copy link
Author

@tkossak tkossak commented May 4, 2020

After debugging into pyrogram i found that I actually don't need indexing to insert links, I can use msg.text.markdown for it. But I leave this thread open in case someone can confirm or deny that this indexing error on msg.text is actually a bug. I found out that the strange encoding is introduced in:

text=(
Str(message.message).init(entities) or None
if media is None or web_page is not None
else None
),

Before this statement the message.message value works with indexing as expected in contrast to the new text attribute. The culprit seems to be Str.__getitem__() method:

def __getitem__(self, item):
return parser_utils.remove_surrogates(parser_utils.add_surrogates(self)[item])

but I didn't debug it further.

That explains why it works with iterating over Str and doesn't work with indexing - only __getitem__() is overriden for Str subclass but not __iter__()).

I don't know why __getitem__() needs to "add surrogates" and then remove them to return a character.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.