Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create analyzer accepting any random param and returning that in response as well #85710

Open
amitmbm opened this issue Apr 6, 2022 · 12 comments · May be fixed by #89112
Open

Create analyzer accepting any random param and returning that in response as well #85710

amitmbm opened this issue Apr 6, 2022 · 12 comments · May be fixed by #89112
Labels
>bug good first issue low hanging fruit help wanted adoptme :Search/Analysis How text is split into tokens Team:Search Meta label for search team

Comments

@amitmbm
Copy link
Contributor

amitmbm commented Apr 6, 2022

Elasticsearch Version

8.1.1

Installed Plugins

None

Java Version

bundled

OS Version

macos

Problem Description

Create a custom analyzer using below request, Note foo as an param to analyzer with bar value.

PUT customanlyzerindex
{
    "settings": {
        "analysis": {
            "analyzer": {
                "my_analyzer": {
                    "type": "custom",
                    "tokenizer": "standard",
                    "foo": "bar"
                }
            }
        }
    }
}

Index is created successfully, and foo param is returned in the _settings API response.

GET customanlyzerindex/_settings

{
    "customanz": {
        "settings": {
            "index": {
                "routing": {
                    "allocation": {
                        "include": {
                            "_tier_preference": "data_content"
                        }
                    }
                },
                "number_of_shards": "1",
                "provided_name": "customanz",
                "creation_date": "1649229792216",
                "analysis": {
                    "analyzer": {
                        "my_analyzer": {
                            "type": "custom",
                            "foo": "bar",
                            "tokenizer": "standard"
                        }
                    }
                },
                "number_of_replicas": "1",
                "uuid": "zo-99vRVQD6igTzbyFMNKA",
                "version": {
                    "created": "8010199"
                }
            }
        }
    }
}

Steps to Reproduce

Same as problem Description

Logs (if relevant)

N/A

@martijnvg martijnvg added the :Search/Highlighting How a query matched a document label Apr 6, 2022
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Apr 6, 2022
@elasticmachine
Copy link
Collaborator

elasticmachine commented Apr 6, 2022

Pinging @elastic/es-search (Team:Search)

@amitmbm
Copy link
Contributor Author

amitmbm commented Apr 6, 2022

@martijnvg this is not related to highlighting, instead analysis. Hence correct tag should be ::Search/analysis

@romseygeek romseygeek added :Search/Analysis How text is split into tokens and removed :Search/Highlighting How a query matched a document labels Apr 6, 2022
@javanna javanna added good first issue low hanging fruit help wanted adoptme labels Apr 19, 2022
@ShailyTyagi
Copy link

ShailyTyagi commented Apr 26, 2022

I am interested in contributing - could somebody give me any useful pointers to be able to pick this issue up? I have gone through the readme(s)
@amitmbm

@javanna
Copy link
Member

javanna commented Apr 26, 2022

Hello @ShailyTyagi I believe https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/index/analysis/AnalysisRegistry.java is where we load the analyzers/normalizers/etc configured in the index settings. There we look at type and name but we seem to be silently ignoring any additional parameter, while we would like to instead throw exception whenever an unsupported parameter is provided.

@EverWinter23
Copy link

EverWinter23 commented May 2, 2022

Hi @javanna @amitmbm , I found it interesting and was looking into it, and it seems like CustomAnalyzerProvider assigns the settings without validating. How should I proceed? I don't see any module being used to validate analyzer settings.

    public CustomAnalyzerProvider(IndexSettings indexSettings, String name, Settings settings) {
        super(indexSettings, name, settings);
        this.analyzerSettings = settings; // {"bar":"foo","foo":"bar","tokenizer":"standard","type":"custom"}
    }

@amitmbm
Copy link
Contributor Author

amitmbm commented May 5, 2022

@javanna would you be able to guide @EverWinter23 and @ShailyTyagi here, At first look it doesn't seems to be a small code change, but I might be completely wrong here.
Also would be more than happy to contribute here.

@EverWinter23
Copy link

EverWinter23 commented May 11, 2022

@amitmbm Any suggestions on how to move forward with it?

@amitmbm
Copy link
Contributor Author

amitmbm commented May 12, 2022

@EverWinter23 Sorry I wouldn't be able to help here, I guess someone from Elastic would be the best to provide input here

@amitmbm
Copy link
Contributor Author

amitmbm commented Jun 2, 2022

@javanna its been quite sometime, and a few users are willing to work on the issue, would be great if community can get some pointers on what needs to be done here.

@javanna
Copy link
Member

javanna commented Jun 9, 2022

thanks for the ping @amitmbm !

I am not super familiar with this area, but I had a look and while it may look simpler to just validate that only supported keys are used when parsing the analysis settings, we currently have no way to determine which keys are supported by which analyzer/filter/etc, which is complex especially given that custom components can be plugged in.

It all starts at AnalysisRegistry where we read settings from a map , from which we read the type and then delegate to the available provider the parsing of its corresponding settings. In other places we have a contract that when a setting is read from the map it is removed, so that at the end we can verify that there are no unsupported parameters left which were provided in the request. This becomes though quite a big infra change. Maybe instead of requiring changes to all of the providers, we could wrap the settings into a settings implementation which removes at get transparently, but that means that the same parameter can't be read multiple times. @romseygeek do you have thoughts on this? Could you double check that I am not missing anything?

@KirtiKamal
Copy link

KirtiKamal commented Jul 4, 2022

can any one help me to find out issues I am new to Open Source and want to find my first issue how can I find

@likzn likzn linked a pull request Aug 4, 2022 that will close this issue
@alesblaze01
Copy link

alesblaze01 commented Aug 5, 2022

Hi,
I am new to open-source this is my first open-source project i am checking and looking forward to start making contributions to it, if anyone can mentor me ,it will be very helpful for me, would like to work on this issue!

Thanks.

@javanna javanna self-assigned this Sep 9, 2022
@javanna javanna removed their assignment Oct 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug good first issue low hanging fruit help wanted adoptme :Search/Analysis How text is split into tokens Team:Search Meta label for search team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants