Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upOptimize TupleDomainFilter for NOT IN #13843
Open
Assignees
Labels
Comments
|
Do we want to create new class that extends BytesValues/BigintValues or we just add the flag to BytesValues/BigintValues class @mbasmanova @bhhari |
|
Also for BigintValues, now we have BigintValuesUsingHashTable and BigintValuesUsingBitmask, do we only extend the function to BigintValuesUsingHashTable? @mbasmanova @bhhari |
|
@kewang1024 Ke, I'd start by adding a flag to BytesValues and BigintValuesXxx classes. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
TupleDomainFilters for IN predicates are optimized to use hash tables for lookups and offer O(1) performance, but NOT IN filters are not optimized and give O(n) performance. Yet, there is no reason why IN predicate optimization couldn't apply to NOT IN. We could extend BytesValues and BigintValues to add a flag indicating that this is a NOT filter and add logic to com.facebook.presto.orc.TupleDomainFilterUtils#toFilter to set this flag.
See com.facebook.presto.orc.TupleDomainFilter.BytesValues and com.facebook.presto.orc.TupleDomainFilter.BigintValues#BigintValues
CC: @prestodb/aria