Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"RangeError: Maximum call stack size exceeded" When concatenating to a function with a large array #389

Open
vans163 opened this issue Sep 6, 2017 · 6 comments
Labels

Comments

@vans163
Copy link
Contributor

@vans163 vans163 commented Sep 6, 2017

EDIT: I tested recursive functions and they seem unaffected by this. It seems isolated only to Enum.filter.

Maximum callstack size on chrome is 10402. Calling Enum.filter on collections of size 1000+ ends up crashing with this

Uncaught (in promise) RangeError: Maximum call stack size exceeded

I think its to do with how the code got generated.

    def react_dom([]), do: []
    def react_dom([nil|t]), do: react_dom(t)
    def react_dom(dom) do
        [tag|dom] = dom
        tag = fix_binary(tag)

        {attributes, dom} = case dom do
            [] -> {%{},[]}
            [map|rdom] when is_map(map) -> {fix_attributes(map),rdom}
            _ -> {%{},dom}
        end

        case dom do
            [v] when v != nil and (is_atom(v) or is_binary(v) or is_integer(v))->
                v = fix_binary(v)
                {tag,attributes,v}

            children when is_list(children) ->
                children = Enum.filter(children, &(&1!=nil))
                children = :lists.map(&(react_dom(&1)), children)
                {tag,attributes,children}
        end
    end

The very end has a filter children = Enum.filter(children, &(&1!=nil)), it crashses inside the filter.

const [children1] = ElixirScript.Core.Patterns.match(ElixirScript.Core.Patterns.variable('children'), Elixir.Enum.__load(Elixir).filter(children0, (...__function_args__) => {
    function recur(...__function_args__) {
        let __arg_matches__ = null;

       //CRASH IS INSIDE HERE START
        if ((__arg_matches__ = ElixirScript.Core.Patterns.match_or_default([ElixirScript.Core.Patterns.variable('x1')], __function_args__, (x10) => {
       //CRASH IS INSIDE HERE END
            return true;
        })) !== null) {
            let [x10] = __arg_matches__;

            return x10 != null;
        }

        throw new ElixirScript.Core.Patterns.MatchError(__function_args__);
    }

    return ElixirScript.Core.Functions.trampoline(new ElixirScript.Core.Functions.Recurse(recur.bind(null, ...__function_args__)));
}));

It seems as if 10+ callstack frames are being allocated per Enum.filter iteration there.

        tenk = Enum.reduce(1..500, [], &(&2++[&1]))
        filt = Enum.filter(tenk, &(&1!=nil))
        :console.log("okay!")

        tenk = Enum.reduce(1..1000, [], &(&2++[&1]))
        :console.log("okay2!")
        filt = Enum.filter(tenk, &(&1!=nil))
        :console.log("not okay :(")

See for yourself with the directly above code.

@vans163 vans163 changed the title Recursion fails on large collections, danger ahead Enum.filter fails on large collections, danger ahead Sep 6, 2017
@bryanjos
Copy link
Collaborator

@bryanjos bryanjos commented Sep 7, 2017

It's possible it is hitting some recursive function in JavaScript, but will have to do more investigating to know for sure

@bryanjos bryanjos added the kind:bug label Sep 7, 2017
@bryanjos bryanjos added this to the 0.31.0 milestone Sep 7, 2017
@bryanjos
Copy link
Collaborator

@bryanjos bryanjos commented Sep 9, 2017

The problem here is specific to concatenation using the current function. Since the call itself is not in what ElixirScript considers the tail position, it causes a stack overflow. The code is in the generation of Elixir.Enum.filter_list

    function filter_list(...__function_args__) {
        function recur(...__function_args__) {
            let __arg_matches__ = null;

            if ((__arg_matches__ = ElixirScript.Core.Patterns.match_or_default([ElixirScript.Core.Patterns.headTail(ElixirScript.Core.Patterns.variable('head'), ElixirScript.Core.Patterns.variable('tail')), ElixirScript.Core.Patterns.variable('fun')], __function_args__, (head0, tail0, fun0) => {
                return true;
            })) !== null) {
                let [head0, tail0, fun0] = __arg_matches__;

                return ElixirScript.Core.Patterns.defmatch(ElixirScript.Core.Patterns.clause([ElixirScript.Core.Patterns.variable('x576460752303352111')], (x5764607523033521110) => {
                    return new ElixirScript.Core.Functions.Recurse(recur.bind(null, tail0, fun0));
                }, (x5764607523033521110) => {
                    return x5764607523033521110 === null || x5764607523033521110 === false;
                }), ElixirScript.Core.Patterns.clause([ElixirScript.Core.Patterns.variable('_')], () => {
                    // ERROR: stackoverflow causing line here
                    return ElixirScript.Core.Functions.concat(head0, filter_list(tail0, fun0));
                    // END ERROR
                }, () => {
                    return true;
                })).call(this, fun0(head0));
            } else if ((__arg_matches__ = ElixirScript.Core.Patterns.match_or_default([[], ElixirScript.Core.Patterns.variable('_fun')], __function_args__, (_fun0) => {
                return true;
            })) !== null) {
                let [_fun0] = __arg_matches__;

                return [];
            }

            throw new ElixirScript.Core.Patterns.MatchError(__function_args__);
        }

        return ElixirScript.Core.Functions.trampoline(new ElixirScript.Core.Functions.Recurse(recur.bind(null, ...__function_args__)));
    }
@bryanjos
Copy link
Collaborator

@bryanjos bryanjos commented Sep 9, 2017

The error can be resolved with a better way of prepending the head of an item to a list if the list is being generated from a function.

To make it clearer, in Elixir it's this line:
https://github.com/elixir-lang/elixir/blob/75150ec7eb7bb373fdf3c188eb01f0c002035cfe/lib/elixir/lib/enum.ex#L2879

@vans163
Copy link
Contributor Author

@vans163 vans163 commented Sep 9, 2017

The problem is the AST generated from this [head | filter_list(tail, fun)], when compiled to JS right?

[{:|, [],
  [{:head, [], Elixir},
   {:filter_list, [], [{:tail, [], Elixir}, {:fun, [], Elixir}]}]}]

Just needs to transcompile better?

Why I ask, because rewriting Enum.filter/2 for ElixirScript would not fix the core problem?

@bryanjos
Copy link
Collaborator

@bryanjos bryanjos commented Sep 9, 2017

Correct. I'm thinking that rewriting concatenation when a function is last will help.

Still using the breaking case, this:

ElixirScript.Core.Functions.concat(head0, filter_list(tail0, fun0));

Can instead be translated into this:

ElixirScript.Core.Functions.concat(head0, () => { 
  return new ElixirScript.Core.Functions.Recurse(recur.bind(null, tail0, fun0));
});

ElixirScript.Core.Functions.concat can then be updated to call the function if given and append the result. Otherwise, concat as normal.

It's just a thought, but I'll have time to test it out later

@bryanjos
Copy link
Collaborator

@bryanjos bryanjos commented Sep 9, 2017

No success with that approach 😞 . Posting the entire error here.

RangeError: Maximum call stack size exceeded
    at Array.map (<anonymous>)
    at resolveArray (/Users/bryanjos/projects/elixirscript/elixirscript/tmp/integration_overflow.mjs:697:27)
    at buildMatch (/Users/bryanjos/projects/elixirscript/elixirscript/tmp/integration_overflow.mjs:929:12)
    at Object.match_or_default (/Users/bryanjos/projects/elixirscript/elixirscript/tmp/integration_overflow.mjs:1180:26)
    at recur (/Users/bryanjos/projects/elixirscript/elixirscript/tmp/integration_overflow.mjs:24357:63)
    at trampoline$1 (/Users/bryanjos/projects/elixirscript/elixirscript/tmp/integration_overflow.mjs:4071:33)
    at Object.concat (/Users/bryanjos/projects/elixirscript/elixirscript/tmp/integration_overflow.mjs:4113:24)
    at ElixirScript.Core.Patterns.defmatch.ElixirScript.Core.Patterns.clause (/Users/bryanjos/projects/elixirscript/elixirscript/tmp/integration_overflow.mjs:24367:56)
    at /Users/bryanjos/projects/elixirscript/elixirscript/tmp/integration_overflow.mjs:986:23
    at recur (/Users/bryanjos/projects/elixirscript/elixirscript/tmp/integration_overflow.mjs:24372:21)
@bryanjos bryanjos changed the title Enum.filter fails on large collections, danger ahead "RangeError: Maximum call stack size exceeded" When concatenating to a function with a large array Sep 9, 2017
@bryanjos bryanjos removed this from the 0.31.0 milestone Sep 20, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.