Background

Haldane [1] was interested in “familial selection”, where the size of the family is strictly limited by the food resource, and more newborns are produced than can survive to enter the struggle among the members of the same family e.g. [2]. Contrary to Haldane, in the present paper we focus on the opposite situation, when during a period of limited food availability [3, 4], by food sharing the siblings help each other to survive in monogamous and diploid families. Clearly, during starvation, food sharing is an altruistic behavior since it decreases the survival probability of the donor, while increases that of the recipient. Here we focus on the following case: Whenever a sib is starving, all altruistic sibs help the starving one to survive by donating the necessary food such that the total quantity of the donated food does not depend on the number of donors. Suppose the received food increases the survival of the starving sib by b. If there is only one altruistic sib, then it is the only to donate, and the donation reduces its own survival probability by c. If there are n altruistic sibs, then the survival probability of each will be reduced by c/n, since they share the costs. We emphasize that in each given food sharing event, we have an additive cost model [4]. Here we will look for a condition of subsistence of sib altruism.

In Haldane’s familial selection model, full siblings (sharing both parents) interact, thus it clearly belongs to the aegis of kin selection, in the sense of Maynard Smith [5]. Kin selection theory focuses on the genetic relatedness between the recipient and the actor [1, 6, 7]. The heuristic basic idea of the kin selection theory is the inclusive fitness effect: The altruistic gene can increase its evolutionary success when an individual having the altruistic gene promotes the survival of its sibling, who also carries the same altruistic gene, thus the altruistic individual indirectly increases the frequency of the altruistic gene. The well-known mathematical formulation of the idea of this indirect effect is the classical Hamilton’s rule [7]: the altruism will spread if \(\:rb>c\), where \(\:b\) is the fitness benefit to the recipient, \(\:\:c\) is the fitness cost to the actor and \(\:r\) is the coefficient of genetic relatedness between the interacting individuals. We also focus on altruistic helping with benefit \(\:b\) and cost \(\:c\), but not in pairwise interactions. We emphasize that our game is not a standard evolutionary game, like e.g. the well-known Public Goods Game e.g. [8], in which the individuals’ benefit may be realized in fecundity, and all individuals have equal benefit from the altruistic act. Our game is a survival game cf. [9], where only one individual gets the benefit (sibling’s life is rescued by its altruistic siblings), and the costs of helping (decreasing altruistic siblings’ survival rate) are shared by the altruistic siblings (i.e. the altruistic siblings have no direct benefit). We concentrate on survival for two main reasons. Survival is a prerequisite for reproduction, thus helping in survival seems an important evolutionary strategy between siblings. Moreover, in our survival game, during an altruistic act there is no direct benefit for altruistic individuals (contrary to the public goods game); thus, only indirect fitness effects can promote altruism, i.e. rescuing its parent’s altruistic allele. As a result, in the survival game considered here, the direct benefit of an altruistic sibling cannot mask the indirect effect.

Over the last decade, the classical Hamilton’s rule has been a topic of controversy [10,11,12]. Van Veelen made an effort to find the connection between kin and group selection theories, but he pointed out that these theories, in general, are not equivalent [10, 13,14,15]. Furthermore, he was looking for conditions under which the classical Hamilton’s rule gives a right prediction. Based on replicator dynamics, he found that in a linear model (when the fitness effects are additive and homogenous in the number of altruistic siblings) the classical Hamilton’s rule is correct [14, 16]. Although the replicator dynamics can handle asexual populations [17], contrary to van Veelen, we focus on diploid sexual populations.

The group selection method focuses on the advantage of group living [18,19,20]. Haldane’s familial selection is connected to the group selection too, since the interaction happens within families. There are ingredients of the group selection theory which are important in Haldane’s familial selection. The interaction is not well-mixed in the whole population and altruistic act can be a common action of full siblings [4]. Usually, this interaction is given by a multi-player game [21] with the synergetic effect of the number of altruists on the individual fitness of the group (i.e. family) members. However, the other two ingredients of the group selection theory do not occur in Haldane’s familial selection. In group selection theory, there is a direct competition between different groups. Contrary to that, in Haldane’s familial selection there is no competition between different families. Moreover, in group selection theory, the groups are “quasi” stable and the phenotypic composition of the groups is determined by the formation process of the groups. Contrary to that, in Haldane’s familial selection, the surviving newborns leave their family, and form new families according to the mating system. The phenotypic composition of the families is determined by the genetic system [3, 4] and the family’s phenotypic composition is fixed by the genotype of the parents. For the perspective of the present paper, the heuristic basic idea of the group selection theory is that the common action by group members (within groups and between groups) should be evolutionarily advantageous for them. Here we only focus on Haldane’s familiar selection where there is common action within families, but no intra-familiar competitions.

We agree with Van Veelen and his coauthors, who pointed out these theories are not equivalent [12, 22]. However, from the viewpoint of Haldane’s familial selection, the heuristic basic ideas of the above two theories are reasonable at the same time. On the one hand, the indirect fitness effect can work in diploid sexual families. On the other hand, the group effect (common action by altruistic siblings on the survival rate) can also work, because the selection takes place within a family. Since a direct combination of these theories is questionable [12, 22], in the framework of a population genetic model one can bring together the heuristic basic ideas of these theories [3, 4, 23]. Indeed, in monogamous diploid families, the genetic relatedness between full siblings is ½, thus altruistic help can increase the frequency of the altruistic gene indirectly. Furthermore, the survival probability of each sibling depends on its siblings’ common action (in other words, the group effect takes place). To make it clear what the role of the indirect fitness and group effects is, we must follow van Veelen and his coauthors [24], treating population genetic models rigorously.

The basis of our investigation is the orthodox Darwinian view, in the sense that we focus exclusively on the frequency change of genotypes in time. We think focusing on genotypes is the most direct and simplest way to study the evolution in the framework of Haldane’s familial selection. Why do we follow this orthodox Darwinian view? Below we mention some other possibly applicable methods and the reason why we think that they are not useful in Haldane’s familiar selection, but without a complete overview, of course.

The applicability of the Price equation was criticized by [12, 13, 25].

The allele frequency based model depends on Haldane’s familial selection. The reason is the following. Under panmixia the embryos follow Hardy–Weinberg equilibrium, but during maturation time, the survival rate of each juvenile depends on the genotypes of their full siblings, which are determined by their parents’ genotypes. Consequently, the parental population is not in Hardy–Weinberg equilibrium. The main technical problem is that in Haldane’s familial selection model there is no bijection between genotype and phenotype distributions without genetic equilibrium [26]. In mathematical terms: the genotype distribution cannot be reconstructed unequivocally from a given allele distribution when the population is not in Hardy–Weinberg equilibrium. Thus, if the parental population is not in Hardy–Weinberg equilibrium, then an infinite number of genotype (phenotype) distributions correspond to the same allele distribution. The number of genotypes is always higher than the number of alleles. Consequently, according to the dominant-recessive inheritance of altruism, at the same parental allele distribution we have an infinite number of parental genotype distributions, and for different parental distributions the relative frequency of altruistic siblings is quite different in the whole population.Footnote 1 Natural selection operates on the diploid level. Consequently, the natural selection takes place in a higher dimensional genotype space than the allele space. Thus, the consecutive populations are never in Hardy–Weinberg equilibrium; we can only follow the evolutionary change by following the frequency change of genotype distributions. We also mention that Allen and his co-workers [27,28,29,30] built up a general stochastic mathematical framework for natural selection. At each time step they calculated which alleles were replaced by copies of others as a result of interaction, reproduction, mating, and/or death. Contrary to that, in essence they also follow the genotype distribution change from generation to generation. Finally, in the population genetics models, when the juveniles’ survival rates depend on the genotypes of their parents, the state variable of the model must be the genotype distribution [31,32,33,34,35].

The applicability of individual-based fitness methods e.g [36], is ambiguous in Haldane’s familial selection. In diploid sexual populations, the parent population produces each juvenile genotype (see Table 1), since all genotypes produce other genotypes. For instance, in all families founded by two different homozygotes, all the offspring are heterozygotes. In other words, the principle of replicator dynamics, i.e. “i-type only from i-type”, does not hold when we focus on the frequency of genotypes. Moreover, an observer who can identify the different genotypes, can collect all information about natural selection. Thus in Haldane’s familial selection model the production of different genotypes is the key problem [23], and not the individual fitness (see genotype dynamics, Eq. (4) later). We note that when strict reproduction occurs (i.e. i-type only produces i-type), the individual fitness is calculated as the quotient of the total production of the i-type and the total number of i-type individuals. Thus, practically, the production rate is a more basic notion than the average production rate (i.e. individual fitness).

Table 1 Genotype survival table based on the mating table, with phenotypes and Mendelian probabilities \({m}_{k(ij)}\)
Full size table

The direct application of inclusive fitness also depends on Haldane’s familial selection. There are three main reasons for that.

  1. 1.

    In the framework of Haldane’s familial selection model, the interaction takes place in families, so the benefit and cost of siblings depend only on the genotypes of their parents. The phenotypic composition of each family does not depend on the frequencies of the different parent genotypes in the whole population. Thus, the sib’s benefit and the sib’s cost within families are independent of the genotype frequencies in the whole population, and hence they are “independent” of the allele frequencies in the whole population, too.

  2. 2.

    In the present-day theory of kin selection, in terms of general Hamilton’s rule, the parameters of benefit \(\:b\) and cost \(\:c\) are functions of the allele frequencies [6]. In Haldane’s familial selection the average benefit/cost of the juvenile genotypes with respect to the whole population depends on the genotype distribution of the parental population.

  3. 3.

    Moreover, many parental genotype distributions have the same allele distribution. Thus, the allele distribution contains less information about the selection process than the genotype distribution.

  4. 4.

    The general definition of genetic relatedness is the regression of the genotypes of social partners on the genotype of the focal individual [37,38,39]. In the framework of Haldane’s familial selection under monogamy, the genetic relatedness is exactly ½ between interacting full siblings independently of the state of the whole population.

  5. 5.

    We are interested in the condition under which the rare mutant selfish allele cannot invade a resident altruistic homozygote population. When the mutant allele is rare enough, using the mean-field model, we are interested in what population-state-independent condition guarantees that the relative frequency of altruistic homozygotes always increases. In other words, we are looking for a condition that does not depend on the genotype (so on the allele) frequencies, like in the case of the classical Hamilton’s rule.

Of course, we do not claim that the above listed methods cannot give the same results as we get. We only mentioned some points why we follow the most direct, orthodox Darwinian method, i.e. we focus only on the change of genotype frequencies in the whole population.

Why do we focus on diploid populations? Firstly, this is the case that Haldane’s model addressed originally. Furthermore, one may think the results in a haploid population are almost the same as the results in a diploid population. This is one of the possible reasons why authors mostly consider haploid models see e.g. [6, 17]. There is an essential difference between the haploid and diploid model. Namely, each diploid individual has two alleles together determining the phenotype. In other words, in a haploid model there is no Mendelian inheritance, since a haploid individual has only one allele, thus there is no meaning of dominance between alleles. Thus, anyone interested in the effect of Mendelian inheritance must consider diploid populations, as we have done.

In this paper, we focus on diploid, panmictic populations with Mendelian inheritance within monogamous families [3, 4]. There are two main consequences of Mendelian inheritance in a diploid population: Firstly, the genotype determines the phenotype. Secondly, the phenotypic composition of each family is determined by the mating system, the genotypes of parents, and the genotype-phenotype mapping together. In general, the population genetics models predict that the fixation of the altruistic behavior depends on the genotype-phenotype mapping e.g. [3, 40,41,42]. However, in pairwise interaction with additive cost and benefit functions, the population genetics models give the same results as Hamilton’s method [3, 4, 19], but it does not in the non-additive situation [4, 14, 16, 32]. We emphasize that these results are in harmony with the above recalled results by van Veelen and his coauthors, based on replicator dynamics [14, 16, 17].

In a family, not only pairwise interactions are possible. Since we can assume there are more than two siblings in families, we can consider non-pairwise interactions. Remember, in the standard synergetic group effect, the group members’ action is common (non-pairwise), which in a non-linear way determines the benefit of each member of the group [4, 43]. In this paper, we also focus on a non-pairwise interaction, i.e. when full siblings (sharing both parents) share the cost of altruism. Moreover, an additive cost and benefit function determines the survival probability of each full sibling [14]. Shortly, we consider additive and non-pairwise interactions. We emphasize that our model is additive in the sense that both the costs and benefits sum up, but by cost sharing the homogeneity is violated. Therefore, our model is not linear.

Now the main question of this paper arises: What is the role of the classical Hamilton’s rule in diploid Mendelian population with survival cost sharing among altruistic siblings?

Methods: Model for survival cost sharing

We will use the already introduced mating table based population genetic model [3] for a diploid species with internal fertilization. For the reader’s convenience, we present an overview of this model setup.

For the sake of simplicity, we only consider one autosomal locus with two alleles [a] and [A]. Since here the survival rate of each juvenile depends on its parents’ genotypes, the diploid parental population is not in Hardy–Weinberg equilibrium. Consequently [44], the state variables of our model must be the frequencies of genotypes. Let \(\:x=({x}_{1},\:{x}_{2},\:{x}_{3})\) be the frequency vector of genotypes G1 = ([a], [a]), G2 = ([a], [A]) and G3 = ([A], [A]) in the present generation. Then

$$\:x\in\:{S}_{3}:=\left\{\left({u}_{1},\:{u}_{2},\:{u}_{3}\right):{u}_{1}\ge\:0,\:{u}_{2}\ge\:0,\:{u}_{3}\ge\:0,\:{u}_{1}+{u}_{2}+{u}_{3}=1\right\}.$$

Genotype-phenotype mapping. We assume that the genotypes uniquely determine the phenotypes. The homozygotes \(\:{G}_{1}=\left(\left[a\right],\left[a\right]\right)\) and \(\:{G}_{3}=\left(\left[A\right],\left[A\right]\right)\) are altruistic and selfish, resp., and heterozygote \(\:{G}_{2}=\left(\left[a\right],\left[A\right]\right)\) is either altruistic or selfish. We will analyze the following general setup. The phenotype of a heterozygote individual is altruistic with probability \(\:p\in\:\left[0,\:1\right]\), and selfish with the complementary probability \(\:1-p\). Particularly, altruism is recessive, dominant, or intermediate (additive), according to \(\:p=0,\:\:1,\:\:\) or \(\:\frac{1}{2}\). We note that \(\:p\) gives the degree of dominance of the altruistic allele.

Structure of the population

The species is monogamous, thus relatedness between full siblings is ½. The population size N is very large, and we consider panmixia, so \(N/2\) couples are formed at random. We note that the random mating excludes inbreeding, since the population is large enough. Formally, at the population state \(x \in S_3\), the relative frequency of the mating pair with genotypes \(G_i\) and \(G_j\) is equal to \(x_ix_j\) . Moreover, the parents’ genotypes, according to the Mendelian inheritance, determine the genotype of their offspring. Denote by \(m_{k\left(ij\right)}\) the probability that a pair with \(G_i\) and \(G_j\) genotypes will have an offspring with genotype \(G_k\) (see Table 1). Since the genotype determines the phenotype, the phenotypic composition of the family of parents with \(G_i\) and \(G_j\) genotypes follows either binomial or multinomial distribution. Moreover, the relative frequencies of different families depend on the relative frequencies of the parents. In summary, the population structure depends on the mating system, the parents’ genotype distribution \(x\), and the genotype-phenotype mapping at the same time. In our \(n\)-person altruistic interaction there are exactly \(n\) siblings in each family.

Altruistic interaction with survival cost sharing. By food sharing siblings can help each other to survive during an extreme food-poor period. For surviving a period of starvation, only a given quantity of food is needed. The main point here is that, under a series of predator attacks each individual must survive all attacks (so the product of survival probabilities is the final survival rate) [9]. Contrary to that, in starvation, each individual must survive a period; therefore, here the additive model seems more reasonable than the multiplicative one. Thus, we can consider an additive model for survival. Each sibling having at least one altruistic sib receives benefit \(\:b\) altogether, which is collected from its altruistic sibs equally. Thus, in a family with \(\:k\) altruistic and \(\:n-k\) selfish siblings, the payoff of each altruist is equal to \(\:a-\left(n-1\right)c\), if \(\:k=1\) (where \(\:a\) is the baseline survival rate without any extra food from siblings) and to \(\:a+b-\left(k-1\right)\frac{c}{k-1}-\left(n-k\right)\frac{c}{k}=a+b-\frac{n}{k}c\), if \(\:k\ge\:2\). At the same time, each selfish individual’s payoff is \(\:a+b\), if \(\:k\ge\:1\), and \(\:a\), if \(\:k=0\). More precisely, the survival probability function has to be truncated at \(\:0\) and \(\:1\), that is, if our formula gives the value \(\:p\), the survival probability is \(\:\text{max}\left\{\text{min}\right\{1,\:p\},\:0\}\). However, if a single altruistic act has only a relatively small effect on the survival probability of the helper and the beneficiary, then truncation is needed with but a practically negligible probability. For the sake of computational simplicity, we will omit truncation, that is, \(\:\:c\le\:a/\left(n-1\right)\) and \(\:b\le\:1-a\) is assumed. This assumption resembles the usual weak selection, where all individuals perform approximately equal fitness [45, 46]. However, in our model we focus on the survival probability of different types, and we do not need that those are approximately equal. For instance, in our model the different genotypes may have survival rates 3/4, 1/2 and 1/4. Also observe that each offspring’s survival probability only depends on the phenotype of its siblings. In other words, the interaction is well-mixed in the families, but it is not well-mixed with respect to the whole population. We denote by \(\:{n}_{k\left(ij\right)}\) the average number of surviving sibs with genotype \(\:{G}_{k}\) in a family founded by parents with genotypes \(\:{G}_{i}\) and \(\:{G}_{j}\), and we compute all numbers \(\:{n}_{k\left(ij\right)}\) in SI-A.

In Table 1, we present the average survival rate of each genotype in different families. If there is at least one altruistic sibling among the full siblings, then each sibling (whether altruistic or selfish) receives an additional term b to its survival probability. The cost of this increase is equal to \(\:c\), which is shared among the altruistic siblings equally. So, each beneficiary contributes \(\:b-c\) to the cumulative payoff. If \(\:k-1\), there are only \(\:n-1\) beneficiaries, while for \(\:k>1\) all \(\:n\) siblings are beneficiaries.

Based on Table 1, the total number of individuals of genotypes G1, G2 and G3 in the next generation can be given as follows:

$$\:{V}_{1}\left(x\right)=\frac{N}{2}\left({x}_{1}^{2}{n}_{1\left(11\right)}+2{x}_{1}{x}_{2}{n}_{1\left(12\right)}+{x}_{2}^{2}{n}_{1\left(22\right)}\right),$$
$$\:{V}_{2}\left(x\right)=\frac{N}{2}\left({2x}_{1}{x}_{2}{n}_{2\left(12\right)}+2{x}_{1}{x}_{3}{n}_{2\left(13\right)}+{x}_{2}^{2}{n}_{2\left(22\right)}+{2x}_{2}{x}_{3}{n}_{2\left(23\right)}\right),$$
$$\:{V}_{3}\left(x\right)=\frac{N}{2}\left({x}_{2}^{2}{n}_{3\left(22\right)}+2{x}_{2}{x}_{3}{n}_{3\left(23\right)}+{x}_{3}^{2}{n}_{3\left(33\right)}\right).$$

Observe that now the basic assumption of replicator dynamics is not valid, since here the i-th genotype is not necessarily born from i-th genotype. To emphasize this difference, we will call \(\:{V}_{i}\left(x\right)\) the production of the i-th genotype. We will also need the notion of total production of the whole population, namely, \(\:\sum\:_{j=1}^{3}{V}_{j}\left(x\right)\).

Results

Static characterization of evolutionary stability

In this section, we are interested in the conditions under which the altruistic homozygote will make an evolutionarily stable genotype distribution (ESGD). We assume that the mutation is rare enough, as in the standard assumption of evolutionary game theory [47]. Following [48], we will say that a homozygote is an ESGD if a rare mutant genotype cannot invade the resident homozygote population [3]. Mathematically, this means that the relative frequency of the resident homozygote increases from generation to generation, provided mutation is sufficiently rare. Now, we recall the formal definition [3]: The altruistic homozygote G1 is an ESGD if for all \(\:x\) from a neighborhood of \(\:{x}^{*}=(\text{1,0},0)\) we have

$$\:{V}_{1}\left(x\right)>{x}_{1}\sum\:_{i=1}^{3}{V}_{i}\left(x\right).$$

Firstly, we calculate the condition for the evolutionary stability of altruism. Applying a general result (see Theorem 1 [3]), , we find the following (for mathematical details see SI-A). If \(\:p<1\), then the pure homozygote altruistic population is an ESGD provided \(\:{n}_{1\left(11\right)}>2{n}_{2\left(12\right)}\), which reads as

$$\:b>\left(\frac{1-p}{1+p}{\left(\frac{2}{1-p}\right)}^{n-1}+\frac{2p}{1+p}\right)c.$$
(1)

Particularly, if \(\:p=0\), the sufficient condition for the recessive altruism to be evolutionarily stable is

$$\:b>{2}^{n-1}c.$$
(2)

If \(\:p=1\) (altruism is dominant), then \(\:{n}_{1\left(11\right)}=2{n}_{2\left(12\right)}\), therefore we need to check the second-order conditions of the ESGD (Theorem 1 [3]), . In our particular case, the general condition takes the following simpler form: genotype \(\:{G}_{1}\) is evolutionarily stable if

$$\:\frac{{n}_{3\left(22\right)}}{{n}_{1\left(11\right)}}\cdot\:\text{min}\left\{0,\:{n}_{1\left(11\right)}-2{n}_{2\left(13\right)}\right\}+{2n}_{1\left(12\right)}>{n}_{2\left(22\right)}+{n}_{3\left(22\right)}.$$

After some calculus we arrive at the simple condition

$$\:b>\frac{{4}^{n-1}+2}{3}\:c.$$
(3)

It is interesting that this condition coincides with that of the additive case, i.e., substituting \(\:p=1/2\) in Eq. (1).

Now we are in the position to see the effect of genotype-phenotype mapping and that of cost sharing on the subsistence of altruism in Mendelian, diploid populations (see Table 2).

Table 2 Sufficient conditions for altruism to be evolutionarily stable
Full size table

Now the survival probability is an additive function, and the relatedness between full siblings is ½ in each family. Thus, according to the state of art of the earlier results on altruism based on replicator dynamics (see references in the Introduction), one may expect that the classical Hamilton’s rule holds here. On the contrary, in the population genetics model for diploid species, we found this expectation true only when the offspring size was exactly two (see Table 2, \(\:n=2\)), where cost sharing was not possible. We note that when the interaction is pairwise, the population genetics model pointed out that the classical Hamilton’s rule implied the evolutionary stability of altruism [3].

We found that the genotype-phenotype mapping influences the condition of evolutionary stability of altruism (see Table 2). We think that the reason for this comes from the heuristic basic idea of group selection theory. The genotype-phenotype mapping determines the phenotypic composition of the families and the phenotypic distribution determines the number \(\:s\) of altruistic siblings, which determines the survival probability of each family member. Furthermore, we particularly found that when the altruistic allele is recessive, the conditions are less strict for arbitrary offspring size (see Table 2). The intuitive reason for this is that under recessive inheritance, at every mixed population state (\(\:x\ne\:1\)) there are less families having altruistic sibling, thus the selfish phenotype has less possibility for making use of altruistic siblings’ service.

Fixing the benefit \(\:b\), we find that for the evolutionary stability of altruism, the cost must decrease as the family size \(\:n\) increases, assuming cost sharing among altruistic siblings (see Table 2). In other words, survival cost sharing implies stricter conditions for subsistence of altruism than the classical Hamilton’s rule. Thus, the naïve hypotheses that survival cost sharing could help in the fixation of altruism is wrong. This observation is rooted in the following two intuitive reasons: On the one hand, in the stochastic feature of Mendelian inheritance, when there is only one altruistic sib in a family, then it helps \(\:n-1\) selfish siblings, thus its cost is high. On the other hand, there are less altruistic events under survival cost sharing than in pairwise interactions: e.g., when there are three altruistic siblings, then each receives \(\:b\) by cost sharing, but \(\:2b\) in pairwise interactions.

Secondly, we are also interested in what conditions make the indirect net effect of the altruistic interaction remunerative for the altruistic gene [a] (see SI- B). We found that the classical Hamilton’s rule makes the altruistic interaction remunerative for the altruistic gene [a], under all considered genotype-phenotype mappings (i.e., for all \(\:p\in\:\left[\text{0,1}\right]\)), and at all population states (i.e. \(\:x\in\:{S}_{3}\)). Observe that when the altruistic siblings can share the cost, the fact that the interaction is remunerative for the altruistic allele is not sufficient for the altruism to be evolutionarily stable (see Table 2). The reason for this is as follows. If we only concentrate on the fact that, on average, the altruistic interaction can increase the number of altruistic alleles (cf. the classical Hamilton’s rule), then we do not take account of the effect the altruistic interaction has on the number of selfish alleles. It can happen that the altruistic interaction increases the number of selfish alleles much more than that of the altruistic ones.

Thirdly, we found that if \(\:b<2c\), then in all here considered genotype-phenotype mappings, the selfish homozygote is an ESGD (see SI-A). This implies that, if the classical Hamilton’s rule holds, the altruistic gene spreads in the monomorphic selfish population. We emphasize that the stability condition of the selfish homozygote depends neither on the size of the family nor on the genotype-phenotype mapping.

Finally, since the mathematical conditions for altruism and selfishness to be ESGDs contradict each other, there is no bistability here.

In summary, when there are more than two siblings sharing the cost of altruism (i.e., \(\:n\ge\:3\)), the classical Hamilton’s rule is a sufficient condition for the existence of altruism, but it does not imply the evolutionary stability of the altruistic homozygote monomorphic population.

Dynamical characterization of ESGDs

The condition of classical evolutionary stability is a local one, since it rules out that any possible but rare mutant can invade the monomorphic homozygote population. So, the following natural question arises: In the case of Haldane’s familial selection, what dynamics describes the natural selection globally? In [3] we introduced the genotype dynamics

$$\:{\dot{x}}_{i}={V}_{i}\left(x\right)-{x}_{i}\sum\:_{j=1}^{3}{V}_{j}\left(x\right),\:\:\:\:\:\:i\in\:\{1,\:2,\:3\}.$$
(4)

Observe that the genotype dynamics is based on the Darwinian tenet: The frequency of the i-th type increases in the next generation if \(\:{\:V}_{i}\left(x\right)>{x}_{i}\sum\:_{j=1}^{3}{V}_{j}\left(x\right)\). Here \(\:{V}_{i}\left(x\right)\) is the production of the i-th genotype, and \(\:{x}_{i}\sum\:_{j=1}^{3}{V}_{j}\left(x\right)\) is its share in the total production. The following statement was proved (Theorem 2 in [3], SI C): If the altruistic homozygote G1 is an ESGD, then \(\:{x}^{*}=(\text{1,0},0)\) is a locally asymptotically stable fixed point of the genotype dynamics, Eq. (4). Clearly, the simplex of the genotype distributions is positively invariant with respect to the genotype dynamics, and the homozygote states are the only possible fixed points of the genotype dynamics on the border of the simplex of genotype distributions.

Here, we will use genotype dynamics, Eq. (4) to gain some insight into the global behavior of our model. We are interested in what happens if the classical Hamilton’s rule is satisfied, and what happens if it is not. For this purpose, we consider three different genotype-phenotype mappings (altruism is either recessive, dominant, or intermediate). Concerning each genotype-phenotype mapping considered here, we focus on the following three types of numerical examples: 1. The altruistic homozygote is an ESGD. 2. Neither homozygote is an ESGD. 3. The selfish homozygote is an ESGD. We are interested in the survival cost sharing among siblings, therefore we consider \(\:n=4\), since cost sharing can only happen when the offspring size is greater than 2. For a unified representation, in all Figs. 1, 2 and 3 of the following illustrations, the same points of the simplex serve as the initial states of the trajectories: (0.86, 0.07, 0.07), (0.07, 0.86, 0.07), (0.2, 0.3, 0.5), (0.07, 0.07, 0.86), (0.48, 0.04, 0.48), (0.48, 0.48, 0.04) and (0.04, 0.48, 0.48).

Fig. 1
Fig. 1The alternative text for this image may have been generated using AI.
Full size image

Firstly, we focus on the case when the altruistic behavior is recessive, i.e. \(\:p=0\). Figure 1A shows that, being a pure ESGD, the recessive altruistic homozygote population is globally asymptotically stable. Figure 1B shows the case where neither of the homozygote populations is an ESGD. Then there is a unique globally asymptotically stable equilibrium of the genotype dynamics, (0.2208, 0.5360, 0.2432). Figure 1C shows a case where Hamilton’s rule does not hold, and the pure selfish homozygote population is globally asymptotically stable

Fig. 2
Fig. 2The alternative text for this image may have been generated using AI.
Full size image

Secondly, we focus on the case where the altruistic behavior is dominant, i.e. \(\:p=1,\) and \(\:n=4\). Figure 2A shows that, being a pure ESGD, the dominant altruistic homozygote population is globally asymptotically stable. Figure 2B shows that when neither of the homozygote populations is an ESGD, then there is a unique globally stable equilibrium, (0.1280, 0.4407, 0.4313) of the genotype dynamics. Figure 2C shows that when Hamilton’s rule does not hold, then the pure selfish homozygote population is globally asymptotically stable

Fig. 3
Fig. 3The alternative text for this image may have been generated using AI.
Full size image

Thirdly, we focus on the case when the altruistic behavior is intermediate, i.e., \(\:p=\frac{1}{2}\), and \(\:n=4\). Figure 3A shows that the intermediate altruistic homozygote population, being a pure ESGD, is globally asymptotically stable. Figure 3B shows that when neither of the homozygote populations is an ESGD, then there is a unique globally stable equilibrium of the genotype dynamics, (0.1603, 0.4828, 0.3569). Figure 3C shows that when Hamilton’s rule does not hold, then the pure selfish homozygote population is globally asymptotically stable

Discussion

Now we are in the position to summarize our results. In the Results section, we already gave a condition for the altruistic homozygote to be an ESGD, which implies the local asymptotic stability of the pure altruistic homozygote population. Using genotype dynamics, Eq. (4), we illustrated that for all here considered genotype-phenotype mappings, the pure altruistic homozygote population is globally asymptotically stable if the altruistic homozygote is an ESGD (see Figs. 1A, 2A and 3A).

By using genotype dynamics, Eq. (4), we illustrated that if the altruistic and selfish homozygotes are not ESGDs, then in all here considered genotype-phenotype mappings, the altruistic and selfish phenotypes coexist and there is a unique globally asymptotically stable polymorphic state (see Figs. 1B, 2B and 3B). However, these fixed points are different according to the genotype-phenotype mapping.

Furthermore, we illustrated that if the classical Hamilton’s rule does not hold, then the pure selfish homozygote population is globally asymptotically stable (see Figs. 1C, 2C and 3C).

Based on our dynamical studies, in all considered selection situations, it seems that the local stability properties of the homozygotes have an effect on the global behavior of the genotype dynamics.

Conclusions

Haldane’s familial selection model includes heuristic ideas of kin and group selection each at a time.

Concerning the indirect fitness effect of altruism, we found the following:

  1. a)

    The naive hypothesis, that survival cost sharing could help the fixation of altruism, is based on the fact that the survival probabilities of altruistic siblings decrease less. Contrary to that, we have found that if the offspring size increases then the condition of evolutionary stability of altruism becomes stricter (see Table 2).

  2. b)

    The genotype-phenotype mapping has an effect on the condition of the evolutionary stability of altruism. Namely, if altruism is recessive, then it will become fixed under weaker conditions. The intuitive reason for this is that, under recessive inheritance, the average number of altruistic siblings in the whole population is less, thus the selfish phenotype has less possibility for making use of altruistic siblings’ service.

  3. c)

    Contrary to point b), we found a universal condition for the evolutionary stability of selfish behavior (for all here considered genotype-phenotype mappings and arbitrary offspring size). If the classical Hamilton’s rule does not hold, then the selfish behavior is an ESGD. The intuitive reason for this is that the classical Hamilton’s rule guarantees that the net indirect effect of the altruistic interaction is remunerative for the altruist gene. Observe that this intuitive reasoning comes from the kin selection theory.

  4. d)

    In all here considered genotype-phenotype mappings, the classical Hamilton’s rule implies the stability of altruism if and only if there are only two siblings in each family, so when the interaction is pairwise, as cost sharing is not possible.

  5. e)

    Moreover, when the classical Hamilton’s rule holds and the condition of evolutionary stability of altruism does not, then the selfish and altruistic phenotypes coexist.

In summary, when the altruistic siblings share the survival cost, the classical Hamilton’s rule is a sufficient condition for the existence of altruism, but it does not imply the evolutionary stability of the altruistic homozygote.

Group living opens up the opportunity for common action of more than two individuals. Here we focused on the cost sharing among altruistic siblings. We have found that the group living advantage takes place in Haldane’s familial selection model, since we have shown the following:

  1. 1.

    The conditions for the evolutionary stability of altruism are sensitive to the offspring size (see inequalities (1–3)).

  2. 2.

    Our observation that the genotype-phenotype mapping modifies the condition of evolutionary stability of altruism, came true due to the heuristic basic idea of group selection, since the phenotypic composition of each family is determined by this mapping.

In summary, group living (by common action) has an effect on the evolutionary stability of altruism under cost sharing among siblings, and modifies the classical Hamilton’s rule.

The inclusive fitness effect and the group effect together modify the conditions of existence of altruism in diploid populations. Thus, it is safe to say, that the population genetics model can at the same time handle the heuristic basic ideas of kin theory (i.e. the indirect fitness effect) and group theory (the common action of the group members takes an effect on the survival probabilities), but it does not combine the methods and/or models of these theories.

Finally, we point out one of our main observations. Based on the standard replicator dynamics [49] in asexual populations, Van Veelen [14, 16] found that when the individuals’ fitness depends linearly on the costs and benefits, then Hamilton’s rule implies the evolutionary stability of altruism. The linear specification needs additivity and homogeneity in the number of altruistic siblings. In the case of cost sharing the homogeneity condition is not satisfied. This makes it likely that the homogeneity condition might be an important condition for Hamilton’s rule in other cases, too.