Random Variables and Expectations

Review

• Correlation
• Covariance

$${\rho }_{xy}=\frac{{\rho }_x}{{\rho }_x}$$

• Correlation and causation
  • Per capita cheese consumption is highly correlated with “Number of people who died by become tangled by their bedsheets”
  • Popularity of the first name Annabelle with “UFO sightings in Mayland”

Correlation doesn’t imply causation

• Anscombe’s quartet
  • Anscombe created 4 data sets with very similar descriptive statistics (linear correlation coefficients)
  • $\rho$ degree of linear association; $r_s$ degree of monotone association (rank), Chatterjee.
    • The Pearson Correlation Coefficient has limits
  • The data sets appeared to be very different, which we could only talk about because we visualized the data

It illustrates the fact that even though we have the same correlation coefficient the reality behind that number may be really different

• Capital $X$ is a random variable, $x_1$ are realizations or actual numerical values for $X$.

Transclude of Random-Variables-and-Expectations-2024-12-12-05.12.39.excalidraw

The expectation values of the mean $\overline{X}$ is the consistent.

$$E(\overline{X})=E\left[\frac{1}{n}\left(\sum _{i=1}^n{X}_i\right)\right]$$

The same lies for the variance

$$\begin{array}{rl}{\sigma }_{\overline{X}}^2& =Var\left[\frac{1}{n}\left(\sum _{i=1}^n{X}_i\right)\right]\\ & =\sum _{i=1}^n{\left(\frac{1}{n}\right)}^{2}Var(X_i)\\ & =\frac{{\sigma }_X^2}{n}\end{array}$$

${\sigma }_{\overline{X}}^2$

The higher the sample size, the lesser the variance of $\overline{X}$, meaning that the value of the estimator will be more close to the real value of the population mean (the value that it’s trying to estimate)

$${\sigma }_{\bar{X}}^2=\frac{{\sigma }^2}{n}⟹{\sigma }_{\bar{x}}=\frac{\sigma }{\sqrt{n}}$$

As the sample size becomes bigger and bigger, theoretically to such an extent that the sample size is equal to the size of the population, then the variance becomes zero, and thus our estimated $\bar{X}$ is actually the population mean.

• When the sample size increases ($n=5,10,20,50,100$) the distribution of the mean gets concentrated towards the population mean.

Keywords:

• Sampling
• Estimation
• Estimator
• Estimate

What are the desirable properties of these Estimators?

Properties

Consistency

Sufficient condition:

• The estimator is unbiased and its variance tends to $0$ when $n$ becomes large.
• A biased estimator may be consistent (thus, not necessary), if the bias disappears when the size of the sample increases¹

The two conditions can also be stated as

1. $E_{\theta }(T_n)\to \gamma (\theta ),n\to \infty$
2. $Va{r}_{\theta }(T_n)\to 0,n\to \infty$ Then, $T_n$ is a consistent estimator of $\gamma (\theta )$.

Plim definition:

$$\underset{n\to \infty }{\lim }P[|Z_n-\alpha |\ge ϵ]\to 0⟹\text{plim}{Z}_n=\alpha$$

Plim properties

$$\text{plim}(X+Y+Z)=\text{plim}(X)+\text{plim}(Y)+\text{plim}(Z)$$

and 5 more…

If we have several unbiased estimators, how do we choose among them? We simply select the one with the smaller variance (higher precision).

If we have an unbiased estimator with a larger variance, but a biased estimator with a smaller variance, how to we choose?

We can turn to a loss function which is the

Transclude of Random-Variables-and-Expectations-2024-12-12-05.37.40.excalidraw

$$\begin{array}{rl}MSE(Z)& =E[(Z-\theta {)}^2]\\ & =E[(Z-{\mu }_Z+{\mu }_Z-\theta {)}^2]\\ & =\cdots \text{solve using}(a+b{)}^2=a^2+2ab+b^2\text{(most remarkable identity)}\\ & ={\sigma }_Z^2+({\mu }_z-\theta {)}^2\end{array}$$

The MSE is a synthetic measure of the bias and the variance. Looking at it we can see whether the bias-variance tradeoff is worth taking if it leads to a smaller MSE. But again, “best” is subjective…

• Try looking at variance as precision.

Footnotes

1. There are certain theories that are only applicable for large samples ↩