Survey reliability

Surveys are constructed from carefully chosen questions to gather information from some group of people. Quite often, it’s not enough to just ask a single question. Surveys usually consist of multiple related questions that tries to measure a complex human behavior or characteristic, known as constructs. When the responses to these individual questions are taken together, they can be combined to form a score or scale. A survey is reliable if it gives similar results when taken again by a similar group of people.

Cronbach’s Alpha is a way to measure internal consistency reliability of the individual questions (or test items):

$$\alpha =\frac{N\overline{c}}{\overline{v}+(N-1)\overline{c}}$$

, where $\overline{c}$ is the average of the inter-item covariance, and $\overline{v}$ is the average variance.

Let’s consider this sample data, where X1, X2, X3, and X4 are questions that take on integer values from 1 to 5 (Likert). So, all these questions form a scale. There are 10 rows, so 10 people filled out this survey.

$$\begin{array}{|ccccc|}\hline \text{id}& X1& X2& X3& X4\\ 1& 3& 3& 3& 3\\ 2& 3& 3& 4& 3\\ 3& 3& 3& 3& 1\\ 4& 5& 4& 4& 5\\ 5& 2& 1& 4& 4\\ 6& 4& 4& 4& 5\\ 7& 3& 2& 3& 3\\ 8& 3& 4& 3& 3\\ 9& 4& 2& 2& 3\\ 10& 3& 3& 2& 3\\ \hline\end{array}$$

Let’s find $\overline{c}$. The covariance matrix is:

$$\begin{array}{ccccc}& \text{X1}& \text{X2}& \text{X3}& \text{X4}\\ \text{X1}& 0.6778& 0.478& 0.0444& 0.456\\ \text{X2}& 0.4778& 0.989& 0.1333& 0.256\\ \text{X3}& 0.0444& 0.133& 0.6222& 0.489\\ \text{X4}& 0.4556& 0.256& 0.4889& 1.344\end{array}$$

So, $\overline{c}=(0.478+0.0444+0.1333+0.456+0.256+0.489)/6=0.309$, and $\overline{v}=(0.6778+0.989+0.6222+1.344)/4=0.908$.

Then,

$$\alpha =\frac{4\ast 0.309}{0.908+3\ast 0.309}=0.674$$

A quick check in R gives the same result:

Cronbach's alpha for the 'df' data-set

Items: 4
Sample units: 10
alpha: 0.674

How do you use this value? According to George and Mallery (2003), an $\alpha$ >= 0.9 is excellent, >= 0.8 is good, >= 0.7 is acceptable, >= 0.6 is questionable, >= 0.5 is poor, and < 0.5 is unacceptable.