At law
school café (reposted on Tax Prof) Deborah Merritt asks several
questions about The Economic Value of a
Law Degree related to sample size and uncertainty.    We thank Professor Merritt for her comments
and hope they helped clarify the annual results for those who were having
trouble interpreting Figures 5 and 6. In the paper we are careful to display the large confidence intervals for Figure 6, which looks at young law graduates over time, and we avoid drawing any strong conclusions from them.  Also, as we'll discuss below, one can readily reject that Figure 5's ups and downs are just noise. 

This post
includes brief discussions of some of the interesting points raised. 

 The estimates in
the paper don't depend on cyclical law school premia

We want to
be clear that our underlying results do not rely on cyclicality.  SIPP annual estimates do not show a recent
post-recession decline in the overall law graduate earnings premium that needs to be explained.   The recent decline in earnings for law
graduates in our sample is matched by a decline in earnings for bachelor’s
degree holders, and the law graduates retained their relative advantage.  But as one can see in Figure 6, the small
sample for young lawyers makes it hard to be sure about the recent outcomes for
that group in isolation.  Whether the
premium cycles up and down or stays flat, over a lifetime every law grad will
see many such transitions over their life, averaging out over time.

 

Is our overall
sample size big enough?

Yes, our
sample size is more than sufficient to support our conclusions on lifetime
earnings.  The standard errors in Tables
1 to 4 reflect the degree of uncertainty about our estimates, which pool data over
many years to increase precision. The standard errors are very small relative
to our law degree earnings premium coefficient estimates, and our results are
statistically significant well beyond conventional levels of statistical
significance.  Deborah Merritt's
discussion is focused specifically on what we can say about how the premium has
changed over time (Figures 5 and 6).  As
one can see in Figure 5, any changes in that premium have been fairly small
relative to its size.  

 

How strong is the
specific evidence from SIPP for cyclicality of earnings premiums?

Consistent
with cyclicality, there is evidence of fluctuations of the earnings premium
(measured on a percentage basis) in the 1996-2011 period.   Prompted by Deborah Merritt's concerns, we
went ahead and added the joint test statistics to the figures in question.  We can reject the hypothesis that the law
degree earnings premium was the same in all years from 1996-2011
(p<0.001).  In other words,
fluctuations in the point estimate in Figure 5 are not all simply random noise.  Further, we don’t see evidence of a notable
long term upward or downward trend. 
Indeed, despite the occasional fluctuations we think the most noticeable
feature of the law school premium recently is its stability.

Several
previous studies have found evidence of fluctuations in law degree holder
earnings premiums and starting salaries. 
We cite many of these studies in the paper.  It would be a bad idea to extrapolate gloom
or boom from a downward or upward trend in earnings using the last few years of
data.  Trends, even when present, can stop or reverse themselves through dynamic labor market responses or exogenous shocks.  A sustained 85 percent decline in the lifetime earnings premium would be required for our main result–that a law degree is a value-creating investment for most law graduates–to no longer hold true.  Such a steep decline seems unlikely.

Though not
crucial to our inqiury into lifetime earnings, it would be interesting
to know if the premium rises and falls with the business cycle.  Prompted by the interest in this question, we
did some exploratory analysis of data from the much larger, but less precise,
American Community Survey which also seems to be consistent with fairly stable
earnings premiums for recent cohorts of law graduates, but more research on the
question will be useful, especially as
passing time provides us more data.

 

How should we
understand confidence intervals and point estimates?

Professor
Merritt’s description of confidence intervals may seem to suggest that the true
population parameter is equally likely to fall close to the point estimate as
it is at the outer edges at the top or bottom of the confidence interval.  

This interpretation
would be incorrect.  The probability
density is highest at the center of the confidence interval, near the point
estimate, and lowest at the outer edges of the confidence interval.   The point estimate is the best estimate of
the population parameter.

Professor
Merritt’s description also doesn't discuss the relationship between different point estimates, looking instead
only at the confidence interval for each point estimate individually.  In a nutshell, two estimates may have
overlapping confidence intervals and still be statistically separable.

 

How strong is the
evidence for a bi-modal distribution of earnings? 

We don’t
think the evidence for a bimodal distribution of lifetime earnings for law
graduates is very compelling.  Recent
full time starting salaries from NALP are not the same thing as lifetime
earnings because:

• Full time salary excludes those who are working less than full time
• Salaries exclude bonuses, which may be more variable than earnings
• Starting salaries tend to be fairly lockstep compared to later
  earnings
• After the JD II suggests faster growth of earnings (on a percentage
  basis) for graduates of lower ranked schools who have lower average initial
  earnings, which suggests convergence of earnings over time

Because
earnings across people are close to log normally distributed it is typical to
see a few people making a lot more than most people. 

 

Would bimodality
cast doubt on the results of our analysis?

Bimodality
does not really call for change to our approach, even if present.  As the sample gets larger the sampling
distribution is asymptotically normal, so standard errors on our key results
should be consistent.  Regression
techniques are consistent regardless of the underlying distribution, but for
those concerned about a thick right tail, we'd suggest they concentrate on the
results in Tables 1 and 2 that use a log transformation—reducing such
concerns.  Bi-modality in the earnings
distribution would also not change how we did our quantile regressions.   Quantile regressions estimate the earnings
premium at different points in the distribution independent of the shape of the
overall distribution.