The Times today has a write-up of the recent Deborah Jones Merritt study of employment outcomes for the JD Class of '10 in Ohio.  As I described in more detail at the law & econ prof blog a few weeks ago, Prof. Merritt's study has, ahem, merits.  It's a great snapshot of struggling graduates in Ohio, people who deserve our attention and support.  The trouble is that the story reports these findings as thought they told us something about the national law job market.  Merritt's new data are all in Ohio, which may be a systematically different legal market than many others.  Nonetheless, the Times story reports Merritt's findings as though they were representative of the whole country (and also describes the study as "published," when in fact it's an ssrn working paper).  Most troublingly, the Times reports Prof. Merritt's conclusion that "the 2010 class had not recovered in the ensuing years" without any caveats.  

Yet there are several serious caveats that ought to have been offered.  For example, as I read the paper, Merritt 's claim depends entirely on a trend line she draws between 2010 national NALP data (which are based on self-reported survey but supplemented with some follow-up) and 2014 Ohio data (which Merritt hand-collected on the web).   That is not likely to be a persuasive method of measuring employment trends for anyone, whether in Ohio or anywhere else.  It's like comparing 2014 scoring in the NBA against 2010 scoring in college basketball.  Unless you can show a really convincing case for why these groups are actually very similar to each other, the trend line is just random noise.  

Deborah and I had an exchange about these issues on my blog.  She convinced me that the method NALP used for supplementing some of the 2010 data was similar to her method (although that leaves the question whether it makes any sense to compare her results to the bulk of the NALP numbers, which used a quite different method).  But she did not address the issue that the 2010 NALP data were for the whole country, not Ohio, and there is no reason to think that Ohio was similar to the U.S. in 2010 or that its trend since has been similar.  I left our exchange believing she would return to her project and revise it to reflect its serious limitations as a window into national trends.  If that has happened, it is not reflected in the Times story.