Statistics, Politics, and Policy

Comment on “Why and When 'Flawed' Social Network Analyses Still Yield Valid Tests of no Contagion”

Cosma Rohilla Shalizi — Mon, 13 Feb 2012 07:37:56 PST

VanderWeele et al.'s paper is a useful contribution to the on-going scientific conversation about the detection of contagion from purely observational data. It is especially helpful as a corrective to some of the more extreme statements of Lyons (2011). Unfortunately, this paper, too, goes too far in some places, and so needs some correction itself.

Why and When "Flawed" Social Network Analyses Still Yield Valid Tests of no Contagion

Tyler J. VanderWeele et al. — Sat, 04 Feb 2012 14:43:11 PST

Lyons (2011) offered several critiques of the social network analyses of Christakis and Fowler, including issues of confounding, model inconsistency, and statistical dependence in networks. Here we show that in some settings, social network analyses of the type employed by Christakis and Fowler will still yield valid tests of the null of no social contagion, even though estimates and confidence intervals may not be valid. In particular, we show that if the alter's state is lagged by an additional period, then under the null of no contagion, the problems of model inconsistency and statistical dependence effectively disappear which allow for testing for contagion. Our results clarify the setting in which even "flawed" social network analyses are still useful for assessing social contagion and social influence.

Data, Statistics, and Controversy: Making Science Research Data Intelligible

Nell Sedransk et al. — Wed, 18 Jan 2012 11:37:25 PST

Making published, scientific research data publicly available can benefit scientists and policy makers only if there is sufficient information for these data to be intelligible. Thus the necessary meta-data go beyond the scientific, technological detail and extend to the statistical approach and methodologies applied to these data. The statistical principles that give integrity to researchers’ analyses and interpretations of their data require documentation. This is true when the intent is to verify or validate the published research findings; it is equally true when the intent is to utilize the scientific data in conjunction with other data or new experimental data to explore complex questions; and it is profoundly important when the scientific results and interpretations are taken outside the world of science to establish a basis for policy, for legal precedent or for decision-making. When research draws on already public data bases, e.g., a large federal statistical data base or a large scientific data base, selection of data for analysis, whether by selection (subsampling) or by aggregating, is specific to that research so that this (statistical) methodology is a crucial part of the meta-data. Examples illustrate the role of statistical meta-data in the use and reuse of these public datasets and the impact on public policy and precedent.

Major Contributions to Quantitative Economics Sponsored by the Defense Community

Henry Solomon — Fri, 13 Jan 2012 10:35:52 PST

Quantitative economics has advanced dramatically in the past eighty years and much of this took place in the three decades following World War II. Many contributions were initiated and sponsored by the defense community. For the resolution of major military management and planning problems it was essential to create new quantitative economic models and methodologies. Accomplishments served the military while providing substantive and lasting new directions in economics. This paper is devoted to highlighting these results by describing the research on Input-Output economics, linear programming, game theory, decision theory, probabilistic models, and high-speed computing.

Improving Statistical Inference with Clustered Data

Jeffrey J. Harden — Wed, 04 Jan 2012 17:51:42 PST

Political science data often contain grouped observations, which produces unobserved "cluster effects" in statistical models. Typical solutions include (1) ignoring the impact on coefficients and only adjusting the standard errors of generalized linear models (GLM) or (2) addressing clustering in coefficient estimation while relying on a parametric assumption for the cluster effects and/or a large number of clusters for standard errors. I show that both approaches are problematic for inference. Through simulation I demonstrate that multilevel modeling (MLM) and generalized estimating equations (GEE) produce more efficient coefficients than does GLM. Next, I show that commonly-used MLM and GEE standard error methods can be biased downward, while bootstrapping by resampling clusters (BCSE) performs better, even with a misspecified error distribution and/or few clusters. I recommend the use of MLM or GEE to estimate coefficients and BCSE to estimate uncertainty, and show that this approach can produce divergent conclusions in applied research.

Discussion of Alemayehu and Levenstein

Susan S. Ellenberg — Mon, 07 Nov 2011 11:44:31 PST

Alemayehu and Levenstein offer an industry perspective on ghoswriting. They propose some general principles to prevent the types of egregious cases that have been reported, but more specifics are needed to understand how to combat this serious problem.

Comment on Alemayehu and Levenstein “Toward a Pragmatic Policy on Authorship”

Peter A. Lachenbruch — Wed, 26 Oct 2011 13:20:10 PDT

This note comments on the article by Alemayehu and Levenstein. It expands a bit on the work and suggests other directions the work could go in.

Sharper p-Values for Stratified Election Audits

Michael J. Higgins et al. — Wed, 26 Oct 2011 10:23:10 PDT

Vote-tabulation audits can be used to collect evidence that the set of winners of an election (the outcome) according to the machine count is correct — that it agrees with the outcome that a full hand count of the audit trail would show. The strength of evidence is measured by the p-value of the hypothesis that the machine outcome is wrong. Smaller p-values are stronger evidence that the outcome is correct.

Most states that have election audits of any kind require audit samples stratified by county for contests that cross county lines. Previous work on p-values for stratified samples based on the largest weighted overstatement of the margin used upper bounds that can be quite weak. Sharper p-values can be found by solving a 0-1 knapsack problem. For example, the 2006 U.S. Senate race in Minnesota was audited using a stratified sample of 2–8 precincts from each of 87 counties, 202 precincts in all. Earlier work (Stark 2008b) found that the p-value was no larger than 0.042. We show that it is no larger than 0.016: much stronger evidence that the machine outcome was correct.

We also give algorithms for choosing how many batches to draw from each stratum to reduce the counting burden. In the 2006 Minnesota race, a stratified sample about half as large — 109 precincts versus 202 — would have given just as small a p-value if the observed maximum overstatement were the same. This would require drawing 11 precincts instead of 8 from the largest county, and 1 instead of 2 from the smallest counties. We give analogous results for the 2008 U.S. House of Representatives contests in California.

Toward a Pragmatic Policy on Authorship

Demissie Alemayehu et al. — Tue, 25 Oct 2011 17:09:44 PDT

There has been growing concern about ghostwriting practices in peer-reviewed biomedical journals. As a result, various proposals have been put forth to abolish or prevent the practice of ghostwriting. In this article, we review the issues underlying ghostwriting in medical research; highlight the shared responsibilities of the pharmaceutical industry and other trial sponsors, medical centers and journal editors; and propose a roadmap for an effective policy on the ethical, transparent and reliable communication of results of clinical research.

What Can We Predict About Libya and the Arab Spring from Statistical Studies?

Alan M. Green — Fri, 24 Jun 2011 11:21:46 PDT

While research using cross country data and analysis is rife with statistical problems, the best papers do provide valuable insight. This commentary illustrates how an excellent paper on democratization provides a useful “stability threshold” that helps to understand current events in the Middle East, especially the Western intervention in Libya.

Reproducible Research: A Range of Response

David Banks — Fri, 20 May 2011 09:24:40 PDT

This editorial explores statistical methods to improve reproducibility and provide a more reliable foundation for trust in scientific research.

Using a Density-Variation/Compactness Measure to Evaluate Redistricting Plans for Partisan Bias and Electoral Responsiveness

Thomas R. Belin et al. — Wed, 18 May 2011 18:02:06 PDT

The clear association between population density and partisan preference in elections suggests that redistricting plans would be better aligned with principles of partisan fairness if there were a deliberate effort to balance population density across legislative districts. To balance population density without sacrificing geometric compactness, we define a density-variation/compactness (DVC) measure that can serve as a one-number summary of a proposed redistricting plan. After analyzing voter registration data from California to guide the choice of a specific DVC measure, we evaluate its performance in both actual and hypothetical redistricting plans using election data from Texas during the past decade. Using a well-established political-science model of the relationship between legislative representation and the proportion of votes received, higher DVC scores corresponded to estimates of partisan bias with smaller magnitude across a range of redistricting scenarios; meanwhile, contrary to expectations that reduced partisan bias would be accompanied by reduced electoral responsiveness, there was no discernible pattern between DVC scores and estimates of electoral responsiveness. Although there are apt to be multiple considerations in choosing a redistricting plan, we discuss how the use of DVC measures could provide a check on attempts to introduce partisan bias into the redistricting process.

The Spread of Evidence-Poor Medicine via Flawed Social-Network Analysis

Russell Lyons — Wed, 18 May 2011 18:02:03 PDT

The chronic widespread misuse of statistics is usually inadvertent, not intentional. We find cautionary examples in a series of recent papers by Christakis and Fowler that advance statistical arguments for the transmission via social networks of various personal characteristics, including obesity, smoking cessation, happiness, and loneliness. Those papers also assert that such influence extends to three degrees of separation in social networks. We shall show that these conclusions do not follow from Christakis and Fowler's statistical analyses. In fact, their studies even provide some evidence against the existence of such transmission. The errors that we expose arose, in part, because the assumptions behind the statistical procedures used were insufficiently examined, not only by the authors, but also by the reviewers. Our examples are instructive because the practitioners are highly reputed, their results have received enormous popular attention, and the journals that published their studies are among the most respected in the world. An educational bonus emerges from the difficulty we report in getting our critique published. We discuss the relevance of this episode to understanding statistical literacy and the role of scientific review, as well as to reforming statistics education.

Assessing the Early Aberration Reporting System’s Ability to Locally Detect the 2009 Influenza Pandemic

Katie S. Hagen et al. — Wed, 18 May 2011 18:01:56 PDT

The Early Aberration Reporting System (EARS) is used by some local health departments (LHDs) to monitor emergency room and clinic data for disease outbreaks. Using actual chief complaint data from local public health clinics, we evaluate how EARS—both the baseline system distributed by the CDC and two variants implemented by one LHD—perform at locally detecting the 2009 influenza A H1N1 pandemic. We also compare the EARS methods to a CUSUM-based method. We find that the baseline EARS system performed poorly in comparison to one of the LHD variants and the CUSUM-based method. These results suggest that changes in how syndromes are defined can substantially improve EARS performance. The results also show that incorporating algorithms that use more historical data will improve EARS performance for routine surveillance by local health departments.

A Snapshot of the 2008 Election

Andrew Gelman et al. — Mon, 16 Aug 2010 19:27:02 PDT

We present maps of the 2008 presidential voting bases on ethnicity, income, and state, and discuss the challenges involved in statistical modeling and the graphical presentation of the results

Measuring Elusive Populations with Bayesian Model Averaging for Multiple Systems Estimation: A Case Study on Lethal Violations in Casanare, 1998-2007

Kristian Lum et al. — Wed, 21 Jul 2010 07:32:32 PDT

Collecting data for the analysis of past human rights violations is fraught with challenges. For example, individuals from or about whom data should be collected may be displaced, missing, or dead. Some reports of acts may be easier to find than others, and as a result, datasets will be biased toward those cases. These challenges must be overcome in order to create effective official policy for violence mitigation and prevention. This relies on statistical analyses that can meet these challenges.

We propose a combination of Bayesian Model Averaging and Multiple Systems Estimation as an example of this type of analysis, which presents an important advancement in human rights research. In particular, this method allows the use of multiple data sources to estimate the number of undocumented violations – those that have not been recorded by any source.

We present an application of this method in Casanare, Colombia, where we estimate a total (both documented and undocumented) of 5,832 killings (95% credible interval: 3,822, 9,332) and 2,345 disappearances (1,221, 4,901) between 1998 and 2007.

Atmospheric Circulations Do Not Explain the Temperature-Industrialization Correlation

Ross McKitrick — Wed, 07 Jul 2010 02:07:11 PDT

Gridded land surface temperature data products are used in climatology on the assumption that contaminating effects from urbanization, land-use change and related socioeconomic processes have been identified and filtered out, leaving behind a “pure” record of climatic change. But several studies have shown a correlation between the spatial pattern of warming trends in climatic data products and the spatial pattern of industrialization, indicating that local non-climatic effects may still be present. This, in turn, could bias measurements of the amount of global warming and its attribution to greenhouse gases. The 2007 report of the Intergovernmental Panel on Climate Change (IPCC) set aside those concerns with the claim that the temperature-industrialization correlation becomes statistically insignificant if certain atmospheric circulation patterns, also called oscillations, are taken into account. But this claim has never been tested and the IPCC provided no evidence for its assertion. I estimate two spatial models that simultaneously control for the major atmospheric oscillations and the distribution of socioeconomic activity. The correlations between warming patterns and patterns of socioeconomic development remain large and significant in the presence of controls for atmospheric oscillations, contradicting the IPCC claim. Tests for outlier influence, spatial autocorrelation, endogeneity bias, residual nonlinearity and other problems are discussed.