What to Look For in an Economic Impact Study

PDF Format


When handed a tobacco-related study - particularly if it claims that going smokefree hurts business - consider these factors to determine if it is a legitimate study or one funded and created by the tobacco industry or its allies.

What evidence is used in the study?

Is the information anecdotal, or is it based on hard data? Sales tax receipts and employment statistics collected by an objective source, such as a state Board of Equalization or an equivalent state agency, are the only data that measure restaurant sales in an unbiased, accurate fashion. Anecdotal information, such as surveys measure restaurant owners' predictions and perceptions; they generally do not provide data to back up those impressions. Although restaurant owners may perceive a downturn in sales, only sales tax receipts tell the real story.

Are sales figures analyzed for at least one year?

To identify underlying trends and fluctuations in the restaurant business cycle's yearly sales, was the study conducted over a sufficiently lengthy period to make the data meaningful? Economic impact studies should include sales data (by quarter) for several years before a law is enacted, as well as for all quarters after enactment. The economy changes for many reasons: inflation, seasonal variation, and fluctuations due to the weather and other superfluous events. Short-term observations cannot accurately evaluate economic impact. By collecting data for several years, it is possible to identify and quantify these trends and take them into account. An observed decrease in sales data for one or two quarters may simply be a typical downward trend in restaurant sales that occurs every year. Seasonal variation is also common. For example, business may be booming in a ski resort town over the winter months each year, while the rest of the year may be slow. So be sure you are looking at data from the same season (or quarter) when comparing year to year.

Also be on the lookout for delays in filing sales tax and employment statements with the Board of Equalization, as delays artificially decrease the sales figures for the following quarter.

What comparisons are appropriate?

Since figures in a vacuum tell little, are the most useful comparisons being made? Compare bar or restaurant numbers with total retail sales in the area: What's going on in the general economy of the community?

Compare bar and restaurant sales to comparable communities' restaurant sales: What's going on in the hospitality sector in the area?

Where was the study published?

Is the study in a "peer reviewed" journal? "Peer review" is the process by which academic studies are evaluated by independent evaluators. One to three anonymous reviewers, who then provide comments and critiques to the editor, typically review the paper. Peer review substantially increases the chance that a study is not based on a biased sample and inappropriate methods.

Who really conducted the study?

Did a tobacco industry front group conduct or commission the economic impact study? Check the American Nonsmokers' Rights Foundation's Tobacco Industry Tracking Database© at http://www,no-smoke.org/, or call the ANR Foundation at (510) 841-3032 for assistance.