QBoard » Statistical modeling » Stats - Tech » Datasets for Running Statistical Analysis on

Datasets for Running Statistical Analysis on

  • What datasets exist out on the internet that I can run statistical analysis on?
      May 23, 2019 2:50 PM IST
    0
  • The datasets package is included with base R. Run this command to see a full list:

    library(help="datasets")

    Beyond that, there are many packages that can pull data, and many others that contain important data. Of these, you may want to start by looking at the HistData package, which "provides a collection of small data sets that are interesting and important in the history of statistics and data visualization".

    For financial data, the quantmod package provides a common interface for pulling time series data from google, yahoo, FRED, and others:

    library(quantmod)
    
    getSymbols("YHOO",src="google")# from google finance  
    getSymbols("GOOG",src="yahoo") # from yahoo finance  
    getSymbols("DEXUSJP",src="FRED") # FX rates from FRED 

    FRED (the Federal Reserve of St. Louis) is really a landmine of free economic data.

    Many R packages come bundled with data that is specific to their goal. So if you're interested in genetics, multilevel models, etc., the relevant packages will frequently have the canonical example for that analysis. Also, the book packages typically ship with the data needed to reproduce all the examples.

    Here are some examples of relevant packages:

    • alr3: includes data to accompany Applied Linear Regression (http://www.stat.umn.edu/alr)
    • arm: includes some of the data from Gelman's "Data Analysis Using Regression and Multilevel/Hierarchical Models" (the rest of the data and code is on the book's website)
    • BaM: includes data from "Bayesian Methods: A Social and Behavioral Sciences Approach"
    • BayesDA: includes data from Gelman's "Bayesian Data Analysis"
    • cat: includes data for analysis of categorical-variable datasets
    • cimis: from retrieving data from CIMIS, the California Irrigation Management Information System
    • cshapes: includes GIS data boundaries and data
    • ecdat: data sets for econometrics
    • ElemStatLearn: includes data from "The Elements of Statistical Learning, Data Mining, Inference, and Prediction"
    • emdbook: data from "Ecological Models and Data"
    • Fahrmeir: data from the book "Multivariate Statistical Modelling Based on Generalized Linear Models"
    • fEcoFin: "Economic and Financial Data Sets" for Rmetrics
    • fds: functional data sets
    • fma: data sets from "Forecasting: methods and applications"
    • gamair: data for "Generalized Additive Models: An Introduction with R"
    • geomapdata: data for topographic and Geologic Mapping
    • nutshell: contains all the data from the "R in a Nutshell" book
    • nytR: provides access to congressional vote data through the NY Times API
    • openintro: data from the book
    • primer: includes data for "A Primer of Ecology with R"
    • qtlbook: includes data for the R/qtl book
    • RGraphics: includes data from the "R Graphics" book
    • Read.isi: access to old World Fertility Survey data
    This post was edited by Pranav B at September 22, 2020 3:48 PM IST
      May 23, 2019 2:56 PM IST
    0
  • A broad selection on the Web. For instance, here's a massive directory of sports databases (all providing the data free of charge, at least that's my experience). In that directory is databaseBaseball.com, which contains among other things, complete datasets for every player who has ever played professional baseball since about 1915.

    StatLib is an other excellent resource--beautifully convenient. This single web page lists 4-5 line summaries of over a hundred databases, all of which are available in flat-file form just by clicking the 'Table' link at the beginning of each data set summary.

    The base distribution of R comes pre-packaged with a large and varied collection of datasts (122 in R 2.10). To get a list of them (as well as a one-line description):

    data(package="datasets")

    Likewise, most packages come with several data sets (sometimes a lot more). You can see those the same way:

    data(package="latticeExtra")
    data(package="vcd")

    These data sets are the ones mentioned in the package manuals and vignettes for a given package, and used to illustrate the package features.

    A few R packages with a lot of datasets (which again are easy to scan so you can choose what's interesting to you): AER, DAAG, and vcd.

    Another thing i find so impressive about R is its I/O. Suppose you want to get some very specific financial data via the yahoo finance API. Let's say closing open and closing price of S&P 500 for every month from 2001 to 2009, just do this:

    tick_data = read.csv(paste("http://ichart.finance.yahoo.com/table.csv?",
        "s=%5EGSPC&a=03&b=1&c=2001&d=03&e=1&f=2009&g=m&ignore=.csv")) 

    In this one line of code, R has fetched the tick data, shaped it to a dataframe and bound it to 'tick_data' all . (Here's a handy cheat sheet w/ the Yahoo Finance API symbols used to build the URLs as above)

      September 22, 2020 3:45 PM IST
    0
  • A good start to look for economic data are always the following three addresses:

    A nice summary of dataset links for development economists can be found at:

    The World Bank decided last week to open up a lot of its previously non-free datasets and published them online on its revised homepage. The new internet appearance looks pretty nice as well.

      September 22, 2020 3:49 PM IST
    0
  • http://www.data.gov/ probably has something you can use.

    In their catalog of raw data you can set your criteria for the data and find what you're looking for http://www.data.gov/catalog/raw

      September 22, 2020 3:50 PM IST
    0