Business Analytics: Datasets

Providing Tools for Business Analytics Research

Datasets of Note

Global Financial Database
Financial and economic data covering over 150 countries and over 6,500 historical and current data series. Bonds, T-bills, unemployment, GDP, commodity prices, exchange rates, and more, some dating back to 1900. Note: To access, use the “Login Anonymously” option and choose GFDatabase in the left nav bar.
Wharton Research Data Services (WRDS)
WRDS provides access to selected datasets in the fields of finance, accounting, banking, economics, management, marketing and public policy. ACCESS: UC San Diego faculty, staff and students first must sign up for a WRDS account by using the Register link on the landing page. LIMITATIONS: Masters & Undergraduate students do not have access during summer or quarter breaks. UC San Diego affiliates without AD network accounts are ineligible for WRDS accounts (most Extension students are in this category). In those cases, Mergent Online and Global Financial Database are your recommended alternatives.
Currently includes (as of 9/14/2021):
Bank Regulatory Database
Blockholders Database
CBOE (Chicago Board Options Exchange)
S&P Compustat Bank
Compustat Bank - daily updates
S&P Compustat Global
Compustat Global - daily updates
Compustat Legacy - Global
S&P Compustat North America - annual update (current + historical data)
Compustat North America - daily updates (current + historical data)
Compustat Legacy - North America
S&P Compustat North America - monthly update (current + historical data)
S&P Compustat Historical Segments
Compustat Historical Segments - daily updates
Contributed Data - Characteristic Returns
Contributed Data
Contributed Database: Intangible Value
KPSS Technological Innovation
Contributed Database: LIVA
CRSP/Compustat Merged (Annual)
CRSP Indexes (Annual)
CRSP Stock (Annual)
CRSP Treasuries (Annual)
CRSP Ziman Real Estates (Annual)
CRSP Index History Intraday (Quarterly)
Dow Jones Averages & Total Return Indexes
Direct Marketing Educational Foundation
SEC Disclosure of Order Execution Statistics
Fama-French portfolios and factors
Federal Judicial Center Linking
Federal Judicial Center Litigation
Federal Reserve Board Foreign Exchange Rates, Interest Rates, and State Composite Indexes
Macro Finance Society
MSRB - Municipal Securities Transaction Data
OTC Markets End-of-Day Pricing
Philadelphia Stock Exchange Currency Options and Implied Volatility
Publicly Available Data
The Penn World Tables Database
Research Quotient (provided to all academics with a Compustat subscription)
NYSE Trades & Quotes 1993
NYSE Trades & Quotes 1994
NYSE Trades & Quotes 1995
NYSE Trades & Quotes 1996
NYSE Trades & Quotes 2014
NYSE Trades & Quotes - Exists for FK contraint
NYSE Trades and Quotes - for website use
NYSE Millisecond Trades & Quotes 2014
NYSE Millisecond Trades & Quotes 2015
NYSE Millisecond Trades & Quotes 2016
NYSE Millisecond Trades & Quotes 2017
NYSE Short Sales 2005
NYSE Short Sales 2006
NYSE Short Sales 2007
Total q by Luke Taylor
FINRA TRACE Enhanced
FINRA TRACE corporate bond trades
WRDS Research Applications
WRDS Backtest Basic
WRDS Beta Suite
WRDS Efficient Frontier
WRDS EU Short Sales
Event Study Intl using Compustat
Event Study Intraday
Event Study Long Run
Event Study US
WRDS Financial Ratios (base)
WRDS Financial Ratios with CCM
Linking Suite: Bond - CRSP
Linking Suite: TAQ Monthly - CRSP
Linking Suite: TAQ Daily - CRSP
Linking Suite: Supply Chain
WRDS Patent Database
WRDS SAS Visual Analytics
WRDS Company Subsidiaries
WRDS World Indices
MIDAS

Ways to Use WRDS
Overviews of using the web interface, WRDS Cloud and connecting remotely through SSH and programming apps.
Programming @ WRDS
Find out how to access WRDS through SAS, Python, R, MATLAB, and Stata. This page includes details about the connection methods.

Datastream
Access Datastream content online via Refinitiv Workspace. Additionally, you may register for the Datastream add-on to do graphical exploration of trends and relationships between series. Datastream provides time series for over a million financial instruments, securities, and economic indicators from over 175 countries. Workspace content also includes company profiles, ESG performance, indices, bonds, commodities, interest and exchange rates, estimates, futures and options, country macroeconomics, market research and investment house reports. New users complete self-registration here - https://my.refinitiv.com/productregistration.html
ICPSR Data Archive
The Inter-University Consortium for Political and Social Research. World's largest archive of social science data. Election data, opinion polls, more. Special collections on Health, Education, Aging, Crime, Substance Abuse. Raw data; no tables. To download data, create a personal account using your ucsd.edu email address while accessing the database from a UC San Diego computer, UCSD-PROTECTED wireless, or the VPN.
Sage Data (formerly Data-Planet)
Provides easy access to statistical data. Data is downloadable in Excel and CSV, and shapefiles when multiple geographies are covered. Sources include: Bombay Stock Exchange, British Bankers' Association, Chicago Board Options, Exchange, China Data Center, D&B (Dun & Bradstreet), Dave Leip's Atlas of US Presidential Elections, Defense Manpower Data Center, Deutsche Börse Group, Dow Jones, Easy Analytic Software Inc. (EASI), Eurostat, FTSE Group, HSI Services Limited (Hong Kong), International Monetary Fund, London Bullion Market Association , London Platinum & Palladium Market, NASDAQ OMX Group, National Bureau of Economic Research, Nikkei, Organisation for Economic Co-operation and Development, Shanghai Stock Exchange, Standard & Poor's, World Bank, World Resource Institute, Xignite, and many US Federal agencies. See the Data-Planet Libguides for more detail. If you are having trouble connecting, check the status page to see if Sage Data is down.

Nielsen Marketing Data Sets
UC San Diego subscribes to Nielsen marketing data from the Chicago Booth School of Business. The James M. Kilts Center for Marketing at Chicago Booth and the Nielsen Company have partnered to make four datasets available to US-based academic researchers. There are special restrictions on data access and use. Read the instructions before requesting access.
Consumer Complaint Database
From the Consumer Financial Protection Bureau
GovInfo: Bulk Data
Congressional Bills; Bill Status; Bill Summaries; Commerce Business Daily; Code of Federal Regulations (Annual Edition); Electronic Code of Federal Regulations; Federal Register; United States Government Manual; House Rules and Manual; Privacy Act Issuances; Public Papers of the Presidents of the United States; Supreme Court Decisions 1937-1975 (FLITE)
The GDELT Project
Monitors the world's broadcast, print, and web news from nearly every corner of every country in over 100 languages and identifies the people, locations, organizations, counts, themes, sources, emotions, counts, quotes, images and events driving our global society every second of every day, creating a free open platform for computing on the entire world.

Data visualizations for events from 2017-present and up to 250 articles at a time can be accessed by changing the structure of the API url (manually or through web forms) to reflect the search query.

Raw data can be analyzed using SQL queries on Google BigQuery. Researchers with the capacity to work with 2.5TB for each year of data can download the raw data directly from GDELT.

Most documentation is shared on various dates throughout the GDELT blog.
more...less...
The GDELT 1.0 Event Database contains over a quarter-billion records organized into a set of tab-delimited files by date. Through March 31, 2013 records are stored in monthly and yearly files by the date the event took place. Beginning with April 1, 2013, files are created daily and records are stored by the date the event was found in the world's news media rather than the date it occurred (97%+ of events are reported within 24 hours of happening, but a small number of events each day are past events being mentioned for the first time - if an event has been seen before it will not be included again). Files are ZIP compressed in tab delimited format, but named with a ".CSV" extension.

Ken Blake's "A short intro to GDELT" tutorial

GDELT Summary + GDELT APIs
In addition to its datasets, GDELT also offers a number of live realtime JSON APIs offering fulltext search and other capabilities, including DOC, GEO and TV. Explore them using the GDELT Summary web form that offers a non-technical, human-friendly website wrapper around the APIs, showcasing their capabilities.

UK Foreign & Commonwealth Office, Open Source Unit has also created a GDELT query interface for working with the API that ofters searching by theme as well by keyword.

Themes taxonomy

GDELT datasets and specialized datasets (some for download, some only available on Google BigQuery)

Google BigQuery documentation
PubMed Central Open Access subset
Yahoo Webscope Datasets
A reference library of interesting and scientifically useful datasets for non-commercial use by academics and other scientists.

Research Methods & Tools

Sage Research Methods Includes books, reference works and journal articles from across the social sciences research methods, including a large collection of qualitative methods books. The resources cover every aspect of research skills training and can help researchers at all levels through every step of the research process, from coming up with a research question, doing a literature review, planning a project, collecting and analyzing data, and writing up a report, dissertation or thesis. Also included are SRM Cases (stories of how real research projects were conducted) and SRM Datasets (a collection of teaching datasets and instructional guides that give students a chance to learn data analysis by practicing themselves).

Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts (Justin Grimmer and Brandon M. Stewart )
Neil Caren - Learning Python for Social Scientists
Includes examples from all kinds of data sources
Ryan Shaw - Text-mining as a Research Tool in the Humanities and Social Sciences

Text Mining and Visualization by Markus Hofmann (Editor); Andrew Chisholm (Editor) This book introduces text mining by using popular open-source tools: KNIME, RapidMiner, Weka, R, and Python. The contributors, who are experienced with text mining and open-source software, explain how text data are gathered and processed from a wide variety of sources, including books, server access logs, websites, social media sites, and message boards. Each chapter presents a case study that you can follow as part of a step-by-step, reproducible example. You can also easily apply and extend the techniques to other problems.
Call Number: O'Reilly e-book
ISBN: 9781482237573
Publication Date: 2016
Mapping texts: computational text analysis for the social sciences by Dustin S. Stoltz & Marshall A. Taylor
Call Number: Oxford Univ. Press e-book
ISBN: 9780197756904
Publication Date: 2024
The authors offer a practical introduction to computational text analysis with step-by-step guidance on how to conduct text analysis workflows in R. When mapping texts, the goal is not necessarily to find these meaningful needles in the haystack, but instead to create reductions of the text to document patterns. The focus is on social science questions and applications, and the book covers understanding the theories of language embedded in text analysis, all the way to more advanced and cutting-edge techniques.

APIs & buik downloads from scholarly sources

Alpha Vantage Open Stock APIs
Free programmatic access to stocks, ETFs, mutual funds, foreign exchange rates, financial reports from SEC filings, and over 50 derived technical indicators. Free API key allows up to 500 API calls per day by default. More information: (1) introduction to stock APIs, (2) official API documentation
arXiv API
API access to Cornell's open-access arXiv e-print repository, used primarily by physics, mathematics and computer science communities to share cutting-edge research.
Crossref REST API - Text and data mining for researchers
Crossref is a not-for-profit membership organization for scholarly publishing who works to make content easy to find, cite, link, and assess. The Crossref API allows researchers to easily harvest full-text documents from all participating members, regardless of whether the content is open access or subscription. Explanatory slides
Common Crawl
Open repository of web crawl data that can be accessed and analyzed by anyone.
OECD data API
The OECD dataset APIs let you programmatically access a selection of our top used datasets covering data for OECD countries and selected non-member economies. OECD datasets are dynamically updated.
UN Comtrade Web Services
Access data from the United Nations Commodity Trade Statistics database, including International Merchandise Trade Statistics (IMTS) and the work of the International Merchandise Trade Statistics Section (IMTSS) of the United Nations Statistics Division. Sign in required to access developer tools including API.
Worldbank APIs
Three APIs to provide access to different datasets: one for Indicators (or time series data), one for Projects (or data on the World Bank’s operations), and one for the World Bank financial data (World Bank Finances API).
Public Datasets
A github 'awesome' list of curated public datasets on various topics and areas including corpora and social networking data.
Finding Data & Statistics:APIs
This page covers an extensive list of API sources from the UCSD Library's full list of data sources.
Data USA
Data visualization using public US government data. More about the API here..

GIS and Mapping Data/Software

SimplyAnalytics
SimplyAnalytics is a web-based mapping application that lets users quickly create professional-quality thematic maps and reports using demographic, business, and marketing data for the United States. Access limited to 5 simultaneous users. Try again later if refused. Includes EASI/MediaMark Research Incorporated (MRI) and Simmons LOCAL data packages. (formerly known as SimplyMap).

ArcGIS
The predominate software in GIS is ArcGIS by Esri. It can be expensive and has a learning curve for users, but is extremely robust and as noted, is predominate in the GIS industry. Can be used in a desktop, server or web environment.
ArcGIS Business Analyst
ArcGIS Business Analyst provides location-based intelligence for planning, site selection, and customer segmentation. Combine demographic, lifestyle, and spending data with map-based analytics for accurate reports and dynamic presentations. It is available as a web browser SaaS solution or as a desktop-based tool. First you should register for an ArcGIS account here.
ESRI Map Book Gallery
Browse these annual volumes to get ideas. They include a 'best in breed' compilation from users, and cover topics from business and economics as well as government, health, naturual resources, transportation, utilities and more.

Refinitiv's ESG Data

The Refinitiv ESG data is hosted on Refinitiv Workspace. Once you register, go to the Apps Library and type ESG to see what is available to use.

LSEG Refinitiv Workspace
Although Workspace primarily provides financial and economic data, news, equity reports, and company & industry analysis, it also features unique content about companies’ ESG performance, sustainable leadership monitoring (SLM), and peer analysis. First-time users register here, https://my.refinitiv.com/productregistration.html (ESG data is in the basic license, called “Workspace for Students”).
Workspace is only available to UC San Diego students, faculty, and staff; only for non-commercial, academic research. It replaces ThomsonONE/Investext and Datastream, and now works across several browsers.

Register only once; otherwise login to Workspace using the main link. You may use web or desktop app version (includes the Workspace add-in for Excel), once you are registered.

Numeric & Text Mining

ProQuest TDM Studio
More than 200 licensed ProQuest content products, including government, archival, dissertation, and news databases, are available for analysis with R or Python through ProQuest TDM Studio. Contact Stephanie Labou (slabou@ucsd.edu) for additional assistance. Includes the ability to text mine the Wall Street Journal.
To create an account:
1. Go to https://tdmstudio.proquest.com
2. Click “Create an account” button
3. Use your UCSD email address to create your account.
Gale Digital Scholar Lab
Explore UCSD holdings from Gale Primary Sources using digital humanities text and data mining tools. No coding required! Rediscover and interpret the past through analysis and visualization of historical texts, including newspapers, books, archival collections, and more. (Create your personal DSL account online to begin selecting and analyzing materials. Be sure to be on VPN if off campus for account to work properly.) Learn more via tutorials and recorded webinars