Eight Tips for Working with Big Data

Administrator — Fri, 16 Nov 2012 16:15:00 GMT

By Abigail Lefkowitz, Data Analyst at MaassMedia, LLC

When describing a set of data, the word "big" doesn't just refer to size. "Big" implies that the data comes from multiple sources and is too complex overall for conventional business intelligence (BI) software to process in a practical amount of time and with sufficient computing memory.

It has been shown that the amount of data in the world is growing at a rate of 50 percent each year and more than doubling every two years. Since data volumes are not getting any smaller, it is imperative that organizations attempting to gain knowledge and insights from their data establish a methodology for storing, analyzing, and interpreting big data.

When you are ready to delve into your data, keep these tips in mind:

1. Start small, think big

Sampling is widely accepted as a preferred standard practice. Sampling takes a random, unbiased selection of individuals from a population to estimate the characteristics of that population. A statistically significant sample should produce results similar to that of the entire population.

First, choose the appropriate sample size to ensure statistical significance. Stated simply, a result is statistically significant if it is unlikely to happen by chance. Using multiple, but independent samples of the same size will increase the statistical significance. Next, randomly select that number of data points from the population and carry out the analysis as planned.

2. Use a unique identifier to connect different sources of data

It is advisable to link multiple data sources using some common denominator in order to create a more complete picture of visitor activity and set the stage for more in-depth, transformative insights. To connect data from various sources, each source must assign a unique ID common to every customer or visitor. Web analytics platforms, such as Google Analytics and Adobe SiteCatalyst allow unique IDs to pass through visitor cookies that are assigned when someone visits a website. These IDs can be linked to any other source of data that captures the same unique ID.

For example, a website utilizing a Web-based survey can assign each survey respondent a unique ID that gets passed from the site into your Web analytics platform. As a result, responses to the survey can be connected to behavioral Web data such as page views, time on site, etc. Without having both the unique survey ID and visitor ID, these types of merges would not be possible.

3. ETL - Extract, transform and load

To obtain your data, you will need to extract it from each source and store it in a place with sufficient space, such as an in-house FTP server or data warehouse. Unstructured data needs to be transformed into a consistent format that allows for data analysis. Removing duplicates, grouping values and concatenating variables are a few examples of possible transformations. When performing these manipulations on millions of rows of data, using structured query language (SQL) can cut down on both processing and overall analysis time.

After extracting and transforming the data, it must be loaded into a storage location or data warehouse. It is recommended that the data be formatted prior to being loaded into the data warehouse for easier access in the future.

4. Select appropriate business intelligence tools

Standard software, like Microsoft Excel, does not have the capacity to manipulate big data and carry out simple calculations in a timely manner because of the data's volume and complex nature, coupled with the fact that it is commonly stored in separate silos with limited accessibility. Microsoft Access is an example of a powerful database that allows you to join and link datasets, especially when connecting unique IDs from different sources.

You will also need two additional analysis tools—one that can perform in-depth analysis and one that can produce visually appealing graphs and charts. It is difficult to find a program that can tackle both effectively. SAS and SPSS/IBM are two examples of statistical analysis programs with excellent data mining capabilities, while Tableau is a tool that produces attractive and dynamic images.

5. Incorporate demographic data

Overlaying demographic information with your dataset is helpful and highly recommended to better understand who your target audience(s) might be. FICO and Experian collect a variety of consumer data scores based on credit card transactions and demographic status. For example, FICO can help you predict whether you will be taking your medication every day based on certain variables such as length of time living in one location and number of years owning a particular car. Again, pending a unique ID, the demographic information can be linked to the existing dataset.

6. Hire a "data scientist"

Making sense of big data can be daunting. Instead of taking it on alone, you might be better off hiring experienced analysts who are comfortable working with massive sets of data. Employing one or more people who can dedicate their time to analysis will allow for smarter and more accurate, meaningful and transformative findings in a shorter amount of time. Look for analysts who are knowledgeable about storage, manipulation, importing and exporting, merging, sub-setting, hypothesis testing and mining for insights.

7. If possible, automate reports

When working with data that is continuously streaming in, automated reporting can save both time and money. Instead of manually reformatting the data each time a new chunk comes in, set up an automatic report with a filter that captures only the necessary variables and manipulates them in such a way that the data is more streamlined to the end goal.

Reports must first be customized to the preference of the end user, and then they can be automatically delivered to one or more recipients at regular intervals. Adobe SiteCatalyst and Google Analytics are two examples of Qeb analytics platforms that can automate daily, weekly and monthly reports.

8. 75 percent investigation, 25 percent discovery

As with any type of analysis, it is a good rule of thumb to spend approximately 75 percent of your time testing pre-developed hypotheses and the other 25 percent poking around in the data with no specific direction in the hopes of uncovering nuggets of knowledge. It is likely that at least one hypothesis you have formulated will lead to additional analytical questions about the scope of the original hypothesis. Therefore, the majority of the time spent testing hypotheses will inevitably lead to hidden findings. Devote a smaller fraction of your time slicing and dicing the data in unexpected segments because many times buried insights can be found when looking at familiar information in an unfamiliar way.

Incorporate Your Business at Go Daddy

Pete Prestipino — Tue, 10 Nov 2009 17:00:00 GMT

In addition to domain registration and hosting services, Go Daddy is now offering Incorporation Services, enabling users its services to transition a business into a corporation or LLC.

Users log in to the service and answer a series of questions, then Go Daddy files the necessary papers on their behalf. After several weeks, incorporation documents are sent back to the user. Each application is reviewed by experienced professionals to make sure it is submitted completely and accurately. Incorporation Services comes in two levels offering everything required to incorporate a business. The Deluxe plan delivers a Federal Employer Identification Number, as well as expedited shipping.

"If you haven't incorporated your businesses, you're leaving yourself and your family open to unnecessary risk," said Go Daddy CEO and Founder Bob Parsons. "Go Daddy wanted to help make the whole incorporation process affordable and painless, which is why we’re offering Incorporation Services."

While I would not expect a rush of GoDaddy customers to use the service, it is much less expensive that retaining an attorney to do virtually the same work. There are alternative however including some well respected services such as LegalZoom and BizFilings.

Stay up to date on the latest Internet trends:
Request a professional subscription to Website Magazine,
the most popular print publication on Web success.

'Net Features : LLC

Eight Tips for Working with Big Data

Incorporate Your Business at Go Daddy