Chapter 1: Understanding the Challenge of Large Data Sets

Researchers often face the daunting task of managing extensive data sets. The reason for this is clear: having more data leads to a deeper comprehension of the phenomena being examined. However, as these data sets grow larger, they become increasingly complex and unwieldy.

An effective way to tackle this challenge is to learn how to automatically divide these large data sets into smaller, more manageable files. Each file can then focus on a specific aspect of the study, making it easier to analyze. Automating this process significantly reduces the effort involved.

In this article, we will guide you through this procedure, providing practical Python code (compatible with Python 2.7) and contextualizing it within a real-world problem often encountered in scientific and engineering fields. To grasp the concepts better, refer to "The Automated Performance Map Creation Tutorial."

If you wish to reinforce your learning, you can download the accompanying data set. This resource will allow you to validate your code and verify your results, ensuring that you are mastering the process effectively.

Importing Required Python Packages

Every Python script begins with statements that bring in the needed packages. For our data splitting script, we will utilize:

• Pandas: This is the premier library for data analysis in Python, allowing you to load data into DataFrames (essentially tables) and providing a rich set of tools for data manipulation. For more information, check out the book "Python for Data Analysis."
• os: This package enables interaction with the operating system, allowing us to create new directories.
• Bokeh: A robust plotting library for Python that enables automatic plot generation and user interaction during data analysis. For further insights, refer to "Hands-On Data Visualization with Bokeh."

To import these packages, include the following lines at the beginning of your program:

import pandas as pd

import os

from bokeh.plotting import figure, save, gridplot, output_file, ColumnDataSource

from bokeh.models import HoverTool

Note that we import Pandas as pd for convenience and that specific functions from Bokeh are also included.

Reading Data Files

Pandas offers a powerful function for importing data sets called read_csv. To read a file, you call pandas.read_csv, specify the file path, and assign it to a variable. While there are additional modifiers available for customization, we will keep it simple for now.

You will need to import two files. The first is the data set itself. If you downloaded the accompanying data set, it should be titled 'COP_HPWH_f_Tamb&Tavg.csv'. If it's saved in the folder C:UsersYourNameDocumentsAutomatedDataAnalysis, you can import it using the following command:

Data = pd.read_csv(r'C:UsersYourNameDocumentsAutomatedDataAnalysisCOP_HPWH_f_Tamb&Tavg.csv')

Executing this code will save the data set to the variable Data, allowing you to leverage all of Pandas' analytical capabilities.

Next, you will need a table detailing the tests contained in the dataset. It is beneficial to create this table yourself for learning purposes. The data set includes results from three tests, each conducted at different ambient temperatures. Generate a table with the necessary information and save it as 'Test_Plan.csv' in the same directory as your data set.

After creating and saving the table, load it into Python using:

Test_Plan = pd.read_csv(r'C:UsersYourNameDocumentsAutomatedDataAnalysisTest_Plan.csv')

Identifying Test Boundaries

To distinguish where one test ends and the next begins, you will need to understand the test procedures. Specifically, you need to determine a condition indicating the conclusion of a test.

In this case, we are analyzing data from tests involving heat pump water heaters (HPWH), which consume electricity to heat water. Each test ceases when the device stops consuming electricity.

To identify these transitions, calculate the difference between the electricity consumption in each row and that of the previous row. A negative result indicates that the HPWH has reduced its electricity usage, signaling the end of a test.

You can achieve this with the .shift() function on the data set, as shown here:

Data['P_Elec_Shift (W)'] = Data['P_Elec (W)'].shift(periods=-1)

This will create two columns that illustrate the HPWH's electricity consumption.

Splitting the Data File

To split the data file effectively, follow these steps:

1. Iterate through each entry in the End_Of_Tests list.
2. Create a new DataFrame for each test's data.
3. Identify the conditions of the test.
4. Save the data into a new file with a descriptive name.

The iteration can be accomplished with a simple for loop:

for i in range(len(End_Of_Tests)):

This loop will run for each test in your data set. The next step involves extracting the relevant data into a new DataFrame, File_SingleTest, based on the indices in End_Of_Tests.

Identifying Each Test's Conditions

Once you have a DataFrame for a specific test, you need to determine which test it corresponds to. You can do this by examining the ambient temperature during the test using:

Temperature_Ambient = File_SingleTest['T_Amb (deg F)'][-50:].mean()

This will calculate the average temperature for the last 50 entries of the test data, which helps in matching it to the test plan.

Saving the Data to a New File

Now that you have a DataFrame with the results from a single test, along with its conditions, you can save it to a new file. First, ensure that the folder where you wish to save the data exists:

Folder = 'C:UsersYourNameDocumentsAutomatingDataAnalysisFiles_IndividualTests'

if not os.path.exists(Folder):

os.makedirs(Folder)

Then, save your DataFrame using:

Filename_SingleTest = f"PerformanceMap_HPWH_{int(Temperature_Ambient)}.csv"

File_SingleTest.to_csv(Folder + Filename_SingleTest, index=False)

Final Steps and Result Verification

With your script complete, you can run it to generate the desired output. The next step is to verify the results by comparing the files in your new folder against the test plan.

To check your results, ensure that:

1. Each test has a corresponding file in the output folder.
2. The conditions specified in the file names match those in the test plan.

You can also create plots of the data for visual inspection using Bokeh.

File_SingleTest['Time_SinceStart (min)'] = File_SingleTest.index * 10. / 60.

p1 = figure(width=800, height=400, x_axis_label='Time (min)', y_axis_label='Temperature (deg F)')

p1.circle(File_SingleTest['Time_SinceStart (min)'], File_SingleTest['T_Amb (deg F)'], legend='Ambient Temperature', color='red')

Finally, save the plot in the same directory:

output_file(Folder + 'PerformanceMap_HPWH_T_Amb=' + str(int(Temperature_Ambient)) + '.html', title='Temperatures')

save(p)

Conclusion

This article is the first step in a tutorial series aimed at teaching you how to automate scientific data analysis. Now that you have successfully split a large data file into manageable parts, you can move forward with processing these files and conducting further analyses.

For more information on subsequent steps, refer to the other articles in this series.