When it comes to data analysis, descriptive statistics is an important step to understand the basic characteristics of data and to summarize it. The Python library pandas, which is widely used for data manipulation and analysis, provides a powerful method called describe() for this purpose. It’s a part of the DataFrame class and can be used to generate descriptive statistics that summarize the central tendency, dispersion, and shape of a dataset’s distribution, excluding NaN values.
The describe() function is very handy when it comes to getting a quick overview of the dataset. It provides a tabular summary of the data which includes information such as count, mean, standard deviation, minimum and maximum values, as well as the quantiles of the data.
import pandas as pd
# Creating a sample DataFrame
data = {
'age': [21, 22, 23, 24, 25, 26, 27, 28, 29, 30],
'salary': [39000, 40000, 41000, 42000, 43000, 44000, 45000, 46000, 47000, 48000]
}
df = pd.DataFrame(data)
# Using describe() on the DataFrame
description = df.describe()
print(description)
This simple code snippet creates a pandas DataFrame with a sample data and then applies the describe() method to it. The output will give you a statistical summary of both ‘age’ and ‘salary’ columns. Knowing how to use the describe() function effectively can save a lot of time in data analysis and can help in making informed decisions based on the data.
Overview of Descriptive Statistics
Descriptive statistics are measures that summarize important features of data, often with a single number. They’re used to present quantitative descriptions in a manageable form. Some basic descriptive statistics include measures of central tendency like mean, median, and mode, which represent the center point of the data set. Measures of variability or dispersion such as standard deviation, variance, range, and interquartile range, provide insights on how spread out the data is around the central value.
Let’s take a deeper look into these measures:
- It shows the number of non-null entries in the data.
- The average of all non-null observations.
- A measure of the amount of variation or dispersion in a set of values.
- The lowest value in the dataset.
- The lower quartile or the 25th percentile. 25% of the data values are below this value.
- The median or the 50th percentile. This is the middle value of the dataset.
- The upper quartile or the 75th percentile. 75% of the data values are below this value.
- The highest value in the dataset.
These statistical measures are incredibly useful in various scenarios. For instance, mean and median can tell us about the average value and the central tendency of the data, but they can be affected by outliers. In contrast, the mode is not affected by outliers, but it might not be useful for all distributions. Understanding the spread of data through standard deviation and interquartile range can help identify outliers and understand the reliability of the mean.
Let’s apply these concepts to our sample dataframe:
# Calculating mean of 'age' and 'salary'
mean_age = df['age'].mean()
mean_salary = df['salary'].mean()
print("Mean Age:", mean_age)
print("Mean Salary:", mean_salary)
# Calculating median of 'age' and 'salary'
median_age = df['age'].median()
median_salary = df['salary'].median()
print("Median Age:", median_age)
print("Median Salary:", median_salary)
# Calculating standard deviation of 'age' and 'salary'
std_age = df['age'].std()
std_salary = df['salary'].std()
print("Standard Deviation of Age:", std_age)
print("Standard Deviation of Salary:", std_salary)
By understanding and calculating these descriptive statistics, we can gain insights into the data’s structure and begin to draw conclusions or ask further questions about the dataset. The describe() function in pandas does all these calculations for us in one go, thus providing a quick and efficient way to get a comprehensive overview of our data.
Understanding the Output of pandas.DataFrame.describe
The count in the output of describe() refers to the number of non-null observations within each column. This can be particularly useful when dealing with large datasets to quickly identify columns which may have missing values.
count_age = df['age'].count()
print("Count for Age:", count_age)
count_salary = df['salary'].count()
print("Count for Salary:", count_salary)
The minimum and maximum values provide a sense of the range of values within the data. Understanding the range can help in identifying outliers and ensuring that the data falls within expected bounds.
min_age = df['age'].min()
max_age = df['age'].max()
print("Age Range:", min_age, "to", max_age)
min_salary = df['salary'].min()
max_salary = df['salary'].max()
print("Salary Range:", min_salary, "to", max_salary)
The 25th, 50th, and 75th percentiles (also known as quartiles) break the data into quarters. The 50th percentile is equivalent to the median, the point at which half of the data values are above and half are below. The lower and upper quartiles provide a sense of the spread of the data, and when compared to the minimum and maximum values, they can give insights into the presence of outliers and the overall distribution shape.
quartiles_age = df['age'].quantile([0.25, 0.5, 0.75])
print("Age Quartiles:")
print(quartiles_age)
quartiles_salary = df['salary'].quantile([0.25, 0.5, 0.75])
print("Salary Quartiles:")
print(quartiles_salary)
Understanding each of these outputs in the context of your specific data is key to making the most of the describe() function. It allows you to quickly assess the central tendencies, variability, and overall distribution of your data set, which are all critical components in the initial stages of data analysis.
Advanced Techniques and Customization Options
While the default settings of describe() are useful for a quick summary, there may be times when you need to customize the output for a more detailed analysis. pandas allows you to tailor the describe() method to fit your specific needs by providing several parameters that can be adjusted.
For example, if you only want to see the descriptive statistics for a particular set of percentiles, you can use the percentiles parameter:
custom_percentiles = df.describe(percentiles=[0.1, 0.5, 0.9]) print(custom_percentiles)
This will give you the 10th, 50th, and 90th percentiles, along with the standard minimum, maximum, and count statistics.
Another customization option is to include or exclude certain data types. By default, the describe() method includes only numeric columns. If you want to include other data types, you can use the include and exclude parameters:
# Including all columns of all types description_all = df.describe(include='all') print(description_all) # Excluding object type columns description_without_objects = df.describe(exclude=[object]) print(description_without_objects)
If you have categorical data, you may want to see the frequency count for the top category or the one that appears most frequently. This can be done by setting the include parameter to object or O:
# Assuming 'department' is a categorical column in the DataFrame description_categorical = df['department'].describe(include=[object]) print(description_categorical)
This will output the count, unique count, top category, and frequency of the top category for the ‘department’ column.
Finally, if you’re working with large DataFrames, the describe() method can take some time to compute. To speed up the process, you can use the include parameter to limit the analysis to a subset of columns:
# Describing only 'age' and 'salary' columns description_subset = df[['age', 'salary']].describe() print(description_subset)
By using these advanced techniques and customization options, you can make the describe() method work more effectively for your specific data analysis requirements. It’s a flexible tool that can be adapted to provide the insights you need to understand your data fully.
