Pandas DataFrame Reference for All Functions

Pandas is a powerful and versatile data analysis and manipulation library for Python. It provides data structures and functions needed to work with structured data seamlessly. Among its core data structures, the DataFrame is one of the most crucial components, allowing users to store and manipulate data in a tabular format, much like a spreadsheet or SQL table. In this article, we will explore the comprehensive reference for all the functions available in Pandas DataFrame.

I. Introduction

A. Overview of Pandas

Pandas was initially developed for financial data analysis but has now become a standard tool for data science across various domains. It allows you to handle and analyze data effortlessly.

B. Importance of DataFrames in data manipulation

The DataFrame provides a rich set of functionalities, including data selection, filtering, grouping, and merging. Understanding how to use DataFrames is vital for mastering data manipulation and analysis in Python.

II. Creating DataFrames

A. From Dictionary

Creating a DataFrame from a dictionary is straightforward and intuitive. Here’s an example:

import pandas as pd

data = {
    'Name': ['John', 'Anna', 'Peter'],
    'Age': [28, 24, 35],
    'City': ['New York', 'Paris', 'Berlin']
}

df = pd.DataFrame(data)
print(df)

B. From Lists

DataFrames can also be created using lists. Here’s how:

data = [['John', 28, 'New York'],
        ['Anna', 24, 'Paris'],
        ['Peter', 35, 'Berlin']]

df = pd.DataFrame(data, columns=['Name', 'Age', 'City'])
print(df)

C. From Numpy Arrays

You can create a DataFrame from a NumPy array:

import numpy as np

data = np.array([['John', 28, 'New York'],
                 ['Anna', 24, 'Paris'],
                 ['Peter', 35, 'Berlin']])

df = pd.DataFrame(data, columns=['Name', 'Age', 'City'])
print(df)

D. From External Files

DataFrames can be created from external files like CSV, Excel, or JSON:

df = pd.read_csv('data.csv')
print(df)

III. Viewing Data

A. head()

The head() function returns the first few rows of the DataFrame:

df.head(2)

B. tail()

The tail() function returns the last few rows:

df.tail(2)

C. info()

The info() method provides concise summary information about the DataFrame:

df.info()

D. describe()

The describe() function generates descriptive statistics:

df.describe()

E. memory_usage()

The memory_usage() method returns the memory consumption of the DataFrame:

df.memory_usage()

F. sample()

To get a random sample of rows from the DataFrame, you can use the sample() function:

df.sample(2)

IV. Selection and Filtering

A. Selecting columns

Columns can be selected by specifying the column name:

df['Name']

B. Selecting rows

Rows can be selected by their index:

df.iloc[0]

C. Conditional selection

Conditional selection allows you to filter DataFrames based on specific criteria:

df[df['Age'] > 25]

D. .loc[] and .iloc[]

The .loc[] and .iloc[] functions can be used for selection based on labels and index positions:

Function	Usage
df.loc[1]	Selects row with index 1 (by label)
df.iloc[1]	Selects row at index 1 (by position)

V. Modifying DataFrames

A. Adding new columns

You can add new columns to a DataFrame easily:

df['Salary'] = [50000, 52000, 60000]

B. Dropping columns

To drop a column:

df.drop('City', axis=1, inplace=True)

C. Dropping rows

You can drop rows based on their index:

df.drop(1, inplace=True)

D. Renaming columns

To rename columns:

df.rename(columns={'Name': 'First Name'}, inplace=True)

VI. Sorting and Ranking

A. sort_values()

The sort_values() method sorts the DataFrame by the specified column:

df.sort_values(by='Age')

B. sort_index()

You can also sort the DataFrame by its index:

df.sort_index()

C. rank()

The rank() method ranks the values in a column:

df['Rank'] = df['Salary'].rank()

VII. Grouping Data

A. groupby()

The groupby() function allows you to group data based on column values:

grouped = df.groupby('City').mean()

B. aggregating data

You can apply aggregation functions after grouping:

grouped = df.groupby('City').agg({'Salary': 'mean', 'Age': 'max'})

C. transforming data

The transform() function applies a transformation to each group:

df['Normalized Salary'] = df.groupby('City')['Salary'].transform(lambda x: x / x.max())

VIII. Merging and Joining

A. merge()

The merge() function allows you to combine two DataFrames:

df_merged = pd.merge(df1, df2, on='common_column')

B. join()

The join() method joins two DataFrames on their indices:

df1.join(df2)

C. concat()

The concat() function can concatenate DataFrames along rows or columns:

pd.concat([df1, df2], axis=0)

IX. Handling Missing Data

A. isnull()

The isnull() function checks for missing values:

df.isnull()

B. notnull()

You can use notnull() to find non-missing values:

df.notnull()

C. dropna()

To drop rows with any missing values:

df.dropna(inplace=True)

D. fillna()

You can fill missing values with a specified value:

df.fillna(0, inplace=True)

X. Dataframe Operations

A. apply()

The apply() function applies a function along the axis of the DataFrame:

df['Adjusted Salary'] = df['Salary'].apply(lambda x: x * 1.1)

B. applymap()

The applymap() function applies a function to each element of the DataFrame:

df.applymap(str)

C. map()

The map() function maps values of a Series:

df['City'] = df['City'].map({'Paris': 'Lyon', 'New York': 'Brooklyn'})

D. replace()

You can use replace() to replace values in the DataFrame:

df['City'].replace('Berlin', 'Hamburg', inplace=True)

XI. DataFrame Attributes and Methods

A. index

The index attribute returns the index of the DataFrame:

df.index

B. columns

The columns attribute returns the column labels:

df.columns

C. dtypes

To check the data types of each column:

df.dtypes

D. shape

The shape attribute returns the dimensions of the DataFrame:

df.shape

E. size

The size attribute returns the total number of elements:

df.size

F. T

The T attribute returns the transpose of the DataFrame:

df.T

XII. Input and Output Operations

A. Reading data

1. read_csv()

Read data from a CSV file:

df = pd.read_csv('data.csv')

2. read_excel()

Read data from an Excel file:

df = pd.read_excel('data.xlsx')

3. read_json()

Read data from a JSON file:

df = pd.read_json('data.json')

B. Writing data

1. to_csv()

Write the DataFrame to a CSV file:

df.to_csv('output.csv', index=False)

2. to_excel()

Write the DataFrame to an Excel file:

df.to_excel('output.xlsx', index=False)

3. to_json()

Write the DataFrame to a JSON file:

df.to_json('output.json')

Q1: What is Pandas used for?

A1: Pandas is used for data manipulation and analysis in Python. It provides data structures like Series and DataFrames that allow for efficient data handling.

Q2: How do I create a DataFrame?

A2: You can create a DataFrame from a dictionary, lists, NumPy arrays, or by loading data from external files.

Q3: What is the difference between .loc[] and .iloc[]?

A3: .loc[] is label-based indexing while .iloc[] is integer-based indexing.

Q4: How do I handle missing data?

A4: You can handle missing data using methods like dropna() to remove them or fillna() to replace them with a specific value.

Q5: What is the purpose of grouping data?

A5: Grouping data allows you to perform operations such as aggregation that summarize the underlying data using functions like sum, mean, etc.

askthedev.com Latest Articles