Chapter 12 pandas DataFrames

run pip3 install pandas or run !pip install pandas on rstudio terminal or mac terminal or jupyter notebook

import pandas as pd 

pd.set_option('display.max_columns', None)

12.1 Data Frames

12.1.1 Data frame creation

Sometimes you may need to create a data frame

## empty data frame
df1 = pd.DataFrame(dtype = 'float64')
df1

## Empty DataFrame
## Columns: []
## Index: []

df2 = pd.DataFrame({'A' : []})
df2

## Empty DataFrame
## Columns: [A]
## Index: []

Is df2 empty?

There is column name but it is still empty.

df2.empty

## True

How to create a sample data frame?

From a dictionary

my_dict = {'Col_1': [1,2,3,4], 
           'Col_2': ['a', 'b', 'c', 'd'],
           'Col_3': 1984}
           
my_df = pd.DataFrame(my_dict)

my_df[:5]

##    Col_1 Col_2  Col_3
## 0      1     a   1984
## 1      2     b   1984
## 2      3     c   1984
## 3      4     d   1984

a_list = [1,2,3,4]
b_list = ['a', 'b', 'c', 'd']
##
df3 = pd.DataFrame({'var1': a_list, 
                    'var2': b_list})
df3

##    var1 var2
## 0     1    a
## 1     2    b
## 2     3    c
## 3     4    d

12.1.2 Read csv files

df_csv = pd.read_csv('__REPO/data/college.csv').iloc[:100, :5]
df_csv[:5]

##        id                             name        city state region
## 0  102669        Alaska Pacific University   Anchorage    AK   West
## 1  101648        Marion Military Institute      Marion    AL  South
## 2  100830  Auburn University at Montgomery  Montgomery    AL  South
## 3  101879      University of North Alabama    Florence    AL  South
## 4  100858                Auburn University      Auburn    AL  South

12.1.3 Data attributes

12.1.3.1 shape of data frame

df_csv.shape

## (100, 5)

print("number of rows:", df_csv.shape[0],
      "\nnumber of columns:", df_csv.shape[1])

## number of rows: 100 
## number of columns: 5

12.1.3.2 Columns

How to get the column names?

–> Column names are stored in columns attribute.

df_csv.columns

## Index(['id', 'name', 'city', 'state', 'region'], dtype='object')

list(df_csv.columns)

## ['id', 'name', 'city', 'state', 'region']

df_csv.columns.to_list()

## ['id', 'name', 'city', 'state', 'region']

for cols in df_csv.columns:
  print(cols)

## id
## name
## city
## state
## region

12.1.3.3 data types in dataframe

df_csv.dtypes

## id         int64
## name      object
## city      object
## state     object
## region    object
## dtype: object

[types for types in df_csv.dtypes]

## [dtype('int64'), dtype('O'), dtype('O'), dtype('O'), dtype('O')]

12.1.3.4 index of the data frame

df_csv[:5]

##        id                             name        city state region
## 0  102669        Alaska Pacific University   Anchorage    AK   West
## 1  101648        Marion Military Institute      Marion    AL  South
## 2  100830  Auburn University at Montgomery  Montgomery    AL  South
## 3  101879      University of North Alabama    Florence    AL  South
## 4  100858                Auburn University      Auburn    AL  South

df_csv.index

## RangeIndex(start=0, stop=100, step=1)

for i in df_csv.index:
  if i < 5:
    print(i)

## 0
## 1
## 2
## 3
## 4

12.1.3.5 assign a new index

df = df_csv.copy()
df.index = df.state

df[:5]

##            id                             name        city state region
## state                                                                  
## AK     102669        Alaska Pacific University   Anchorage    AK   West
## AL     101648        Marion Military Institute      Marion    AL  South
## AL     100830  Auburn University at Montgomery  Montgomery    AL  South
## AL     101879      University of North Alabama    Florence    AL  South
## AL     100858                Auburn University      Auburn    AL  South

12.1.3.6 index to column

df = df_csv.copy()
df['index_column'] = df.index

df[:5]

##        id                             name  ... region index_column
## 0  102669        Alaska Pacific University  ...   West            0
## 1  101648        Marion Military Institute  ...  South            1
## 2  100830  Auburn University at Montgomery  ...  South            2
## 3  101879      University of North Alabama  ...  South            3
## 4  100858                Auburn University  ...  South            4
## 
## [5 rows x 6 columns]

calling keys() function: output is similar to columns attributes

df_csv.keys()

## Index(['id', 'name', 'city', 'state', 'region'], dtype='object')

column.values method returns an array of index.

list(df_csv.columns.values)

## ['id', 'name', 'city', 'state', 'region']

Using tolist() method with values with given the list of columns.

list(df_csv.columns.values.tolist())

## ['id', 'name', 'city', 'state', 'region']

Using sorted() method : sorted() method will return the list of columns sorted in alphabetical order.

sorted(df3)

## ['var1', 'var2']

12.1.4 Create or add a new column

You can create a new column in a pandas DataFrame based on the values of existing columns using various methods, such as assignment, the apply() function, or vectorized operations.

Here are some examples:

12.1.5 Explore Data

#recipes.head()