How do I reshape this dataset in Python pandas?

Question

Say I have a dataset like this:

is_a  is_b  is_c  population infected
1     0     1     50         20
1     1     0     100        10
0     1     1     20         10
...

How do I reshape it to look like this?

feature  0       1 
a        10/20   30/150
b        20/50   20/120
c        10/100  30/70
...

In the original dataset, I have features a, b, and c as their own separate columns. In the transformed dataset, these same variables are listed under column feature, and two new columns 0 and 1 are produced, corresponding to the values that these features can take on.

In the original dataset where is_a is 0, add infected values and divide them by population values. Where is_a is 1, do the same, add infected values and divide them by population values. Rinse and repeat for is_b and is_c. The new dataset will have these fractions (or decimals) as shown. Thank you!

I've tried pd.pivot_table and pd.melt but nothing comes close to what I need.

Answer 1

After doing the wide_to_long , your question is more clear

df=pd.wide_to_long(df,['is'],['population','infected'],j='feature',sep='_',suffix='\w+').reset_index()
df
  population  infected feature is
0          50        20    a   1
1          50        20    b   0
2          50        20    c   1
3         100        10    a   1
4         100        10    b   1
5         100        10    c   0
6          20        10    a   0
7          20        10    b   1
8          20        10    c   1

df.groupby(['feature','is']).apply(lambda x : sum(x['infected'])/sum(x['population'])).unstack()
is      0         1
feature
a     0.5  0.200000
b     0.4  0.166667
c     0.1  0.428571

Answer 2

I tried this on your small dataframe, but I am not sure it will work on a larger dataset.

dic_df = {}
for letter in ['a', 'b', 'c']: 
    dic_da = {}
    dic_da[0] = df[df['is_'+str(letter)] == 0].infected.sum()/df[df['is_'+str(letter)] == 0].population.sum()
    dic_da[1] = df[df['is_'+str(letter)] == 1].infected.sum()/df[df['is_'+str(letter)] == 1].population.sum()
    dic_df[letter] = dic_da
    dic_df
dic_df_ = pd.DataFrame(data = dic_df).T.reset_index().rename(columns= {'index':'feature'})

feature 0   1
0   a   0.5 0.200000
1   b   0.4 0.166667
2   c   0.1 0.428571

Answer 3

Here, DF would be your original DataFrame

Aux_NewDF = [{'feature': feature, 
               0       : '{}/{}'.format(DF['infected'][DF['is_{}'.format(feature.lower())]==0].sum(), DF['population'][DF['is_{}'.format(feature.lower())]==0].sum()), 
               1       : '{}/{}'.format(DF['infected'][DF['is_{}'.format(feature.lower())]==1].sum(), DF['population'][DF['is_{}'.format(feature.lower())]==1].sum())} for feature in ['a','b','c']] 



NewDF = pd.DataFrame(Aux_NewDF)