data.table Reshaping, stacking and splitting Going from long to wide format using dcast
Example
Casting: The Basics
Casting is used to transform data from long to wide format.
Starting with a long data set:
DT = data.table(ID = rep(letters[1:3],3), Age = rep(20:22,3), Test = rep(c("OB_A","OB_B","OB_C"), each = 3), Result = 1:9)
We can cast our data using the dcast function in data.table. This returns another data.table in wide format:
dcast(DT, formula = ID ~ Test, value.var = "Result")
ID OB_A OB_B OB_C
1: a 1 4 7
2: b 2 5 8
3: c 3 6 9
class(dcast(DT, formula = ID ~ Test, value.var = "Result"))
[1] "data.table" "data.frame"
Casting a value
A value.var argument is necessary for a proper cast - if not provided dcast will make an assumption based on your data.
dcast(DT, formula = ID ~ Test, value.var = "Result")
ID OB_A OB_B OB_C
1: a 1 4 7
2: b 2 5 8
3: c 3 6 9
ID OB_A OB_B OB_C
1: a 20 20 20
2: b 21 21 21
3: c 22 22 22
Multiple value.vars can be provided in a list
dcast(DT, formula = ID ~ Test, value.var = list("Result","Age"))
ID Result_OB_A Result_OB_B Result_OB_C Age_OB_A Age_OB_B Age_OB_C
1: a 1 4 7 20 20 20
2: b 2 5 8 21 21 21
3: c 3 6 9 22 22 22
Formula
Casting is controlled using the formula argument in dcast. This is of the form ROWS ~ COLUMNS
dcast(DT, formula = ID ~ Test, value.var = "Result")
ID OB_A OB_B OB_C
1: a 1 4 7
2: b 2 5 8
3: c 3 6 9
dcast(DT, formula = Test ~ ID, value.var = "Result")
Test a b c
1: OB_A 1 2 3
2: OB_B 4 5 6
3: OB_C 7 8 9
Both rows and columns can be expanded with further variables using +
dcast(DT, formula = ID + Age ~ Test, value.var = "Result")
ID Age OB_A OB_B OB_C
1: a 20 1 4 7
2: b 21 2 5 8
3: c 22 3 6 9
dcast(DT, formula = ID ~ Age + Test, value.var = "Result")
ID 20_OB_A 20_OB_B 20_OB_C 21_OB_A 21_OB_B 21_OB_C 22_OB_A 22_OB_B 22_OB_C
1: a 1 4 7 NA NA NA NA NA NA
2: b NA NA NA 2 5 8 NA NA NA
3: c NA NA NA NA NA NA 3 6 9
#order is important
dcast(DT, formula = ID ~ Test + Age, value.var = "Result")
ID OB_A_20 OB_A_21 OB_A_22 OB_B_20 OB_B_21 OB_B_22 OB_C_20 OB_C_21 OB_C_22
1: a 1 NA NA 4 NA NA 7 NA NA
2: b NA 2 NA NA 5 NA NA 8 NA
3: c NA NA 3 NA NA 6 NA NA 9
Casting can often create cells where no observation exists in the data. By default this is denoted by NA, as above. We can override this with the fill= argument.
dcast(DT, formula = ID ~ Test + Age, value.var = "Result", fill = 0)
ID OB_A_20 OB_A_21 OB_A_22 OB_B_20 OB_B_21 OB_B_22 OB_C_20 OB_C_21 OB_C_22
1: a 1 0 0 4 0 0 7 0 0
2: b 0 2 0 0 5 0 0 8 0
3: c 0 0 3 0 0 6 0 0 9
You can also use two special variables in the formula object
.represents no other variables...represents all other variables
dcast(DT, formula = Age ~ ., value.var = "Result")
Age .
1: 20 3
2: 21 3
3: 22 3
dcast(DT, formula = ID + Age ~ ..., value.var = "Result")
ID Age OB_A OB_B OB_C
1: a 20 1 4 7
2: b 21 2 5 8
3: c 22 3 6 9
Aggregating our value.var
We can also cast and aggregate values in one step. In this case, we have three observations in each of the intersections of Age and ID. To set what aggregation we want, we use the fun.aggregate argument:
#length
dcast(DT, formula = ID ~ Age, value.var = "Result", fun.aggregate = length)
ID 20 21 22
1: a 3 0 0
2: b 0 3 0
3: c 0 0 3
#sum
dcast(DT, formula = ID ~ Age, value.var = "Result", fun.aggregate = sum)
ID 20 21 22
1: a 12 0 0
2: b 0 15 0
3: c 0 0 18
#concatenate
dcast(DT, formula = ID ~ Age, value.var = "Result", fun.aggregate = function(x){paste(x,collapse = "_")})
ID 20 21 22
1: a 1_4_7
2: b 2_5_8
3: c 3_6_9
We can also pass a list to fun.aggregate to use multiple functions
dcast(DT, formula = ID ~ Age, value.var = "Result", fun.aggregate = list(sum,length))
ID Result_sum_20 Result_sum_21 Result_sum_22 Result_length_20 Result_length_21 Result_length_22
1: a 12 0 0 3 0 0
2: b 0 15 0 0 3 0
3: c 0 0 18 0 0 3
If we pass more than one function and more than one value, we can calculate all combinations by passing a vector of value.vars
dcast(DT, formula = ID ~ Age, value.var = c("Result","Test"), fun.aggregate = list(function(x){paste0(x,collapse = "_")},length))
ID Result_function_20 Result_function_21 Result_function_22 Test_function_20 Test_function_21 Test_function_22 Result_length_20 Result_length_21
1: a 1_4_7 OB_A_OB_B_OB_C 3 0
2: b 2_5_8 OB_A_OB_B_OB_C 0 3
3: c 3_6_9 OB_A_OB_B_OB_C 0 0
Result_length_22 Test_length_20 Test_length_21 Test_length_22
1: 0 3 0 0
2: 0 0 3 0
3: 3 0 0 3
where each pair is calculated in the order value1_formula1, value1_formula2, ... , valueN_formula(N-1), valueN_formulaN.
Alternatively, we can evaluate our values and functions one-to-one by passing 'value.var' as a list:
dcast(DT, formula = ID ~ Age, value.var = list("Result","Test"), fun.aggregate = list(function(x){paste0(x,collapse = "_")},length))
ID Result_function_20 Result_function_21 Result_function_22 Test_length_20 Test_length_21 Test_length_22
1: a 1_4_7 3 0 0
2: b 2_5_8 0 3 0
3: c 3_6_9 0 0 3
Naming columns in the result
By default, column name components are seperated by an underscore _. This can be manually overridden using the sep= argument:
dcast(DT, formula = Test ~ ID + Age, value.var = "Result")
Test a_20 b_21 c_22
1: OB_A 1 2 3
2: OB_B 4 5 6
3: OB_C 7 8 9
dcast(DT, formula = Test ~ ID + Age, value.var = "Result", sep = ",")
Test a,20 b,21 c,22
1: OB_A 1 2 3
2: OB_B 4 5 6
3: OB_C 7 8 9
This will seperate any fun.aggregate or value.var we use:
dcast(DT, formula = Test ~ ID + Age, value.var = "Result", fun.aggregate = c(sum,length), sep = ",")
Test Result,sum,a,20 Result,sum,b,21 Result,sum,c,22 Result,length,a,20 Result,length,b,21 Result,length,c,22
1: OB_A 1 2 3 1 1 1
2: OB_B 4 5 6 1 1 1
3: OB_C 7 8 9 1 1 1
