```{r, include=FALSE}
knitr::opts_chunk$set(cache=TRUE, autodep=TRUE, cache.comments=TRUE)
```

Name:  
Andrew ID:  
Collaborated with:  

On this homework, you can collaborate with your classmates, but you must identify their names above, and you must submit **your own** homework as an knitted HTML file on Canvas, by Monday 10pm, next week (July 15).

States data set
===

Below we construct a data frame, of 50 states x 10 variables. The first 8 variables are numeric and the last 2 are factors. The numeric variables here come from the built-in `state.x77` matrix, which records various demographic factors on 50 US states, measured in the 1970s. You can learn more about this state data set by typing `?state.x77` into your R console. 

```{r}
state_df <- data.frame(state.x77, Region = state.region, 
                       Division = state.division)
```

Basic data frame manipulations
===

- **1a.** Add a column to `state_df`, containing the state abbreviations that are stored in the built-in vector `state.abb`. Name this column `Abbr`. You can do this in (at least) two ways: by using a call to `data.frame()`, or by directly defining `state_df$Abbr`. Display the first 3 rows and all 11 columns of the new `state_df`.


- **1b.** Remove the `Region` column from `state_df`. You can do this in (at least) two ways: by using negative indexing, or by directly setting `state_df$Region` to be `NULL`. Display the first 3 rows and all 10 columns of `state_df`.


- **1c.** Add two columns to `state_df`, containing the x and y coordinates (longitude and latitude, respectively) of the center of the states, that are stored in the (existing) list `state.center`. Hint: take a look at this list in the console, to see what its elements are named. Name these two columns `Center_x` and `Center_y`. Display the first 3 rows and all 12 columns of `state_df`.


- **1d.** Make a new data.frame which contains only those states whose longitude is less than -100. Do this in two different ways and check that they are equal to each other, using an appropriate function call.


Prostate cancer data set
===

Let's return to the prostate cancer data set that we looked at in the lab/homework from Week 2 (taken from the book [The Elements of Statistical Learning](http://statweb.stanford.edu/~hastie/ElemStatLearn/)). Below we read in a data frame of 97 men x 9 variables. You can remind yourself about what's been measured by looking back at the lab/homework (or by visiting the URL linked above in your web browser, clicking on "Data" on the left-hand menu, and clicking "Info" under "Prostate"). 

```{r}
pros_data <- 
  read.table("https://raw.githubusercontent.com/benjaminleroy/36-350-summer-data/master/Week1/pros.dat")
```

Practice with the apply family
===

- **2a.** Using `sapply()`, calculate the mean of each variable. Also, calculate the standard deviation of each variable. Each should require just one line of code. Display your results.


- **2b.** Now, use `lapply()` to perform t-tests for each variable in the data set, between SVI and non-SVI groups. To be precise, you will perform a t-test for each variable excluding the SVI variable itself. For convenience, we've defined a function `t_test_by_ind()` below, which takes a numeric variable `x`, and then an indicator variable `ind` (of 0s and 1s) that defines the groups. Run this function on the columns of `pros_data`, and save the result as `tests`. What kind of data structure is `tests`? Print it to the console.

```{r}
t_test_by_ind <- function(x, ind) {
  stopifnot(all(ind %in% c(0, 1)))
  return(t.test(x[ind == 0], x[ind == 1]))
}
```


- **Challenge.** Using an appropriate apply function again, extract the p-values from the `tests` object you created in the last question, with just a single line of code. Hint: run the command `"[["(pros_data, "lcavol")` in your console---what does this do?


Rio Olympics data set
===

We're going to examine data from the 2016 Summer Olympics in Rio de Janeiro, taken from https://github.com/flother/rio2016 (itself put together by scraping the official Summer Olympics website for information about the athletes). Below we read in the data and store it as `rio`.

```{r}
rio <- read.csv("https://raw.githubusercontent.com/benjaminleroy/36-350-summer-data/master/Week2/rio.csv")
```

More practice with data frames and apply
===

- **3a.** Call `summary()` on `rio` and display the result. Is there any missing data?


- **3b.** Use `rio` to answer the following questions. How many athletes competed in the 2016 Summer Olympics? How many countries were represented? What were these countries, and how many athletes competed for each one? Which country brought the most athletes, and how many was this? 


- **3c.** How many medals of each type---gold, silver, bronze---were awarded at this Olympics? Is this result surprising, and can you explain what you are seeing?


- **3d.** Create a column called `total` which adds the number of gold, silver, and bronze medals for each athlete, and add this column to `rio`. Which athlete had the most number of medals and how many was this? Which athlete had the most silver medals and how many was this? (Ouch! So close, so many times ...) In the case of ties, here, display all the relevant athletes. 


- **3e.** Using `tapply()`, calculate the total medal count for each country. Save the result as `total_by_nat`, and print it to the console. Which country had the most number of medals, and how many was this? How many countries had zero medals? **Challenge**: among the countries that had zero medals, which had the most athletes, and how many athletes was this? (Ouch!)


Some advanced practice with apply
===

- **4a.** The variable `date_of_birth` contains strings of the date of birth of each athlete. Use text processing commands to extract the year of birth, and create a new numeric variable called `age`, equal to 2016 - (the year of birth). (Here we're ignoring days and months for simplicity.) Add the `age` variable to the `rio` data frame. variable  Who is the oldest athlete, and how old is he/she? Youngest athlete, and how old is he/she? In the case of ties, here, display all the relevant athletes. **Challenge:** Answer the same questions, but now only among athletes who won a medal.


- **4b.** Create a new data.frame called `sports`, which we'll populate with information about each sporting event at the Summer Olympics. Initially, define `sports` to contain a single variable called `sport` which contains the names of the sporting events in alphabetical order. Then, add a column called `n_participants` which contains the number of participants in each sport. Use one of the apply functions to determine the number of gold medals given out for each sport, and add this as a column called `n_gold`. Using your newly created `sports` data frame, calculate the ratio of the number of gold medals to participants for each sport. Which sport has the highest ratio? Which has the lowest?


Plotting tools 
===

Below, we read in a data set, as in lab, that is related to shark attacks. The data is taken from Kaggle and was originally compiled by the global shark attack file.  [More information is available here.](https://www.kaggle.com/teajay/global-shark-attacks)

```{r}
shark_attacks <- read.csv("https://raw.githubusercontent.com/benjaminleroy/36-350-summer-data/master/Week2/shark-attacks.csv", as.is = TRUE)
```

- **5a.** As we did in Lab (2.2, Question 3a),  define `attack_time` to be the `Date` column of `shark_attacks`. Define a character vector `attack_year` to contain the first 4 characters of an entry of `attack_time`. Hint: hopefully you haven't forgotten ... use `substr()` here. Finally, convert `attack_year` into a numeric vector. Do a similar thing for `attack_month`. Then append these onto the shark_data matrix (as `year` and `month`). We will use these added columns later.


  - **5b.** Additionally create a new column to the `shark_attacks` data frame named `decade` that contains the decade the attacked happened (try the `floor` function). Make a new data.frame called `shark_attacks_above` which only contains individuals with ages not equal to `-5`. Using one of the apply functions and `shark_attacks_above`, compute the average age of the victim for each decade and storage it in `average_age_by_decade`. 


One can alternatively use the `round` function as suggested in the prompt. Due to ambiguity related to rounding off at 5, the following line yields a few decades which are different from the ones above.


- **5c.** Make a barchart to inform yourself of the number of reported attacks per decade (use the `shark_attacks_above` data frame). With visual in mind, do you have any comments about the values in `average_age_by_decade`?


- **5d.** In lab we examined the distribution of Age with a histogram and conditional on gender in the scatter plot. Use `geom_boxplot` to examine the distributions of Age conditional on gender (continue using the `shark_attacks_above` data frame). Try adding `coord_flip` to your figure as well. How could we make the boxplot just have 1 boxplot (plot this as well)?


- **5e.** Remember in class that we showed how to label all parts of the plot using using  `+ labs()` update the plots in this problem with quality labels (in their respective locations - not in **5e.**).

More plots
===

Below you'll find a data set that contains a subset of all the passengers on the titanic, information about each of them and if they survived. The dataset was taken from kaggle, and you can find more information about the dataset here: [https://www.kaggle.com/c/titanic/data](https://www.kaggle.com/c/titanic/data).

```{r}
titanic <- read.csv("https://raw.githubusercontent.com/benjaminleroy/stat315summer_data/master/assignments/assignment03/titanic.csv")
```

- **6a.** Does class (`Pclass`) effect if you survived or not? Below is a stacked barchart that tries to answer this question. Replace `factor(Survived)` with just `Survived`. What happened - and why do you think this happened?

```{r}
library(ggplot2)
ggplot(data = titanic,
       aes(x = Pclass, fill = factor(Survived))) + geom_bar()
```

- **6b.** In `geom_bar` you can also change the `position` to `='fill'` or `='dodge'`. Do so for the above graphic and describe what these 2 variants do. Which do you think best answers the question in part **6a.** - that is "Does class (`Pclass`) effect if you survived or not?".


- **6c.** During the time of the titanic, there was a code of conduct called [Woman and Children First](https://www.wikiwand.com/en/Women_and_children_first). As seen in lecture, facet (hint: `facet_grid`) on the gender of the individual to see how gender relates to the interaction of class and survival (please use use the default `position`). Specifically try both `~Sex` and `Sex~.` - select the one that makes the relationship clearer? We can also add marginals by doing `+ facet_grid(...,margin = TRUE)` - try it.


- **6f** The `volcano `object in R is a matrix of dimension 87 x 61. It is a digitized version of a topographic map of the Maungawhau volcano in Auckland, New Zealand. To visualize in `ggplot` we need to do a slight transformation of the object into a different format (*we'll learn how `tidyr::gather` works next week*). Examine both `volcano` and the data.frame `my_volcano` created below. Express in words what data transformation occured. Next, using the `geom_tile` create an image of the volcano (hint: `fill = value`). Note that first row of the volcano matrix is actually to the row at the bottom of the outputted display - do a minor correction to the aesthetics for the row to rotate the image to the correct orientation.

```{r}
library(tidyr)
my_volcano <- tidyr::gather(data.frame(row = 1:nrow(volcano), volcano), 
                            key = "column", value = "value", -row)
my_volcano$column <- as.numeric(substr(my_volcano$column, 2, nchar(my_volcano$column))) 
```


Shakespeare and overlaid histograms (challenge)
===

- **7.** Return to the Shakespeare data set from lab/homework in Week 1 (taken from Project Gutenberg). Following the commands you worked out in lab/homework, extract the text for at least two of Shakespeare's plays. Then using `table()`, compute counts of the word lengths, separately, for each play you are considering. Produce a plot that displays histograms of the word lengths---i.e., one histogram for each play, overlaid. In the `geom` have `aes(..., y= ..density..)` (so that all histograms are on the probability scale, rather than the frequency scale). Set the title and axes labels appropriately. Use transparent colors. Describe any differences/similarities that you are seeing between plays, according to Shakepeare's word length useage.

*Notes: *
1. the standard for `geom_histogram`'s y is "`..count..`" .
2. `geom_histogram(position="identity")` makes stacked histograms non-stacked.

Reading (challenge 2)
===
Go to the Homework 2 assignment page and download and read "Tufte Chapter 9". This chapter is about making better graphics. Have you ever visualized something that could have been better off with done with a table of a few values? How about the "Unfriendly" graphic attributes on page 183? If you remember such a thing - how might you update it with the recommendations in class and this chapter? This book was written a while ago - what thoughts hold up (in your opinion) and which don't. That is: what seems less important to you than you expect Tufte would have thought (or visa versa).


