Appendix G — Solutions to Selected Exercises

Warning
  • Solutions are incomplete and need to be corrected!
  • They serve as a starting point for the final solution.

G.1 Data-Driven Modeling and Optimization

G.1.1 Histograms

Solution G.1 (Density Curve).

  • We can calculate propabilities.
  • We only need two parameters (the mean and the sd) to form the curve -> Store data more efficently
  • Blanks can be filled

G.1.2 The Normal Distribution

Solution G.2 (TwoSDAnswer). 95%

Solution G.3 (OneSDAnswer). 68%

Solution G.4 (ThreeSDAnswer). 99,7%

Solution G.5 (DataRangeAnswer). 80 - 120

Solution G.6 (PeakHeightAnswer). low

G.1.3 The mean, the media, and the mode

G.1.4 The exponential distribution

G.1.5 Population and Estimated Parameters

Solution G.7 (ProbabilityAnswer). 50%

G.1.6 Calculating the Mean, Variance and Standard Deviation

Solution G.8 (MeanDifferenceAnswer). If we have all the data, \(\mu\) is the population mean and x-bar is the sample mean. We don’t have the full information.

Solution G.9 (EstimateMeanAnswer). Sum of the values divided by n.

Solution G.10 (SigmaSquaredAnswer). Variance

Solution G.11 (EstimatedSDAnswer). The same as the normal standard deviation, but using n-1.

Solution G.12 (VarianceDifferenceAnswer). \(n\) and \(n-1\)

Solution G.13 (ModelBenefitsAnswer).

  • Approximation
  • Prediction
  • Understanding

Solution G.14 (SampleDefinitionAnswer). It’s a subset of the data.

G.1.7 Hypothesis Testing and the Null-Hypothesis

Solution G.15 (RejectHypothesisAnswer). It means the evidence supports the alternative hypothesis, indicating that the null hypothesis is unlikely to be true.

Solution G.16 (NullHypothesisAnswer). It’s a statement that there is no effect or no difference, and it serves as the default or starting assumption in hypothesis testing.

Solution G.17 (BetterDrugAnswer). By conducting experiments and statistical tests to compare the new drug’s effectiveness against the current standard and demonstrating a significant improvement.

G.1.8 Alternative Hypotheses, Main Ideas

G.1.9 p-values: What they are and how to interpret them

Solution G.18 (PValueIntroductionAnswer). We can reject the null hypothesis. We can make a decision.

Solution G.19 (PValueRangeAnswer). It can only be between 0 and 1.

Solution G.20 (PValueRangeAnswer). It can only be between 0 and 1.

Solution G.21 (TypicalPValueAnswer). The chance that we wrongly reject the null hypothesis.

Solution G.22 (FalsePositiveAnswer). If we have a false-positive, we succeed in rejecting the null hypothesis. But in fact/reality, this is false -> False positive.

G.1.10 How to calculate p-values

Solution G.23 (CalculatePValueAnswer). Probability of specific result, probability of outcome with the same probability, and probability of events with smaller probability.

Solution G.24 (SDCalculationAnswer). 7 is the SD.

Solution G.25 (SidedPValueAnswer). If we are not interested in the direction of the change, we use the two-sided. If we want to know about the direction, the one-sided.

Solution G.26 (CoinTestAnswer). TBD

Solution G.27 (BorderPValueAnswer). TBD

Solution G.28 (OneSidedPValueCautionAnswer). If you look in the wrong direction, there is no change.

Solution G.29 (BinomialDistributionAnswer). TBD

G.1.11 p-hacking: What it is and how to avoid it

Solution G.30 (PHackingWaysAnswer).

  • Performing repeats until you find one result with a small p-value -> false positive result.
  • Increasing the sample size within one experiment when it is close to the threshold.

Solution G.31 (AvoidPHackingAnswer). Specify the number of repeats and the sample sizes at the beginning.

Solution G.32 (MultipleTestingProblemAnswer). TBD

G.1.12 Covariance

Solution G.33 (CovarianceDefinitionAnswer). Formula

Solution G.34 (CovarianceMeaningAnswer). Large values in the first variable result in large values in the second variable.

Solution G.35 (CovarianceVarianceRelationshipAnswer). Formula

Solution G.36 (HighCovarianceAnswer). No, size doesn’t matter.

Solution G.37 (ZeroCovarianceAnswer). No relationship

Solution G.38 (NegativeCovarianceAnswer). Yes

Solution G.39 (NegativeVarianceAnswer). No

G.1.13 Pearson’s Correlation

Solution G.40 (CorrelationValueAnswer). Recalculate

Solution G.41 (CorrelationRangeAnswer). From -1 to 1

Solution G.42 (CorrelationFormulaAnswer). Formula

G.1.14 Boxplots

Solution G.43 (UnderstandingStatisticalPower). It is the probability of correctly rejecting the null hypothesis.

Solution G.44 (DistributionEffectOnPower). Power analysis is not applicable.

Solution G.45 (IncreasingPower). By taking more samples.

Solution G.46 (PreventingPHacking). TBD

Solution G.47 (SampleSizeAndPower). The power will be low.

G.1.15 Power Analysis

Solution G.48 (MainFactorsAffectingPower). The overlap (distance of the two means) and sample sizes.

Solution G.49 (PowerAnalysisOutcome). The sample size needed.

Solution G.50 (RisksInExperiments). Few experiments lead to very low power, and many experiments might result in p-hacking.

Solution G.51 (StepsToPerformPowerAnalysis).

  1. Select power
  2. Select threshold for significance (alpha)
  3. Estimate the overlap (done by the effect size)

G.1.16 The Central Limit Theorem

Solution G.52 (CentralLimitTheoremAnswer). TBD

G.1.17 Boxplots

Solution G.53 (MedianAnswer). The median.

Solution G.54 (BoxContentAnswer). 50% of the data.

G.1.18 R-squared

Solution G.55 (RSquaredFormulaAnswer). TBD

Solution G.56 (NegativeRSquaredAnswer). If you fit a line, no, but there are cases where it could be negative. However, these are usually considered useless.

Solution G.57 (RSquaredCalculationAnswer). TBD

G.1.18.1 The main ideas of fitting a line to data (The main ideas of least squares and linear regression.)

Solution G.58 (LeastSquaresAnswer). It is the calculation of the smallest sum of residuals when you fit a model to data.

G.1.19 Linear Regression

G.1.20 Multiple Regression

G.1.21 A Gentle Introduction to Machine Learning

Solution G.59 (RegressionVsClassificationAnswer). Regression involves predicting continuous values (e.g., temperature, size), while classification involves predicting discrete values (e.g., categories like cat, dog).

G.1.22 Maximum Likelihood

Solution G.60 (LikelihoodConceptAnswer). The distribution that fits the data best.

G.1.23 Probability is not Likelihood

Solution G.61 (ProbabilityVsLikelihoodAnswer). Likelihood: Finding the curve that best fits the data. Probability: Calculating the probability of an event given a specific curve.

G.1.24 Cross Validation

Solution G.62 (TrainVsTestDataAnswer). Training data is used to fit the model, while testing data is used to evaluate how well the model fits.

Solution G.63 (SingleValidationIssueAnswer). The performance might not be representative because the data may not be equally distributed between training and testing sets.

Solution G.64 (FoldDefinitionAnswer). TBD

Solution G.65 (LeaveOneOutValidationAnswer). Only one data point is used as the test set, and the rest are used as the training set.

G.1.25 The Confusion Matrix

Solution G.66 (ConfusionMatrixAnswer). TBD

G.1.26 Sensitivity and Specificity

Solution G.67 (SensitivitySpecificityAnswer1). TBD

Solution G.68 (SensitivitySpecificityAnswer2). TBD

G.1.27 Bias and Variance

Solution G.69 (BiasAndVarianceAnswer). TBD

G.1.28 Mutual Information

Solution G.70 (MutualInformationExampleAnswer). TBD

G.1.29 Principal Component Analysis (PCA)

Solution G.71 (WhatIsPCAAnswer). A dimension reduction technique that helps discover important variables.

Solution G.72 (screePlotAnswer). It shows how much variation is defined by the data.

Solution G.73 (LeastSquaresInPCAAnswer). No, in the first step it tries to maximize distances.

Solution G.74 (PCAStepsAnswer).

  1. Calculate mean
  2. Shift the data to the center of the coordinate system
  3. Fit a line by maximizing the distances
  4. Calculate the sum of squared distances
  5. Calculate the slope
  6. Rotate

Solution G.75 (EigenvaluePC1Answer). Formula (to be specified).

Solution G.76 (DifferencesBetweenPointsAnswer). No, because the first difference is measured on the PC1 scale and it is more important.

Solution G.77 (ScalingInPCAAnswer). Scaling by dividing by the standard deviation (SD).

Solution G.78 (DetermineNumberOfComponentsAnswer). TBD

Solution G.79 (LimitingNumberOfComponentsAnswer).

  1. The dimension of the problem
  2. Number of samples

G.1.30 t-SNE

Solution G.80 (WhyUseTSNEAnswer). For dimension reduction and picking out the relevant clusters.

Solution G.81 (MainIdeaOfTSNEAnswer). To reduce the dimensions of the data by reconstructing the relationships in a lower-dimensional space.

Solution G.82 (BasicConceptOfTSNEAnswer).

  1. First, randomly arrange the points in a lower dimension
  2. Decide whether to move points left or right, depending on distances in the original dimension
  3. Finally, arrange points in the lower dimension similarly to the original dimension

Solution G.83 (TSNEStepsAnswer).

  1. Project data to get random points
  2. Set up a matrix of distances
  3. Calculate the inner variances of the clusters and the Gaussian distribution
  4. Do the same with the projected points
  5. Move projected points so the second matrix gets more similar to the first matrix

G.1.31 K-means clustering

Solution G.84 (HowKMeansWorksAnswer).

  1. Select the number of clusters
  2. Randomly select distinct data points as initial cluster centers
  3. Measure the distance between each point and the cluster centers
  4. Assign each point to the nearest cluster
  5. Repeat the process

Solution G.85 (QualityOfClustersAnswer). Calculate the within-cluster variation.

Solution G.86 (IncreasingKAnswer). If k is too high, each point would be its own cluster. If k is too low, you cannot see the structures.

G.1.32 DBSCAN

Solution G.87 (CorePointInDBSCANAnswer). A point that is close to at least k other points.

Solution G.88 (AddingVsExtendingAnswer). Adding means we add a point and then stop. Extending means we add a point and then look for other neighbors from that point.

Solution G.89 (OutliersInDBSCANAnswer). Points that are not core points and do not belong to existing clusters.

G.1.33 K-nearest neighbors

Solution G.90 (AdvantagesAndDisadvantagesOfKAnswer).

  • k = 1: Noise can disturb the process because of possibly incorrect measurements of points.
  • k = 100: The majority can be wrong for some groups. It is smoother, but there is less chance to discover the structure of the data.

G.1.34 Naive Bayes

Solution G.91 (NaiveBayesFormulaAnswer). TBD

Solution G.92 (CalculateProbabilitiesAnswer). TBD

G.1.35 Gaussian Naive Bayes

Solution G.93 (UnderflowProblemAnswer). Small values multiplied together can become smaller than the limits of computer memory, resulting in zero. Using logarithms (e.g., log(1/2) -> -1, log(1/4) -> -2) helps prevent underflow.

G.1.36 Trees

Solution G.94 (Tree Usage). Classication, Regression, Clustering

Solution G.95 (Tree Usage). TBD

Solution G.96 (Tree Feature Importance). The most important feature.

Solution G.97 (Regression Tree Limitations). High dimensions

Solution G.98 (Regression Tree Score). SSR + alpha * T

Solution G.99 (Regression Tree Alpha Value Small). The tree is more complex.

Solution G.100 (Regression Tree Increase Alpha Value). We get smaller trees

Solution G.101 (Regression Tree Pruning). Decreases the complexity of the tree to enhance performance and reduce overfitting

G.2 Machine Learning and Artificial Intelligence

G.2.1 Backpropagation

Solution G.102 (ChainRuleAndGradientDescentAnswer). Combination of the chain rule and gradient descent.

Solution G.103 (BackpropagationNamingAnswer). Because you start at the end and go backwards.

G.2.2 Gradient Descent

Solution G.104 (GradDescStepSize). learning rate x slope

Solution G.105 (GradDescIntercept). Old intercept - step size

Solution G.106 (GradDescIntercept). When the step size is small or after a certain number of steps

G.2.3 ReLU

Solution G.107 (Graph ReLU). Graph of ReLU function: f(x) = max(0, x)

G.2.4 CNNs

Solution G.108 (CNNImageRecognitionAnswer).

  • too many features for input layer -> high memory consumption
  • always shift in data
  • it learns local informations and local correlations

Solution G.109 (CNNFiltersInitializationAnswer). The filter values in CNNs are randomly initialized and then trained and optimized through the process of backpropagation.

Solution G.110 (CNNFilterInitializationAnswer). The filter values in CNNs are initially set by random initialization. These filters undergo training via backpropagation, where gradients are computed and used to adjust the filter values to optimize performance.

Solution G.111 (GenNNStockPredictionAnswer). A limitation of using classical neural networks for stock market prediction is their reliance on fixed inputs. Stock market data is dynamic and requires models that can adapt to changing conditions over time.

G.2.5 RNN

Solution G.112 (RNNUnrollingAnswer). In the unrolling process of RNNs, the network is copied and the output from the inner loop is fed into the second layer of the copied network.

Solution G.113 (RNNReliabilityAnswer). RNNs sometimes fail to work reliably due to the vanishing gradient problem (where gradients are less than 1) and the exploding gradient problem (where gradients are greater than 1). Additionally, reliability issues arise because the network and the weights are copied during the unrolling process.

G.2.6 LSTM

Solution G.114 (LSTMSigmoidTanhAnswer). The sigmoid activation function outputs values between 0 and 1, making it suitable for probability determination, whereas the tanh activation function outputs values between -1 and 1.

Solution G.115 (LSTMSigmoidTanhAnswer). State how much of the long term memory should be used.

Solution G.116 (LSTMGatesAnswer). An LSTM network has three types of gates: the forget gate, the input gate, and the output gate. The forget gate decides what information to discard from the cell state, the input gate updates the cell state with new information, and the output gate determines what part of the cell state should be output.

Solution G.117 (LSTMLongTermInfoAnswer). Long-term information is used in the output gate of an LSTM network.

Solution G.118 (LSTMUpdateGatesAnswer). In the input and forget gates.

G.2.7 Pytorch/Lightning

Solution G.119 (PyTorchRequiresGradAnswer). In PyTorch, requires_grad indicates whether a tensor should be trained. If set to False, the tensor will not be trained.

G.2.8 Embeddings

Solution G.120 (NN STrings). No, they process numerical values.

Solution G.121 (Embedding Definition). Representation of a word as a vector.

Solution G.122 (Embedding Dimensions). We can model similarities.

G.2.9 Sequence to Sequence Models

Solution G.123 (LSTM). Because they are able to consider “far away” information.

Solution G.124 (Teacher Forcing). We need to force the correct words for the training.

Solution G.125 (Attention). Attention scores compute similarities for one input to the others.

G.2.10 Transformers

Solution G.126 (ChatGPT). Decoder only.

Solution G.127 (Translation). Encoder-Decoder structure.

Solution G.128 (Difference Encoder-Decoder and Decoder Only.).

  • Encoder-Decoder: self-attention.
  • Decoder only: masked self-attention.

Solution G.129 (Weights).

  • a: Randomly
  • b: Backpropagation

Solution G.130 (Order of Words). Positional Encoding

Solution G.131 (Relationship Between Words). Masked self-attention which looks at the previous tokens.

Solution G.132 (Masked Self Attention). It works by investigating how similar each word is to itself and all of the proceeding words in the sentence.

Solution G.133 (Softmax). Transformation to values between 0 and 1.

Solution G.134 (Softmax Output). We create two new numbers: Values – like K and Q with different weights. We scale these values by the percentage. -> we get the scaled V´s

Solution G.135 (V´s). Lastly, we sum these values together, which combine separate encodings for both words relative to their similarities to “is”, are the masked-self-attention values for “is”.

Solution G.136 (Residual Connections). They are bypasses, which combine the position encoded values with masked-self-attention values.

Solution G.137 (Generate Known Word in Sequence).

  • Training
  • Because it is a Decoder-Only transformer used for prediction and the calculations that you need.

Solution G.138 (Masked-Self-Attention Values and Bypass). We use a simple neural network with two inputs and five outputs for the vocabulary.