Matrix Algebra

<p>Part I Linear Algebra</p> <p>1 Basic Vector/Matrix Structure and Notation </p> <p>1.1 Vectors </p> <p>1.2 Arrays </p> <p>1.3 Matrices</p> <p>1.4 Representation of Data</p> <p>2 Vectors and Vector Spaces</p> <p>2.1 Operations on Vectors </p> <p>2.1.1 Linear Combinations and Linear Independence</p> <p>2.1.2 Vector Spaces and Spaces of Vectors</p> <p>2.1.3 Basis Sets for Vector Spaces</p> <p>2.1.4 Inner Products</p> <p>2.1.5 Norms</p> <p>2.1.6 Normalized Vectors</p> <p>2.1.7 Metrics and Distances </p> 2.1.8 Orthogonal Vectors and Orthogonal Vector Spaces<p></p> <p>2.1.9 The “One Vector”</p> <p>2.2 Cartesian Coordinates and Geometrical Properties of Vectors</p> <p>2.2.1 Cartesian Geometry</p> <p>2.2.2 Projections</p> <p>2.2.3 Angles between Vectors </p> <p>2.2.4 Orthogonalization Transformations; Gram-Schmidt .</p> <p>2.2.5 Orthonormal Basis Sets</p> <p>2.2.6 Approximation of Vectors </p> <p>2.2.7 Flats, Affine Spaces, and Hyperplanes</p> <p>2.2.8 Cones</p> <p>2.2.9 Cross Products in IR3</p> <p>2.3 Centered Vectors and Variances and Covariances of Vectors</p> <p>2.3.1 The Mean and Centered Vectors</p> 2.3.2 The Standard Deviation, the Variance, andScaled Vectors <p></p> <p>2.3.3 Covariances and Correlations between Vectors</p> <p>Exercises</p> <p>3 Basic Properties of Matrices </p> <p>3.1 Basic Definitions and Notation</p> <p>3.1.1 Matrix Shaping Operators</p> <p>3.1.2 Partitioned Matrices </p> <p>3.1.3 Matrix Addition</p> <p>3.1.4 Scalar-Valued Operators on Square Matrices:The Trace</p> <p>3.1.5 Scalar-Valued Operators on Square Matrices:The Determinant </p> <p>3.2 Multiplication of Matrices and Multiplication ofVectors and Matrices </p> <p>3.2.1 Matrix Multiplication (Cayley) </p> 3.2.2 Multiplication of Matrices with Special Patterns<p></p> <p>3.2.3 Elementary Operations on Matrices</p> <p>3.2.4 The Trace of a Cayley Product that Is Square</p> <p>3.2.5 The Determinant of a Cayley Product of Square Matrices </p> <p>3.2.6 Multiplication of Matrices and Vectors</p> <p>3.2.7 Outer Products</p> <p>3.2.8 Bilinear and Quadratic Forms; Definiteness </p> <p>3.2.9 Anisometric Spaces </p> <p>3.2.10 Other Kinds of Matrix Multiplication </p> <p>3.3 Matrix Rank and the Inverse of a Matrix </p> <p>3.3.1 The Rank of Partitioned Matrices, Products of Matrices, and Sums of Matrices</p> <p>3.3.2 Full Rank Partitioning </p> 3.3.3 Full Rank Matrices and Matrix Inverses <p></p> <p>3.3.4 Full Rank Factorization </p> <p>3.3.5 Equivalent Matrices</p> <p>3.3.6 Multiplication by Full Rank Matrices </p> <p>3.3.7 Gramian Matrices: Products of the Form ATA</p> <p>3.3.8 A Lower Bound on the Rank of a Matrix Product</p> <p>3.3.9 Determinants of Inverses </p> <p>3.3.10 Inverses of Products and Sums of Nonsingular Matrices </p> <p>3.3.11 Inverses of Matrices with Special Forms </p> <p>3.3.12 Determining the Rank of a Matrix </p> <p>3.4 More on Partitioned Square Matrices: The Schur Complement </p> <p>3.4.1 Inverses of Partitioned Matrices</p> <p>3.4.2 Determinants of Partitioned Matrices </p> 3.5 Linear Systems of Equations<p></p> <p>3.5.1 Solutions of Linear Systems </p> <p>3.5.2 Null Space: The Orthogonal Complement </p> <p>3.6 Generalized Inverses</p> <p>3.6.1 Special Generalized Inverses; The Moore-Penrose Inverse</p> <p>3.6.2 Generalized Inverses of Products and Sums of Matrices </p> <p>3.6.3 Generalized Inverses of Partitioned Matrices </p> <p>3.7 Orthogonality</p> <p>3.8 Eigenanalysis; Canonical Factorizations</p> <p>3.8.1 Basic Properties of Eigenvalues and Eigenvectors </p> <p>3.8.2 The Characteristic Polynomial </p> <p>3.8.3 The Spectrum </p> <p>3.8.4 Similarity Transformations </p> 3.8.5 Schur Factorization <p></p> <p>3.8.6 Similar Canonical Factorization; Diagonalizable Matrices</p> <p>3.8.7 Properties of Diagonalizable Matrices </p> <p>3.8.8 Eigenanalysis of Symmetric Matrices</p> <p>3.8.9 Positive Definite and Nonnegative Definite Matrices </p> <p>3.8.10 Generalized Eigenvalues and Eigenvectors</p> <p>3.8.11 Singular Values and the Singular Value Decomposition (SVD)</p> <p>3.9 Matrix Norms </p> <p>3.9.1 Matrix Norms Induced from Vector Norms </p> <p>3.9.2 The Frobenius Norm — The “Usual” Norm</p> <p>3.9.3 Other Matrix Norms</p> <p>3.9.4 Matrix Norm Inequalities</p> 3.9.5 The Spectral Radius<p></p> <p>3.9.6 Convergence of a Matrix Power Series</p> <p>3.10 Approximation of Matrices</p> <p>Exercises</p> <p>4 Vector/Matrix Derivatives and Integrals </p> <p>4.1 Basics of Differentiation</p> <p>4.2 Types of Differentiation </p> <p>4.2.1 Differentiation with Respect to a Scalar </p> <p>4.2.2 Differentiation with Respect to a Vector</p> <p>4.2.3 Differentiation with Respect to a Matrix</p> <p>4.3 Optimization of Scalar-Valued Functions</p> <p>4.3.1 Stationary Points of Functions</p> <p>4.3.2 Newton’s Method</p> <p>4.3.3 Least Squares</p> <p>4.3.4 Maximum Likelihood </p> <p>4.3.5 Optimization of Functions with Constraints </p> &lt;4.3.6 Optimization without Differentiation <p></p> <p>4.4 Integration and Expectation: Applications to Probability Distributions</p> <p>4.4.1 Multidimensional Integrals and Integrals InvolvingVectors and Matrices</p> <p>4.4.2 Integration Combined with Other Operations </p> <p>4.4.3 Random Variables and Probability Distributions</p> <p>Exercises</p> <p>5 Matrix Transformations and Factorizations</p> <p>5.1 Linear Geometric Transformations</p> <p>5.1.1 Transformations by Orthogonal Matrices </p> <p>5.1.2 Rotations </p> <p>5.1.3 Reflections </p> 5.1.4 Translations; Homogeneous Coordinates<p></p> <p>5.2 Householder Transformations (Reflections)</p> <p>5.3 Givens Transformations (Rotations)</p> <p>5.4 Factorization of Matrices </p> <p>5.5 LU and LDU Factorizations</p> <p>5.6 QR Factorization</p> <p>5.6.1 Householder Reflections to Form the QR Factorization </p> <p>5.6.2 Givens Rotations to Form the QR Factorization</p> <p>5.6.3 Gram-Schmidt Transformations to Form theQR Factorization</p> <p>5.7 Factorizations of Nonnegative Definite Matrices </p> <p>5.7.1 Square Roots</p> <p>5.7.2 Cholesky Factorization </p> <p>5.7.3 Factorizations of a Gramian Matrix</p> 5.8 Nonnegative Matrix Factorization<p></p> <p>5.9 Other Incomplete Factorizations</p> <p>Exercises</p> <p>6 Solution of Linear Systems</p> <p>6.1 Condition of Matrices</p> <p>6.1.1 Condition Number</p> <p>6.1.2 Improving the Condition Number</p> <p>6.1.3 Numerical Accuracy </p> <p>6.2 Direct Methods for Consistent Systems</p> <p>6.2.1 Gaussian Elimination and Matrix Factorizations </p> <p>6.2.2 Choice of Direct Method</p> <p>6.3 Iterative Methods for Consistent Systems</p> <p>6.3.1 The Gauss-Seidel Method withSuccessive Overrelaxation</p> 6.3.2 Conjugate Gradient Methods for SymmetricPositive Definite Systems<p></p> <p>6.3.3 Multigrid Methods</p> <p>6.4 Iterative Refinement</p> <p>6.5 Updating a Solution to a Consistent System</p> <p>6.6 Overdetermined Systems; Least Squares </p> <p>6.6.1 Least Squares Solution of an Overdetermined System </p> <p>6.6.2 Least Squares with a Full Rank Coefficient Matrix</p> <p>6.6.3 Least Squares with a Coefficient MatrixNot of Full Rank</p> <p>6.6.4 Updating a Least Squares Solution of anOverdetermined System</p> <p>6.7 Other Solutions of Overdetermined Systems</p> <p>6.7.1 Solutions that Minimize Other Norms of the Residuals </p> <p>6.7.2 Regularized Solutions</p> 6.7.3 Minimizing Orthogonal Distances <p></p> <p>Exercises</p> <p>7 Evaluation of Eigenvalues and Eigenvectors</p> <p>7.1 General Computational Methods</p> <p>7.1.1 Numerical Condition of an Eigenvalue Problem</p> <p>7.1.2 Eigenvalues from Eigenvectors and Vice Versa </p> <p>7.1.3 Deflation</p> <p>7.1.4 Preconditioning</p> <p>7.1.5 Shifting</p> <p>7.2 Power Method</p> <p>7.3 Jacobi Method </p> <p>7.4 QR Method</p> <p>7.5 Krylov Methods</p> <p>7.6 Generalized Eigenvalues</p> <p>7.7 Singular Value Decomposition</p> <p>Exercises</p> Part II Applications in Data Analysis<p></p> <p>8 Special Matrices and Operations Useful in Modeling andData Analysis</p> <p>8.1 Data Matrices and Association Matrices</p> <p>8.1.1 Flat Files </p> <p>8.1.2 Graphs and Other Data Structures</p> <p>8.1.3 Term-by-Document Matrices</p> <p>8.1.4 Probability Distribution Models </p> <p>8.1.5 Derived Association Matrices</p> <p>8.2 Symmetric Matrices and Other Unitarily Diagonalizable Matrices</p> <p>8.2.1 Some Important Properties of Symmetric Matrices</p> <p>8.2.2 Approximation of Symmetric Matrices and an Important Inequality</p> <p>8.2.3 Normal Matrices </p> 8.3 Nonnegative Definite Matrices; Cholesky Factorization <p></p> <p>8.4 Positive Definite Matrices</p> <p>8.5 Idempotent and Projection Matrices</p> <p>8.5.1 Idempotent Matrices </p> <p>8.5.2 Projection Matrices: Symmetric Idempotent Matrices </p> <p>8.6 Special Matrices Occurring in Data Analysis </p> <p>8.6.1 Gramian Matrices </p> <p>8.6.2 Projection and Smoothing Matrices</p> <p>8.6.3 Centered Matrices and Variance-Covariance Matrices </p> <p>8.6.4 The Generalized Variance </p> <p>8.6.5 Similarity Matrices</p> <p>8.6.6 Dissimilarity Matrices</p> <p>8.7 Nonnegative and Positive Matrices </p> 8.7.1 Properties of Square Positive Matrices<p></p> <p>8.7.2 Irreducible Square Nonnegative Matrices </p> <p>8.7.3 Stochastic Matrices</p> <p>8.7.4 Leslie Matrices</p> <p>8.8 Other Matrices with Special Structures</p> <p>8.8.1 Helmert Matrices </p> <p>8.8.2 Vandermonde Matrices</p> <p>8.8.3 Hadamard Matrices and Orthogonal Arrays</p> <p>8.8.4 Toeplitz Matrices </p> <p>8.8.5 Circulant Matrices</p> <p>8.8.6 Fourier Matrices and the Discrete Fourier Transform </p> <p>8.8.7 Hankel Matrices</p> <p>8.8.8 Cauchy Matrices</p> <p>8.8.9 Matrices Useful in Graph Theory</p> <p>8.8.10 M-Matrices</p> Exercises<p></p> <p>9 Selected Applications in Statistics</p> <p>9.1 Multivariate Probability Distributions</p> <p>9.1.1 Basic Definitions and Properties</p> <p>9.1.2 The Multivariate Normal Distribution</p> <p>9.1.3 Derived Distributions and Cochran’s Theorem</p> <p>9.2 Linear Models</p> <p>9.2.1 Fitting the Model</p> <p>9.2.2 Linear Models and Least Squares</p> <p>9.2.3 Statistical Inference</p> <p>9.2.4 The Normal Equations and the Sweep Operator </p> <p>9.2.5 Linear Least Squares Subject to LinearEquality Constraints</p> <p>9.2.6 Weighted Least Squares </p> 9.2.7 Updating Linear Regression Statistics<p></p> <p>9.2.8 Linear Smoothing</p> <p>9.2.9 Multivariate Linear Models</p> <p>9.3 Principal Components</p> <p>9.3.1 Principal Components of a Random Vector</p> <p>9.3.2 Principal Components of Data</p> <p>9.4 Condition of Models and Data </p> <p>9.4.1 Ill-Conditioning in Statistical Applications</p> <p>9.4.2 Variable Selection</p> <p>9.4.3 Principal Components Regression </p> <p>9.4.4 Shrinkage Estimation</p> <p>9.4.5 Statistical Inference about the Rank of a Matrix</p> <p>9.4.6 Incomplete Data</p> <p>9.5 Optimal Design</p> 9.6 Multivariate Random Number Generation <p></p> <p>9.7 Stochastic Processes </p> <p>9.7.1 Markov Chains </p> <p>9.7.2 Markovian Population Models</p> <p>9.7.3 Autoregressive Processes </p> <p>Exercises</p> <p>Part III Numerical Methods and Software</p> <p>10 Numerical Methods</p> <p>10.1 Digital Representation of Numeric Data </p> <p>10.1.1 The Fixed-Point Number System </p> <p>10.1.2 The Floating-Point Model for Real Numbers</p> <p>10.1.3 Language Constructs for Representing Numeric Data</p> <p>10.1.4 Other Variations in the Representation of Data;Portability of Data</p> <p>10.2 Computer Operations on Numeric Data </p> 10.2.1 Fixed-Point Operations <p></p> <p>10.2.2 Floating-Point Operations </p> <p>10.2.3 Exact Computations </p> <p>10.2.4 Language Constructs for Operations onNumeric Data</p> <p>10.3 Numerical Algorithms and Analysis </p> <p>10.3.1 Error in Numerical Computations</p> <p>10.3.2 Efficiency </p> <p>10.3.3 Iterations and Convergence</p> &lt;10.3.4 Other Computational Techniques <p></p> <p>Exercises </p> <p>11 Numerical Linear Algebra</p> <p>11.1 Computer Representation of Vectors and Matrices </p> <p>11.2 General Computational Considerations forVectors and Matrices</p> 11.2.1 Relative Magnitudes of Operands <p></p> <p>11.2.2 Iterative Methods </p> <p>11.2.3 Assessing Computational Errors </p> <p>11.3 Multiplication of Vectors and Matrices</p> <p>11.4 Other Matrix Computations</p> <p>Exercises</p> <p>12 Software for Numerical Linear Algebra</p> <p>12.1 General Considerations</p> <p>12.2 Libraries</p> <p>12.2.1 BLAS</p> <p>12.2.2 Level 2 and Level 3 BLAS and Related Libraries </p> <p>12.2.3 Libraries for High Performance Computing </p> <p>12.2.4 Matrix Storage Modes</p> <p>12.2.5 Language-Specific Libraries</p> <p>12.2.6 The IMSLTM Libraries</p> 12.3 General Purpose Languages<p></p> <p>12.3.1 Programming Considerations </p> <p>12.3.2 Modern Fortran</p> <p>12.3.3 C and C++ </p> <p>12.3.4 Python &lt;12.4 Interactive Systems for Array Manipulation </p><p></p> <p>12.4.1 R </p> <p>12.4.2 MATLABR and Octave</p> <p>12.4.3 Other Systems</p> <p>12.5 Software for Statistical Applications</p> <p>12.6 Test Data</p> <p>Exercises</p> <p>Appendices and Back Matter</p> <p>A Notation and Definitions </p> <p>A.1 General Notation </p> <p>A.2 Computer Number Systems</p> A.3 General Mathematical Functions and Operators<p></p> <p>A.4 Linear Spaces and Matrices</p> <p>A.5 Models and Data</p> <p>B Solutions and Hints for Selected Exercises</p> <p>Bibliography</p> <p>Index</p>