DOMAIN DECOMPOSITION TECHNIQUES AND DISTRIBUTED PROGRAMMING IN COMPUTATIONAL FLUID DYNAMICS

Transcripción

1 DOMAIN DECOMPOSITION TECHNIQUES AND DISTRIBUTED PROGRAMMING IN COMPUTATIONAL FLUID DYNAMICS by Rodrigo Rafael Paz A dissertation submitted to the Postgraduate Department of the FACULTAD DE INGENIERÍA Y CIENCIAS HÍDRICAS for partial fulfillment of the requirements for the degree of DOCTOR IN ENGINEERING Field of Computational Mechanics of the UNIVERSIDAD NACIONAL DEL LITORAL 2006

2

3 A Eliana y Guadalupe, a mis Padres Néstor y Susana, a mi Hermana Lici y a la memoria de mi Abuelo Agustín.

4

5 Acknowledgments I will always be in debt to Mario Storti for his advise, support and kindly guide during the elaboration of this thesis at CIMEC. I have had the privilege to work and teach with him along these years. Also, I would like to remark that Mario gives special and dedicated support to every PhD student and researcher at CIMEC laboratory. He can stay by your side (stuck on a chair) for hours discussing an idea or debugging a (frequently unfriendly) code as he would be the writer. I would like to express my deepest appreciation to Prof. Sergio Idelsohn for constant encouragement. Prof. Idelsohn has given me special participation in one of the most important projects in which the CIMEC laboratory has been involved. Special thanks to Norberto Nigro for very insightful discussions and intense collaboration. Beto had always been interested on my work. The research documented in this dissertation has been supported by Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET), the national council for the research in Argentina. I would like to name Professor Vitoriano Ruas from the Laboratoire de Modélisation en Mécanique, Université Pierre et Marie Curie (Paris VI) and Professor Grigori Panasenko from the Equipe d Analyse Numérique, Université de Saint-Etienne, who gave me special support during my stage at Paris and Saint-Etienne. It has been very fruitful to work with them. I am grateful to Carlos Méndez for revising the manuscript of this thesis, for his useful advices and for the amusing conversations at the river shore in Santa Fe while eating the well-known choris. To all my friends at CIMEC. I have had wonderful days working with them in an enlightening environment. To my friends, always. Finally, I would like to express my deepest thanks to Eliana and Guadalupe, for being always with me, for the love, forbearance and unconditional support. To my father and mother, Néstor and Susana, and to my sister Lici. They have always taught me the importance of studying and the freedom that a person needs to do what he believes. To my grandfather Agustín, I have spent my happiest days with him. My family is the energy that moves me through life. Deo Gratias.

6

7 Agradecimientos Estoy inmensamente agradecido a Mario Storti por su dirección, guía, dedicación y ayuda; dejándome ir siempre en la dirección en que me sentía con mayor confianza. También por haber confiado en mí para dar clases junto a él en la facultad. A Sergio Idelsohn por creer en mí y darme participación dentro de los proyectos en que trabajé durante la tesis. A Norberto Nigro por las discusiones y charlas, y por haberse interesado siempre en lo que estaba trabajando. Quiero agradecerle a CONICET por su programa de soporte para las carreras de doctorado. Un agradecimiento especial al Profesor Vitoriano Ruas del Laboratoire de Modélisation en Mécanique, Université Pierre et Marie Curie (Paris VI) y al Profesor Grigori Panasenko del Equipe d Analyse Numérique, Université de Saint-Etienne, por el inmenso apoyo brindado durante mi estadía en Paris y Saint-Etienne. El trabajo con ellos fue muy enriquecedor. Agradezco a Carlos Méndez por la revisión del manuscrito de esta tesis y por las excelentes discusiones que hemos tenido a lo largo de estos años. A todos los amigos del CIMEC con los que aprendí y me divertí en un ambiente muy grato. A mis amigos, siempre. Finalmente quiero agradecer a Eliana y a Guadalupe por estar a mi lado siempre, por el cariño y apoyo incondicional y la paciencia grande que tienen. A mis Padres Néstor y Susana y a mi Hermana Lici por haberme inculcado la importancia del estudio y la libertad que una persona necesita para hacer lo que cree. A mi abuelo Agustín que siempre me hizo feliz. Ellos son el motor indispensable para seguir siempre adelante. Deo Gratias.

8

9 Author s Legal Declaration This dissertation have been submitted to the Postgraduate Department of the Facultad de Ingeniería y Ciencias Hídricas for partial fulfillment of the requirements for the degree of Doctor in Engineering - Field of Computational Mechanics of the Universidad Nacional del Litoral. A copy of this document will be available at the University Library and it will be subjected to the Library s legal normative. Some parts of the work presented in this thesis have been (or are going to be) published in the following journals: International Journal for Numerical Methods in Engineering, International Journal for Numerical Methods in Fluids, Journal of Parallel and Distributed Computing, Journal of Sound and Vibration, Journal of Computational Methods in Science and Engineering and Journal of Computational Physics. Any comment about the ideas and topics discussed and developed through this document will be highly appreciated. Rodrigo Rafael PAZ c Copyright by Rodrigo Rafael PAZ 2006 All Rights Reserved

10

11 Introduction The large spread in length and time scales present in Computational Fluid Dynamics (CFD) problems and its interaction with solid or elastic bodies (e.g., coupled surfacesubsurface flows, high speed wind flows around complex bodies, non-linear fluid-structure interactions) requires a high degree of refinement in the finite element mesh and, then, requires very large computational resources. The solution of Large Scale CFD problems has a particular challenge that is the efficient use of the allowable computational resources [LTV97, SP96]. If no suitable numerical techniques are used to reduce, optimize and/or simplify the problem at hand it may be necessary to increase computational resources in order to handle the problem. Newer technologies and even faster and powerful (super-)computers make that the problems to be resolved be even larger and complex (i.e., larger domains, larger number of degree of freedom s (dof s), models with increasing number of evolution variables, coupled interacting fields). That is why the mathematical models used nowadays could be more complex and complete (from a physical point of view) making the simulations extensive and complicated. The constraint on the allowable computer resources is always present and that is the reason of the urgent development and verification of solution techniques that exploit efficiently the potential of new computers and the possibility to obtain solutions of high quality [PNS06] in a affordable simulation time (i.e., CPU time). This thesis has been conceived based on that fact. During last decades there have been developed and tested a wide diversity of linear system solvers and they have been applied to the resolution of real world physics problems by means of the discretization of coupled (or not) sets of non-linear Partial Differential Equations (PDE s) via the Finite Element Method (FEM), the Finite Difference Method (FDM) and/or Finite Volume Method (FVM). Since no long time ago (even at present days), it has been preferred the direct solution of these systems of equations instead of iterative schemes due to its higher robustness and its predictable behavior. Nevertheless, the increasing number of iterative techniques and the proposed improvements, jointly with the need to solve larger problems in Computational Mechanics area have bent to the use xi

12 xii of this kind of schemes and the development of newer ones. This trend has been taking place since early seventies when two crucial developments marked up an inflection point in the solution techniques of large scale systems of equations. On of these was the advantage of using low density or low sparsity matrices that arise in FEM (as well as in FDM and FVM) when discretized PDE s are stated. The other was the development of iterative methods over the Krylov space (or sub-space) such as Conjugate Gradients (CG) and Generalized Minimal Residuals (GMRes) [SS86, Saa00]. Gradually, iterative methods (and their variants such as preconditioning) have attained popularity and have begun to be extensively used by the scientific community and software developers. Particularly, there is a vast amount of written work on the use of CG methods for solving large scale coercive systems such as those resulting from the discretization of linear elastic problem, potential flows and heat conduction, among others. Nowadays, very large scale systems that arise in the context of FEM treatment of nonlinear transient governing equations are solved in high performance computers (parallel and vectorized architectures) by means of iterative methods due to less requirements in communications between processor than those required by direct methods (such as LU decomposition, multifrontal methods). The iterative Substructuring Method, or Domain Decomposition Method (DDM) and iteration over the Schur Complement Matrix on non-overlapping sub-domains, lead to a reduced system better suited (i.e., lower condition number κ(a) and better eigenvalue distribution) for Krylov-based iterative solution than the global solution. In Schur complement domain decomposition general method, the condition number is lowered ( 1/h vs 1/h 2 for the global system, h being the mesh size) and the computational cost per iteration is not so high once the sub-domain matrices have been factorized. Iterative substructuring methods rely on a non-overlapping partition into sub-domains (substructures). The efficiency of these methods can be further improved by using preconditioners [LTV97]. Once the degrees of freedom inside the substructures have been eliminated by block Gaussian elimination (or other algorithm), a preconditioner for the resulting Schur complement system is built with matrix blocks relative to a decomposition of interface finite element functions into subspaces related to geometrical objects (vertices, edges, faces, single substructures) or simply by the coefficients of sub-domain matrices near the interface. Iterative methods like CG and GMRes are then employed. Early works, such as [BPS86, BPS89], have influenced most of the later work in the field. They proposed two spaces for the coarse problem. One of their coarse spaces is given in terms of the averages of the nodal values over the entire substructure boundaries Ω i. The other space is defined by extending the wire basket (we recall that the wire basket is

13 xiii the union of the boundaries of the faces which separate the substructures) values as a two dimensional discrete harmonic function onto the faces, and then as a discrete harmonic function into the interiors of the sub-domains. For self-adjoint positive semidefinite problems, Neumann-Neumann preconditioner is the most classical one. From a mathematical point of view, the preconditioner is defined by approximating the inverse of the global Schur complement matrix by the weighted sum of local Schur complement matrices. From a physical point of view, Neumann-Neumann preconditioner is based on splitting the flux applied to the interface in the preconditioning step and solving local Neumann problems in each sub-domain. This strategy is good only for symmetric operators. Another family of DDM, the overlapping Schwarz domain decomposition schemes, have also been extensively used in computational mechanics. A good introduction and applications of these methods is presented by Smith and coworkers in Reference [SBrG96]. In the CFD area, Rachowicz [Rac97] applied successfully the GMRes solver with a domain decomposition Schwarz-type preconditioner in the solution of hypersonic high Reynolds number flows with strong shock-boundary layer interaction. The main purpose of the present thesis work is the efficient solution of large scale problems arising in Computational Fluid Dynamics challenge problems, the proposition of new ideas in preconditioning techniques, the implementation of such ideas in a parallel multiphysics C++ code using the message passing paradigm via MPI/PETSc libraries [GLS94, BGCMS04] and its evaluation on a Beowulf class cluster [SSBS99]. These topics are presented in the first part of this work. The second part is devoted to the application of the algorithm proposed in the first part ( I) to the solution of more general/complex problems like the wave absorption on fictitious boundaries and the resolution of fluid-structure problems in the supersonic regime of a compressible fluid flow.

14

15 Introducción La diversidad de escalas de tiempo y de espacio presentes en problemas relacionados con la mecánica de fluidos y su interacción con cuerpos sólidos (e.g., problemas de la hidrología superficial y subterránea acoplados o no, flujo de viento alrededor de cuerpos, edificios o vehículos, etc.) requiere un alto grado de refinamiento en las mallas utilizadas en el método de elementos finitos, y por lo tanto, demanda grandes recursos computacionales. La solución de problemas en gran escala en la mecánica computacional tiene un desafío particular y es el de utilizar eficientemente los recursos disponibles [LTV97, SP96]. Si no se utilizan adecuadas técnicas numéricas para reducir, optimizar y/o simplificar el problema, es menester contar con grandes recursos computacionales para tratar el problema. Por otro lado, el auge de computadoras cada vez más rápidas y con mayor capacidad de cálculo hace que los problemas que se quieren resolver sean cada vez más grandes y complejos (i.e., mayores y más variadas escalas, acople de distintos campos, modelos que tengan en cuenta otras variables y su evolución e interacción con las demás, etc.). Es así que los modelos matemáticos son cada vez más complejos y sofisticados, haciendo que las simulaciones de los sistemas resultantes sean extensas y complicadas. La restricción sobre los recursos computacionales disponibles está siempre presente y por eso la urgencia en el desarrollo y verificación de técnicas de solución capaces de explotar eficientemente el potencial de las modernas computadoras y la posibilidad de obtener soluciones de buena calidad en un tiempo aceptable de simulación (tiempo de CPU). La presente tesis nace de esta necesidad. Durante varias décadas se han desarrollado y probado técnicas concernientes a la solución de problemas lineales que son resultado de la aplicación del método de elementos finitos (MEF) a ecuaciones diferenciales en derivadas parciales (EDDP) que tratan de describir un conjunto de eventos de la física (e.g., mecánica de cuerpos sólidos, dinámica estructural, dinámica de fluidos, etc.). Hasta no hace mucho tiempo, la solución directa de estos sistemas era preferida a la solución iterativa debido a su mayor robustez y al carácter predictivo de su comportamiento. Sin embargo, la gran cantidad de técnicas iterativas que han sido desarrolladas, conjuntamente con la necesidad de resolver sistemas de ecuaciones xv

16 xvi cada vez más grandes en diferentes arquitecturas han dado como resultado una inclinación al uso de este tipo de técnicas y al desarrollo de nuevas. Esta tendencia se viene dando desde 1970 cuando dos importantes desarrollos marcaron un punto de inflexión en la solución de grandes sistemas de ecuaciones. Uno fue la explotación de la baja densidad (por sparsity, matrices ralas, matrices con tasa de llenado baja) de los sistemas que resultan de la aplicación del MEF (como así también del método de diferencias finitas MDF) a las EDDP. El otro fue el desarrollo de métodos tales como los de Krylov (o métodos tipo gradientes conjugados precondicionados). Gradualmente los métodos iterativos (precondicionamiento e iteración en el espacio de Krylov) comenzaron a aproximarse en calidad a las soluciones provistas por métodos directos. Particularmente, mucho se ha escrito sobre el método de gradientes conjugados precondicionado para sistemas lineales simétricos que resultan de operadores simétricos (e.g., elasticidad lineal y no lineal, flujo potencial, etc.). Hoy, los grandes sistemas de ecuaciones obtenidos de las EDDP no lineales mediante el MEF para problemas transitorios en dos y tres dimensiones donde pueden haber varias incógnitas por nodo, son resueltos con métodos iterativos en computadoras de alta performance (arquitecturas paralelas o vectoriales) debido a que requieren mucha menor comunicación entre los procesadores que la necesaria en métodos directos donde la solución de cada una de las incógnitas esta acoplada con las demás. El método de subestructuración (o método de descomposición de dominios e iteración sobre la matriz complemento de Schur para dominios no solapados) conduce a sistemas reducidos mejor condicionados para la solución mediante métodos de Krylov. El número de condición de estos problemas se ve disminuido en un factor 1/h ( 1/h vs 1/h 2 para el sistema global, siendo h la dimensión característica de la malla) y el costo computacional por iteración no se ve encarecido debido a que las matrices correspondientes a los grados de libertad de los subdominios (grados de libertad interiores) ya han sido factorizadas. La eficiencia de estos métodos puede ser mejorada mediante el uso de precondicionadores [Meu99, Man93, BPS86, Cro02]. Diferentes técnicas de precondicionamiento han sido propuestas y la reducción del número de condición de las matrices ha sido demostrada en el marco de ecuaciones diferenciales lineales elípticas (e.g., precondicionadores del tipo wire basket, Neumann-Neumann y sus variantes para los problemas de elasticidad y flujo de Stokes). En este trabajo se buscará solucionar eficientemente los sistemas de ecuaciones provenientes de la discretización mediante el MEF o MDF de ecuaciones diferenciales no lineales en derivadas parciales que representan modelos numéricos de problemas reales (como los descriptos arriba) considerados un desafío para los métodos computacionales actuales. El

17 xvii objetivo es también el desarrollo de un código de elementos finitos orientado a objetos (que reduce drásticamente las dependencias de implementación entre subsistemas y que conduce al principio de reusabilidad de diseños de interfaces) que resuelva problemas de la mecánica de fluidos computacional en gran escala en forma distribuida mediante la técnica de paso de mensajes (MPI/PETSc [GLS94, BGCMS04]). Esta técnica es ampliamente explotable en arquitecturas de computadoras paralelas, tales como la de clusters Beowulf [SSBS99]. En la primer parte de esta tesis serán expuestos y desarrollados los tópicos relacionados con los métodos de descomposición de dominios y su desempeño en problemas clásicos de la mecánica de fluidos computacional. La segunda parte está dedicada a la aplicación del algoritmo propuesto en la primera parte ( I) a la solución de problemas más generales/complejos como lo es la absorción de ondas en fronteras ficticias y la solución de problemas de interacción fluido/estructura para el flujo supersónico de un fluido compresible.

18

19 Contents I Domain Decomposition Methods 1 1 Preliminaries Solution of Linear Systems Perturbation Theory Condition Number Preconditioning Basic Iterative Method Optimal Iteration Methods The Interface Strip Preconditioner for Domain Decomposition Methods Introduction Schur Complement Domain Decomposition Method The Steklov Operator Eigenvalues of Steklov Operator Preconditioners for the Schur Complement Matrix The Neumann-Neumann Preconditioner The Interface Strip Preconditioner (ISP) The Advective-Diffusive Case Implementation of the Neumann-Neumann Preconditioner The Balancing Neumann-Neumann Version The Interface Strip Preconditioner: Solution of the Strip Problem Implementation Details of the IISD Solver Classical Overlapping Domain Decomposition Method: Alternating Schwarz Methods Conclusions xix

20 xx CONTENTS 3 Numerical Tests Numerical Examples in Sequential Environments The Poisson s Problem The Scalar Advective-Diffusive Problem The Hypersonic Flow Over a Flat Plate Test Numerical Examples in Parallel Environment The Poisson s Problem The Scalar Advective-Diffusive Problem The Coupled Hydrological Flow Model The Stokes Flow in a Long Horizontal Channel The Viscous Incompressible Navier-Stokes Flow Around an Infinite Cylinder Navier-Stokes Flow Using the Fractional Step Scheme. The Lid Driven Cavity The Wind Flow Around a 3D Immersed Body. The AHMED Model Conclusions II Applications and Usage 99 4 Dynamic Boundary Conditions in CFD Introduction General Advective-Diffusive Systems of Equations Linear Advection-Diffusion Model Gas Dynamic Equations Shallow Water Equations Channel Flow Variational Formulation Absorbing Boundary Conditions Advective-Diffusive Systems in 1D Linear 1D Absorbing Boundary Conditions Multidimensional Problems Absorbing Boundary Conditions for Nonlinear Problems Riemann Based Absorbing Boundary Conditions Absorbing Boundary Conditions Based on Last State Imposing Nonlinear Absorbing Boundary Conditions

21 CONTENTS xxi Numerical Example. Viscous Compressible Subsonic Flow Over a Parabolic Bump Dynamically Varying Boundary Conditions Varying Boundary Conditions in External Aerodynamics Aerodynamics of Falling Objects Conclusions Strong Coupling Strategy for Fluid-Structure Interaction Problems in Supersonic Regime Via Fixed Point Iteration Introduction Strongly Coupled Partitioned Algorithm Via Fixed Point Iteration Notes on the Fluid/Structure Interaction (FSI) Algorithm Description of Test Case Dimensionless Parameters Houbolt s Model FSI Code Results Stability of the Weak/Strong Staged Coupling Outside the Flutter Region Conclusions III Final Conclusions Overview and Final Remarks 161 IV Appendix 163 A Functional Spaces 165 A.1 Some Used Sobolev Spaces A.2 Extension to Vector-Valued Functions B Resumen extendido en castellano 169 B.1 El Método de Descomposición de Dominios en Mecánica de Fluidos Computacional B.2 Ecuaciones de Gobierno B.2.1 Propiedades Continuas de los Fluidos B.2.2 Campos Lagrangianos y Eulerianos B.2.3 La Ecuación de Continuidad

22 xxii CONTENTS B.2.4 La Ecuación de Cantidad de Movimiento B.2.5 Las Ecuaciones de Navier-Stokes en Sistemas de Referencia No Inerciales B.2.6 Las Ecuaciones de Navier-Stokes Incompresibles B.3 Formulación de otros Modelos Matemáticos a Tratar B.3.1 Problemas Hidrológicos B.4 Computación de Alta Performance B.4.1 Resolución Numérica del Modelo de CFD/Hidrología Superficial y Subterránea B.4.2 Solución de Grandes Sistemas de Ecuaciones B.4.3 Métodos de Descomposición de Dominio B.4.4 Precondicionamiento B.4.5 Implementación Operativa del Cluster B.5 Algunas Definiciones Topológicas B.6 Dominio Lipschitz, Frontera Lipschitz B.7 Función Lipschitz B.8 Problemas Bien Planteados en el Sentido de Hadamard

23 List of Tables 2.1 Condition number for the Steklov operator and several preconditioners (mesh: elements, strip: 5 layers of nodes) Condition number for the Steklov operator and several preconditioners (mesh: elements, strip: 10 layers of nodes) CPU time and memory requirements per proc. for Poisson problem (mesh elements). Note: * in table means iteration failed to converge to a specified tolerance in a maximum of 200 its CPU time and memory requirements per proc. for advective-diffusive problem (mesh elements). Note: * in table means iteration failed to converge to a specified tolerance in a maximum of 200 its CPU time and memory requirements for Saint-Venant equations (mesh elements). Note: * in table means iteration failed to converge to a specified tolerance in a maximum of 400 its B.1 Algoritmo Gradiente Conjugado Precondicionado xxiii

24

25 List of Figures 1.1 Families of Solvers: Direct and Iterative Solvers Families of Solvers: Domain Decomposition Solvers Aleksei Nikolaevich Krylov ( ) Carl Gustav Jacob Jacobi ( ) Johann Carl Friedrich Gauß ( ) André-Louis Cholesky ( ) Pafnuty Lvovich Chebyshev ( ) Lewis Fry Richardson ( ) Cornelius Lanczos ( ) Issai Schur ( ) Carl Gottfried Neumann ( ) Domain Decomposition Johann Peter Gustav Lejeune Dirichlet ( ) Joseph Louis Lagrange ( ) Simon-Denis Poisson ( ) Vladimir Andreevich Steklov ( ) Pierre Simon Laplace ) Eigenfunctions of Schur complement matrix with 2 sub-domains Eigenfunctions of Schur complement matrix with 9 sub-domains Eigenfunctions of Schur complement matrix with 2 sub-domains and advection (global Péclet 5) Eigenvalues of Steklov operators and preconditioners for the Laplace operator (P e = 0) and symmetric partitions (L 1 = L 2 = L/2, b = 0.1L) Eigenvalues of Steklov operators and preconditioners for the Laplace operator (P e = 0) and non-symmetric partitions (L 1 = 0.75L, L 2 = 0.25L, b = 0.1L) xxv

26 xxvi LIST OF FIGURES 2.14 Eigenvalues of Steklov operators and preconditioners for the advectiondiffusion operator (Pe = 5) and symmetric partitions (L 1 = L 2 = L/2, b = 0.1L) Eigenvalues of Steklov operators and preconditioners for the advectiondiffusion operator (Pe = 50) and symmetric partitions (L 1 = L 2 = L/2, b = 0.1L) Robert Lee Moore ( ) Roger Penrose (1931 ) Strip Interface problem IISD decomposition by sub-domains. Actual decomposition Non local element contribution due to bad partitioning Hermann Amandus Schwarz ( ) Leonhard Euler ( ) Solution of Poisson s problem Solution of advective-diffusive problem Problem definition Claude Louis Marie Henri Navier ( ) George Gabriel Stokes ( ) Leopold Kronecker ( ) Osborne Reynolds ( ) Ernst Mach ( ) Skin friction coefficient Stanton number Solution of Poisson s problem (mesh elements) Solution of advective-diffusive problem (mesh elements) Iteration counts for advective-diffusive problem (mesh elements) Adhémar Jean Claude Barré de Saint-Venant ( ) Stream/Aquifer coupling Iteration counts for Saint-Venant system of equations (mesh elements) Solution of Saint-Venant system of equations (mesh elements) Iteration counts for the coupled flow Soybean location Difference in phreatic levels for both cases Aquifer State at t=2 years