Dr. Francisco Prieto Castrillo. Coordinador de la Unidad de Ciencia y Tecnología

Transcripción

1 Modelling the GRID as a Complex System Dr. Francisco Prieto Castrillo. Coordinador de la Unidad de Ciencia y Tecnología Modelling the GRID as a Complex System Francisco Prieto Castrillo

2 Can the GRID be modelled? GRIDS y COMPLEJIDAD CENTRO EXTREMEÑO DE TECNOLOGÍAS AVANZADAS

3 Research in GRID/Distributed computing Modelling the GRID as a Complex System Francisco Prieto Castrillo

4 But TO MODEL WHAT? GRIDS y COMPLEJIDAD CENTRO EXTREMEÑO DE TECNOLOGÍAS AVANZADAS

5 THE GRID!!!! GRIDS y COMPLEJIDAD CENTRO EXTREMEÑO DE TECNOLOGÍAS AVANZADAS

6 BIRTH OF THE GRID ALICE (ion-ion, plasma q-g) ATLAS (Higgs, dark matter) CMS (Higgs, dimensiones add) LHCb (Anti-materia) Modelling the GRID as a Complex System Francisco Prieto Castrillo

7 Where is GRID mostly used Computational electro-magnetism Particle physics Neuroscience Earth sciences Plasma physics Modelling the GRID as a Complex System Francisco Prieto Castrillo

8 WORLD SCALE EXPERIMENTS The Pierre Auger experiment JEM-EUSO Japanese Experiment Module Extreme Universe Space Observatory" Virtual Observatory Modelling the GRID as a Complex System Francisco Prieto Castrillo

9 High Energy Physics Nuclear physics at plasma devices Modelling the GRID as a Complex System Francisco Prieto Castrillo

10 ALSO... Human Genome Project (Open Science GRID y TeraGrid) fmri: Dynamics of Cortical Change CardioGRID Modelling the GRID as a Complex System Francisco Prieto Castrillo

11 ACTORS A B C D E Community Application workflow The distributed environment Modelling the GRID as a Complex System Francisco Prieto Castrillo

12 Distributed computing environment categorization orkflow = { A1 A2... AN } Type sequential parallel Local Non-local Centralized Hierarchy based A1 Distributed A2 A1 A2... AN AN A1 A2 A3 A4 A2 A2 A2 A2 A2 examples GRID Cloud Computing Desktop-grids P2P Computing Modelling the GRID as a Complex System Francisco Prieto Castrillo

13 Distributed computing environment categorization GRID Cloud Computing P2P Computing PU W MS W MS PU PU PU C CE CE CE W W W W W W W EW N N N N N N N N desktop-grids (parasitic computing) GRID + Virtualization PU PU PU BOINC Sever Software Hardware Napster Gnutella... Client 1 Client 2 Client N Modelling the GRID as a Complex System Francisco Prieto Castrillo

14 GRID COMPUTING Accessible resources for a VO WMS WMS CE CE CE CE W N W N W N W N W N W N W N W N Admin Domain 1 Admin Domain 2 Modelling the GRID as a Complex System Francisco Prieto Castrillo

15 GRID COMPUTING GRID resources & services Communications Network Modelling the GRID as a Complex System Francisco Prieto Castrillo

16 Academic research networks GEANT in EUROPE REDCLARA in LA RedIris GÉANT2 connects European National Research and Education Networks (NRENs)

17 GRID COMPUTING Users Middleware GRID User Interface (UI) Information Services (IS) Computing resources (ROC/RC) Storage Elements (SE) Computing Elements (CE) Workload Management System (WMS) Virtual Organization (VO) Data Management System (DMS) Worker Nodes Storage Units Modelling the GRID as a Complex System Francisco Prieto Castrillo

18 GRID COMPUTING Regional Operations Centre (ROC) COMMUNICATION NETWORK VO ALICE wn s VO CSS wn s UI Resource Centre (RC) CE SE GRID SERVICES CE SE Resource Centre (RC) wn s CE SE Modelling the GRID as a Complex System Francisco Prieto Castrillo

19 GRID COMPUTING glite GRID certificate User account MATCH- MAKING Job requirements publish status records all GRID events Job requirements publish status publish status Grid Gate: Interface to the cluster Local Resource Management System (LRMS) WN s Computing Resources Modelling the GRID as a Complex System Francisco Prieto Castrillo

20 SOME GRID COMPUTING PROJECTS AP GRID Asia-Pacific Grid D4Science DIstributed collaboratories Infrastructure on Grid ENabled Technology 4 Science DEISA Distributed European Infrastructure for Supercomputing Application EELA-2 E-science for Europe and Latin America EGEE Enabling Grids for E-sciencE EGI_DS European Grid Initiative Design Stud EUAsiaGrid Collaboration between Europe and Asia EU-IndiaGrid Collaboration between Europe and Indi GridPP Grid for UK Particle Physic LCG Worldwide LHC Computing Grid NextGRID Supporting mainstream use of grid AstroGrid: astronomy BIRN: human disease CaBIG: cancer Earth System Grid: climate ECOGrid: economics FusionGrid: fusion energy NEESit : earthquakes Virolab: infectious disease...@home: volunteer computin NorduGrid Grids in the Nordic region Open Science Grid Open grid infrastructure for collaborative science Modelling the GRID as a Complex System Francisco Prieto Castrillo

21 MODULO 1: ADAPTACION DE APLICACIONES CIENTIFICAS A GRID 1.4 DESPLIEGUE DE WORKFLOWS DE APLICACIONES EN ENTORNOS GRID I Etapas de despliegue de workflow en un entorno distribuído 1) Determinar colaboracion/software en la Red de VO s disponible 2) Resource Discovery para una VO 3) Diseño del workflow de producción Particionamiento Gestión de persistencia de datos Modelo colaborativo 4) Despliegue del workflow de ejecución Generación de wrappers y scripts Pruebas en entornos de testeo Producción científica

22 MODULO 1: ADAPTACION DE APLICACIONES CIENTIFICAS A GRID 1.4 DESPLIEGUE DE WORKFLOWS DE APLICACIONES EN ENTORNOS GRID II 1) Partición de la aplicación Determinar el ejecutable y datos de entrada y salida típicos Partición en datos Partición en procesos Determinar qué constituye un ciclo de ejecución elemental en máquina local El ciclo de ejecución se dividirá en procesos elementales. Ver qué constituye un proceso elemental Los datos de entrada y salida se dividen en datos elementales. Fijar qué constituye un dato elemental Cómo se relacionan los datos entre si en tiempo de ejecución Qué dependencias externas (librerías, software) hay INPUT SPACE 2) Gestión de la persistencia de datos Cual es el tamaño típico de los datos de entrada salida A1 A2... AN A1... AN 3) Modelo colaborativo Qué datos/resultados interesa federar OUTPUT SPACE

23 MODULO 1: ADAPTACION DE APLICACIONES CIENTIFICAS A GRID 1.4 DESPLIEGUE DE WORKFLOWS DE APLICACIONES EN ENTORNOS GRID III Grado de dependencia entre procesos durante ejecución Procesos Independientes Job Collections Parametric Jobs Procesos dependientes conjuntos de jobs independientes se optimiza el tiempo de ejecucion por usar la misma autenticacion (via delegación de proxy) para todos los jobs de la colección conjuntos de jobs independientes parametrizados por metavariables A1 A2... AN DAG Jobs MPI Jobs parallel programming A B MPI: Existe comunicacion entre procesos A y B en tiempo de ejecución B A C Grado de Interaccion con el usuario durante el proceso de ejecución Interactive Jobs Los jobs abren un canal de comunicacion en tiempo real con un host remoto (ej, UI) y las salidas estandard del job se redirigen a ese host A DAG: B y C no se ejecutan hasta concluido A

24 as you see...grids are very COMPLEX SYSTEMS!!!! Interacting communities in a collaborative way (or not!) Hierarchical structure spread across admin domains Resources are being added/removed dynamically Heterogeneous in resources (different type of nodes) Workflows may be DAGs Different interacting communication networks (with complex topology!!)...weighted network Queueing processes (need of traffic models)...basically stochastic Modelling the GRID as a Complex System Francisco Prieto Castrillo

25 Research: basically from community, system and application point of view 1) Analysis and Modelling of distributed computing environments through a complex systems perspective. Researchers: Francisco Prieto Castrillo (CETA-CIEMAT), Elena Rey Espada (CETA-CIEMAT) 2) Muti-objective Optimization for Grid Scheduling through bio-inspired algorithms. Researchers: María Arsuaga Ríos (CETA-CIEMAT), Francisco Prieto Castrillo (CETA-CIEMAT) y Miguel Angel Vega Rodríguez (UNEX). 3) Application Workflow Optimization in Distributed Computing Environments.Researchers: María Botón Fernández (CETA-CIEMAT), Francisco Prieto Castrillo (CIEMAT) y Miguel Angel Vega Rodríguez (UNEX). A B C D E Modelling the GRID as a Complex System Francisco Prieto Castrillo

26 Where is the DATA???

27 Preliminary DATA (RTM) Ilijašić y Saitta. Characterization of a computational grid as a complex system. Proceedings of the 6th international conference industry session on Grids meets autonomic computing (2009) pp DATA: RTM 15 Gb Logs/ 20 Meses

28 Preliminary DATA (RTM) Ilijašić y Saitta. Characterization of a computational grid as a complex system. Proceedings of the 6th international conference industry session on Grids meets autonomic computing (2009) pp DATA: RTM 15 Gb Logs/ 20 Meses

29 Preliminary DATA (RTM) Costa et al. Complex Grid Computing. arxiv (2003) vol. cond-mat.stat-mech

30 Preliminary DATA (RTM) Node: U+CE Edge: U--CE Weight: Num Jobs sent Grafo bipartito dirigido y con pesos Ilijašić y Saitta. Characterization of a computational grid as a complex system. Proceedings of the 6th international conference industry session on Grids meets autonomic computing (2009) pp DATA: RTM 15 Gb Logs/ 20 Meses

31 Our approach 1) Generate different communication network topologies (router level) Homogeneous network 2) Map GRID services (basically Brokers and resources) to that topology Homogeneous resources (equally distributed capacity) 3) Map tasks to Brokers and gauge speedup, efficiency etc. M independent tasks

32 Our approach

33 Our approach NODE: routers EDGES: Comm Connection MAP: Brokers + CE --> TOPOLOGY

34 Two possible approaches Task Space (J) Broker Space (B) Resource Space (C) Rb queue Resource queue

35 There are many ways for a given VO to use the GRID configuration space and now map..m tasks into Nb Brokers So there are ways of deploying M tasks through Nb brokers into Nc resources Most users do not fully benefit from distributed paradigm!!

36 1) Markov Modulated Process FINITE STATE MACHINE s... At each system state s, a tasks partition is being mapped to Nc(s) resources through Nb(s) brokers M tasks Compute speed-up f as a random variable

37 How to model GRID times for a given state machine Basically depends mostly on network topology and traffic model at global level depends on queueing model and computing capacities at resource level In this regard, the problem can be seen as a Modulated Markov Process MMP. Challenge: Model the MMP with and gauge f(s) for different topologies

38 M tasks Partition m tasks go to Broker Bk Furthermore...at partition level

39 A reasonable assumption for communication times in order to highlight the topology effect Markov Modulated topological distance between current broker/resource pair What about

40 (M/M/s) model. Multi-server queue configurations/servers s1 job space queue s2 Poisson stream FCFS RS (Others..) service rates

41 Deterministic model At t0, grid is empty and service times are FCFS at CE level Reach stationary state after transient state with the task load process Study mean properties then

42 Muti-objective Optimization for Grid Scheduling through bio-inspired algorithms.

43 Thanks! Francisco Prieto Castrillo Centro Extremeño de Tecnologías Avanzadas CIEMAT (Centro de Investigaciones Engergéticas, Mediambientales y Tecnológicas) ( Ministerio de Ciencia e Innovación

44 Conventual de San Francisco, Sola 1, Trujillo Teléfono: Fax: Modelling the GRID as a Complex System Francisco Prieto Castrillo