Session: Herramientas para facilitar la administración y el desarrollo con DB2 LUW. Ana Rivera IBM 22/OCTUBRE/2013 DB2 LUW

Transcripción

1 Session: Herramientas para facilitar la administración y el desarrollo con DB2 LUW. Ana Rivera IBM 22/OCTUBRE/2013 DB2 LUW 1

2 El día a día de un DBA. Cada día hay más trabajo que hacer y menos recursos Sería bueno tener algún aviso previo antes de que todo entre en crisis y suene el teléfono rojo Cuando hay problemas de rendimiento siempre me llaman aunque no sea un problema de base de datos. Tengo que justificar que la base de datos no es el culpable. Realizar tuning lleva tiempo y es necesario tener skills adecuados. Si tuviera tiempo podría hacerlo, pero nunca es posible Tenemos requerimientos desde el área de negocio para priorizar aplicaciones críticas y/o grupos de usuarios. Cómo lo hago? Por dónde empiezo? Mi entorno es muy complejo, gran número de bases de datos, diferentes versiones Cómo gestionarlas? 2

3 Mejorar la Gestión y el Rendimiento de forma eficaz. Integrar las necesidades del negocio. Monitorizar KPIs para ser capaces de dar respuesta a bajadas del rendimiento que impactan en el usuario final. Monitorizar y generar informes sobre los tiempos de respuesta de los SLA. Gestionar los recursos de acuerdo con las prioridades del negocio. Mejorar la eficacia. Aislar los problemas disminuye el tiempo de resolución. Herramientas que faciliten la administración y configuración de entornos complejos. Mejorar el rendimiento de las aplicaciones. Obtener recomendaciones sobre cargas de trabajo, optimización de estadísticas, crear los índices adecuados. Cargas de trabajo : No solo queries aisladas. 3

4 Mejorar la Gestión y el Rendimiento de forma eficaz. Prevención, prevención, prevención Evitar problemas de rendimiento con análisis de patrones y capacity plannning. Ayudar a desarrolladores a identificar problemas en las queries, realizar tuning antes de implementar, comparar y validar resultados. Mejorar la implementación de cambios con herramientas que permitan realizar pruebas con cargas de trabajo real. 4

5 Optim Tools Incluidas en DB2 Advanced Edition IBM Data Studio InfoSphere Optim Performance Manager Extended Edition InfoSphere Optim Query Workload Tuner InfoSphere Optim Configuration Manager InfoSphere Data Architect (10 authorized users) InfoSphere Optim purequery Runtime for Linux, UNIX, and Windows Advanced Recovery Tools IBM Recovery Expert Infosphere Optim High Performance Unload DB2 Merge Backup Otras Infosphere Optim Capture and Replay 5

6 Database and Performance Management Solutions IBM Data Studio Optim Query Workload Tuner Optim Performance Manager Optim purequery Optim Configuration Manager 6

7 Administración y desarrollo: DATA STUDIO Control Center desaparece Cómo administramos a partir de ahora? DATA STUDIO: Primera versión : Versión actual: 4.1. Descarga sin coste desde developer works Principales Tareas Manage database objects Manage data Configure instances and databases Backing up and restoring databases or table spaces Manage database connections Manage applications Analyze queries using Visual Explain to look at access plans Command Editor Job Manager Health Monitoring 7

8 Administración y desarrollo: DATA STUDIO Principales Tareas (continuación) Routine development and debugging purequery UI Built on Eclipse Flexible UI Shell Sharing : Diferentes productos integrados: IBM InfoSphere Data Architect (IDA), InfoSphere Optim Query Workload Tuner (OQWT), IBM Rational Application Developer (RAD) Integración con otras herramientas: InfoSphere Optim Performance Manager (OPM) -> Data Studio Tune Queries Gather metrics 8

9 Administración y desarrollo: DATA STUDIO Qué me descargo? Desarrollo Query tuning Administración de la bbdd y objetos. Generación de scripts. Embedded browser Health monitoring Alert notification Task management List utilities and applications Browser dashboard 9

10 Administración y desarrollo: DATA STUDIO 10

11 Administración y desarrollo: DATA STUDIO 11

12 Administración y desarrollo: DATA STUDIO 12

13 Administración y desarrollo: DATA STUDIO. BLU Acceleration Object Management Create/Alter/Drop column-organized tables just like row-organized tables DFT_TABLE_ORG 13 13

14 Administración y desarrollo: DATA STUDIO Nickname wizard IBM Corporation

15 Administración y desarrollo: DATA STUDIO 15

16 Análisis de Rendimiento: Data Studio + OQWT Cliente Data Studio : Basado en Eclipse. Cliente común for DB2 LUW and DB2 z/os Licencia Activación de la licencia en la base de datos que deseamos monitorizar Desde Data Studio: Activar las funcionalidades de tuning en cada base de datos: Perspectiva Data: Conexión a la base de datos: Configure Monitoring Activate Full Set of Tuning Features Una vez activado: Start Tuning. Data Studio Activate license on server DB2 z/os DB2 LUW 16 16

17 Tuning: Optim Query Workload Tuner El proceso de rendimiento y análisis de querys es iterativo: 17 17

18 Optim Query Workload Tuner Primera fase: Capturar el Workload Status 2. Capture 3. Manage 4. Invoke 5. Review 6. Compare 18

19 Optim Query Workload Tuner Tuning and Analyze. 19

20 Optim Query Workload Tuner Tuning and Analyze. 20

21 Optim Query Tuner Report 21

22 Optim Query Tuner Report 1. Problemas encontrados/recomendaciones 22

23 Optim Query Tuner Report Recomendaciones 23

24 Optim Query Tuner Report Query Formateada 24

25 OQWT. Test Candidate Indexes What if 25

26 OQWT. Test Candidate Indexes What if 26

27 OQWT. Test Candidate Indexes What if 27

28 Optim Query Workload Tuner 4.1 Soporte a BLU Workload Table Organization Advisor Determines which row organized tables are to be converted to column organization Selects the conversion which will lead to the best estimated workload performance improvement Workload Test Candidate Table Organization feature Allows the user to select a set of row organized tables Determines the estimated workload performance improvement for this input set of tables when they are converted to column organization IBM Corporation

29 Workload Table Organization Advisor Displays performance improvement Lists all tables in the workload and ones that should be converted Displays the current and recommended organization for each table as well as other properties including: The number of references of the table in the workload Findings of why a table was or was not recommended, and any warnings if a conversion is made Lists the statements in the workload affected by the table conversion IBM Corporation

30 Workload Test Candidate Table Organization Select the tables to convert virtually to column store tables Select tables from a list of all tables in the workload Select Test Candidate Table Organization to execute a workload explain Output the result in the Review view The output details are similar to the advisor IBM Corporation

31 Indicación de la Organización de la Tabla en el apartado de Query Formatter Query formatter reflects organization for tables used in the query under the label ORGANIZATION IBM Corporation

32 Visual Explain Visual Explain reflects organization as well DB2 for LUW 2013 IBM Corporation

33 Mejoras en la recomendación de estadísticas. Statistics advice based on frequency of modification Table stats is obsolete % changed in table is high i.e. IUDM / Cardinality is high DB2 for LUW 2013 IBM Corporation

34 Tuning: Optim Query Workload Tuner Resumen OQWT permite realizar de manera eficaz el proceso de tuning: Identificando los workload críticos (diferentes opciones para obtener el workload) Proporcionando un informe con los resultados del análisis. Mostrando los problemas y aconsejando soluciones para mejorar el rendimiento. Funcionalidades de comparación de planes de acceso. Validando las acciones a tomar gracias a la funcionalidad: What if 34 34

35 MONITORIZACIÓN: OPM EE Integrar las necesidades del negocio. Monitorizar KPIs para ser capaces de dar respuesta a bajadas del rendimiento que impactan en el usuario final. Monitorizar y generar informes sobre los tiempos de respuesta de los SLA. Por usuario/aplicativo/servidor de aplicaciones Gestionar los recursos de acuerdo con las prioridades del negocio. Mejorar la eficacia. Aislar los problemas disminuye el tiempo de resolución. Herramientas que faciliten la administración y configuración de entornos complejos. Prevención, prevención, prevención Evitar problemas de rendimiento con análisis de patrones y capacity plannning. 35

36 Optim Performance Manager Architecture DB2 Clients Performance Monitor entire landscape of databases and clients Health Administration Monitoring configuration User privileges Real-time Optim Performance Manager Server User Interaction Interactive Dashboards Health summary Performance overview History Background operation Performance Repository Canned Reports SQL analysis Capacity planning Database overview History collection Configurable sampling Configurable setails Data aggregation 15 minutes, 1 hour, 1 day Data pruning Configurable retention by aggregation Alerting Events, thresholds, health status, user scripts Notify via & SNMP Detailed dashboards 36 36

37 Identificar los problemas antes de que impacten al negocio: ALERTAS Aggregated alerts 37 24x7, always-on monitoring Web based dashboards available anywhere KPIs for server and response time metrics Workload-specific templates State, threshold, and userdefined alerts Cross-database summary, alert history, alert details Customized , SMTP, SNMP notifications Drill through to detailed data for alerted condition 37

38 Diágnostico de los Problemas. Desde la alerta navegar al panel con información en detalle Revisar el contexto de la alerta y analizar el problema Locking information: events, waiters, blockers Analyze locking event 38

39 Ejemplo: Análisis de bloqueos. View Lock holder Lock waiters Locked object Application details SQL statement Take action Force application Stop statement Tune query 39

40 Visión general de la base de datos. Get more context of system activity Set baselines using the time slider to set reference point View significant deviations from the baseline Set baselines See deviations. Hover to see values. View range Time slider control View activity by resource 40 40

41 Y si el problema no está en la Base de datos? Extended Insight Most database monitors focus only on the database Extended Insight looks beyond the database Monitor workload response time See where transactions spend time View database time spent analysis 41

42 Transaction Response Time Monitoring By Workload Percent of transactions not meeting response time objectives Time spent distribution 42 42

43 Analyze Workload Shows where workload is spending time Provides detail on queries, clients, and partitions or members Click to tune workload Details for selected object or attribute 43 Click to view, tune or other actions 43

44 Reports para prevenir problemas. Tune proactively Identify heavy hitter SQL Get expert advice Compare results Add value to developers Identify hot spots for developers Give them tuning advice Plan for growth Assess workload and storage growth trends Identify storage savings Allocate resources by business priority Ensure critical work has resource preference Improve server utilization Protect data server from overload Monitor and manage configuration changes 44

45 Optim Configuration Manager Understand Your Environment Explore client and database server properties Create a baseline inventory of clients for upgrades Control Clients Redirect client connections seamlessly for high availability or migration Utilize WLB resources efficiently on purescale Note: Needs OCM client install Solve Problems Faster Prevent problems by tracking and alerting configuration changes Debug problems faster by finding configuration changes relevant to performance issues Optimize Storage Consumption Report compression opportunities, storage trapped in pages and unused objects Manage data migration with DB2 v10 multi-temp storage 45 "IBM InfoSphere Optim Configuration Manager is a great tool for exploiting both the Adaptive Compression and Multi-Temperature Data Management features in DB2. I would definitely use its features to find contiguous space on disks for better I/O performance and allocate the most efficient storage devices for the most demanded data. --Juvenal Garcia Cuevas, Consulting Manager, SBRT Computing 45

46 Impacto en los entornos productivos de los cambios en el negocio. Growth Additional users, data and application tasks HW or SW platform changes Patches, platform switch, configurations, parameters, schemas and more HW or SW upgrades or migrations OS upgrades, fixpacks HW or SW consolidation or retirements Application deployment, enhancements or changes 46

47 Impacto en los entornos productivos de los cambios en el negocio. Channels Business Units Data Systems Direct Finance Administration DB CRM App DB DW App Core Systems ERP Call Centers Sales & Marketing Product Management DB CRM DB CRM App DB App DB DB App DW App Core Systems Core Systems CRM CRM Internet Consulting Services DB CRM App DB DB App Core Systems ERP Business Partners New Business Development DB CRM App DB DW App Core Systems 47

48 Consecuencias de implementaciones en Producción de cambios no testeados adecuadamente. Loss of customer satisfaction Higher costs Delays due to extra testing cycles Inability to deliver required functionality; Missed services level agreements; Inability to process transactions Time consuming process of rolling back changes; Inability to identify the source of the problem; Manual process of modifying test scripts No methods for fully testing databases workloads 48

49 Testing with workload record and replay Minimize differences between simulated lab conditions and production Capture Replay Complex production environment with thousands of users Mirror production environment in test 49

50 Capture and replay example scenarios Migration: Upgrade the database, OS or apply a fixpack Understand potential performance impacts, identify bottlenecks or rebind impacts Migration: Move data to a new database platform e.g. Oracle to DB2 Ensure workloads will run successfully and maintain/improve performance New Application: Run a new application along side production Identify any response time impact, lock contention or other bottlenecks More Data: Test impact of increased data volume Maintain stable performance as the data increases More Users: Speed up replay to simulate increase in concurrent users Continue to meet SLAs as the number of users increases Tuning: Physical database design changes for example new indexes Identify how a new index will impact existing access plans and overall performance 50

51 IBM InfoSphere Optim Query Capture and Replay Capture production workloads and replay them in testing environments Requirements Record and replay SQL Application Source Minimize unexpected production problems Shorten testing cycles Develop more realistic database testing scenarios Record InfoSphere Optim Query Capture and Replay Play Target Benefits Identify database problems sooner with validation reports and performance tuning Use actual production workloads for testing rather than fabricated scenarios Extend quality testing efforts to include the data layer 51

52 InfoSphere Optim Query Capture and Replay Architecture Overview Remote Applications S-TAP Local Applications SQL Source Installed on source Intercept traffic SQL S-TAP OQCR server Software component running in appliance Processes, replays and analyzes workloads Query Capture and Replay (OQCR) Server User interfaces Browser-based Administrative command line interface SQL S-TAP Target 52

53 IBM InfoSphere Optim Query Capture and Replay Limit the use of laborious database test script creation and load simulators by leveraging actual production workloads for testing Deploy a single database testing solution across heterogeneous systems Accelerate project delivery by identifying and correcting potential problems sooner Develop more streamlined, accurate database tests Compliment existing database tools to pinpoint database problems faster to improve performance Solve problems more rapidly with deep diagnostics and validation reports 53

54 Workflow: IBM InfoSphere Optim Query Capture and Replay Capture Prepare Replay Compare and Analyze 54

55 Example: Migration from Oracle DB to DB2 Production AIX v5.3 Oracle v10 InfoSphere Optim Query Capture and Replay Record Test AIX v6.1 DB2 v10 Capture Play Prepare Replay Compare and Analyze Record Oracle v10 production activity Prepare test data on the test system Replay the production activity on DB2 v10 test system Compare Oracle v10 production capture to DB2 v10 replay Analyze results 55

56 DB2-to-DB2 Capture and Replay Ejemplo Scenario: Assess the impact of a query workload tuning activity Process flow Capture workload in production environment Prepare test database environment Produce baseline in test Implement query workload tuning recommendations in test environment and analyze impact; repeat tuning if necessary InfoSphere Capture Replay Production environment Record Test environment Play 56

57 Step 1: Capturing the Workload S-TAP Captures all inbound database traffic Applies optionally basic filtering Remote Applications Sends collected information to the OQCR server Local Applications SQL Production Database OQCR server stores collected information SQL S-TAP Output: captured workload, metadata required to replay the workload and metrics Not captured but required for replay: database content, such as configuration, catalog, data InfoSphere Capture Replay server Record 57

58 Step 1: Capturing the Workload Start capture Identify where to capture Specify when to capture Capture records inbound local and remote database traffic Capture can be stopped at any time Capture ignores SQL associated with in-flight connections because correct replay cannot be guaranteed 58

59 Reviewing the Captured Workload Capture report provides aggregated workload information: SQL count, unique SQL statements, transaction count, metrics, 59

60 Step 2: Preparing the Replay Workload replay requires Workload transformation (into an optimized replay format ) Replay database setup Production Database Workload transformation can also address some database differences Setup replay database such that it approximates (as much as necessary) the production database InfoSphere Capture Replay Server Transform Workload Clone/setup database environment Test Database 60

61 Step 2: Preparing the Replay Transform workload: Map to replay database, map schema and user credentials 61

62 Reviewing the Prepared Workload Transform report provides aggregated information about the replay-ready workload SQL count, unique SQL statements, transaction count, metrics, 62

63 Step 3: Replaying the Workload Replay engine executes transformed workload Matches original SQL execution characteristics (concurrency, sequence) as much as possible JDBC is used to process workload Replay options Match original speed Faster (if possible) Slower S-TAP on replay database records all database traffic and collects metrics to support analysis InfoSphere Capture Replay server JDBC SQL S-TAP Test Database First replay(s) establish baseline; subsequent replays are used to assess the impact of changes 63

64 Step 4: Comparing and Analyzing Results Reports provide insights into how changes to the data server environment (or workload) impact workload replay accuracy and performance Replay accuracy: does the replay approximate the original workload characteristics sufficiently? SQL and transaction reports highlight performance differences between two workload executions InfoSphere Capture Replay server Captured or replayed workload Analysis Replayed workload Analyze relevant differences and assess whether follow-up actions are necessary Repeat prepare, replay, compare and analyze steps if necessary 64

65 Analyzing Replay Accuracy Summary report highlights workload execution differences Return codes Rows fetched Rows modified New SQL Data value differences are ignored Drill-through reports provide details Reports also available for transactions 65

66 Reviewing Replay Results Drill-through reports display aggregated information for each unique SQL Execution count, rows fetched or updated, average and total response time 66

67 Reviewing Replay Results More details are available for fastest and slowest SQL executions 67

68 Analyzing Performance Differences Performance overview highlights SQL improvements or regressions 68

69 Inspecting Workload Characteristics Charts visualize workload characteristics 69

70 Extending performance analysis with database tools Capture Prepare Replay Compare and Analyze 70

71 Optim Tools Integración de Herramientas. Optim Performance Manager Optim Performance Manager Optim Configuration Manager Network 1. Identify Users 2. Diagnose Application Servers DBMS & OS 4. Prevent All of the above 3. Solve 71 Optim Query Workload Tuner Data Studio and purequery Runtime 71

72 RESUMEN La utilización de las herramientas de Optim nos permiten: Facilitar la administración de múltiples entornos. Gestionar los cambios y facilitar las implementaciones en producción Mejorar el rendimiento de las aplicaciones mediante análisis avanzado. Identificación temprana de los problemas Integración de las diferentes herramientas. 72

73 Links de utilidad Redbook OPM + OQWT Artículos, tutoriales, demos, videos en developer works por producto: OPM: OQWT: OCM: Optim Capture and Replay 73

74 Session Title Ana Rivera Matías IBM 74