Métodos Adaptativos de Minería de Datos y Aprendizaje para Flujos de Datos.

Transcripción

1 Métodos Adaptativos de Minería de Datos y Aprendizaje para Flujos de Datos. Albert Bifet LARCA: Laboratori d Algorismica Relacional, Complexitat i Aprenentatge Departament de Llenguatges i Sistemes Informàtics Universitat Politècnica de Catalunya Junio 2009, Santander

2 Minería de Datos y Aprendizaje para Flujos de Datos con Cambio de Concepto Extraer información de secuencia potencialmente infinita de data datos que varian con el tiempo usando pocos recursos La Desintegración de la Persistencia de la Memoria Salvador Dalí usando ADWIN ADaptive Sliding WINdow: Ventana deslizante adaptativa sin parámetros 2 / 29

3 Minería de Datos y Aprendizaje para Flujos de Datos con Cambio de Concepto Extraer información de secuencia potencialmente infinita de data datos que varian con el tiempo usando pocos recursos La Desintegración de la Persistencia de la Memoria Salvador Dalí usando ADWIN ADaptive Sliding WINdow: Ventana deslizante adaptativa sin parámetros 2 / 29

4 Minería de Datos Masivos Explosión de Datos en los últimos años : 100 millones búsquedas por día : 20 millones transacciones por día 1,000 millones de transacciones de tarjetas de credito por mes 3,000 millones de llamadas telefónicas diarias en EUA 30,000 millones de s diarios, 1,000 millones de SMS Tráfico de redes IP: 1,000 millones de paquetes por hora por router 3 / 29

5 2007 Minería de Datos Masivos Datos Masivos Universo Digital: 281 exabytes (mil millones de gigabytes) La cantidad de información creada excedió el almacenaje disponible por primera vez Green Computing Estudio y práctica de como usar recursos informáticos eficientemente. Algorithmic Efficiency Una de las principales maneras de hacer Green Computing 4 / 29

8 Minería de Datos Masivos Koichi Kawana Simplicidad significa conseguir el máximo efecto con los mínimos medios. Donald Knuth... we should make use of the idea of limited resources in our own education. We can all benefit by doing occasional "toy" programs, when artificial restrictions are set up, so that we are forced to push our abilities to the limit. 5 / 29

9 Minería de Datos Masivos Koichi Kawana Simplicidad significa conseguir el máximo efecto con los mínimos medios. Donald Knuth... we should make use of the idea of limited resources in our own education. We can all benefit by doing occasional "toy" programs, when artificial restrictions are set up, so that we are forced to push our abilities to the limit. 5 / 29

10 Introducción: Data Streams Data Streams Secuencia potencialmente infinita Gran cantidad de datos: espacio sublineal Gran velocidad de llegada: tiempo sublineal por ejemplo Cada vez que un elemento de un data stream se ha procesado, se descarta o se archiva Puzzle: Encontrar números que faltan Sea π una permutación of {1,...,n}. Sea π 1 la permutación π con un elemento que falta. π 1 [i] llega en orden creciente Tarea: Determinar el número que falta 6 / 29

11 Introducción: Data Streams Data Streams Secuencia potencialmente infinita Gran cantidad de datos: espacio sublineal Gran velocidad de llegada: tiempo sublineal por ejemplo Cada vez que un elemento de un data stream se ha procesado, se descarta o se archiva Puzzle: Encontrar números que faltan Sea π una permutación of {1,...,n}. Sea π 1 la permutación π con un elemento que falta. π 1 [i] llega en orden creciente Tarea: Determinar el número que falta 6 / 29

12 Introducción: Data Streams Data Streams Secuencia potencialmente infinita Gran cantidad de datos: espacio sublineal Gran velocidad de llegada: tiempo sublineal por ejemplo Cada vez que un elemento de un data stream se ha procesado, se descarta o se archiva Puzzle: Encontrar números que faltan Sea π una permutación of {1,...,n}. Sea π 1 la permutación π con un elemento que falta. π 1 [i] llega en orden creciente Usar un vector n-bit para memorizar todos los numeros (espacio O(n) ) Tarea: Determinar el número que falta 6 / 29

13 Introducción: Data Streams Data Streams Secuencia potencialmente infinita Gran cantidad de datos: espacio sublineal Gran velocidad de llegada: tiempo sublineal por ejemplo Cada vez que un elemento de un data stream se ha procesado, se descarta o se archiva Puzzle: Encontrar números que faltan Sea π una permutación of {1,...,n}. Sea π 1 la permutación π con un elemento que falta. π 1 [i] llega en orden creciente Data Streams: espacio O(log(n)). Tarea: Determinar el número que falta 6 / 29

14 Introducción: Data Streams Data Streams Secuencia potencialmente infinita Gran cantidad de datos: espacio sublineal Gran velocidad de llegada: tiempo sublineal por ejemplo Cada vez que un elemento de un data stream se ha procesado, se descarta o se archiva Puzzle: Encontrar números que faltan Sea π una permutación of {1,...,n}. Sea π 1 la permutación π con un elemento que falta. π 1 [i] llega en orden creciente Almacenar n(n + 1) 2 π 1 [j]. j i Tarea: Determinar el número que falta 6 / 29

15 Problema Introducción: Data Streams 12,35,21,42,5,43,57,2,45,67 Dados n números no ordenados, encontrar un número que esté en la mitad superior de la lista ordenada. Algoritmo 2,5,12,21,35 42,43,45,57,67 Elegir k números aleatoriamente. Devolver el número mayor. Análisis La probabilidad de que la solución sea incorrecta es la probabilidad de que todos los k números estén en la mitad inferior : (1/2) k Para tener probabilidad δ usaremos k = log 1/δ muestras 7 / 29

18 Outline 1 Introduction 2 ADWIN : Concept Drift Mining 3 Hoeffding Adaptive Tree 4 Conclusions 8 / 29

19 Data Streams Data Streams At any time t in the data stream, we would like the per-item processing time and storage to be simultaneously O(log k (N,t)). Approximation algorithms Small error rate with high probability An algorithm (ε,δ) approximates F if it outputs F for which Pr[ F F > εf ] < δ. 9 / 29

20 Data Streams Approximation Algorithms Frequency moments Frequency moments of a stream A = {a 1,...,a N }: F k = where f i is the frequency of i in the sequence, and k 0 v i=1 F 0 : number of distinct elements on the sequence F 1 : length of the sequence F 2 : self-join size, the repeat rate, or as Gini s index of homogeneity Sketches can approximate F 0,F 1,F 2 in O(logv + logn) space. f k i Noga Alon, Yossi Matias, and Mario Szegedy. The space complexity of approximation the frequency moments / 29

21 Data Streams Approximation Algorithms Sliding Window We can maintain simple statistics over sliding windows, using O( 1 ε log2 N) space, where N is the length of the sliding window ε is the accuracy parameter M. Datar, A. Gionis, P. Indyk, and R. Motwani. Maintaining stream statistics over sliding windows / 29

28 Data Mining Algorithms with Concept Drift No Concept Drift Concept Drift input DM Algorithm Counter 5 Counter 4 Counter 3 output input DM Algorithm Static Model output Counter 2 Counter 1 Change Detect. 13 / 29

29 Data Mining Algorithms with Concept Drift No Concept Drift Concept Drift DM Algorithm DM Algorithm input Counter 5 output input Estimator 5 output Counter 4 Estimator 4 Counter 3 Estimator 3 Counter 2 Estimator 2 Counter 1 Estimator 1 13 / 29

30 Time Change Detectors and Predictors: A General Framework Estimation x t Estimator 14 / 29

31 Time Change Detectors and Predictors: A General Framework Estimation x t Estimator Alarm Change Detect. 14 / 29

32 Time Change Detectors and Predictors: A General Framework Estimation x t Estimator Alarm Change Detect. Memory 14 / 29

33 Window Management Models W = Equal & fixed size subwindows [Kifer+ 04] Equal size adjacent subwindows [Dasu+ 06] Total window against subwindow [Gama+ 04] ADWIN: All Adjacent subwindows / 29

47 Example W = W 0 = 1 Algorithm ADWIN ADWIN: ADAPTIVE WINDOWING ALGORITHM 1 Initialize Window W 2 for each t > 0 3 do W W {x t } (i.e., add x t to the head of W ) 4 repeat Drop elements from the tail of W 5 until ˆµ W0 ˆµ W1 ε c holds 6 for every split of W into W = W 0 W 1 7 Output ˆµ W 16 / 29

48 Example Algorithm ADWIN W = W 0 = 1 W 1 = ADWIN: ADAPTIVE WINDOWING ALGORITHM 1 Initialize Window W 2 for each t > 0 3 do W W {x t } (i.e., add x t to the head of W ) 4 repeat Drop elements from the tail of W 5 until ˆµ W0 ˆµ W1 ε c holds 6 for every split of W into W = W 0 W 1 7 Output ˆµ W 16 / 29

52 Example Algorithm ADWIN W = W 0 = W 1 = ADWIN: ADAPTIVE WINDOWING ALGORITHM 1 Initialize Window W 2 for each t > 0 3 do W W {x t } (i.e., add x t to the head of W ) 4 repeat Drop elements from the tail of W 5 until ˆµ W0 ˆµ W1 ε c holds 6 for every split of W into W = W 0 W 1 7 Output ˆµ W 16 / 29

56 Algorithm ADWIN Example W = ˆµ W0 ˆµ W1 ε c : CHANGE DET.! W 0 = W 1 = ADWIN: ADAPTIVE WINDOWING ALGORITHM 1 Initialize Window W 2 for each t > 0 3 do W W {x t } (i.e., add x t to the head of W ) 4 repeat Drop elements from the tail of W 5 until ˆµ W0 ˆµ W1 ε c holds 6 for every split of W into W = W 0 W 1 7 Output ˆµ W 16 / 29

57 Algorithm ADWIN Example W = Drop elements from the tail of W W 0 = W 1 = ADWIN: ADAPTIVE WINDOWING ALGORITHM 1 Initialize Window W 2 for each t > 0 3 do W W {x t } (i.e., add x t to the head of W ) 4 repeat Drop elements from the tail of W 5 until ˆµ W0 ˆµ W1 ε c holds 6 for every split of W into W = W 0 W 1 7 Output ˆµ W 16 / 29

58 Algorithm ADWIN Example W = Drop elements from the tail of W W 0 = W 1 = ADWIN: ADAPTIVE WINDOWING ALGORITHM 1 Initialize Window W 2 for each t > 0 3 do W W {x t } (i.e., add x t to the head of W ) 4 repeat Drop elements from the tail of W 5 until ˆµ W0 ˆµ W1 ε c holds 6 for every split of W into W = W 0 W 1 7 Output ˆµ W 16 / 29

59 Algorithm ADWIN [BG07] ADWIN has rigorous guarantees (theorems) On ratio of false positives On ratio of false negatives On the relation of the size of the current window and change rates Other methods in the literature: [Gama+ 04], [Widmer+ 96], [Last 02] don t provide rigorous guarantees. 17 / 29

60 Algorithm ADWIN [BG07] Theorem At every time step we have: 1 (Few false positives guarantee) If µ t remains constant within W, the probability that ADWIN shrinks the window at this step is at most δ. 2 (Few false negatives guarantee) If for any partition W in two parts W 0 W 1 (where W 1 contains the most recent items) we have µ W0 µ W1 > ε, and if ε 4 3max{µ W0, µ W1 } ln 4n min{n 0,n 1 } δ then with probability 1 δ ADWIN shrinks W to W 1, or shorter. 18 / 29

62 Classification Data set that describes features for deciding if it is spam. Example Contains Domain Has Time Money type attach. received spam yes com yes night yes yes edu no night yes no com yes night yes no edu no day no no com no day no yes cat no day yes Assume we have to classify the following new instance: Contains Domain Has Time Money type attach. received spam yes edu yes day? 20 / 29

63 Classification Assume we have to classify the following new instance: Contains Domain Has Time Money type attach. received spam yes edu yes day? 20 / 29

64 Decision Trees Basic induction strategy: A the best decision attribute for next node Assign A as decision attribute for node For each value of A, create new descendant of node Sort training examples to leaf nodes If training examples perfectly classified, Then STOP, Else iterate over new leaf nodes 21 / 29

65 Hoeffding Tree / CVFDT Hoeffding Tree : VFDT Pedro Domingos and Geoff Hulten. Mining high-speed data streams With high probability, constructs an identical model that a traditional (greedy) method would learn With theoretical guarantees on the error rate 22 / 29

66 VFDT / CVFDT Concept-adapting Very Fast Decision Trees: CVFDT G. Hulten, L. Spencer, and P. Domingos. Mining time-changing data streams It keeps its model consistent with a sliding window of examples Construct alternative branches as preparation for changes If the alternative branch becomes more accurate, switch of tree branches occurs 23 / 29

67 Decision Trees: CVFDT No theoretical guarantees on the error rate of CVFDT CVFDT parameters : 1 W : is the example window size. 2 T 0 : number of examples used to check at each node if the splitting attribute is still the best. 3 T 1 : number of examples used to build the alternate tree. 4 T 2 : number of examples used to test the accuracy of the alternate tree. 24 / 29

68 Decision Trees: Hoeffding Adaptive Tree Hoeffding Adaptive Tree: replace frequency statistics counters by estimators don t need a window to store examples, due to the fact that we maintain the statistics data needed with estimators change the way of checking the substitution of alternate subtrees, using a change detector with theoretical guarantees Summary: 1 Theoretical guarantees 2 No Parameters 25 / 29

69 What is MOA? {M}assive {O}nline {A}nalysis is a framework for online learning from data streams. It is closely related to WEKA It includes a collection of offline and online as well as tools for evaluation: boosting and bagging Hoeffding Trees with and without Naïve Bayes classifiers at the leaves. 26 / 29

70 Ensemble Methods abifet/moa/ New ensemble methods: ADWIN bagging: When a change is detected, the worst classifier is removed and a new classifier is added. Adaptive-Size Hoeffding Tree bagging 27 / 29

72 Conclusions Adaptive and parameter-free methods based in replace frequency statistics counters by ADWIN don t need a window to store examples, due to the fact that we maintain the statistics data needed with ADWINs using ADWIN as change detector with theoretical guarantees, Summary: 1 Theoretical guarantees 2 No parameters needed 3 Higher accuracy 4 Less space needed 29 / 29