Applying Scrum in an Interdisciplinary Project using ...

Applying Scrum in an Interdisciplinary Project using Big Data, Internet of Things, and Credit Cards Rene  Esteves  Maria,  Luiz  Antonio  Rodrigues  Junior,  Luiz  Eduardo  Guarino  de  Vasconcelos,  Adriano  Fonseca   Mancilha  Pinto,  Paulo  Takachi  Tsoucamoto,  Henrique  Nunweiler  Angelim  Silva,  Airton  Lastori,  Adilson   Marques  da  Cunha,  Luiz  Alberto  Vieira  Dias     Computer  Science  Division     AeronauIcs  InsItute  of  Technology,  ITA     Sao  Jose  dos  Campos,     Brazil     {rene.stvs,  lrodrigues5150,  du.guarino,  amancilha,  ptakachi,  henriqueprogramacao,  alastori}@gmail.com,   {cunha,  vdias}@ita.br  

Agenda 1.  2.  3.  4. 

INTRODUCTION     BACKGROUND     TAILORING  SCRUM  FOR  ACADEMIC  PURPOSES     THE  IPBL*  

               The  project  development  and  its  challenges    

5.  CONCLUSION      

  (*)  IPBL  =  Interdisciplinary  Problem  Based  Learning  

1. INTRODUCTION Motivation

◦ According   to   Gartner,   McKinsey,   and   IDC,   the   Internet   of   Things   (IoT)   represents  the  technology  trend  for  next  years   ◦ In  2014,  more  than  12  billion  devices  were  connected  to  the  Internet     ◦ Consequently,  the  data  volume  has  increased  exponenIally  

1. INTRODUCTION The Project

  During   the   first   semester   of   2014,   courses   at   the   Brazilian   AeronauIcs   InsItute   of   Technology   (InsItuto   Tecnologico   de   AeronauIca   -­‐   ITA   -­‐   Brazil)  have  produced,  as  a  capstone  project,  a  prototype  named  BDI-­‐C3*:     ◦ Big  Data,  Internet  of  Things,  and  Credit  Cards     ◦ The   BDI-­‐C3   was   academically   developed   as   a   Proof   of   Concept   (PoC),   using  the  approach  of  IPBL  (Interdisciplinary  Problem  Based  Learning)       (*)  in  Portuguese,  Banco  de  Dados,  Internet  das  Coisas  e  Cartoes  de  Credito  –  BDI-­‐C3  

2. BACKGROUND Big Data Age

◦ Big   Data   are   large   pools   of   data   that   can   be   captured,   communicated,  aggregated,  stored,  and  analyzed   ◦ Big  Data  can  be  describe  by  5  V’s     ◦  Volume   ◦  Velocity     ◦  Variety     ◦  Value     ◦  Veracity  

2. BACKGROUND The Internet of Things (IoT)

◦ Internet   of   Things   (IoT)   is   a   compuIng   concept   that   describes   a   future   where   everyday   physical   objects   will   be   able   to   be   connected   to   the   Internet,   idenIfy   themselves,   and   interact   to   other  devices    

                                                                                                                                       Source:  techpedia.com,  2014    

2. BACKGROUND Agile Method Scrum

◦ Scrum   is  a  framework  with  roots  from  Control  Theory,  which  has   been  used  for  years,  to  address  complex  adapIve  problems     ◦ It   was   used   in   order   to   manage   sogware   development,   its   processes,  techniques,  and  best  prac@ces  

3. TAILORING SCRUM FOR ACADEMIC PURPOSES The Project Settings

◦ The   BDI-­‐C3   project   conducted   during   17   weeks,   has   involved   approximately  forty  grad  students  and  two  professors,  within  a   set  of  three  courses  taught  at  the  Graduate  Program  in  Electronic   Engineering  and  Computer  Science  at  ITA   ◦ Students   of   the   Tes@ng   Course   (CE-­‐229)   acted   as   POs   and   Testers,   while   students   from   the   other   courses   (Database   Systems   Project   –   CE-­‐240   and   Informa@on   Technologies   –   CE-­‐245)  have  acted  as  Development  Team  members  

3. TAILORING SCRUM FOR ACADEMIC PURPOSES Communication

◦ Poten@al   communica@on   problems   were   mi@gated  with   the   use   of  collaboraIve  and  free  tools  available  on  the  Internet,  such  as   Google   Docs   Suite,   Google   Drive,   Google   Hangouts,   Skype,   Github,  Trello,  among  others     ◦ To   centralize   shared   informaIon,   a   website   for   the   project   was   created  using  Google  Sites  

4. THE IPBL

Sprint 0 - Safe Project •  Sprint   0   is   a   kind   of   project-­‐before-­‐the-­‐ project   where   POs   can   work   toward   creaIng   a n   i n i I a l   p r o d u c t   backlog,   and   team   members   sehng   up   t h e   t e c h n i c a l   environment     •  TesIng   students   have   created   the   product   vision   as   an   important   agile  arIfact  

4. THE IPBL

Sprint 0 - Forming Phase •  In  order  to  help   students  to   understand  the   problem,  the  first   version  of  a  possible   architecture  design   was  generated  

4. THE IPBL

Sprint 1 - Proposal for the Development Enviroment  

4. THE IPBL

Sprint 1 - Storming Phase   ◦ This   Sprint   aimed   to   define   replica@on   mechanism   of   data   between  data  sources  (structured  to  unstructured)     ◦  This   challenge   was   overcome   with   the   use   of   a   tool   called   MySQL   Applier   for  Hadoop  (Happlier)    

◦ Another   goal   of   this   sprint   was   the   automaIc   generaIon   of   graphics,   in   order   to   help   fraud   analysts   to   detect   suspected   transacIons     ◦  This  goal  was  achieved  by  integraIng  the  architecture  with  the  R  System     ◦  In   order   to   idenIfy   suspected   transacIons,   the   upper   limit   from   boxplot,   Ls,   was  used:  

4. THE IPBL

Sprint 2 - Norming Phase ◦  During   the   2nd   Sprint   Planning,   four   features   were   proposed,   in   order   to   improve   students  technology  knowledge     ◦  All  these  features  were  documented  and  available  on  the  project  portal    

◦  The   first   feature   generated   a   comprehensive   documenta@on   on   the   required   technologies  used  in  the  VM  Cloudera  to  create  a  BDI-­‐C3  project  environment     ◦  This   documentaIon   was   collaboraIvely   generated   by   students   who   gained   knowledge   on   the   used  technologies  (Java,  Eclipse  IDE,  MySQL,  among  others)  during  the  project  development    

◦  The   second   feature   was   responsible   for   creaIng   an   account   on   Github   where   the   project   development   was   centralized   (e.g.   database   scripts,   R   scripts,   Java   applica@on)     ◦  In  order  to  assist  students,  a  complete  tutorial  on  how  to  use  the  Github  and  the  Git  with  Eclipse   IDE  was  prepared  

4. THE IPBL

Sprint 2 - Norming Phase ◦  The  result  of  the  third  feature  was  a  dataset  based  on  the  relaIonal  database  model   from  Sprint  1     ◦  The   dataset   loaded   the   main   tables   of   the   MySQL   database   related   to   transacIons   (i.e.   transacIons,  fraud,  locaIon,  media,  browser,  client  enIIes)     ◦  Each  data  transacIon  had  a  georeferenced  iden@fica@on  and  credit  card  number     ◦  The   transacIons   were   first   persisted   on   a   relaIonal   MySQL   database   and   then   instantly   replicated  to  the  HDFS  in  CSV  format,  via  MySQL  Hadoop  Applier    

◦  The   fourth   feature   has   helped   on   implemenIng   the   concept   of   IoT   creaIng   an   e-­‐ Commerce    

◦  The   Java   with   a   Spring   Framework   applica@on   was   defined,   having   integraIon   tools   that   facilitate  and  accelerate  the  development     ◦  As  Maven  does  to  manage  the  applicaIon  lifecycle  and  dependencies  control  and  Hibernate   also  does  for  data  persistence     ◦  The  MVC  (Model  View  Controller)  architecture  and  REST  API  were  also  used  

4. THE IPBL

Sprint 2 - Architecture •  Ager  features’   compleIon,  it   was  possible  to   generate  the   final  version  of   the  project   architecture  

_  

4. THE IPBL

Sprint 3 - Performing Phase ◦  In   order   to   improve   communicaIon   between   teams,   the   pracIce   of   Scrum   of   Scrums  was  used     ◦  3rd  Sprint  aimed  to  integrate  all  the  technologies  involved  in  the  project  into   a  single  applica@on     ◦  The  groups  were  divided  according  to  the  following  specialIes     ◦  (i)  e-­‐Commerce  (i.e.  Team  1),  working  with  REST  architecture     ◦  (ii)  Hive  (i.e.  Team  2),  analyzing  data  from  HDFS,  in  a  transparent  way  to  the  users     ◦  (iii)   Team   3   analyzed   the   client   fraud   contestaIons,   using   Hive   and   Happlier,   with   real   Ime   data  replicaIon     ◦  (iv)  Team  4  used  R  scripts  in  Java,  in  order  to  present  StaIsIcal  Analysis     ◦  ( v )   A r c g i s   O n l i n e   w a s   t h e   d u t y   o f   T e a m   5 ,   p r e s e n I n g   real  Ime  fraud  idenIficaIon  in  world  maps  

4. THE IPBL

Sprint 3 - Performing Phase ◦ The  tools  used  for  each  team  are  shown  

4. THE IPBL

The Final Presentation ◦ On   June   18,   2014,   the   students   performed   a   final   project   presentaIon  to  an  audience  composed  of  the  professors  and  guest   from  financial  insItuIons     ◦  The  alendees  have  appointed  as  a  posiIve  highlight,  the  use  of  free  tools   for  development  and  management   ◦  They  also  highlighted  the  transparent  use  of  Big  Data  technologies,  making  it   an  excellent  tool  for  analysts  operaIons  on  financial  transacIons  

5. CONCLUSION Inspection and Adaptation

◦ The   Sprint   RetrospecIves   was   remotely   conducted   using   the   Google  Drive     ◦  1st   Retrospec@ve   -­‐   students   described   that   the   internal   communicaIon   of   some  groups  was  impaired     ◦  2nd   Retrospec@ve   -­‐   it   was   noIced   that   the   developing   process   has   improved,  from  suggesIons  on  the  1st  RetrospecIve     ◦  3rd  Retrospec@ve   -­‐   students,   unanimously   agreed   that   the   Scrum   of   Scrums   pracIces  have  improved  the  project  development  

5. CONCLUSION Final Considerations

◦  The  experience  of  parIcipaIng  in  the  BDI-­‐C3  project  was  rewarding  in  terms   of   learning,   use   of   Big   Data   technologies,   Internet   of   Things,   and   agile   methods     ◦  This  successful  result  was  derived  from    

◦  (i)  the  use  of  a  IPBL  aligned  with  real  market  needs     ◦  (ii)  the  integra@on  of  students’  teams  from  other  graduate  courses     ◦  (iii)  the  use  of  agile  methods     ◦  (iv)  the  focus  on  sogware  working  delivery     ◦  (v)  the  choice  of  open-­‐source  tools  proved  to  be  a  success  on  large  scale  

◦  It   is   suggested   the   exploitaIon   of   specific   tools   to   provide   test   automa@on   for  Big  Data  (e.g.  MRunit  for  Map  Reduce  Tests)     ◦  In  this  way,  it  is  possible  to  explore  and  verify  that  these  tools  produce  safer  applicaIons  

ACKNOWLEDGMENTS The BDI-C3 Project 2014

  The  authors  of  this  paper  would  like  to  thank  the  Brazilian   AeronauIcs  InsItute  of  Technology  (ITA)  and  its  Electronic   and  Computer  Engineering  Graduate  Program  for  their   support,  during  the  development  of  this  BDI-­‐C3  project  

REFERENCES Gartner.  Published  in  STANFORD,  Conn.,  June  27,  2011.  Last  access:  09/09/2014.  Available  at:        

McKinsey.  McKinsey  Global  InsItute.  Ten  IT-­‐enabled  business  trends  for  the  decade  ahead.  May  2013.       IDC.  IDC’s  Top  10  Global  IT  Trends.  January  27,  2014.  Last  access:  17/08/2014.  Available  at:         Techopedia.  Internet  of  things  (IoT).  Last  access:  06/10/2014.  Available  at:         Schwaber,  Ken;  Beedle,  Mike.  "Agile  Sogware  Development  with  Scrum."  (2002).       Shayib,  Mohammed  A.  Applied  StaIsIcs,  1st  EdiIon.  1st  EdiIon.  2013.