What is the Current State of the art open source frameworks for Big data?


In the beginning Hadoop was simply about batch processing and the distributed file system. However rapid developments in the technology state of the art has brought us to the much talked about Lambda Architecture.

There is no silver bullet even in the big data world with all its advantages over traditional Data Management systems like OLTP (Online Transaction Processing), Data Warehouses running Online Analytical Processing and object and relational models. More on that shortly.


We have arrived today at a place where the Hadoop ecosystem makes it possible to handle unstructured and structured data in a single system. Also, it has the led to capabilities of stored data on a scale not seen before making archiving less relevant. Today your archive can be online and accessible almost instantly allowing you to do analysis on historical data as well as live data in near real time. The bleeding edge of open source is the lambda architecture that combines batch processing approach with a speed and services layer. We are witnessing a convergence of approaches borrowed from SOA, Batch and traditional data management systems. Let’s look at this in some detail.

NAICS Codes: 

541511, 541512, 541513, 541519, 541611, 541618, 541620, 541690, 611420, 511210, 518210, 541430, 541720, 541990, 561110

Where We Work Includes: 

  • Department of State

  • U.S. Commission on Civil Rights

  • FTI Consulting

  • SD Solutions, LLC.

  • DHS FEMA

© 2019 END-TO-END COMPUTING. ALL RIGHTS RESERVED.