Data in the public sector continues to grow exponentially and agencies are being challenged to find ways to leverage big data in the support of government mission outcomes. In my big data blog earlier this year, I mentioned several emerging trends that are driving the adoption of new approaches to data analytics across the federal government – including the volume, velocity and variety (both structured and unstructured data, including full motion video, emails, voice, social networks, sensor-enabled facilities, web and biometrics data). Agencies have started taking big data seriously and I am seeing this technology take a firm hold in the public sector.
In March 2012, the Obama Administration launched a plan to spend $200M every year on funding “big data” research and development projects which would be divided among agencies like NIH, DoD, DoE and USGA. The big data funding is expected to accelerate the research of big data technology solutions. Take a minute to read the highlights of the various ongoing federal programs that tap into the opportunities provided by the advances in big data technological solutions and innovations.
Earlier this month, NetApp and MeriTalk released a report on the big data gaps in public sector based on a survey of IT professionals from various agencies. This report had many useful nuggets of information, but the following pieces stand out.
- Improving overall agency efficiency is the top advantage of big data, followed by improving speed/accuracy of decisions and the ability to forecast.
- Most agencies do not have sufficient data storage/access, computational power and personnel to take advantage of the big data opportunity.
- While federal IT professionals agree there are many benefits to big data, the technology and applications needed to successfully leverage big data are still emerging.
- The amount of government data has and will continue to grow by an average of 64% in the next two years. Nearly a third of agency data is unstructured and therefore substantially less useful unless it can be properly mined.
Big data technology provides significant advantages for agencies – including fighting crime, providing event-driven operational intelligence, controlling cyber threats using sensor data, geospatial analytics, infrastructure protection and counter terrorism analytics, as well as detecting fraud, waste and abuse in government programs. There is a definite need for organizations to be able to combine disparate data sources and be able to draw meaningful scientific correlations and predictions. Technologies supporting big data have emerged and become widely available. Storage prices have come down, processing power has grown exponentially, and new technologies like massively parallel processing, in-memory database, flash based solid state storage, high speed infiniband networks, hadoop based distributed file systems, virtualization and cloud computing are making big data solutions a reality. Are you ready to leverage the power of big data in your agency? It is time to start thinking about big data and how it can help your agency support mission outcomes. Let me know your thoughts. Follow me on Twitter at @GTSI_Architect.