Cybersecurity and the Internet of Things are increasingly uncomfortable bedfellows. We’ve blogged before on the the security gaps that already exist as a result of connecting yesterday’s technology to the Internet. A recent article by Colin Wood published on goes several steps further and brings us up to date. The Importance of Cybersecurity in the Age of the Cloud and Internet of Things discusses how Government IT organizations are losing control of their technology with the move towards the Cloud and BYOD, and away from the secure, internal data center and vetted equipment. Organizations are realizing that cybersecurity attacks will happen, and that what is required is a shift in emphasis from prevention to response. Based on research carried out by Gartner, by 2020, 60 percent of an organization’s security IT budget will be allocated to rapid detection and response. The paradigm is described as moving from ‘incident response’ to ‘continuous response’, and requires the introduction of new IT technologies – Big Data technologies.

Both Hadoop and stream processing already have a significant role to play in the detection and response to cyber attacks. And much as many vendors in both areas would have you believe, this isn’t an either-or situation. The Internet of Things generates semi-structured event data from sensors, apps,  Cloud and Internet server infrastructure. Hadoop enables organizations to store and process Internet of Things data streams, and to apply pattern detection and what-if scenarios. This enables the signatures for new attacks to be identified much more quickly that is possible today. Perhaps in the order of hours rather than weeks.

However, this is still not real-time which is where stream processing comes in to play. Stream processing is based on the concept of “continuous processing” – each and every new record or event is processed as it arrives, and all results updated incrementally based on that event. Patterns and exceptions can be identified and actions taken in less than a second.

Which is great, but stream processing excels at real-time responses to known patterns, rather than the ad-hoc mining of data streams to identify new patterns. Hence the complementary Big Data architecture of stream processing on Hadoop. Which is where SQLstream excels. Our streaming adapter for Hadoop enables data streams and streaming analytics to be ingested into Hadoop at very fast rates, and for the resulting Hadoop analysis, pattern and trend data to be joined against incoming data.

The move from Incident Response to Continuous Response is therefore one of technology advances, from traditional RDBMS technology to Hadoop-based storage platforms, and by extending IT architectures to the very edge of the network and connecting directly to the data sources using stream processing. Of course other weapons in the armory are equally important, in particular the need for standards, but Big Data will provide the data processing infrastructure for the Internet of Things.