Real-time Traffic App Innovation Powered by Stream Processing Platform in the Cloud
Apps have been the single most important IT innovation of recent times. Apps bridge the gap between users and data, providing immediate access to information when it’s most convenient, presented in a way that’s accessible and useful. However, app development, and in particular the backend platform capability, has struggled to keep up with user expectation. Remember the days of the first mobile Internet based on WAP technology? A lesson in over-hype and expectation mismatch.
Traffic apps have been leading the way in app uptake, helping travelers and commuters plan (and replan) their journeys. Yet how many times have you used an app to plan your route, selected the option with the shortest journey time and the least congestion, yet you still arrive late? That’s precisely the problem that Transport for New South Wales (TfNSW), the Australian government agency responsible for managing the road and public transportation networks in Sydney and the New South Wales (NSW) region, Australia, has been addressing in partnership with SQLstream.
Driving Innovation
The initiative began in late 2014 as a code Hackathon for real-time traffic app development. Of the 50 or so teams entered, eight of the most promising and innovative ideas were selected by TfNSW and sponsored through to launch early in 2015. The first five apps have been launched, with more to follow, each focusing on different aspects of the traveler experience. Go to the rms.nsw.gov.au website for a full description of available apps and App Store download links.
Real-time Traffic App – End to End Journey Time
- Snarl gives up-to-date information about road incidents and congestion across the road network and close to the customer’s travel route or location.
- TripGo is for customers who want to find the most efficient journey from A-to-B using road and public transport options and designs smart itineraries around information on home, work and calendar events.
- Auto is for customers who have a number of locations they often travel to and want to see the current travel time to help them plan when to leave.
- Plan A helps customers with a busy schedule who want up-to-date information on congestion, suggested travel times and want to be alerted when to leave to ensure they arrive on time.
- EasyDrop is for commercial delivery drivers who want an interactive tool to manage their daily delivery schedule, plan for the fastest routes and be up-to-date with traffic conditions.
So if you’re in the New South Wales area, and you’re trying a new app that’s now offering accurate, real-time updates, and suddenly you start arriving on time as planned, it’s likely you are using one of TfNSW’s new road apps that are powered by SQLstream’s stream-processing enabled, real-time data hub and Big Data applications.
The IT Architecture for the Next Generation of Real-time App Platforms
The next generation of web app platforms must scale to deliver accurate, real-time information to a rapidly increasing app community, and scale to support many apps on the same core platform architecture. Conventional web architectures offer on-demand access but can only deliver results based on historical content that is database-driven. As has been the case with traditional enterprise data management architectures, a data stream management platform offers real-time scalability for larger data management applications.
That’s not to say that existing data warehouses are redundant, far from it. The future is an integrated, unified data management architecture that utilizes the strengths of the best mature technologies.
TTDS – Real-time Data Hub for Web Apps – Architecture
The major components of the enterprise web architecture supporting the Travel Time Data System (TTDS) as deployed by TfNSW is as follows:
- Data Warehouse. The master record is maintained in the data warehouse(s), in this case SQLServer, and is the platform for data mining, trend analytics and enterprise reporting. The SQLServer data warehouse is populated from a number of sources, including existing enterprise systems, as well operational traffic data and streaming congestion analytics streamed through in real-time from the data stream management platform.
- Real-time Data Hub for Enterprise Apps. Real-time operational intelligence platform is built on Guavus SQLstream with the SmartCity StreamApp. Operational users gain actionable insights through real-time map-based dashboards and alerts. Analytics are the result of the streaming joins of input vehicle GPS and sensor data streams from over 10,000 vehicles, joining with the historical trend data from the SQLServer data warehouse.
- Scalable Web API Platform for Real-time Access. Web application access for Smartphone apps is provided through a REST API. The API supports real-time queries for journey time updates, traffic flow and incident information as well as crowd-sourced. Additional web servers are added to the cluster as app uptake increases.
- Real-time Middleware. Inter-application and distributed server cluster scalability requires a middleware-based architecture (Apache Kafka).
Real-time Traffic App API
App developers have access to the Mobile Application Streaming API which is based on HTTP 1.1. The API connects to SQLstream’s Real-time App Platform and supports the following HTTP requests. The API is serviced based on the current state of the road network, traffic flow and incidents, and the current calculations for journey travel times.
| Config API: | Returns API URIs and client configuration values. |
| Route API: | Accepts end to end journey route path and departure/arrival time, and returns the route travel time and current incidents on that path. |
| Progress API: | Accepts device position, speed and heading and returns travel time updates and current incidents on that path. |
| Events API: | Returns all incidents on the road network. |
Third party developers also have access to the Server API for developing their own on-premise or cloud-based SQLstream applications. The Server API delivers the real-time traffic data feed over a federated connection to the third party Guavus SQLstream system.
Real-time Travel Time Calculations
Journey time planning with real-time updates is the most popular category of traffic apps. Accurate journey times must be calculated based on the actual traffic on the road at that point in time. This is where most device and web apps fall down. Using crowdsourced data and historical trend information alone is insufficient. The Travel Time Data System takes a different approach based on a combination of vehicle GPS data feeds and crowdsourced app data, with traffic flow and congestion alerts calculated in real-time over the entire road network.
The SmartCity StreamApp includes real-time traffic flow and congestion analytics dashboards that are available for internal use, showing the entire road network on interactive, map-based dashboards (click on the thumbnail above for an example). However, app developers are now able to offer travelers access to the same data through their smartphones. The following example is the web-based version (link to www.snarl.com.au) of Snarl’s app for up-to-date information on road incidents and congestion across the road network.
Snarl’s Real-time Traffic App (Web Version)
The road network is represented as a GIS database of 10 meter road segments, with a total of 80 million road segments for the New South Wales network. The SQLstream platform updates traffic flow and congestion incident predictions over road segments simultaneously and in real-time for every new data record, where the average current speed for each road segment is calculated as follows.
- End to end journey times are calculated as a function of all 10 meter road segments in an end to end route.
- Vehicle GPS data is used to calculate average speed where a vehicle has travelled across that road segment in the past 15 minute.
- If no vehicle has travelled over a road segment in the past 15 minutes, the 28 day historical average for a comparable 15 minute time period is used for that road segment. That is, the next best choice for Saturday at 9am would be the average for that 9am / 15 minute period for all previous Saturdays within a 28-day window.
- In the unlikely event that no vehicles have been recorded over a road segment in the past 28 days, the posted speed limit is used.
The minimum and maximum travel times vary based on the above calculations and the variance. The variance is calculated where at least two vehicle reports have been reported over any 10m road segment.
The real-time updates are streamed to both the operations centers, where the current traffic flow and predictions alerts are displayed on color-coded, map-based displays, and also used to service requests in real-time from the smartphone apps through the Mobile Application Streaming API.
What do these new apps mean for commuters?
The next generation of real-time traffic apps offer commuters with accurate and reliable journey time and other traffic-related information. Moving away from a reliance on historical and crowd-sourced data means greater reliability, accuracy and timeliness. However, the real-time processing required also requires a change in mindset. By extending the traditional enterprise data management architecture to embrace the benefits of data stream management, a real-time platform for app support has been delivered that scales with both data growth and app uptake, to the benefit of all concerned.