00:01Earlier this morning, we saw an architecture to perform analysis on social media streams. Real-time analysis on social media.

00:09Now we're going to look at another big data architecture and how to efficiently analyze more than 3.5 billion records.

00:18So please welcome Mansour Raad.

00:21Thank you John. Working with big data is challenging because of the volume of the data, the velocity of the data...

00:31...and the variety of the data. I'd like to share with you today two things that you can do with the ArcGIS tools...

00:37...that you already have. One, spatially analyzing big data, and two, spatially querying billions of records.

00:48Let's start with the big data analysis. ArcGIS Online processes 1.7 billion records every month. What I'm showing you here... a density map for all the street map requests in the month of February. What we see is we see of course...

01:06...a lot of activity around the urban areas. There was a lot of activity in Germany, but we can see that there's a hot spot... the Southeast Asia region. And actually, there's a reason behind that. It's because we've been updating the streets...

01:22...for that region recently. Lot of activities in Japan; lot of activities in Thailand. And again, one of the many reasons...

01:31...because of that, is because the streets are now annotated in the local language. Let's go now to Europe.

01:40We see a lot of activities in Germany and in London, but something very interesting happens if I switch to analyzing...

01:48...all the topo map requests. We see a lot of activity again in the Netherlands and in London, and there is a reason behind this.

01:58And that is because the local agencies are producing high-quality topo maps. In England, it is the Ordnance Survey...

02:08...and in the Netherlands, it's the Dutch cadastral agency. Basically, if you produce high-quality maps and you share it...

02:18...if you build it, they will come and get it. Now, we've been analyzing big data. I'd like to share something with you...

02:26...that I'm very passionate about. And that is querying billions of records. If you remember a couple of weeks ago...

02:35...a meteor blasted through the atmosphere in Russia. I'd like to do a spatial analysis over the area for all the map requests... the area and the following couple of days to see if there's any trends. So I'm going to be looking for the sixteenth...

02:54...the seventeenth, and the eighteenth. Let me run this. While it is running, let me share with you the back-end architecture.

03:05I have a Hadoop cluster of 18 data nodes on commodity machines running MapReduce, Hive, Pig, and Impala.

03:15Using an ArcGIS Python geoprocessing script, I'm instructing Hadoop to perform a spatial and temporal distributed analysis...

03:26...over 3.5 billion records. When the result comes back, I'm telling ArcGIS to perform a density analysis in such a way...

03:35...I can see hot spots on the map. What used to take hours to run now will take minutes, and the reason behind this is that because...

03:47...we're taking the program and we're sending it to the data rather than taking the data and bringing it to the program.

03:55You can see we've returned 1.1 million records in 48 seconds. Pretty impressive. And now we're doing the density analysis.

04:05And we can see the trends on the map. Nothing unusual over Moscow. Lots of map requests over Moscow.

04:15But if you look to the east of it, we can see a lot of map requests. Why? Because that was the area that was affected...

04:23...with the meteor. So to recap. Two things that you can do today with the tools that you already have.

04:30You can spatially analyze big data, and you can spatially query billions of records. Again with the ArcGIS tools...

04:37...that you already have. Back to you, John.

04:45Thanks, Mansour. You sound like my college professor. Sounds really, really smart, talking about things...

04:51...that I don't totally understand, but I think I definitely understand what used to take me hours and hours and days to do... just did in a matter of seconds by using all those acronyms and all those technologies and putting them all together.

05:04Is that right? Yes, and they are real, as I keep saying. You know, I geeked out, I know, because I am very passionate about it... know, and I wanted to share this with all of you, but it is real. Okay, thanks Mansour.

Copyright 2014 Esri
Auto Scroll (on)Enable or disable the automatic scrolling of the transcript text when the video is playing. You can save this option if you login

Big Data in ArcGIS

Mansour Raad demonstrates the ArcGIS tools for analyzing and querying big data and billions of records.

  • Recorded: Feb 25th, 2013
  • Runtime: 05:17
  • Views: 730
  • Published: Mar 5th, 2013
  • Night Mode (Off)Automatically dim the web site while the video is playing. A few seconds after you start watching the video and stop moving your mouse, your screen will dim. You can auto save this option if you login.
  • HTML5 Video (Off) Play videos using HTML5 Video instead of flash. A modern web browser is required to view videos using HTML5.
Download VideoDownload this video to your computer.
<Embed>Customize the colors and use the HTML code to include this video on your own website
Start From:
Player Color:

Right-click on these links to download and save this video.


Be the first to post a comment
To post a comment, you'll need to login.
If you don't have an Esri Global Login ID, please register here.