00:01Yeah, my name's Bruce Harold. I'm the product manager at E-s-r-i, Esri, for the Data Interoperability extension.

00:06First qualification is not being able to mumble when you say the name of the extension.

00:11With me today is Dale Lutz, vice president of software development and one of the cofounders of Safe Software…

00:16…and together, we engineered the Data Interop extension.

00:21So what we're going to…we've got a lot of ground to cover today.

00:23Hopefully, the gods of the Internet willing, we'll be able to show you a lot of material.

00:27And if you have a burning question, just pipe up with it during the show.

00:30Otherwise, we're going to have enough time for questions at the end.

00:32And of course, we're more than willing to follow up with you after the show, with your own issues to chase.

00:40So what we're going to do, a little double act here.

00:42Dale is going to do most of the talking; I'll interject if he lies or forgets something.

00:46Otherwise, I'll be doing most of the demonstrations, and he might do…pipe in then and again with stuff he likes to show about XML.

00:53So I think without further ado… Can everybody hear me? And sound check from Dale.

00:58I think we're good on my side. Sound check? Yeah.

01:00Alright. I'll face forward to not talk to Dale sideways so I don't lose the mic when I'm on.

01:04So I think we're ready to go. Take it away, Dale.

01:07Okay, thanks, Bruce. So just before we get started here, I'd like to get a feeling of the audience members here.

01:12How many of you are already using either FME…

01:15I see at least one knows what I'm going to ask. Two.

01:17…are already using FME or Data Interop extension? Okay, so we've got a fair number of folks that are new to us, and that's great.

01:25Thanks very much for making the time.

01:27I guess I should also say that you're going to be treated today not to just one interesting accent.

01:32You've got a New Zealand accent and also a Canadian accent, so apologies in advance if you have to sort through that.

01:39So just some background, Bruce already mentioned that Esri and Safe worked together to build a Data Interoperability extension…

01:48…I guess, almost six years ago. As far as I can recall, it was 2005 when it was first born.

01:53And it was a marriage really of FME and, at that time, ArcGIS 9.

01:59And so we brought the data format support that we had inside of our FME product…

02:05…together with the extensibility of ArcGIS 9 at the time to allow ArcGIS 9 to read formats that we supported…

02:13…that weren't in ArcGIS to do quick translations and, as well, to do more interesting stuff, which is what we call spatial ETL…

02:21…or rearranging, restructuring data as it moves from one format to another or indeed, within the Esri world to the Esri world.

02:32And as a footnote, in case I forget, you can use Data Interoperability extension, for example…

02:38…to prep data to be used in the Community Maps Program.

02:42So you're starting inside a geodatabase; you need to get to a different data model.

02:46The Data Interop extension is a very good thing to use for that.

02:49And actually, we have a webinar coming up on that in a couple of weeks' time.

02:52If anyone's interested, talk to me afterwards or come by our booth.

02:57So lots of data formats are supported.

03:01What a lot of folks don't realize is that besides just reading directly different formats…

03:06…you can use the Data Interoperability extension to integrate data.

03:10So you can take 2D things and make 3D things.

03:13You can bring in CAD and do very sophisticated, fine-grained things on CAD files…

03:20…extracting block attributes or setting extended entity data.

03:23All that sort of stuff just within the CAD world can be done.

03:27And we can add textures or images to 3D objects as you write them to multipatches.

03:33And if there is time at the very, very end, I may give you a sneak peek in 10.1.

03:38We can do a bunch of stuff with lidar that complements the LAS support that's in ArcGIS 10.1 very nicely.

03:45The Data Interop can clip lidar. It can slice it. It can split it based on returns.

03:51All those sorts of things can be done as well as you prep data for later use inside of ArcGIS.

03:57And don't get me started on XML. Our cofounder at Safe, Don, is a huge XML fan.

04:03He likes to go to…I'm not joking. Goes to [unintelligible] like this and says, "Who here loves XML? I do."

04:08I'm not quite the same as him, but we can do almost any kind of XML that you could imagine…

04:15…can be solved with Data Interop to break it apart or produce it. That can be done.

04:21So kinds of tasks, bringing things together.

04:24You might use Data Interop to migrate once from somewhere to somewhere else.

04:29More often, though, people use it to continually move data around within their organization.

04:35You can use it to do QA of data.

04:37It's a very convenient tool to set up rules that say, Look it, I'm going to run my input through here…

04:43…and all the stuff that's got bad attributes in this way or that way, or maybe the geometry is malformed in some way…

04:49…I can fish that stuff out.

04:51You can send data to other folks. I think Bruce might treat us to a change detection example. Is that true?

04:57Yes, the rumors are true.

04:59So you can tell what's changed. You can manage query layers, so you can load data to be used later in query layers.

05:07Hands up, who's using query layers? Anybody using them?

05:10So there's direct use of spatial types in the DBMS. So that's something.

05:15Because the Data Interop extension can read and write directly to a number of different spatial databases in their native form.

05:24So not only is it an extension, but if you just install it, you get some stuff for free that's just for the privilege of installing it.

05:34So if you have to ever deal with GML, probably if you're from North America, you don't have that happen too much…

05:40…but some of our European friends have that, so they can read and write what's called GML simple features.

05:45And you can also read and write…I'm sorry; read, access WFS, Web Feature Services that belong to the OGC spec.

05:55And new, I think, in 10…was that when this business of cached and noncached came in, Bruce?

06:01Yeah. Yeah. If your WFS server is an ArcGIS Server, then the ObjectID is exposed in the WFS string…

06:09…and if the client, in other words, Data Interop sees that and you turn on the switch, then the features won't be cached.

06:15The time won't be taken to cache them. So each map refresh will be a new mouthful of data.

06:21One of the interesting places that we at Safe Software live in the food chain is that we have to sit in the middle…

06:25…between these different systems that might have different approaches to the world.

06:30And in the Esri world, there's very much a view that every row has a unique integer ID.

06:36That's fine in the Esri world, except when we try to bring stuff in from, say, WFS; they don't have that property.

06:42And so that puts us in the uncomfortable spot of trying to fit a square peg in a round hole, and there's a…

06:49…the only way we can win is either (a) come up with a unique integer ID, so beg the person that's making the WFS to have one…

06:55…or, if it comes from ArcGIS, it will have one, or (b) we suck the whole contents of that WFS down locally…

07:03…perhaps only a search window gets sucked down locally, give it all IDs, and then ArcGIS can do its thing.

07:09But we give you the power to basically decide how you want to fit that square peg in a round hole.

07:15So Bruce, I think you're going to switch me off here somehow.

07:18You're gone. Am I gone? There you go; just like that.

07:22So everybody, it was of course ArcCatalog application; I just want to show you that the extension…

07:28…the Data Interoperability extension is available but not enabled.

07:31So it's not there, it didn't cost us any money. We're using the F word…it's free so far; you just have to install.

07:37And I've navigated in the browse tree down to some GML data.

07:41So all of the data sources that are visible to Data Interop look like a feature dataset.

07:49You see this icon here where my mouse is moving around? That's the kind of a feature dataset icon.

07:54Inside there, the system is seeing some polygon data, and you can see the map of New Zealand…

08:00…and it can treat it like you normally can in ArcCatalog.

08:04So what's happened behind the scenes is the GML file has been locally cached…

08:07…into something which ArcGIS can read as a thing called a plug-in data source.

08:12Now, sad fact of life is that data is fundamentally evil, and some things can happen with some formats.

08:19And here's one of them that happens with GML or can happen with GML, is the x's have been swapped.

08:24So it can't, doesn't get much worse than that. Unless you can go through to the antipodes of New Zealand…

08:30…which happens to be Spain, and you cross your eyes and you'll see the data okay, you're toast at this point.

08:35So if you do purchase--so getting away from the F word there--the extension…

08:40…you get some more switches and capabilities to fix problems like that.

08:44And that's one little way you can play with GML, so locally important…

08:48…perhaps less so in the US, but quite important in jurisdictions in Europe and in other countries.

08:54So of course, a fancy way to consume GML is as a Web Feature Service, WFS data.

09:01And here I've got an extent of where I used to live in New Zealand…

09:04…and I'll zoom in on this map and show you some of the data that's coming down the wire there in a moment.

09:11I'm navigating now in the Catalog window in ArcMap to the interop connections that I've made.

09:16So they're accessible to me anywhere in my…any session; actually just to my login account.

09:22So for this one here, I've got…oh, let me see. This one here, let's look at the properties for this connection.

09:30So it'll take a second before it comes up with the Properties window.

09:35I look at the properties there, I can see an ugly-looking URL to a dataset…

09:39…but there are other parameters available in that WFS connection.

09:42And you see in this case I've used a search envelope.

09:45So while at the back end on the server there are many millions of features, I've plugged in here an extent which I want to use as a filter.

09:51Also I can say like how many features maximum do I want to download and cache at any one time. I'm saying there 50,000.

09:58So I'll cancel out of there and go back to the Catalog window.

10:05And I'll say for this WFS service, let's look at its connection properties. Go back to this tag again.

10:18I haven't used a window, but what I have done is used a filter expression.

10:23So my XML skills are rudimentary at best.

10:28The one behind me, but I did manage to parrot off a filter expression I found on some document somewhere…

10:33…and I'm saying the attribute value, attribute field Affected Surveys has to be this value here…

10:40…which stands for deposited plan, and then some number, and the land district has to be North Auckland…

10:47…because this survey number is not unique across the whole country.

10:52So the net result of doing those little pieces of work with a WFS connection is my data's streamed down to my client [unintelligible]...

11:03…can go away while it makes its mind up, and when I've loaded the data into ArcMap, it will behave like any other layer.

11:11I will show you when it comes back to me. I've made a definition query, I've symbolized the data on attribute field…

11:17…the attributes are typed, they have a schema like anything else. Still working on that.

11:24And I can use them in geoprocessing as well as mapmaking.

11:28So for each session, the data is, if noncached, downloaded for me and cached locally and will become available in my session.

11:37Think that's going to take a while to return after its excursion down the web there, so we might…

11:43Shall we flip back? Yeah, we'll flip back. We'll see it when we come back. We'll finish that later on. Dale's back on.

11:48Okay, let's see what happens. There we go.

11:51So just to kind of summarize this section, GML and WFS comes free, but you have to install.

11:56And I noticed on the ArcGIS forums in the last couple days, somebody had built one of these things as Bruce was showing…

12:03…published that map document to their ArcGIS Server and it didn't work.

12:07Well, what they forgot to do is with ArcGIS Server, they had to install Data Interop extension.

12:10You don't have to license it, but it's got to of course be installed.

12:14Simple parameters are there, and you can do the stuff that Bruce just showed you.

12:18So the next section we're going to do is some of the simple things you can do with the Data Interop…

12:24…and that is the direct read of many different formats.

12:28So when we can tell what the format is, based on its file extension, we'll just read it as if it was a native ArcGIS format.

12:36So things like MapInfo files, things like--let me think about an example like…

12:43I can't use the CAD examples 'cause we can connect to those, but large number of regional formats, KF85.

12:51You can kind of see here even SketchUp files, we can directly read those 3D different formats like that, 3DS files. Yep?

12:59[Inaudible audience question]

13:05Answer that one? Yeah. You think you have a description object in KML data, yes, it's preserved…

13:10…but even better, with Data Interop extension, if your KML data has a schema object…

13:15…that is preserved and used to type the attributes that are seen in ArcGIS.

13:19[Audience question] And that's at version 10?

13:21Yes, it's in the released product now.

13:23Yep, yep. So, yeah, you can see there the KML is in one of them as well.

13:27So anyway, you can do that. You can also extend. You can add your own extension in there.

13:34There's people that might have used MGE; if any of you are old like me, you might have seen MGE used.

13:39They use kind of funny file extensions. You can make that work in there too.

13:44And so when you're doing this, you can just directly use any of those things as if they were native ArcGIS tables…

13:50…and so you can run them right into geoprocessing.

13:52You can display them, you can run them in ArcPy.

13:56I know Bruce is a big ArcPy fan, so you can use them in there just like that.

14:01And you can also make a connection, and Bruce was showing that with WFS, to a format.

14:08And you need to do that when either the file extension isn't what you'd expect it to be or you want to set some parameters.

14:17So you can use Data Interop, for example, to read CAD files directly…

14:22…and you have a bunch of options at your fingertips there that extend what functionality is in the base ArcGIS product.

14:28So you can split by level names or by geometry type. You can do that kind of thing.

14:33You can also, in one connection, type in

14:43You don't have to actually split them out by their file names.

14:46And you can set a coordinate system there as well.

14:49So Bruce is showing on this slide, using it to connect to SQLite, for example, that's an open source database package…

14:57…or to…you can just about do any of these formats in that way.

15:02And then once you have these connections, you see them in the part of your ArcCatalog tree where…

15:10…underneath that Interoperability Connections heading.

15:14Oh, yeah. I want to get one other thing there.

15:17Bruce mentions web feeds, so you can use this to connect to GeoRSS URLs.

15:23You can also pull KML off of a URL in that way, RSS feeds.

15:28You can also make a connection to a database, which when we first did this was very critical.

15:33Now that there's query layers, it's less likely that you'd actually come through this way…

15:37…unless it was a database that we support that isn't supported through query layers. Right.

15:43And so I think, Bruce, we're going to hand it back to you to show us your magic.

15:48One of these times, I'm going to forget to switch the video.

15:52Okay, so here we are in somewhere in Texas, I think, from memory.

15:56We have Texans in the back of the room. Oh, do we? Does that look like Texas to you?

16:00[Inaudible audience comments]

16:01I'd have to do some… That's a pretty good clue, isn't it?

16:03I'd have to do some reverse geocoding, double-check that.

16:06But as I mentioned when I was showing the GML data, we can browse on disk and directly read foreign to ArcGIS…foreign formats.

16:15And here's a couple of examples here. Floodplains.mif. Now, MIF is MapInfo Interchange Format.

16:22Like it's their equivalent of, I don't know what. Shapefiles. It's an open published format from their point of view.

16:29So I can just drag in that data into my document here, my map document here, and I've got the floodplain data.

16:36Now, embedded in MIF is coordinate system information as well, so it's analogous to our data types in ArcGIS.

16:45Traveling…here's another one. This is a TAB. Again, a MapInfo format I think.

16:50And I can just drag in the subdivision polygons there, and they'll turn up and join the data quite happily in the right place.

16:58It's not always as simple as navigating disk to find foreign format data 'cause some of it's quite complex.

17:05And we were talking just a second ago about the requirement for connections to be made.

17:12And connections have the ability to set properties which you need to read data.

17:16Dale mentioned EMG [sic] format, which is Modular GIS Environment from Intergraph, right?

17:23And I think I have some data here that is that, LinesPoint.

17:31It's indexing that data, so it should turn up somewhere in the same area.

17:38Where'd it go, Bruce? I don't know. That's a good question. Sometimes things don't work.

17:44We'll try that again. No, say it ain't so. Yeah…lines. There might not be points.

17:51Try one more time. Yeah, there were yesterday.

17:54No, you need LotPoints if you want points, Bruce.

18:00Trouble. Sometimes these things don't work as advertised.

18:01No, go for LotPointsPoint. I'll take LotPointsPoint for a hundred.

18:04Oh, I see, Gotcha. Alright, now we go. Use the arrow. LotPointsPolygon…

18:09And you might want LotLinesLine.

18:12This format is just so confusing. I'll take LotLinesLine for 200. Okay, yeah, he's a winner.

18:18Anyway, we got there in the end. Sorry Identify didn't query.

18:23So in other words, in browsing some foreign formats, what I was going to get to, talking about the… Thanks, Dale.

18:28…the connection details is some of them can be quite complex.

18:34So not just a path to some data, but they pull tricks like their data store can't have any extension at all. In this case, it's PIR.

18:43And so of course Windows Explorer is not going to be able to recognize data…

18:47…by its extension type and a whole raft of other parameters.

18:50Like in this format, the attributes for the data are stored in a remote DBMS. I choose it and set the parameters to read it.

18:59Anyway, never let this ornery, evil data stop you; just give the extension and learn a few tricks…

19:05…about how to read data that you're getting and have to live with and just get on with life.

19:11So I'm going to get rid of these things that Dale made me add to the map here…

19:17…and remind everybody that we don't always have the luxury of data being a file on disk.

19:26Sometimes it's a DBMS we connect to, and sometimes it's a feed off the web.

19:30Now, I bought my wife an iPhone recently…

19:33…and she came bustling in the room a few days ago and said, "There's a tsunami alert, a tsunami alert."

19:38I said, well, I live 150 kilometers from the ocean, which is…I live in Redlands. 00:19:42

19:46Turns out she was talking about New Zealand there was a tsunami alert, because that's where I'm from…

19:50…and we both lived there until a few years back. So, well, what do we do?

19:55So if we want to investigate the relative life of what's happening with earthquakes and stuff…

20:00…we can go to the US Geological Survey website and we can have a look at an RSS feed they have on there.

20:06They've got a number of them.

20:08I'm connecting here. We're looking at the path to an XML file which is the RSS feed, which shows the current state…

20:14…and I'm in real time pretty much; I don't know how often this is refreshed, but pretty regularly.

20:19You can look at all the earthquakes that are happening all the time.

20:23So back in my map here, I made an RSS connection to this data.

20:32I can see it just reads from the web all that XML data and displays it on the map without my having to do anything.

20:39Now the tsunami alert that my wife was particularly worried about was for this part of the world.

20:44 It's off a tiny little island which is so small it's not even showing up right now called Raoul Island where I have been diving.

20:52I can tell you there's a lot of sharks and a lot of turtles.

20:55And the sharks and turtles will have been worried about this one here, a 7.6.

21:01And that resulted in a two-meter wave on the shore of Raoul Island…

21:06…which is approximately half the size of the normal waves there anyway, so that wasn't any much of a panic.

21:11One thing I might notice, though, and this will lead on with what to do about this kind of issue later on…

21:15…is that this is not ideal for mapping.

21:17They've confounded the magnitude of the quake and its location in the one field…

21:23… and they even stuck a character in front of the magnitude that's not even a number…

21:26…and other nasty stuff which you're going to have to deal with at some point if you want to make a good map.

21:30But that's for a later show in about 10 minutes, so with that, I think I'll hand it back to Dale after showing you direct read of data.

21:37Okay. So while it settles back there, what you just saw was directly reading data that's from a bunch of different formats…

21:44…without doing any kind of translation at all; it hits it live off the disk.

21:48Particularly useful if you don't want to take ownership of data.

21:52So in other words, some other group in your office is using another product; they've got their stuff sitting there.

21:57You might want to use that stuff.

21:59Traditionally you might imagine you'd do a translation into geodatabase and refresh it every week…

22:04…but you can instead hit it and read it live, and each time you go to look, it's getting the most recent version.

22:12So, next we'll talk about quick things.

22:17So we're building up to the exciting conclusion here, getting more and more sophisticated as we go.

22:22So the next step here that Bruce has highlighted is that we should look at the toolbox that comes…

22:27…with the Data Interoperability extension that has exactly two tools in it, Quick Import and Quick Export.

22:33So you can go out to just about any format, or you can come in from almost any format.

22:38And again, you can do these things from within a model or interactively, one at a time, however you might want to do it.

22:44So actually we have seen many customers put a quick export at the end of a big model…

22:51…because they're going to send the results of the analysis out to their neighbor who uses MapInfo or something like that.

22:59And so, yeah, I'm saying it can be done in a model, and you can do multiple files in and out at the same time.

23:09And so, Bruce, are you ready to take the helm? I am. Here you go.

23:14I had it wrong before; it wasn't Texas before, but it is now. I just double-check and reverse geocode.

23:19We're somewhere in Austin, Texas, and this is a pack.

23:22Some twitcher out there has gone and mapped the location of bird nests and how many eggs are in them…

23:29…and unbelievably, the offset of the eggs from the center of the nest. So it's just going a little bit too far.

23:36But regardless we've received this data in a SQLite database file. It's just a file with a .db extension; sometimes .sl3 extensions.

23:43I really love this format. It's a high-performance, single file DBMS. It's support for its built-in to Python.

23:52I've used it in script tools to hit it as a temporary data store. It's great for cross-DBMS queries.

23:59And even better, we can map it using Data Interop.

24:04One problem though. Here, let's go look at where the nests are, let's drag them in.

24:08Looks like it should be point data, and it looks like it should be point data there too, but in fact, it's not.

24:16It's just tabular data with x and y.

24:18Have no fear. Of course, in ArcGIS, we can turn x and y fields into geometry, but not if the system already thinks it's got geometry.

24:26So we're kind of toast here, so what to do?

24:31The problem we have is that in the morass of format supports that are out there…

24:35…it's not always possible to reliably tell what geometry's in a table. There can be many, and the default is point.

24:42So what we're going to do is we're going to bring the data quickly into a geodatabase.

24:48So my input, birds.db file, is going to be quick imported into a geodatabase.

24:55And there's a model-only tool called the Select Data tool, which will retrieve a feature class or table from a workspace…

25:04…and in my case, I'm going to retrieve both the nests and eggs ones, and I'm adding to display.

25:11So I'm just going to run the model, and the Quick Import's running back there and Select Data will run momentarily. There it goes.

25:21In a few seconds, we've got that. I won't save the change.

25:26Back over here, hey, presto! Eggs and nests are now tabular data sources, quickly imported in here.

25:32So I'm going to say let's display x,y data. I'm going to say…

25:37I happen to have a…this one. Can't remember which one it is.

25:43Yep. That one? I think so. Yeah, it'll do.

25:47Things will turn up or I'll have to use the other one. Yes, there we go.

25:49There's the nests now displayed as an event layer.

25:52I can now do something else that's geodatabase-y like…

25:55…add a relate from--we happen to have nest ID in the table that we've now put it into.

26:05Did I do that right? Yeah. I think so. You've got to change that one to… Eggs, eggs, yeah.

26:10Brain's going to sleep here. Nest ID. Nest eggs; everyone likes to have those.

26:17So it's remarkably, this species of bird, whatever, is very sociable; the nests are all handily convenient.

26:24To identify, I can see that that one had a bunch of eggs in it; one above didn't.

26:31So we're doing geodatabase-y things after quickly importing a foreign format virtually painlessly so…

26:36…that, ladies and gentlemen, is quick import.

26:40Good. And we're back, and so just to very…you know, they're very straightforward tools.

26:48They let you work very conveniently within the ArcGIS geoprocessing framework with all this foreign data…

26:54…give you a way to bring it in so you can work with it.

26:57You can also send it back out again if you need to. And as Bruce points out, you can use ModelBuilder and also Python.

27:05So now do yourself stuff.

27:08We have this ability to make your own format by piecing together some of the primitives within the Data Interoperability extension.

27:17And so you build this workflow using the Workbench tool, can it, save it, and now it appears as if it is a brand-new format.

27:26And so, Bruce, I'm giving it back to you real fast. There you go. What have you got for us this time?

27:34Okay. I guess it's my fault because I built the slides, huh?

27:38Remember when we looked at the earthquake feed, we didn't find the data ideal for mapping purposes.

27:44Right, great example.

27:45Get x,y's on the page, but we couldn't symbolize by magnitude; it's an obvious one that you'd want to do, so what to do?

27:56So back in the Catalog tree here, you can add interoperability connections, and you come up with this dialog here.

28:05Lets you choose the format you're going to need for the connection to the data, and then you add in, say, a path or a URL to the data.

28:13If you need to…you get the last 10 I think formats that you've used as your default choices in the pull-down there.

28:21If you need to, you go to what's called the Format Gallery, and you get the selection of options that you're licensed to get.

28:31Now, notice some buttons down here. You can say New Custom Format or Import. Okay?

28:38So it's like a cooking show, that's Dale's analogy. Here is one we did earlier; it is for this purpose, the USGS quakes.

28:46When I click on that, the system knows that that's a custom format that I've built…

28:51…and so it gives me the option to edit that format.

28:53So that's rather than making it from scratch and possibly having finger trouble…

28:57…I'm going to go and edit that custom format and show you what I did.

29:02Now remember when we inspected the data briefly in the previous map, we had two attribute fields.

29:08Was it Title and Content? Title had M then a number, comma, then a location in words, and the other one had a date in words.

29:21So our goal is to extract useful attribution from those two character fields and turn them into proper data.

29:32This is, if you're not already a user of Data Interoperability, what we're seeing now is the Workbench application…

29:38…which is installed with the product.

29:40We're going to go into a lot more detail with the more powerful demos coming up…

29:43…but this, so we'll walk you through this one, since we seem to be making quite good time.

29:48So what happens, the old snake chain here is going on, we've got data being read in with this thing here…

29:54…and data being written out with that thing down there. They're called readers and writers, so that's not too hard.

30:00What do I want to do with the data that's come in?

30:02The first thing I want to do is split that field that had data with a comma in it into two elements.

30:08And I'm splitting a field called Title with a comma character, and it's going to output a list.

30:16That's a foreign concept to ArcGIS, a list within a row. In ArcGIS, we have a georelational model.

30:23Within the Workbench application, we can break that model and we can carry it through all sorts of data like…

30:29…and structures like XML documents, PDFs. In this case, we're talking about lists.

30:35So I'm splitting that character field into two bits.

30:39I'm getting the bit before the comma then with that transformer there, and I'm giving it a name Magnitude…

30:48…and I'm deleting the M which prepends the data.

30:51I go on and do the same thing for the bit after the comma and give it a name; I'm calling it Location.

30:57And last step all in my custom format definition here, I'm formatting that date string there into…

31:04…data that's suitable to be pumped into a date type field, or time stamp field.

31:10I'm constraining my output to be the entry points, so any point geometry, and I'm saying my Magnitude…

31:16…I'm going to make it a float field. Location's going to be staying the same; Time is going to be a date-time.

31:22So I've split Title and Content into three fields.

31:25Magnitude is a number, Location is a character field of 30 characters, and Time is a date-time field.

31:33So that's making a custom format; it's as simple as getting some transformers together in Workbench application.

31:44And I've got custom quake feed here. If I just open it up, I will do. Oops, sorry. Do it up here.

31:53The data will be reread in real time. I add it to my map there.

31:58Here come the points. This time I can see the fields which I made for myself there with my custom format…

32:06…and I'm going to symbolize on Magnitude. I'll give it some…oops, I missed.

32:14Magnitude, I could easily derive the diameter of the dots by that as well.

32:20So okay, so now I've got a thematic map.

32:23So that's one example of a custom format, handily rearranging some free data and giving me the schema that I'm shooting for…

32:31…to consume it ideally inside ArcGIS.

32:35So I think that's all I wanted to show with the custom formats. Dale?

32:39Okay. So the good thing is that…to remember about these custom formats…

32:44…really we're layering on some transformation, on top of an existing format.

32:48In this case, Bruce was starting with GeoRSS which just had a description and a title.

32:53He knew there was goodies inside there; he wanted to split it out.

32:56He could then make the kind of schema he wanted the rest of the world to see, the data structure the rest of the world to see…

33:01…and then that flow would happen every time that custom format is used.

33:06And he can take that custom format and give it to his friends, share it with his organization…

33:12…doesn't have to ever create it again.

33:14And you could layer this on top of, let's say, some CAD files.

33:17In older…in days of old, people used to encode interesting things inside of color and line thickness, and so on.

33:23You could set up rules like this that when you're displaying and reading the CAD files from your engineering group…

33:29…they appear in ArcGIS using vocabulary you'd understand like "two-lane road," "paved," and so on…

33:37…instead of "blue, thin." So this is the kind of thing that can be done.

33:43Alright. For our next section, which Bruce has entitled Cool Stuff, we're going to talk about spatial ETL.

33:50And this is the part, honestly, where the real value comes.

33:54And actually, sitting in the back of the room are my friends from San Antonio Water District.

33:58Have you given your talk yet?

34:00[Audience comment] No, tomorrow at three-fifteen.

34:01Tomorrow at when?

34:03[Audience comment] Tomorrow at three-fifteen.

34:04Okay. So and what's it called? [Inaudible audience comment] Okay, I'll let you work on that until later.

34:08But nonetheless, if you want to see how they have used this part of the Data Interoperability extension…

34:13…to make a big business difference in their organization, you should go to their talk tomorrow.

34:18But really, this is where the main power comes.

34:21Yes, solving all these format things is wonderful and helps in many cases, but the real value comes…

34:28…when you start to apply the kind of technology Bruce just showed to all kinds of data transformation things.

34:34And you can get stuff done faster than you would if you were writing code to do this, which is usually the other choice.

34:40If you're going to start parsing field, rearranging things, and so on, you can use the spatial ETL tools of Data Interop…

34:47…or you can write Python code or, even worse, C++ code.

34:50And so using these tools can make you much, much more productive.

34:56So there's…inside of the Data Interop are these things we call transformers. These are operations that alter data.

35:03And again, I'll invite you down at our booth at 1206, the Safe Software booth…

35:08…we have a little handout that's a catalog of all the transformers.

35:11And if you're interested in seeing the scope of what's there, come on by, grab that. It makes wonderful late-night reading.

35:17But you can go through this catalog; there's about 400 of them now, a bit overwhelming…

35:21…but they do everything from string parsing to doing 3D extrusions and everything in between.

35:27And that's all that's there.

35:28The other thing that's key is that many inputs are possible.

35:31I can bring a CAD file and an orthophoto and somehow do something with that and produce a geodatabase.

35:37I can rearrange many different input types, and I can also output to many things.

35:43So I could output a geodatabase and a 3D PDF if that's what I felt like.

35:47You're not really constrained in terms of what's coming in and out.

35:52So you graphically design these workflows; Bruce showed this already a little bit.

35:57You experiment. It's very easy to pop these things down, run a partial translation, see where you are.

36:02There is a thing in here we call the visualizer that you can connect anywhere in the flow and hit Run…

36:07…and then it pops us and shows you what would be at that stage.

36:12And in 10.1, you can actually debug these things by watching each feature go through one at a time.

36:17Am I allowed to talk about 10.1? Yes. Okay, good.

36:20Safe doesn't let me talk about FME 2012, but at least I can talk about 10.1 here.

36:25So, yeah, you start with something, you transform it, and you put it out.

36:28And that's where the ETL comes from--extract from somewhere, transform, and load.

36:33And we think that the T is the most important aspect of this whole thing.

36:39So as Bruce says, it's a pipeline. The readers read things, there's geometry and attributes.

36:44Then we transform things, and then we write things. And so that's the very useful way…

36:50And as you're crafting these things, you often don't care about where you're going to write it until the very end.

36:56So you start with your reader, you start adding transformations and visualizers making sure…

37:00…yes, I'm weeding through the data, I'm throwing things out I don't want, I'm getting what I want…

37:04…yep, looks good, looks good, looks good.

37:05Okay, at the very end, you say, Now where do I want to output this?

37:08Because really, the format is somewhat irrelevant. You just can decide that at the very, very end, and then you say…

37:14…okay, yes, I wanted to put this out to shapefiles today, and maybe I'll also add, for good measure, file geodatabase.

37:20That's a decision you can make at the very end.

37:22I think the format can be dynamic.

37:24Yes, that's also true. You can defer the decision on the format until run time.

37:29So you can make a geoprocessing tool that one of the parameters that it asks the user is…

37:34…What the heck format do you want this in when I'm done?

37:36And you fill it in, and then it'll just do the right thing.

37:39So you can dynamically change that.

37:41Inside of the tool, the thing that enables that, for those that are interested, is something we call a generic writer.

37:47For that matter, we have a generic reader as well, which will read anything.

37:51You can…it'll guess by the file extension or you can ask the user, come on, give me a hint here; tell me what to read.

37:57But you can make one workspace that can take data from anywhere as well.

38:05So sometimes there are situations where you might have these graphs that only have one thing.

38:11So in this case, Bruce is doing a change detection.

38:14He's reading two versions of something and then comparing them and then outputting to three different tables.

38:22And all the work is happening in one.

38:25I think the most complicated workspace I'm aware of has around 4,000 transformers in it…

38:30…and it's to do with German cadastral data model transformation from a government format into geodatabase.

38:36But most of them are like 5 to 15 transformers.

38:42And there's ways of structuring these things to group them, and so on.

38:47So, yes, many, many of them, they're categorized in this way…

38:50…and just casting your eyes over the categories gives you an idea of some of the kinds of things that we can do.

38:56So we can filter data; we can throw stuff out that's too small, too short, you don't like, has the wrong attribute values, whatever.

39:03People do write translations where they hope nothing comes out. So why would that be?

39:10What they're doing is throwing away all the good data, and whatever comes out is the bad stuff.

39:14So you might have a CAD group that is supposed to be following a spec.

39:18You can write one of these tools that only outputs the things that aren't to spec…

39:22…then you give that back to the CAD group and you say, Keep fixing this until I get nothing.

39:26When I get nothing, then we're done. That's a common kind of scenario.

39:31There's handfuls of transformers to do with KML. What else should I hit on here?

39:35The 3D things for creating 3D out of 2D, we can do that sort of stuff.

39:41String manipulation, lots of good tools there for concatenating things together, splitting them apart, parsing…

39:47…matching unregular expressions, all those kinds of tools.

39:50If some of you are Perl or AWK fans, there's a lot of influence of those tools in these things as well.

39:58In fact, at one point, it was called the Grepper. And we used to say, Parse one for the Grepper…

40:02…but we changed that now to be the String Searcher, so people that aren't UNIX hacks would know what that is.

40:07But you can basically match to find…I only want to get the data where the attribute name…

40:14…starts with the letter A and ends with the letter zed, and I don't care how many characters are between that.

40:18You can write those sorts of things very, very quickly, so…

40:25And what Dale just missed that I pulled a trick on him…

40:28Oh, you did? Did it change? Oh.

40:29See, if you put on a search term in the bottom of the window there for a search for a keyword, then you'll get search results there.

40:36So that's analogous to the Search window in ArcGIS.

40:40That's right. And Bruce, I see you brought up your favorite too, the Python ones.

40:43That's right. Well, I only could fit two responses in the bottom of the picture so…

40:47I see. I see. So you can actually write a tiny bit of Python that would get executed on every feature as it goes by.

40:55So in case we've missed some kind of obscure operation you want to do, you can do that.

41:00But Bruce has actually used this to do parallel processing, among other things.

41:03Yes, right. I will show you that in a little while.

41:06Okay. So how do you get into this stuff?

41:09You right-click on a toolbox and you say New Spatial ETL Tool, and it'll pop up, walk you through a wizard, and then off you go.

41:17One trick. I was doing a bunch of work with this prepping for a talk I gave at the Lidar Summit…

41:22…and I found that if I know that I'm starting with lidar and ending in lidar, I don't want to go through that wizard every time…

41:28…I actually found it easier to copy an old one and just paste it. That's another way.

41:32Once you get one workspace, you can just kind of copy and paste them rather than going through the thing all the time.

41:37And then, to run them, you can double-click them from the toolbox and they'll ask you for parameters…

41:43…or you can run them inside the Workbench tool, or you can embed them in ModelBuilder.

41:47I think that pretty much…and you can probably call them from a Python script as well.

41:51So there's a variety of ways you can run these things.

41:54I will highlight this idea. Inside of the Workbench tool, you can publish certain parameters that will become…

42:01…parameters of the geoprocessing tool like any other geoprocessing tool parameter.

42:06So you can ask this or that. I want to know a buffer width, I want to know a tolerance, whatever it is.

42:11I can publish those things, and then the user can fill those in from within the geoprocessing framework.

42:18And so I'm going to let Bruce show us this stuff at this point. Thank you, Dale.

42:25I'm sorry about the map here. We're constrained to the screen resolution we can project, and it's only 1024 x 768.

42:33Anyway, what we have here, the blue lines, are in fact polygon data.

42:38I've symbolized them with no fill, so I can go down, click on the topmost layer.

42:44Essentially, these are postal zones in an area of the world.

42:48Now, I subscribe to this data, and it gets refreshed a couple times a year, and here's a look at what happens, right.

42:57They change the boundaries of the data between these postal zones, or suburbs they are, and sometimes they add new ones.

43:04So the real estate people love this because some suburbs are worth more than others. But anyway, that's a fact of life.

43:10So if your job is to deal with this data, a very powerful analysis that you frequently want to do is change detection.

43:18You'll need eyes like a hawk or some other tools that you write yourself to do that normally…

43:24…but with the Data Interop extension, it's a very simple process.

43:27I want to show more than one aspect of this analysis, and I'll bring up query layers as an output of this exercise in a minute.

43:37So there they are; there're…what is it? Five and eight, so curious number of months apart, but it's a year apart, more or less.

43:45And we've got some data there of the two different deliveries.

43:48So I'm going to open up my ETL tool. I'll say Edit.

43:52And this is the one that the screen shot was taken from for the slide a few sessions back.

43:58So on my left here, I have my readers, my inputs, and they're the two feature classes that you saw in ArcMap…

44:05…and they are coming from an Esri geodatabase, file-based geodatabase, so…

44:12And the feature types there, there they are.

44:14They have the same schema, at least things don't change that much on us in this case, though they may.

44:21What I want to point out is we have an interesting requirement to output the data.

44:26I want to send it to SQL Server Spatial, and on my laptop here, I'm running SQL Express 2008, which supports spatial data types.

44:35So I'm not writing the data out in my answers to a geodatabase at all; it's going to a DBMS somewhere that I have connectivity to.

44:43Now the change detection is done by this one transformer, very powerful one, in the Workbench here.

44:49And these transformers, you set their behavior by clicking on the button there with the [unintelligible] in it, the little dots there.

44:56I can choose to do my change detection with or without geometry, and if it is geometry, it could be 3D.

45:05I can…lineate geometry matching relates to whether or not you consider lines…

45:09…which are otherwise identical but have opposite direction, to be the same.

45:13And I'm choosing which attributes to match on.

45:16So changes in an ID attribute and a suburb name attribute from my list there will be considered a change.

45:25So that takes a minute and 40 seconds to run, and I won't do it because we have a good 10 minutes' material…

45:31…of a subsequent demo before question time.

45:34But what happens is the unchanged data gets piped to one feature class…I beg your pardon.

45:40…one table in the DBMS out here; the adds get piped to another table in the DBMS…

45:45…and the deletes get piped to another table in the DBMS.

45:49So that's the result I'm looking for…

45:51…and it just gets dumped out in this really simple workspace with a very powerful transformer within it.

45:57The readers and writers, when you expand the little plus sign on them, you can see the schema that's in the data.

46:04So I think I've just kept all the inputs that are there, some system ones there I might've dropped.

46:09But the green icon against a row means that we were able to feed data to that row in the output.

46:19So let's just consider that we've run that, and we're going to go now to the SQL Express DBMS where the output was pumped to.

46:32So I've got tables there, and I can, I guess, add a table in from there and see where [unintelligible] adds.

46:43I can open it, and it's got the same attribution as the file geodatabase data had, except it's got a geometry column…

46:54…which has BLOB data in it.

46:55So I can't consume that as a feature class on its own, but I can, if I go to the File menu, Add Data > Add a Query Layer.

47:05So I've set up my OLL DB connection as a source of query layers in ArcMap here. I'll say this.

47:15Okay? So I'm making a SQL expression.

47:19This expression must resolve to a set of rows with a common geometry type like polygons and common coordinate system.

47:28In my case, all my data does.

47:31And I just have to give the thing a name.

47:33I can validate the expression so it gets tested that it's going to return some rows. I can say Next.

47:40I can say okay, let's say, ID is…I'm going to use ID as an object ID rather than have one be synthesized for me.

47:49The spatial reference, I agree with what the system's seeing in the DBMS so I'm good to go. I can say Finish.

47:58So let's go to…so put max…

48:04So there, retrieved as a query layer in ArcMap, is output to a spatial DBMS type…

48:11…generated by a very simple workbench from Esri geodatabase inputs.

48:18So that's I think a powerful analysis that was simple to do and will be useful if you're in the business of managing query layers.

48:27And people who manage query layers include people who do publication of data to the great wide world…

48:34…that are perhaps not Esri users or just purely DBAs.

48:40Now I'm waiting for…here we go.

48:43Here's the grand finale. So what is this mess?

48:48What this is, is all the US domestic flights in July 2010 going from A to B.

48:55The symbology is driven by the length of the flight, so the red ones are long…

48:59…and it goes through orange and yellow and then to blue; they're the shortest ones, the small hops.

49:06Now, I learned so much about US domestic flights building this part of the demo.

49:12So in round numbers, there are 570,000 flights.

49:16Now I downloaded that data from a website run by an organization called the Bureau of Transportation Statistics.

49:23For some reason, their site's called RITA.

49:25Now, this is publicly available data. The only gotcha is it comes to you as a CSV file.

49:31You fill in the form with what you want, like as I said, Give me everything for July 2010…

49:37…so down came a hundred and something megs of data, which is fine.

49:41That's okay, we're not scared of data because we have Data Interop.

49:45So here's the CSV file. And remember, CSV file, if it's appropriately structured, looks like a spreadsheet to Excel.

49:51So here's what we're seeing.

49:54I initially started off on this demo thinking, well, I'll find out how many ways are there to get to San Diego taking two hops…

50:01…but there are over 7,000, so it didn't make a good map.

50:04So I went back. So let's just see what we can do otherwise.

50:07So the flight date is my column here; 570,000 rows and change, and they're all in July 2010.

50:15The most recent data is three or seven months out of date.

50:19What does it know about these flights spatially? Only an airport code.

50:24So this is from Minneapolis, right? To or from Columbus, Ohio, with another half a million rows underneath it.

50:34Well, that's all well and good, but I can't make a map out of that.

50:39So I went to the FAA site and rummaged around, and I found an airports database…

50:45…where I could download an Excel spreadsheet, a real one, and had airport codes as one of the columns in the spreadsheet…

50:55…and further along amongst the very large number of fields is a latitude and a longitude.

51:04Well, that's good. The issue there, though, is that those are not numbers.

51:11No GIS is going to be able to read that string with the dashes and the N and the W as coordinates.

51:18So we have a little bit of task ahead of us, but not insurmountable.

51:23So I'll quit Excel there and go back to ArcMap. So what did I do?

51:28What I wanted to achieve is what you're seeing there, and I'm going to do something else with it in a little while.

51:34I'll first go back to the ETL tool I built to bring this data together.

51:43Now, this is a little bit bigger than the other ones we've seen so far this afternoon…

51:50…but at the end of the day, not very scary compared to some of the 4,000-plus ones which are out there in the wild.

51:58So same story. I've got a reader which is looking at my CSV data, so it's all up here.

52:03Remember there are a lot of fields there including, amongst other things…

52:07…airport codes with departures took off and flights arrived.

52:14I forgot to show you that there's a time stamp in that data for when wheels up and when wheels down…

52:21…but that's local time, which is not good because we got at least three time zones in domestic…

52:26Oh, okay, so a lot more when you considered Alaska.

52:29I can tell you how many flights go to Adak Bay in the Aleutian Islands in July without any trouble at all. So that's all good.

52:36And I've got my FAA data which is my spreadsheet that had the coordinates for the airports; that's down there too.

52:47I had another input which I neglected to show you in the map, but I may as well while we're here…

52:53…a polygon shapefile of US time zones, so I'll identify that's…whatever it is. Minus 6 from GMT, central [unintelligible].

53:05I've colored them by the offset from GMT, or UTC as it's more correctly known these days.

53:13So taking all this data as my challenge, twice for the FAA data, I pulled that coordinate data apart…

53:24…by splitting it on the hyphens; grabbing hold of the coordinates, the parts there…

53:30…and turning them into a real number, a decimal calculator, decimal degree. Excuse me; I'm going to get some more water.

53:38For the spreadsheet of CSV data of flights, I used these tools here to add the x,y coordinates onto the stream of data.

53:50So that gave a start point and end point to each of the flights.

53:55And then my only remaining task was to turn the local time values for wheels up and wheels down into a common time frame…

54:04…and I chose UTC time, so what time it is over in London essentially.

54:09And to do that work, I used a Python caller which ran this function.

54:14So it imports the date-time module, gets hold of some of the feature attributes, does some time arithmetic…

54:22…and sets some feature attributes, in this case, a departure UTC time and the arrival UTC time.

54:31So I think maybe this takes 10 minutes to run with that 5-, 600,000 features. That gives me the data I'm looking for.

54:38Over here, I'll show you the writer briefly, and I have added through my work previously not only geometry…

54:44…to make it a two-point line but UTC time stamps when the wheels left the tarmac and when they got back down on the ground.

54:53You're sorting as well, Bruce?

54:54Oh, yes. I sorted on flight length so that when I displayed it in ArcMap, the little ones wouldn't get buried below the big ones.

55:02Excuse me. So here we are there.

55:08We have the data nicely symbolized, but in ArcGIS 10, we support time properties on feature classes.

55:18And I can enable the time in this feature class so that features know they only exist between two time stamps…

55:26…in my case, the fields I added.

55:27So departure UTC time and arrival UTC time are the life span…the life expect…no, the life span of each of the features.

55:37So I can say that's good. And once you…I'll just escape the drawing of that. 00:55:48

55:50I'm going to bring up the animation dialog.

55:54And I'm going to, in my case, I'm going to animate all the flights in the US in July 2010 on one-hour interval…

56:03…and show you the net result of my analysis as we run.

56:08So what I've chosen to display in the animation dialog here is central time zone time for all the flights.

56:16So give it a second to get its…everything in memory.

56:19So here we are. Nine p.m. central time, 30th of June, we find there's a lot of flights in the air, even short flights.

56:29And as you move to 11 p.m., later in the day, we see the East Coast ends up going to sleep…

56:36…and the West Coast is still alive; there's people still in the air.

56:39At every time of day, there's the long flights, the red ones, that are there.

56:43Now I want you to watch as we go through the data. West Coast, everyone gets to land and go to bed at 4 a.m. central time.

56:49Five a.m., nothing's happening except the long flights.

56:52East Coast wakes up at 5 a.m. central time. Everyone's in the air getting to their business.

56:59The activity sweeps across the country till we get to middle of the day. It's getting very, very busy indeed.

57:08What I want you to notice is what happens between 7 and 8 p.m.

57:11So it's going to take a second while the animation gets to that time of day. We have a little bit of time; we're just about…

57:21We're on good for go time there.

57:26What's interesting part of this analysis, I have a theory as to why what you're going to see happen does happen.

57:35So I get into 4 p.m. central, 5 p.m. We're looking for 7 p.m., 7 to 8 p.m. One more hour. Wait for it.

57:49The skies are empty at 7 p.m. to 8 p.m. central time every day of the week.

57:55And I, being the diehard stickler for discipline and hardworking that I am, I sat on the balcony of my hotel room…

58:02….which overlooks Lindbergh Field, with a six-pack last night, and between 7 and 8 p.m., there were no flights.

58:08I think it's dinner hour for the pilots. What else can it be?

58:12So that is what I really wanted to find out.

58:14I think it's a data problem, Bruce, but…

58:16Yeah. I've watched the whole month go past, from 7 to 8 p.m. And I did watch the airfield.

58:22I can see Lindbergh Field from my little balcony.

58:25So anyway, that's a very powerful analysis done with publicly available data and a not very complex workbench and workflow…

58:34…and it's an answer that we could do a lot more with in ArcGIS, that's for sure. Okay.

58:40So with that, I'll hand it back to Dale to wrap up and take questions.

58:43Okay. So in summary, then, you saw how the spatial ETL tool can be used…

58:50… to create these dataflows that operate at a very fine grain.

58:54So it says here it's analogous to ModelBuilder except we work feature by feature in there.

58:59We're not working a whole table at a time.

59:01Individual features come through, we can pick things apart, route them different ways, do things like that.

59:06It's more close to writing code than it is to ModelBuilder, which pieces these transformations together in a larger context.

59:16So you can manipulate things individually as it goes through.

59:21So just to wrap up, a couple things. Who here has to deal with XML data at all? Anybody? There's a few of you.

59:29So the Data Interop, especially at 10.1, is a very powerful XML machine. I won't…

59:35Bruce knows that I have a little gig where I sing I Know an Old Lady Who Swallowed a Fly, but I think I'll spare you that.

59:40But nonetheless, if you're familiar with that song, it's about how stuff gets nested one inside the other, one inside the other.

59:46That's how XML often is, and you can deal with that using the spatial ETL tool to break that apart and get at the pieces you care about.

59:54Thinking about XML in a traditional, relational way is a very difficult thing to do, so we've made that easier for folks.

1:00:03You can use this kind of stuff to integrate data.

1:00:05We've had many customers doing integrations from various counties together into a statewide Department of Homeland Security thing.

1:00:12I know Indiana did something like that.

1:00:15More recently, many people that are submitting data to the Community Maps Program…

1:00:20…are using the Data Interop to rearrange their data to fit into the data model that the Community Maps Program wants.

1:00:27I know this morning at the local government thing…

1:00:29…there was a lot of talk about using this technology to take whatever your data model…

1:00:34…local government that you're using today is, into the Esri model…

1:00:38…and then all those applications that they have will work immediately. That kind of thing.

1:00:42And again, I'll mention that we have a webinar coming up about this kind of stuff.

1:00:46Come by our booth at 1206, and we can give you the details on when that is in a couple weeks.

1:00:51I also need to point out that ArcGIS Server supports the Interop extension.

1:00:55So you make these tools, you can make maps that refer to foreign data types, push it down there, it'll all work.

1:01:02In 10.1, yes, it is 64 bit straight through, so the Data Interop is…our stuff can work in 64 bit just fine.

1:01:11Coming in 10.1. If you're a Data Interop customer, you're in for a treat…

1:01:15…because you have two full years of development that's going to debut in 10.1.

1:01:20So FME, we rerelease our FME product every year, and depending on where that lines up…

1:01:25…it may or may not come at the same time as an ArcGIS release.

1:01:30But this time around, both FME 2011 and FME 2012 functionality is going to roll out with Data Interop in 10.1.

1:01:37So there's some new cloud formats, there's some very powerful SQL transformers; again, the XML things are there.

1:01:45But the biggest thing that's not on here that I want to just quickly show; I've got a couple minutes…

1:01:48…is the fact that we'll be doing lidar in 10.1. And so I'll give you an example.

1:01:54I've got this little project where I've got an area of interest that I've been told to work with.

1:02:00So it's a CAD file; that's my area. And someone has given me a directoryful of x,y,z files…

1:02:08…which, if I take a look inside, they look like this. And there's a lot of lidar around like this.

1:02:13So I can take this, and what I'm supposed to do is produce a LAS file for only the area of interest that we wanted for my end user.

1:02:21And so I can make a spatial ETL tool that looks like this; it reads those x,y,z's, combines them into one point cloud…

1:02:28…clips them to that area, and outputs it to a lidar file, or a LAS file.

1:02:34And at the same time, I'll put out a LAS directory file, or a .lasd, that ArcGIS 10.1 wants to use.

1:02:40And then for an added bonus, I'll also go ahead and make a TIN out of the LAS…

1:02:46…the lidar data, and I'll take an orthophoto of the same area…

1:02:51…and I'll slap that onto the TIN and then output that as a 3D PDF to give to my end person.

1:02:57And so when I go and do this, I get this 3D PDF here, which I can spin around, do whatever, that's fine.

1:03:06I can give that to whoever I want.

1:03:08And I'll go back to ArcCatalog where I can see that when I'm done I've output the results, and here they are.

1:03:16Here's my LAS file being shown, and I can spin that around in 3D, and that is a full LAS file that I could give to somebody else.

1:03:24And that, that was starting with x,y,z data, but we can clip LAS files, we can split them, we can reproject them…

1:03:32…all kinds of things like that as you prepare the lidar for use in ArcGIS.

1:03:36My favorite, I won't show the whole workflow, but I will show the end result.

1:03:40I start with a LAS file and some road lines, and I want to end up producing cross sections.

1:03:49These are the lines that are parallel to my…or perpendicular to my road that I'm interested in.

1:03:55I'm seeing this in 3D; let's go back to just geography to take a look.

1:03:58That's the road, here's the cross sections, here's sort of the area where we're doing this project…

1:04:03…and here is a set of lidar files that I've output from a big one that are just cut along the distance or along the thing.

1:04:13Because these things are much easier…

1:04:14These are all individual LAS files which I can then work with…

1:04:18…without carrying the whole weight of the million-or-more-point overall lidar file.

1:04:23Again, not that hard to do in the Data Interop extension.

1:04:26So that's coming all in 10.1, and we'll be very happy to show that.

1:04:32I'll also mention that Safe does a road show every spring.

1:04:35We're going to be going around in April of 2012 to probably about 20 cities in North America…

1:04:41…talking about the new things in FME 2012, but those new things are also in Data Interop.

1:04:47And Data Interop users are welcome to come and eat our food and have fun with us and learn the latest things as well…

1:04:52…so watch for that. Safe.com is our website; we will be coming around, like I say, to 20 cities in North America next April…

1:04:59…and we'll be talking lots about 10.1 there as well.

1:05:02So there's some free courses all the time.

1:05:06You can get eval software if you want to.