00:01Yeah, my name's Bruce Harold. I'm the product manager at E-s-r-i, Esri, for the Data Interoperability extension.
00:06First qualification is not being able to mumble when you say the name of the extension.
00:11With me today is Dale Lutz, vice president of software development and one of the cofounders of Safe Software…
00:16…and together, we engineered the Data Interop extension.
00:21So what we're going to…we've got a lot of ground to cover today.
00:23Hopefully, the gods of the Internet willing, we'll be able to show you a lot of material.
00:27And if you have a burning question, just pipe up with it during the show.
00:30Otherwise, we're going to have enough time for questions at the end.
00:32And of course, we're more than willing to follow up with you after the show, with your own issues to chase.
00:40So what we're going to do, a little double act here.
00:42Dale is going to do most of the talking; I'll interject if he lies or forgets something.
00:46Otherwise, I'll be doing most of the demonstrations, and he might do…pipe in then and again with stuff he likes to show about XML.
00:53So I think without further ado… Can everybody hear me? And sound check from Dale.
00:58I think we're good on my side. Sound check? Yeah.
01:00Alright. I'll face forward to not talk to Dale sideways so I don't lose the mic when I'm on.
01:04So I think we're ready to go. Take it away, Dale.
01:07Okay, thanks, Bruce. So just before we get started here, I'd like to get a feeling of the audience members here.
01:12How many of you are already using either FME…
01:15I see at least one knows what I'm going to ask. Two.
01:17…are already using FME or Data Interop extension? Okay, so we've got a fair number of folks that are new to us, and that's great.
01:25Thanks very much for making the time.
01:27I guess I should also say that you're going to be treated today not to just one interesting accent.
01:32You've got a New Zealand accent and also a Canadian accent, so apologies in advance if you have to sort through that.
01:39So just some background, Bruce already mentioned that Esri and Safe worked together to build a Data Interoperability extension…
01:48…I guess, almost six years ago. As far as I can recall, it was 2005 when it was first born.
01:53And it was a marriage really of FME and, at that time, ArcGIS 9.
01:59And so we brought the data format support that we had inside of our FME product…
02:05…together with the extensibility of ArcGIS 9 at the time to allow ArcGIS 9 to read formats that we supported…
02:13…that weren't in ArcGIS to do quick translations and, as well, to do more interesting stuff, which is what we call spatial ETL…
02:21…or rearranging, restructuring data as it moves from one format to another or indeed, within the Esri world to the Esri world.
02:32And as a footnote, in case I forget, you can use Data Interoperability extension, for example…
02:38…to prep data to be used in the Community Maps Program.
02:42So you're starting inside a geodatabase; you need to get to a different data model.
02:46The Data Interop extension is a very good thing to use for that.
02:49And actually, we have a webinar coming up on that in a couple of weeks' time.
02:52If anyone's interested, talk to me afterwards or come by our booth.
02:57So lots of data formats are supported.
03:01What a lot of folks don't realize is that besides just reading directly different formats…
03:06…you can use the Data Interoperability extension to integrate data.
03:10So you can take 2D things and make 3D things.
03:13You can bring in CAD and do very sophisticated, fine-grained things on CAD files…
03:20…extracting block attributes or setting extended entity data.
03:23All that sort of stuff just within the CAD world can be done.
03:27And we can add textures or images to 3D objects as you write them to multipatches.
03:33And if there is time at the very, very end, I may give you a sneak peek in 10.1.
03:38We can do a bunch of stuff with lidar that complements the LAS support that's in ArcGIS 10.1 very nicely.
03:45The Data Interop can clip lidar. It can slice it. It can split it based on returns.
03:51All those sorts of things can be done as well as you prep data for later use inside of ArcGIS.
03:57And don't get me started on XML. Our cofounder at Safe, Don, is a huge XML fan.
04:03He likes to go to…I'm not joking. Goes to [unintelligible] like this and says, "Who here loves XML? I do."
04:08I'm not quite the same as him, but we can do almost any kind of XML that you could imagine…
04:15…can be solved with Data Interop to break it apart or produce it. That can be done.
04:21So kinds of tasks, bringing things together.
04:24You might use Data Interop to migrate once from somewhere to somewhere else.
04:29More often, though, people use it to continually move data around within their organization.
04:35You can use it to do QA of data.
04:37It's a very convenient tool to set up rules that say, Look it, I'm going to run my input through here…
04:43…and all the stuff that's got bad attributes in this way or that way, or maybe the geometry is malformed in some way…
04:49…I can fish that stuff out.
04:51You can send data to other folks. I think Bruce might treat us to a change detection example. Is that true?
04:57Yes, the rumors are true.
04:59So you can tell what's changed. You can manage query layers, so you can load data to be used later in query layers.
05:07Hands up, who's using query layers? Anybody using them?
05:10So there's direct use of spatial types in the DBMS. So that's something.
05:15Because the Data Interop extension can read and write directly to a number of different spatial databases in their native form.
05:24So not only is it an extension, but if you just install it, you get some stuff for free that's just for the privilege of installing it.
05:34So if you have to ever deal with GML, probably if you're from North America, you don't have that happen too much…
05:40…but some of our European friends have that, so they can read and write what's called GML simple features.
05:45And you can also read and write…I'm sorry; read, access WFS, Web Feature Services that belong to the OGC spec.
05:55And new, I think, in 10…was that when this business of cached and noncached came in, Bruce?
06:01Yeah. Yeah. If your WFS server is an ArcGIS Server, then the ObjectID is exposed in the WFS string…
06:09…and if the client, in other words, Data Interop sees that and you turn on the switch, then the features won't be cached.
06:15The time won't be taken to cache them. So each map refresh will be a new mouthful of data.
06:21One of the interesting places that we at Safe Software live in the food chain is that we have to sit in the middle…
06:25…between these different systems that might have different approaches to the world.
06:30And in the Esri world, there's very much a view that every row has a unique integer ID.
06:36That's fine in the Esri world, except when we try to bring stuff in from, say, WFS; they don't have that property.
06:42And so that puts us in the uncomfortable spot of trying to fit a square peg in a round hole, and there's a…
06:49…the only way we can win is either (a) come up with a unique integer ID, so beg the person that's making the WFS to have one…
06:55…or, if it comes from ArcGIS, it will have one, or (b) we suck the whole contents of that WFS down locally…
07:03…perhaps only a search window gets sucked down locally, give it all IDs, and then ArcGIS can do its thing.
07:09But we give you the power to basically decide how you want to fit that square peg in a round hole.
07:15So Bruce, I think you're going to switch me off here somehow.
07:18You're gone. Am I gone? There you go; just like that.
07:22So everybody, it was of course ArcCatalog application; I just want to show you that the extension…
07:28…the Data Interoperability extension is available but not enabled.
07:31So it's not there, it didn't cost us any money. We're using the F word…it's free so far; you just have to install.
07:37And I've navigated in the browse tree down to some GML data.
07:41So all of the data sources that are visible to Data Interop look like a feature dataset.
07:49You see this icon here where my mouse is moving around? That's the kind of a feature dataset icon.
07:54Inside there, the system is seeing some polygon data, and you can see the map of New Zealand…
08:00…and it can treat it like you normally can in ArcCatalog.
08:04So what's happened behind the scenes is the GML file has been locally cached…
08:07…into something which ArcGIS can read as a thing called a plug-in data source.
08:12Now, sad fact of life is that data is fundamentally evil, and some things can happen with some formats.
08:19And here's one of them that happens with GML or can happen with GML, is the x's have been swapped.
08:24So it can't, doesn't get much worse than that. Unless you can go through to the antipodes of New Zealand…
08:30…which happens to be Spain, and you cross your eyes and you'll see the data okay, you're toast at this point.
08:35So if you do purchase--so getting away from the F word there--the extension…
08:40…you get some more switches and capabilities to fix problems like that.
08:44And that's one little way you can play with GML, so locally important…
08:48…perhaps less so in the US, but quite important in jurisdictions in Europe and in other countries.
08:54So of course, a fancy way to consume GML is as a Web Feature Service, WFS data.
09:01And here I've got an extent of where I used to live in New Zealand…
09:04…and I'll zoom in on this map and show you some of the data that's coming down the wire there in a moment.
09:11I'm navigating now in the Catalog window in ArcMap to the interop connections that I've made.
09:16So they're accessible to me anywhere in my…any session; actually just to my login account.
09:22So for this one here, I've got…oh, let me see. This one here, let's look at the properties for this connection.
09:30So it'll take a second before it comes up with the Properties window.
09:35I look at the properties there, I can see an ugly-looking URL to a dataset…
09:39…but there are other parameters available in that WFS connection.
09:42And you see in this case I've used a search envelope.
09:45So while at the back end on the server there are many millions of features, I've plugged in here an extent which I want to use as a filter.
09:51Also I can say like how many features maximum do I want to download and cache at any one time. I'm saying there 50,000.
09:58So I'll cancel out of there and go back to the Catalog window.
10:05And I'll say for this WFS service, let's look at its connection properties. Go back to this tag again.
10:18I haven't used a window, but what I have done is used a filter expression.
10:23So my XML skills are rudimentary at best.
10:28The one behind me, but I did manage to parrot off a filter expression I found on some document somewhere…
10:33…and I'm saying the attribute value, attribute field Affected Surveys has to be this value here…
10:40…which stands for deposited plan, and then some number, and the land district has to be North Auckland…
10:47…because this survey number is not unique across the whole country.
10:52So the net result of doing those little pieces of work with a WFS connection is my data's streamed down to my client [unintelligible]...
11:03…can go away while it makes its mind up, and when I've loaded the data into ArcMap, it will behave like any other layer.
11:11I will show you when it comes back to me. I've made a definition query, I've symbolized the data on attribute field…
11:17…the attributes are typed, they have a schema like anything else. Still working on that.
11:24And I can use them in geoprocessing as well as mapmaking.
11:28So for each session, the data is, if noncached, downloaded for me and cached locally and will become available in my session.
11:37Think that's going to take a while to return after its excursion down the web there, so we might…
11:43Shall we flip back? Yeah, we'll flip back. We'll see it when we come back. We'll finish that later on. Dale's back on.
11:48Okay, let's see what happens. There we go.
11:51So just to kind of summarize this section, GML and WFS comes free, but you have to install.
11:56And I noticed on the ArcGIS forums in the last couple days, somebody had built one of these things as Bruce was showing…
12:03…published that map document to their ArcGIS Server and it didn't work.
12:07Well, what they forgot to do is with ArcGIS Server, they had to install Data Interop extension.
12:10You don't have to license it, but it's got to of course be installed.
12:14Simple parameters are there, and you can do the stuff that Bruce just showed you.
12:18So the next section we're going to do is some of the simple things you can do with the Data Interop…
12:24…and that is the direct read of many different formats.
12:28So when we can tell what the format is, based on its file extension, we'll just read it as if it was a native ArcGIS format.
12:36So things like MapInfo files, things like--let me think about an example like…
12:43I can't use the CAD examples 'cause we can connect to those, but large number of regional formats, KF85.
12:51You can kind of see here even SketchUp files, we can directly read those 3D different formats like that, 3DS files. Yep?
12:59[Inaudible audience question]
13:05Answer that one? Yeah. You think you have a description object in KML data, yes, it's preserved…
13:10…but even better, with Data Interop extension, if your KML data has a schema object…
13:15…that is preserved and used to type the attributes that are seen in ArcGIS.
13:19[Audience question] And that's at version 10?
13:21Yes, it's in the released product now.
13:23Yep, yep. So, yeah, you can see there the KML is in one of them as well.
13:27So anyway, you can do that. You can also extend. You can add your own extension in there.
13:34There's people that might have used MGE; if any of you are old like me, you might have seen MGE used.
13:39They use kind of funny file extensions. You can make that work in there too.
13:44And so when you're doing this, you can just directly use any of those things as if they were native ArcGIS tables…
13:50…and so you can run them right into geoprocessing.
13:52You can display them, you can run them in ArcPy.
13:56I know Bruce is a big ArcPy fan, so you can use them in there just like that.
14:01And you can also make a connection, and Bruce was showing that with WFS, to a format.
14:08And you need to do that when either the file extension isn't what you'd expect it to be or you want to set some parameters.
14:17So you can use Data Interop, for example, to read CAD files directly…
14:22…and you have a bunch of options at your fingertips there that extend what functionality is in the base ArcGIS product.
14:28So you can split by level names or by geometry type. You can do that kind of thing.
14:33You can also, in one connection, type in
14:43You don't have to actually split them out by their file names.
14:46And you can set a coordinate system there as well.
14:49So Bruce is showing on this slide, using it to connect to SQLite, for example, that's an open source database package…
14:57…or to…you can just about do any of these formats in that way.
15:02And then once you have these connections, you see them in the part of your ArcCatalog tree where…
15:10…underneath that Interoperability Connections heading.
15:14Oh, yeah. I want to get one other thing there.
15:17Bruce mentions web feeds, so you can use this to connect to GeoRSS URLs.
15:23You can also pull KML off of a URL in that way, RSS feeds.
15:28You can also make a connection to a database, which when we first did this was very critical.
15:33Now that there's query layers, it's less likely that you'd actually come through this way…
15:37…unless it was a database that we support that isn't supported through query layers. Right.
15:43And so I think, Bruce, we're going to hand it back to you to show us your magic.
15:48One of these times, I'm going to forget to switch the video.
15:52Okay, so here we are in somewhere in Texas, I think, from memory.
15:56We have Texans in the back of the room. Oh, do we? Does that look like Texas to you?
16:00[Inaudible audience comments]
16:01I'd have to do some… That's a pretty good clue, isn't it?
16:03I'd have to do some reverse geocoding, double-check that.
16:06But as I mentioned when I was showing the GML data, we can browse on disk and directly read foreign to ArcGIS…foreign formats.
16:15And here's a couple of examples here. Floodplains.mif. Now, MIF is MapInfo Interchange Format.
16:22Like it's their equivalent of, I don't know what. Shapefiles. It's an open published format from their point of view.
16:29So I can just drag in that data into my document here, my map document here, and I've got the floodplain data.
16:36Now, embedded in MIF is coordinate system information as well, so it's analogous to our data types in ArcGIS.
16:45Traveling…here's another one. This is a TAB. Again, a MapInfo format I think.
16:50And I can just drag in the subdivision polygons there, and they'll turn up and join the data quite happily in the right place.
16:58It's not always as simple as navigating disk to find foreign format data 'cause some of it's quite complex.
17:05And we were talking just a second ago about the requirement for connections to be made.
17:12And connections have the ability to set properties which you need to read data.
17:16Dale mentioned EMG [sic] format, which is Modular GIS Environment from Intergraph, right?
17:23And I think I have some data here that is that, LinesPoint.
17:31It's indexing that data, so it should turn up somewhere in the same area.
17:38Where'd it go, Bruce? I don't know. That's a good question. Sometimes things don't work.
17:44We'll try that again. No, say it ain't so. Yeah…lines. There might not be points.
17:51Try one more time. Yeah, there were yesterday.
17:54No, you need LotPoints if you want points, Bruce.
18:00Trouble. Sometimes these things don't work as advertised.
18:01No, go for LotPointsPoint. I'll take LotPointsPoint for a hundred.
18:04Oh, I see, Gotcha. Alright, now we go. Use the arrow. LotPointsPolygon…
18:09And you might want LotLinesLine.
18:12This format is just so confusing. I'll take LotLinesLine for 200. Okay, yeah, he's a winner.
18:18Anyway, we got there in the end. Sorry Identify didn't query.
18:23So in other words, in browsing some foreign formats, what I was going to get to, talking about the… Thanks, Dale.
18:28…the connection details is some of them can be quite complex.
18:34So not just a path to some data, but they pull tricks like their data store can't have any extension at all. In this case, it's PIR.
18:43And so of course Windows Explorer is not going to be able to recognize data…
18:47…by its extension type and a whole raft of other parameters.
18:50Like in this format, the attributes for the data are stored in a remote DBMS. I choose it and set the parameters to read it.
18:59Anyway, never let this ornery, evil data stop you; just give the extension and learn a few tricks…
19:05…about how to read data that you're getting and have to live with and just get on with life.
19:11So I'm going to get rid of these things that Dale made me add to the map here…
19:17…and remind everybody that we don't always have the luxury of data being a file on disk.
19:26Sometimes it's a DBMS we connect to, and sometimes it's a feed off the web.
19:30Now, I bought my wife an iPhone recently…
19:33…and she came bustling in the room a few days ago and said, "There's a tsunami alert, a tsunami alert."
19:38I said, well, I live 150 kilometers from the ocean, which is…I live in Redlands. 00:19:42
19:46Turns out she was talking about New Zealand there was a tsunami alert, because that's where I'm from…
19:50…and we both lived there until a few years back. So, well, what do we do?
19:55So if we want to investigate the relative life of what's happening with earthquakes and stuff…
20:00…we can go to the US Geological Survey website and we can have a look at an RSS feed they have on there.
20:06They've got a number of them.
20:08I'm connecting here. We're looking at the path to an XML file which is the RSS feed, which shows the current state…
20:14…and I'm in real time pretty much; I don't know how often this is refreshed, but pretty regularly.
20:19You can look at all the earthquakes that are happening all the time.
20:23So back in my map here, I made an RSS connection to this data.
20:32I can see it just reads from the web all that XML data and displays it on the map without my having to do anything.
20:39Now the tsunami alert that my wife was particularly worried about was for this part of the world.
20:44 It's off a tiny little island which is so small it's not even showing up right now called Raoul Island where I have been diving.
20:52I can tell you there's a lot of sharks and a lot of turtles.
20:55And the sharks and turtles will have been worried about this one here, a 7.6.
21:01And that resulted in a two-meter wave on the shore of Raoul Island…
21:06…which is approximately half the size of the normal waves there anyway, so that wasn't any much of a panic.
21:11One thing I might notice, though, and this will lead on with what to do about this kind of issue later on…
21:15…is that this is not ideal for mapping.
21:17They've confounded the magnitude of the quake and its location in the one field…
21:23… and they even stuck a character in front of the magnitude that's not even a number…
21:26…and other nasty stuff which you're going to have to deal with at some point if you want to make a good map.
21:30But that's for a later show in about 10 minutes, so with that, I think I'll hand it back to Dale after showing you direct read of data.
21:37Okay. So while it settles back there, what you just saw was directly reading data that's from a bunch of different formats…
21:44…without doing any kind of translation at all; it hits it live off the disk.
21:48Particularly useful if you don't want to take ownership of data.
21:52So in other words, some other group in your office is using another product; they've got their stuff sitting there.
21:57You might want to use that stuff.
21:59Traditionally you might imagine you'd do a translation into geodatabase and refresh it every week…
22:04…but you can instead hit it and read it live, and each time you go to look, it's getting the most recent version.
22:12So, next we'll talk about quick things.
22:17So we're building up to the exciting conclusion here, getting more and more sophisticated as we go.
22:22So the next step here that Bruce has highlighted is that we should look at the toolbox that comes…
22:27…with the Data Interoperability extension that has exactly two tools in it, Quick Import and Quick Export.
22:33So you can go out to just about any format, or you can come in from almost any format.
22:38And again, you can do these things from within a model or interactively, one at a time, however you might want to do it.
22:44So actually we have seen many customers put a quick export at the end of a big model…
22:51…because they're going to send the results of the analysis out to their neighbor who uses MapInfo or something like that.
22:59And so, yeah, I'm saying it can be done in a model, and you can do multiple files in and out at the same time.
23:09And so, Bruce, are you ready to take the helm? I am. Here you go.
23:14I had it wrong before; it wasn't Texas before, but it is now. I just double-check and reverse geocode.
23:19We're somewhere in Austin, Texas, and this is a pack.
23:22Some twitcher out there has gone and mapped the location of bird nests and how many eggs are in them…
23:29…and unbelievably, the offset of the eggs from the center of the nest. So it's just going a little bit too far.
23:36But regardless we've received this data in a SQLite database file. It's just a file with a .db extension; sometimes .sl3 extensions.
23:43I really love this format. It's a high-performance, single file DBMS. It's support for its built-in to Python.
23:52I've used it in script tools to hit it as a temporary data store. It's great for cross-DBMS queries.
23:59And even better, we can map it using Data Interop.
24:04One problem though. Here, let's go look at where the nests are, let's drag them in.
24:08Looks like it should be point data, and it looks like it should be point data there too, but in fact, it's not.
24:16It's just tabular data with x and y.
24:18Have no fear. Of course, in ArcGIS, we can turn x and y fields into geometry, but not if the system already thinks it's got geometry.
24:26So we're kind of toast here, so what to do?
24:31The problem we have is that in the morass of format supports that are out there…
24:35…it's not always possible to reliably tell what geometry's in a table. There can be many, and the default is point.
24:42So what we're going to do is we're going to bring the data quickly into a geodatabase.
24:48So my input, birds.db file, is going to be quick imported into a geodatabase.
24:55And there's a model-only tool called the Select Data tool, which will retrieve a feature class or table from a workspace…
25:04…and in my case, I'm going to retrieve both the nests and eggs ones, and I'm adding to display.
25:11So I'm just going to run the model, and the Quick Import's running back there and Select Data will run momentarily. There it goes.
25:21In a few seconds, we've got that. I won't save the change.
25:26Back over here, hey, presto! Eggs and nests are now tabular data sources, quickly imported in here.
25:32So I'm going to say let's display x,y data. I'm going to say…
25:37I happen to have a…this one. Can't remember which one it is.
25:43Yep. That one? I think so. Yeah, it'll do.
25:47Things will turn up or I'll have to use the other one. Yes, there we go.
25:49There's the nests now displayed as an event layer.
25:52I can now do something else that's geodatabase-y like…
25:55…add a relate from--we happen to have nest ID in the table that we've now put it into.
26:05Did I do that right? Yeah. I think so. You've got to change that one to… Eggs, eggs, yeah.
26:10Brain's going to sleep here. Nest ID. Nest eggs; everyone likes to have those.
26:17So it's remarkably, this species of bird, whatever, is very sociable; the nests are all handily convenient.
26:24To identify, I can see that that one had a bunch of eggs in it; one above didn't.
26:31So we're doing geodatabase-y things after quickly importing a foreign format virtually painlessly so…
26:36…that, ladies and gentlemen, is quick import.
26:40Good. And we're back, and so just to very…you know, they're very straightforward tools.
26:48They let you work very conveniently within the ArcGIS geoprocessing framework with all this foreign data…
26:54…give you a way to bring it in so you can work with it.
26:57You can also send it back out again if you need to. And as Bruce points out, you can use ModelBuilder and also Python.
27:05So now do yourself stuff.
27:08We have this ability to make your own format by piecing together some of the primitives within the Data Interoperability extension.
27:17And so you build this workflow using the Workbench tool, can it, save it, and now it appears as if it is a brand-new format.
27:26And so, Bruce, I'm giving it back to you real fast. There you go. What have you got for us this time?
27:34Okay. I guess it's my fault because I built the slides, huh?
27:38Remember when we looked at the earthquake feed, we didn't find the data ideal for mapping purposes.
27:44Right, great example.
27:45Get x,y's on the page, but we couldn't symbolize by magnitude; it's an obvious one that you'd want to do, so what to do?
27:56So back in the Catalog tree here, you can add interoperability connections, and you come up with this dialog here.
28:05Lets you choose the format you're going to need for the connection to the data, and then you add in, say, a path or a URL to the data.
28:13If you need to…you get the last 10 I think formats that you've used as your default choices in the pull-down there.
28:21If you need to, you go to what's called the Format Gallery, and you get the selection of options that you're licensed to get.
28:31Now, notice some buttons down here. You can say New Custom Format or Import. Okay?
28:38So it's like a cooking show, that's Dale's analogy. Here is one we did earlier; it is for this purpose, the USGS quakes.
28:46When I click on that, the system knows that that's a custom format that I've built…
28:51…and so it gives me the option to edit that format.
28:53So that's rather than making it from scratch and possibly having finger trouble…
28:57…I'm going to go and edit that custom format and show you what I did.
29:02Now remember when we inspected the data briefly in the previous map, we had two attribute fields.
29:08Was it Title and Content? Title had M then a number, comma, then a location in words, and the other one had a date in words.
29:21So our goal is to extract useful attribution from those two character fields and turn them into proper data.
29:32This is, if you're not already a user of Data Interoperability, what we're seeing now is the Workbench application…
29:38…which is installed with the product.
29:40We're going to go into a lot more detail with the more powerful demos coming up…
29:43…but this, so we'll walk you through this one, since we seem to be making quite good time.
29:48So what happens, the old snake chain here is going on, we've got data being read in with this thing here…
29:54…and data being written out with that thing down there. They're called readers and writers, so that's not too hard.
30:00What do I want to do with the data that's come in?
30:02The first thing I want to do is split that field that had data with a comma in it into two elements.
30:08And I'm splitting a field called Title with a comma character, and it's going to output a list.
30:16That's a foreign concept to ArcGIS, a list within a row. In ArcGIS, we have a georelational model.
30:23Within the Workbench application, we can break that model and we can carry it through all sorts of data like…
30:29…and structures like XML documents, PDFs. In this case, we're talking about lists.
30:35So I'm splitting that character field into two bits.
30:39I'm getting the bit before the comma then with that transformer there, and I'm giving it a name Magnitude…
30:48…and I'm deleting the M which prepends the data.
30:51I go on and do the same thing for the bit after the comma and give it a name; I'm calling it Location.
30:57And last step all in my custom format definition here, I'm formatting that date string there into…
31:04…data that's suitable to be pumped into a date type field, or time stamp field.
31:10I'm constraining my output to be the entry points, so any point geometry, and I'm saying my Magnitude…
31:16…I'm going to make it a float field. Location's going to be staying the same; Time is going to be a date-time.
31:22So I've split Title and Content into three fields.
31:25Magnitude is a number, Location is a character field of 30 characters, and Time is a date-time field.
31:33So that's making a custom format; it's as simple as getting some transformers together in Workbench application.
31:44And I've got custom quake feed here. If I just open it up, I will do. Oops, sorry. Do it up here.
31:53The data will be reread in real time. I add it to my map there.
31:58Here come the points. This time I can see the fields which I made for myself there with my custom format…
32:06…and I'm going to symbolize on Magnitude. I'll give it some…oops, I missed.
32:14Magnitude, I could easily derive the diameter of the dots by that as well.
32:20So okay, so now I've got a thematic map.
32:23So that's one example of a custom format, handily rearranging some free data and giving me the schema that I'm shooting for…
32:31…to consume it ideally inside ArcGIS.
32:35So I think that's all I wanted to show with the custom formats. Dale?
32:39Okay. So the good thing is that…to remember about these custom formats…
32:44…really we're layering on some transformation, on top of an existing format.
32:48In this case, Bruce was starting with GeoRSS which just had a description and a title.
32:53He knew there was goodies inside there; he wanted to split it out.
32:56He could then make the kind of schema he wanted the rest of the world to see, the data structure the rest of the world to see…
33:01…and then that flow would happen every time that custom format is used.
33:06And he can take that custom format and give it to his friends, share it with his organization…
33:12…doesn't have to ever create it again.
33:14And you could layer this on top of, let's say, some CAD files.
33:17In older…in days of old, people used to encode interesting things inside of color and line thickness, and so on.
33:23You could set up rules like this that when you're displaying and reading the CAD files from your engineering group…
33:29…they appear in ArcGIS using vocabulary you'd understand like "two-lane road," "paved," and so on…
33:37…instead of "blue, thin." So this is the kind of thing that can be done.
33:43Alright. For our next section, which Bruce has entitled Cool Stuff, we're going to talk about spatial ETL.
33:50And this is the part, honestly, where the real value comes.
33:54And actually, sitting in the back of the room are my friends from San Antonio Water District.
33:58Have you given your talk yet?
34:00[Audience comment] No, tomorrow at three-fifteen.
34:01Tomorrow at when?
34:03[Audience comment] Tomorrow at three-fifteen.
34:04Okay. So and what's it called? [Inaudible audience comment] Okay, I'll let you work on that until later.
34:08But nonetheless, if you want to see how they have used this part of the Data Interoperability extension…
34:13…to make a big business difference in their organization, you should go to their talk tomorrow.
34:18But really, this is where the main power comes.
34:21Yes, solving all these format things is wonderful and helps in many cases, but the real value comes…
34:28…when you start to apply the kind of technology Bruce just showed to all kinds of data transformation things.
34:34And you can get stuff done faster than you would if you were writing code to do this, which is usually the other choice.
34:40If you're going to start parsing field, rearranging things, and so on, you can use the spatial ETL tools of Data Interop…
34:47…or you can write Python code or, even worse, C++ code.
34:50And so using these tools can make you much, much more productive.
34:56So there's…inside of the Data Interop are these things we call transformers. These are operations that alter data.
35:03And again, I'll invite you down at our booth at 1206, the Safe Software booth…
35:08…we have a little handout that's a catalog of all the transformers.
35:11And if you're interested in seeing the scope of what's there, come on by, grab that. It makes wonderful late-night reading.
35:17But you can go through this catalog; there's about 400 of them now, a bit overwhelming…
35:21…but they do everything from string parsing to doing 3D extrusions and everything in between.
35:27And that's all that's there.
35:28The other thing that's key is that many inputs are possible.
35:31I can bring a CAD file and an orthophoto and somehow do something with that and produce a geodatabase.
35:37I can rearrange many different input types, and I can also output to many things.
35:43So I could output a geodatabase and a 3D PDF if that's what I felt like.
35:47You're not really constrained in terms of what's coming in and out.
35:52So you graphically design these workflows; Bruce showed this already a little bit.
35:57You experiment. It's very easy to pop these things down, run a partial translation, see where you are.
36:02There is a thing in here we call the visualizer that you can connect anywhere in the flow and hit Run…
36:07…and then it pops us and shows you what would be at that stage.
36:12And in 10.1, you can actually debug these things by watching each feature go through one at a time.
36:17Am I allowed to talk about 10.1? Yes. Okay, good.
36:20Safe doesn't let me talk about FME 2012, but at least I can talk about 10.1 here.
36:25So, yeah, you start with something, you transform it, and you put it out.
36:28And that's where the ETL comes from--extract from somewhere, transform, and load.
36:33And we think that the T is the most important aspect of this whole thing.
36:39So as Bruce says, it's a pipeline. The readers read things, there's geometry and attributes.
36:44Then we transform things, and then we write things. And so that's the very useful way…
36:50And as you're crafting these things, you often don't care about where you're going to write it until the very end.
36:56So you start with your reader, you start adding transformations and visualizers making sure…
37:00…yes, I'm weeding through the data, I'm throwing things out I don't want, I'm getting what I want…
37:04…yep, looks good, looks good, looks good.
37:05Okay, at the very end, you say, Now where do I want to output this?
37:08Because really, the format is somewhat irrelevant. You just can decide that at the very, very end, and then you say…
37:14…okay, yes, I wanted to put this out to shapefiles today, and maybe I'll also add, for good measure, file geodatabase.
37:20That's a decision you can make at the very end.
37:22I think the format can be dynamic.
37:24Yes, that's also true. You can defer the decision on the format until run time.
37:29So you can make a geoprocessing tool that one of the parameters that it asks the user is…
37:34…What the heck format do you want this in when I'm done?
37:36And you fill it in, and then it'll just do the right thing.
37:39So you can dynamically change that.
37:41Inside of the tool, the thing that enables that, for those that are interested, is something we call a generic writer.
37:47For that matter, we have a generic reader as well, which will read anything.
37:51You can…it'll guess by the file extension or you can ask the user, come on, give me a hint here; tell me what to read.
37:57But you can make one workspace that can take data from anywhere as well.
38:05So sometimes there are situations where you might have these graphs that only have one thing.
38:11So in this case, Bruce is doing a change detection.
38:14He's reading two versions of something and then comparing them and then outputting to three different tables.
38:22And all the work is happening in one.
38:25I think the most complicated workspace I'm aware of has around 4,000 transformers in it…
38:30…and it's to do with German cadastral data model transformation from a government format into geodatabase.
38:36But most of them are like 5 to 15 transformers.
38:42And there's ways of structuring these things to group them, and so on.
38:47So, yes, many, many of them, they're categorized in this way…
38:50…and just casting your eyes over the categories gives you an idea of some of the kinds of things that we can do.
38:56So we can filter data; we can throw stuff out that's too small, too short, you don't like, has the wrong attribute values, whatever.
39:03People do write translations where they hope nothing comes out. So why would that be?
39:10What they're doing is throwing away all the good data, and whatever comes out is the bad stuff.
39:14So you might have a CAD group that is supposed to be following a spec.
39:18You can write one of these tools that only outputs the things that aren't to spec…
39:22…then you give that back to the CAD group and you say, Keep fixing this until I get nothing.
39:26When I get nothing, then we're done. That's a common kind of scenario.
39:31There's handfuls of transformers to do with KML. What else should I hit on here?
39:35The 3D things for creating 3D out of 2D, we can do that sort of stuff.
39:41String manipulation, lots of good tools there for concatenating things together, splitting them apart, parsing…
39:47…matching unregular expressions, all those kinds of tools.
39:50If some of you are Perl or AWK fans, there's a lot of influence of those tools in these things as well.
39:58In fact, at one point, it was called the Grepper. And we used to say, Parse one for the Grepper…
40:02…but we changed that now to be the String Searcher, so people that aren't UNIX hacks would know what that is.
40:07But you can basically match to find…I only want to get the data where the attribute name…
40:14…starts with the letter A and ends with the letter zed, and I don't care how many characters are between that.
40:18You can write those sorts of things very, very quickly, so…
40:25And what Dale just missed that I pulled a trick on him…
40:28Oh, you did? Did it change? Oh.
40:29See, if you put on a search term in the bottom of the window there for a search for a keyword, then you'll get search results there.
40:36So that's analogous to the Search window in ArcGIS.
40:40That's right. And Bruce, I see you brought up your favorite too, the Python ones.
40:43That's right. Well, I only could fit two responses in the bottom of the picture so…
40:47I see. I see. So you can actually write a tiny bit of Python that would get executed on every feature as it goes by.
40:55So in case we've missed some kind of obscure operation you want to do, you can do that.
41:00But Bruce has actually used this to do parallel processing, among other things.
41:03Yes, right. I will show you that in a little while.
41:06Okay. So how do you get into this stuff?
41:09You right-click on a toolbox and you say New Spatial ETL Tool, and it'll pop up, walk you through a wizard, and then off you go.
41:17One trick. I was doing a bunch of work with this prepping for a talk I gave at the Lidar Summit…
41:22…and I found that if I know that I'm starting with lidar and ending in lidar, I don't want to go through that wizard every time…
41:28…I actually found it easier to copy an old one and just paste it. That's another way.
41:32Once you get one workspace, you can just kind of copy and paste them rather than going through the thing all the time.
41:37And then, to run them, you can double-click them from the toolbox and they'll ask you for parameters…
41:43…or you can run them inside the Workbench tool, or you can embed them in ModelBuilder.
41:47I think that pretty much…and you can probably call them from a Python script as well.
41:51So there's a variety of ways you can run these things.
41:54I will highlight this idea. Inside of the Workbench tool, you can publish certain parameters that will become…
42:01…parameters of the geoprocessing tool like any other geoprocessing tool parameter.
42:06So you can ask this or that. I want to know a buffer width, I want to know a tolerance, whatever it is.
42:11I can publish those things, and then the user can fill those in from within the geoprocessing framework.
42:18And so I'm going to let Bruce show us this stuff at this point. Thank you, Dale.
42:25I'm sorry about the map here. We're constrained to the screen resolution we can project, and it's only 1024 x 768.
42:33Anyway, what we have here, the blue lines, are in fact polygon data.
42:38I've symbolized them with no fill, so I can go down, click on the topmost layer.
42:44Essentially, these are postal zones in an area of the world.
42:48Now, I subscribe to this data, and it gets refreshed a couple times a year, and here's a look at what happens, right.
42:57They change the boundaries of the data between these postal zones, or suburbs they are, and sometimes they add new ones.
43:04So the real estate people love this because some suburbs are worth more than others. But anyway, that's a fact of life.
43:10So if your job is to deal with this data, a very powerful analysis that you frequently want to do is change detection.
43:18You'll need eyes like a hawk or some other tools that you write yourself to do that normally…
43:24…but with the Data Interop extension, it's a very simple process.
43:27I want to show more than one aspect of this analysis, and I'll bring up query layers as an output of this exercise in a minute.
43:37So there they are; there're…what is it? Five and eight, so curious number of months apart, but it's a year apart, more or less.
43:45And we've got some data there of the two different deliveries.
43:48So I'm going to open up my ETL tool. I'll say Edit.
43:52And this is the one that the screen shot was taken from for the slide a few sessions back.
43:58So on my left here, I have my readers, my inputs, and they're the two feature classes that you saw in ArcMap…
44:05…and they are coming from an Esri geodatabase, file-based geodatabase, so…
44:12And the feature types there, there they are.
44:14They have the same schema, at least things don't change that much on us in this case, though they may.
44:21What I want to point out is we have an interesting requirement to output the data.
44:26I want to send it to SQL Server Spatial, and on my laptop here, I'm running SQL Express 2008, which supports spatial data types.
44:35So I'm not writing the data out in my answers to a geodatabase at all; it's going to a DBMS somewhere that I have connectivity to.
44:43Now the change detection is done by this one transformer, very powerful one, in the Workbench here.
44:49And these transformers, you set their behavior by clicking on the button there with the [unintelligible] in it, the little dots there.
44:56I can choose to do my change detection with or without geometry, and if it is geometry, it could be 3D.
45:05I can…lineate geometry matching relates to whether or not you consider lines…
45:09…which are otherwise identical but have opposite direction, to be the same.
45:13And I'm choosing which attributes to match on.
45:16So changes in an ID attribute and a suburb name attribute from my list there will be considered a change.
45:25So that takes a minute and 40 seconds to run, and I won't do it because we have a good 10 minutes' material…
45:31…of a subsequent demo before question time.
45:34But what happens is the unchanged data gets piped to one feature class…I beg your pardon.
45:40…one table in the DBMS out here; the adds get piped to another table in the DBMS…
45:45…and the deletes get piped to another table in the DBMS.
45:49So that's the result I'm looking for…
45:51…and it just gets dumped out in this really simple workspace with a very powerful transformer within it.
45:57The readers and writers, when you expand the little plus sign on them, you can see the schema that's in the data.
46:04So I think I've just kept all the inputs that are there, some system ones there I might've dropped.
46:09But the green icon against a row means that we were able to feed data to that row in the output.
46:19So let's just consider that we've run that, and we're going to go now to the SQL Express DBMS where the output was pumped to.
46:32So I've got tables there, and I can, I guess, add a table in from there and see where [unintelligible] adds.
46:43I can open it, and it's got the same attribution as the file geodatabase data had, except it's got a geometry column…
46:54…which has BLOB data in it.
46:55So I can't consume that as a feature class on its own, but I can, if I go to the File menu, Add Data > Add a Query Layer.
47:05So I've set up my OLL DB connection as a source of query layers in ArcMap here. I'll say this.
47:15Okay? So I'm making a SQL expression.
47:19This expression must resolve to a set of rows with a common geometry type like polygons and common coordinate system.
47:28In my case, all my data does.
47:31And I just have to give the thing a name.
47:33I can validate the expression so it gets tested that it's going to return some rows. I can say Next.
47:40I can say okay, let's say, ID is…I'm going to use ID as an object ID rather than have one be synthesized for me.
47:49The spatial reference, I agree with what the system's seeing in the DBMS so I'm good to go. I can say Finish.
47:58So let's go to…so put max…
48:04So there, retrieved as a query layer in ArcMap, is output to a spatial DBMS type…
48:11…generated by a very simple workbench from Esri geodatabase inputs.
48:18So that's I think a powerful analysis that was simple to do and will be useful if you're in the business of managing query layers.
48:27And people who manage query layers include people who do publication of data to the great wide world…
48:34…that are perhaps not Esri users or just purely DBAs.
48:40Now I'm waiting for…here we go.
48:43Here's the grand finale. So what is this mess?
48:48What this is, is all the US domestic flights in July 2010 going from A to B.
48:55The symbology is driven by the length of the flight, so the red ones are long…
48:59…and it goes through orange and yellow and then to blue; they're the shortest ones, the small hops.
49:06Now, I learned so much about US domestic flights building this part of the demo.
49:12So in round numbers, there are 570,000 flights.
49:16Now I downloaded that data from a website run by an organization called the Bureau of Transportation Statistics.
49:23For some reason, their site's called RITA.
49:25Now, this is publicly available data. The only gotcha is it comes to you as a CSV file.
49:31You fill in the form with what you want, like as I said, Give me everything for July 2010…
49:37…so down came a hundred and something megs of data, which is fine.
49:41That's okay, we're not scared of data because we have Data Interop.
49:45So here's the CSV file. And remember, CSV file, if it's appropriately structured, looks like a spreadsheet to Excel.
49:51So here's what we're seeing.
49:54I initially started off on this demo thinking, well, I'll find out how many ways are there to get to San Diego taking two hops…
50:01…but there are over 7,000, so it didn't make a good map.
50:04So I went back. So let's just see what we can do otherwise.
50:07So the flight date is my column here; 570,000 rows and change, and they're all in July 2010.
50:15The most recent data is three or seven months out of date.
50:19What does it know about these flights spatially? Only an airport code.
50:24So this is from Minneapolis, right? To or from Columbus, Ohio, with another half a million rows underneath it.
50:34Well, that's all well and good, but I can't make a map out of that.
50:39So I went to the FAA site and rummaged around, and I found an airports database…
50:45…where I could download an Excel spreadsheet, a real one, and had airport codes as one of the columns in the spreadsheet…
50:55…and further along amongst the very large number of fields is a latitude and a longitude.
51:04Well, that's good. The issue there, though, is that those are not numbers.
51:11No GIS is going to be able to read that string with the dashes and the N and the W as coordinates.
51:18So we have a little bit of task ahead of us, but not insurmountable.
51:23So I'll quit Excel there and go back to ArcMap. So what did I do?
51:28What I wanted to achieve is what you're seeing there, and I'm going to do something else with it in a little while.
51:34I'll first go back to the ETL tool I built to bring this data together.
51:43Now, this is a little bit bigger than the other ones we've seen so far this afternoon…
51:50…but at the end of the day, not very scary compared to some of the 4,000-plus ones which are out there in the wild.
51:58So same story. I've got a reader which is looking at my CSV data, so it's all up here.
52:03Remember there are a lot of fields there including, amongst other things…
52:07…airport codes with departures took off and flights arrived.
52:14I forgot to show you that there's a time stamp in that data for when wheels up and when wheels down…
52:21…but that's local time, which is not good because we got at least three time zones in domestic…
52:26Oh, okay, so a lot more when you considered Alaska.
52:29I can tell you how many flights go to Adak Bay in the Aleutian Islands in July without any trouble at all. So that's all good.
52:36And I've got my FAA data which is my spreadsheet that had the coordinates for the airports; that's down there too.
52:47I had another input which I neglected to show you in the map, but I may as well while we're here…
52:53…a polygon shapefile of US time zones, so I'll identify that's…whatever it is. Minus 6 from GMT, central [unintelligible].
53:05I've colored them by the offset from GMT, or UTC as it's more correctly known these days.
53:13So taking all this data as my challenge, twice for the FAA data, I pulled that coordinate data apart…
53:24…by splitting it on the hyphens; grabbing hold of the coordinates, the parts there…
53:30…and turning them into a real number, a decimal calculator, decimal degree. Excuse me; I'm going to get some more water.
53:38For the spreadsheet of CSV data of flights, I used these tools here to add the x,y coordinates onto the stream of data.
53:50So that gave a start point and end point to each of the flights.
53:55And then my only remaining task was to turn the local time values for wheels up and wheels down into a common time frame…
54:04…and I chose UTC time, so what time it is over in London essentially.
54:09And to do that work, I used a Python caller which ran this function.
54:14So it imports the date-time module, gets hold of some of the feature attributes, does some time arithmetic…
54:22…and sets some feature attributes, in this case, a departure UTC time and the arrival UTC time.
54:31So I think maybe this takes 10 minutes to run with that 5-, 600,000 features. That gives me the data I'm looking for.
54:38Over here, I'll show you the writer briefly, and I have added through my work previously not only geometry…
54:44…to make it a two-point line but UTC time stamps when the wheels left the tarmac and when they got back down on the ground.
54:53You're sorting as well, Bruce?
54:54Oh, yes. I sorted on flight length so that when I displayed it in ArcMap, the little ones wouldn't get buried below the big ones.
55:02Excuse me. So here we are there.
55:08We have the data nicely symbolized, but in ArcGIS 10, we support time properties on feature classes.
55:18And I can enable the time in this feature class so that features know they only exist between two time stamps…
55:26…in my case, the fields I added.
55:27So departure UTC time and arrival UTC time are the life span…the life expect…no, the life span of each of the features.
55:37So I can say that's good. And once you…I'll just escape the drawing of that. 00:55:48
55:50I'm going to bring up the animation dialog.
55:54And I'm going to, in my case, I'm going to animate all the flights in the US in July 2010 on one-hour interval…
56:03…and show you the net result of my analysis as we run.
56:08So what I've chosen to display in the animation dialog here is central time zone time for all the flights.
56:16So give it a second to get its…everything in memory.
56:19So here we are. Nine p.m. central time, 30th of June, we find there's a lot of flights in the air, even short flights.
56:29And as you move to 11 p.m., later in the day, we see the East Coast ends up going to sleep…
56:36…and the West Coast is still alive; there's people still in the air.
56:39At every time of day, there's the long flights, the red ones, that are there.
56:43Now I want you to watch as we go through the data. West Coast, everyone gets to land and go to bed at 4 a.m. central time.
56:49Five a.m., nothing's happening except the long flights.
56:52East Coast wakes up at 5 a.m. central time. Everyone's in the air getting to their business.
56:59The activity sweeps across the country till we get to middle of the day. It's getting very, very busy indeed.
57:08What I want you to notice is what happens between 7 and 8 p.m.
57:11So it's going to take a second while the animation gets to that time of day. We have a little bit of time; we're just about…
57:21We're on good for go time there.
57:26What's interesting part of this analysis, I have a theory as to why what you're going to see happen does happen.
57:35So I get into 4 p.m. central, 5 p.m. We're looking for 7 p.m., 7 to 8 p.m. One more hour. Wait for it.
57:49The skies are empty at 7 p.m. to 8 p.m. central time every day of the week.
57:55And I, being the diehard stickler for discipline and hardworking that I am, I sat on the balcony of my hotel room…
58:02….which overlooks Lindbergh Field, with a six-pack last night, and between 7 and 8 p.m., there were no flights.
58:08I think it's dinner hour for the pilots. What else can it be?
58:12So that is what I really wanted to find out.
58:14I think it's a data problem, Bruce, but…
58:16Yeah. I've watched the whole month go past, from 7 to 8 p.m. And I did watch the airfield.
58:22I can see Lindbergh Field from my little balcony.
58:25So anyway, that's a very powerful analysis done with publicly available data and a not very complex workbench and workflow…
58:34…and it's an answer that we could do a lot more with in ArcGIS, that's for sure. Okay.
58:40So with that, I'll hand it back to Dale to wrap up and take questions.
58:43Okay. So in summary, then, you saw how the spatial ETL tool can be used…
58:50… to create these dataflows that operate at a very fine grain.
58:54So it says here it's analogous to ModelBuilder except we work feature by feature in there.
58:59We're not working a whole table at a time.
59:01Individual features come through, we can pick things apart, route them different ways, do things like that.
59:06It's more close to writing code than it is to ModelBuilder, which pieces these transformations together in a larger context.
59:16So you can manipulate things individually as it goes through.
59:21So just to wrap up, a couple things. Who here has to deal with XML data at all? Anybody? There's a few of you.
59:29So the Data Interop, especially at 10.1, is a very powerful XML machine. I won't…
59:35Bruce knows that I have a little gig where I sing I Know an Old Lady Who Swallowed a Fly, but I think I'll spare you that.
59:40But nonetheless, if you're familiar with that song, it's about how stuff gets nested one inside the other, one inside the other.
59:46That's how XML often is, and you can deal with that using the spatial ETL tool to break that apart and get at the pieces you care about.
59:54Thinking about XML in a traditional, relational way is a very difficult thing to do, so we've made that easier for folks.
1:00:03You can use this kind of stuff to integrate data.
1:00:05We've had many customers doing integrations from various counties together into a statewide Department of Homeland Security thing.
1:00:12I know Indiana did something like that.
1:00:15More recently, many people that are submitting data to the Community Maps Program…
1:00:20…are using the Data Interop to rearrange their data to fit into the data model that the Community Maps Program wants.
1:00:27I know this morning at the local government thing…
1:00:29…there was a lot of talk about using this technology to take whatever your data model…
1:00:34…local government that you're using today is, into the Esri model…
1:00:38…and then all those applications that they have will work immediately. That kind of thing.
1:00:42And again, I'll mention that we have a webinar coming up about this kind of stuff.
1:00:46Come by our booth at 1206, and we can give you the details on when that is in a couple weeks.
1:00:51I also need to point out that ArcGIS Server supports the Interop extension.
1:00:55So you make these tools, you can make maps that refer to foreign data types, push it down there, it'll all work.
1:01:02In 10.1, yes, it is 64 bit straight through, so the Data Interop is…our stuff can work in 64 bit just fine.
1:01:11Coming in 10.1. If you're a Data Interop customer, you're in for a treat…
1:01:15…because you have two full years of development that's going to debut in 10.1.
1:01:20So FME, we rerelease our FME product every year, and depending on where that lines up…
1:01:25…it may or may not come at the same time as an ArcGIS release.
1:01:30But this time around, both FME 2011 and FME 2012 functionality is going to roll out with Data Interop in 10.1.
1:01:37So there's some new cloud formats, there's some very powerful SQL transformers; again, the XML things are there.
1:01:45But the biggest thing that's not on here that I want to just quickly show; I've got a couple minutes…
1:01:48…is the fact that we'll be doing lidar in 10.1. And so I'll give you an example.
1:01:54I've got this little project where I've got an area of interest that I've been told to work with.
1:02:00So it's a CAD file; that's my area. And someone has given me a directoryful of x,y,z files…
1:02:08…which, if I take a look inside, they look like this. And there's a lot of lidar around like this.
1:02:13So I can take this, and what I'm supposed to do is produce a LAS file for only the area of interest that we wanted for my end user.
1:02:21And so I can make a spatial ETL tool that looks like this; it reads those x,y,z's, combines them into one point cloud…
1:02:28…clips them to that area, and outputs it to a lidar file, or a LAS file.
1:02:34And at the same time, I'll put out a LAS directory file, or a .lasd, that ArcGIS 10.1 wants to use.
1:02:40And then for an added bonus, I'll also go ahead and make a TIN out of the LAS…
1:02:46…the lidar data, and I'll take an orthophoto of the same area…
1:02:51…and I'll slap that onto the TIN and then output that as a 3D PDF to give to my end person.
1:02:57And so when I go and do this, I get this 3D PDF here, which I can spin around, do whatever, that's fine.
1:03:06I can give that to whoever I want.
1:03:08And I'll go back to ArcCatalog where I can see that when I'm done I've output the results, and here they are.
1:03:16Here's my LAS file being shown, and I can spin that around in 3D, and that is a full LAS file that I could give to somebody else.
1:03:24And that, that was starting with x,y,z data, but we can clip LAS files, we can split them, we can reproject them…
1:03:32…all kinds of things like that as you prepare the lidar for use in ArcGIS.
1:03:36My favorite, I won't show the whole workflow, but I will show the end result.
1:03:40I start with a LAS file and some road lines, and I want to end up producing cross sections.
1:03:49These are the lines that are parallel to my…or perpendicular to my road that I'm interested in.
1:03:55I'm seeing this in 3D; let's go back to just geography to take a look.
1:03:58That's the road, here's the cross sections, here's sort of the area where we're doing this project…
1:04:03…and here is a set of lidar files that I've output from a big one that are just cut along the distance or along the thing.
1:04:13Because these things are much easier…
1:04:14These are all individual LAS files which I can then work with…
1:04:18…without carrying the whole weight of the million-or-more-point overall lidar file.
1:04:23Again, not that hard to do in the Data Interop extension.
1:04:26So that's coming all in 10.1, and we'll be very happy to show that.
1:04:32I'll also mention that Safe does a road show every spring.
1:04:35We're going to be going around in April of 2012 to probably about 20 cities in North America…
1:04:41…talking about the new things in FME 2012, but those new things are also in Data Interop.
1:04:47And Data Interop users are welcome to come and eat our food and have fun with us and learn the latest things as well…
1:04:52…so watch for that. Safe.com is our website; we will be coming around, like I say, to 20 cities in North America next April…
1:04:59…and we'll be talking lots about 10.1 there as well.
1:05:02So there's some free courses all the time.
1:05:06You can get eval software if you want to.
An Introduction to Data Interoperability
- Recorded: Jul 14th, 2011
- Runtime: 1:05:08
- Views: 56241
- Published: Sep 22nd, 2011
- Night Mode (Off)Automatically dim the web site while the video is playing. A few seconds after you start watching the video and stop moving your mouse, your screen will dim. You can auto save this option if you login.
- HTML5 Video (Off) Play videos using HTML5 Video instead of flash. A modern web browser is required to view videos using HTML5.