00:01Okay. Anyway. So what we're going to do...we're all impressed that you didn't all go out and have a cheeseburger...
00:05...or something for lunch on a beautiful day like today.
00:08You must be really interested in this stuff.
00:10So we'll try and do it justice for you, make it worth your while.
00:14Listen, things to pay attention to, or what have we been doing on the geodatabase team for the past...
00:18...oh, I don't know, five plus years or so?
00:21What we've been doing, you know, sort of starting at ArcGIS 9.1...
00:25...is we've been really trying to finish up the initial information model of the geodatabase.
00:33At 9.1, remember we were kind of focused on the transportation domain.
00:36That's where the network dataset came into being.
00:39That's for transportation.
00:41At 9.2, it was actually a very significant release for the geodatabase team and ArcGIS, you know, with all the server technology.
00:48But then we also brought out replication.
00:51That's what we use and that's the term we use for sharing data.
00:55While archiving if you have an enterprise geodatabase so you can see what happened in the past.
01:01The spatial types came into being, supported a new terrain model called the terrain.
01:08Also brought out some functionality that's used for maintaining your parcel fabrics.
01:13The cadastral fabric dataset came out at that time.
01:15That's now been rolled into core at 10.
01:18And then also remember we brought out high-precision geometry.
01:22Okay. So there were a lot of changes that happened at 9.2.
01:24At 9.3, we were a little bit more focused on trying to support better distributed workflows...
01:31...as well as improve the versioning experience, okay? with better conflict and change detection management tools.
01:40Okay. So, what are we doing now?
01:43What are the key themes now that we are focusing on, on basically completing the model?
01:49Where again, we continue to evolve and enhance our information model.
01:54We weren't done.
01:55That's sort of like the first big iteration has been completed.
01:59We continue to work on our models, our information model...
02:02...to make it more flexible and powerful for you guys so you can do your work.
02:07One of the things we want to do is also help our data sharing technology, replication...
02:12...to make it more flexible, make it easier to use.
02:16These were based upon comments and feedbacks that we have gotten from you in the user community.
02:21How would you like us to be improving things?
02:24Also a key thing at 10 is we want to make the system even more open and allow people to access the data more directly.
02:35Say you don't want to use ArcObjects.
02:37You don't want to use our desktop applications.
02:40How can you get at your data?
02:43Okay, so we'll talk about some of the technology we have there to support those sorts of user environments.
02:49And then also the never-ending quest to improve our performance and the quality inside the geodatabase.
02:55So, let's start hopping through this stuff.
02:59The big things inside the information model that we addressed, the main topics were our geometric network model.
03:05That's what we use for utilities and the utilities environment, natural resources environment.
03:12The network dataset, I mentioned that a little bit before.
03:14That's our network model for the transportation domain.
03:18Then also topology.
03:21Now, more specifically, with the geometric networks, one of the things that we wanted to do was make it easier for people to create...
03:34...design and create their geometric networks as well as add lots and lots of data to existing geometric networks.
03:42Now, when we initially released the geometric network at 8.0, remember with the first release of ArcGIS?
03:49We had a limit, an internal limit to only being able to build networks that had somewhere on the order of 15 million features.
03:57That was a limitation.
03:58At that time, nobody had networks that big.
04:00You know, the big utility companies, gosh, they had maybe one or two million features in their geometric networks.
04:06So we didn't think it was big deal.
04:08Well, as time has passed, people have gotten bigger and bigger.
04:11Data's just coming at us, you know, this huge pipe.
04:14Just more and more data flowing at us.
04:16So we now need to be able to handle end users that have just monstrously large networks. Okay?
04:22So we've gone and made it such that there is now no limitation on the size of the network that you can initially create.
04:31Let's see, also, we have the...we went and reengineered our wizard for going and defining, creating your geometric network.
04:40That's been simplified and streamlined, sort of made more modern.
04:44That was a nice thing to do it in the 10 time frame.
04:50Now, one of the other interesting things that we've noticed in the user community is, there are a lot of people that with time...
04:57...you know, over the past 10 years, they've been adding more and more data into their existing networks. Okay?
05:04Now, you know, say, for example, maybe you're a utility company and maybe you go and buy the adjacent utility company...
05:11...and you need to get all their millions of features into your network.
05:15Well, previously, we had some...some ways of doing that but it could be a slow process.
05:21So one of the things that we focused on during the 10 development cycle was to create a new geoprocessing tool...
05:29...that will allow you to pour data into existing geometric networks very, very rapidly.
05:36By that we're meaning we're focused on people that need to add millions upon millions of features to an existing geometric network.
05:50This is basically our transportation-focused model.
05:55Now, everyone's seen all the flurry of activity that networks, routing networks, and whatnot have gotten on the Internet and...
06:03...you know, with Google and Microsoft, and others.
06:07You know, this is a very hot and fertile domain right now.
06:10Anyway, what we want to do is take our existing transportation model...
06:15...and continue to improve it and make it even more flexible for our end users.
06:20First thing we did was we made the network dataset model itself, you can now version it.
06:26Okay. It can be versioned.
06:28And much like topology, when you go and edit the features inside your network dataset, inside your transportation network model...
06:36...we'll track the changes and then on a periodic basis, you can then go and have that portion of the network...
06:44...have the connectivity rebuilt. Okay?
06:46This is much like topology with how we track areas that have been edited through dirty area management...
06:52...and we support an incremental validate process. Okay?
06:57So, want to be able to support in the versioned environment partial rebuilds, that's great.
07:02One of the big things that you see these days is being able to support time-dependent traffic data.
07:09By that, you know, say you're trying to do a routing application or doing a...
07:15...figure out the best time to drive from, say, here to L.A. starting at eight o'clock in the morning.
07:20The driving directions that you get or the traffic you encounter at eight o'clock in the morning...
07:24...is very different from the traffic that you'll see, like, at either noon or at five p.m.
07:30You know, we've got the commutes and reverse commutes and whatnot.
07:33Previously, we could only handle, you know, just one, what is that free-flow speed?
07:38And we would give you directions based upon a free-flow speed.
07:41It would not vary by time of day, but now by integrating better with the big data vendors such as NAVTEQ and Tele Atlas...
07:47...we extended the network dataset to support time-of-day-based routing.
07:53So the route that we will provide to you will be different depending upon historical traffic conditions.
08:00You saw on the big stage Matt Crowder, he gave a really nice demo.
08:04He was the one that was talking about LeBron James and Cleveland and health care and all that.
08:11That's the location-allocation solver.
08:13This is a very sophisticated analysis routine that if you have a bunch of customers or end users or places that you have to deliver stuff to...
08:21...where do you optimally allocate the places to distribute from so that you can minimize drive times and be as efficient as possible.
08:29Also Matt showed us the enhanced barriers.
08:32Remember that was the snowstorm sort of on the northeast half of Cleveland.
08:37Now you can have polygons to define barriers rather than just little point barriers.
08:41It's much more flexible.
08:43Also, we enhanced the information model of the network dataset to support something we term in...
08:51...we use, internally was termed a hyperedge, that allows you to do very rapid solve operations...
08:59...network analysis operations that can work on a national or continent-wide basis.
09:05Think about it for a second.
09:06If you start here in San Diego, you want to get to New York City, but you know you're going to be getting on interstate freeways...
09:12...there's still going to be somewhere between 3- and 5,000 street segments, interstate highway segments...
09:18...that you'll have to traverse to get there.
09:20Hyperedge is, what we're doing is, we're taking large collections of these edges or these features...
09:25...linear features that share similar, you know, similar attributes or properties, and we make one big edge.
09:33So then to get from say here to New York City, maybe it's only a hundred hops rather than 3- or 5,000 hops.
09:40That's how we can make things much faster to do these analysis operations. Okay?
09:45That's a very important issue for our user community.
09:50Okay, so we've already talked about this.
09:55And I guess I got ahead of myself a little bit.
09:59Well, here are the further details that you can read at your own leisure.
10:03But, yeah, we're basically talking about the new solvers and the enhanced barriers and the long-distance routing with hyperedges.
10:11So, that's enough PowerPoint chattiness.
10:14We'll see how this stuff really works.
10:19Yep. Okay, thanks, Eric.
10:22So I'm going to...you okay in the back? Good?
10:25I'm going to take you through about three demos that are going to highlight some of the topics that Eric just talked about...
10:31...and show you some of the functionality that the Network Analyst team has been working on and is available in 10.
10:37The first demo that I'd like to show is going to highlight the new editing functionality that we have...
10:42...and the ability to track edits with your network dataset.
10:45So what I have here is a map of Paris, France, and I have a number of locations that I'd like to find the route for.
10:52These might be locations that somebody would like to visit and they want to get the optimal route while they're in Paris.
10:57So I can go and solve on that information.
11:01And you'll see that it returns me back the route from all the locations starting at my hotel.
11:07If we zoom in to my hotel though, I notice that I'm actually missing some roads here.
11:12I don't have the most up-to-date data in my network dataset.
11:15Now previously I'd have to update the data and then rebuild the entire network dataset, and in this case...
11:21...that could take anywhere from 10 to 15 minutes.
11:24But now at 10, like Eric mentioned, we have dirty area tracking with the network dataset, so that makes it a lot easier.
11:30So what I'm going to do is turn on another layer which highlights the roads that I'd like to bring into my network.
11:38I'll select those roads and then I'll copy and paste them into my street feature class.
11:47And when I do that you'll notice that I get these hatched polygons.
11:51Anyone who's familiar with our topology model, these will be very, very familiar to you.
11:55These are dirty areas.
11:57You can see their extent over the features that I imported.
12:01So network datasets are now tracking those edits that you're performing.
12:05So now that I've done this, I can rebuild my network.
12:09And you'll notice that it took just a matter of a couple seconds to rebuild, not 10 or 15 minutes like it would for the entire network.
12:15Again, because it's only highlighting those features under the dirty areas.
12:19And now when I re-solve, you'll notice that instead of heading down south and taking a road by the river Seine...
12:27...it's actually just heading straight north and using those new features immediately...
12:30...in the exact same edit session into which I added them.
12:33So for anybody who's actually editing and maintaining their network data in network datasets...
12:38...this makes it a lot easier to get new edits into your networks and consume them extremely quickly.
12:46So the second demo that I'd like to show, we're going to head back...we're going to head over the Atlantic...
12:50...and we're going to go to Upland, California.
12:52And we're looking at a shot of downtown Upland here.
12:54I'm going to highlight some of the work the network team has done to enhance barriers within Network Analyst.
13:01So I have two stops that I'm going to create a route between, one in the north which is my warehouse depot...
13:05...and one in the south which is a restaurant.
13:09I can go and create that route and then zoom in to downtown Upland where that is.
13:16Now, this route's fine, but I know for example that north of this restaurant on this street there's actually construction going on.
13:24So I actually can't take this route as it exists because they've closed the north side of the street to traffic.
13:29So I'm going to go and place a new barrier on the north side and re-solve.
13:34This is a really nice enhancement that we made to point barriers at 10.0.
13:38Previously, it would restrict traversing across that entire edge.
13:42We're now tracking the percent along individual edges...
13:45...which allows you to more accurately model real-world situations when you're solving.
13:52The second part of this demo highlights line barriers.
13:56Upland, like a lot of cities in California, has a farmer's market where a lot of growers and farmers...
14:02...from around the area bring their vegetables and their produce down to downtown Upland so people can get organic produce.
14:10If I'm traveling to this restaurant on the day that the actual farmer's market is happening...
14:14...I have to take that into consideration.
14:16So if we go and turn on that layer that contains the information of where that farmer's market's occurring...
14:22...you can see that there's a number of streets in red, they're going to be closed to traffic...
14:26...because usually what they'll do is they'll set up shop directly on the street.
14:31So I'd like to actually consider that in my route.
14:33Now previously what I would have to do is actually create a number of line barrier...
14:37...a number of point barriers around the exterior to try to make sure that I could somehow cordon off this area.
14:44But at 10, what I can do is actually load in those line features as barriers to my network and then re-solve on it considering them.
14:57And you'll notice what it does is it actually avoids those streets entirely and comes at my restaurant from the east.
15:03That's avoiding the closed streets and the farmer's network [sic].
15:07The third type of enhancement to the barriers that I'd like to show is the scaled cost.
15:13Matt also showed this in the plenary on Monday.
15:17There's something about when events happen in the downtown core, the slowdown and the traffic congestion that you see...
15:23...isn't just limited to the streets where the event is happening.
15:27It also has an effect that buffers around those streets.
15:30So it's not just enough to take into account that I can't travel on these streets.
15:34I also need to consider that it's going to be slower for me to actually get to this restaurant because of the traffic congestion...
15:40...and just people walking on the street.
15:42So I'm going to go and create a new polygon barrier around this street.
15:51And I'm going to go and set its cost to be two.
15:55And essentially what this means is that any trip that I take within this barrier is going to take twice as long as streets outside the barrier.
16:04And when I re-solve it, hmm, I don't get the answer I'm looking for.
16:13What normally would happen is the actual route would miss the barrier completely...
16:17...but it wouldn't prohibit it from actually traveling inside of it.
16:20It would take the north side of the barrier, avoid it where possible...
16:24...but it would still allow the route to actually travel inside of it and get me to the restaurant.
16:27So what it's doing there is just simply considering the fact that this area represents a slowdown...
16:32...but it doesn't restrict me from traveling in that area.
16:37The last demo I'd like to show is how we've integrated time into the network dataset.
16:42Like Eric mentioned, we're taking account of historic traffic data.
16:46A lot of the data providers that you get your network data from have started to provide historic time traffic data...
16:52...that models what the speeds are on the different roads at different times of day and on different days of the year.
17:00We all know that the route that you're going to take isn't just dependent on the network itself...
17:05...but also the time of day that you're traveling and the day of the week.
17:09So what I have here is a map of downtown San Francisco, and I've got a number of locations that I'd like to solve on.
17:17So when I go and create my route, you'll notice that it's traveling eastward and it gets on Interstate 80 as soon as it can.
17:24Because it's decided that the sooner you can get on the interstate, the faster you'll get to the routes that you want to travel on.
17:31But this is showing me the ideal route.
17:33What if I was actually going to travel on this network and take this route at five p.m.
17:39Well before I do that, let's actually go and take a look at what the data looks like at five p.m.
17:46I'm going to go and use the new time slider window that's available in ArcMap and turn on my network.
17:52When I do that you'll see a number of lines that are highlighted in both green, yellow, and some orange and red.
17:58And if you look down at the bottom, you'll see in my TOC I'm highlighting what those different colors mean...
18:03...where green is free-flowing traffic and red in the extreme is stop and go.
18:09Now what this is showing me is historically what the traffic is like in San Francisco around the time right now...
18:13...just around twelve p.m.
18:15But let's go and take a look at what it's going to look like at five p.m.
18:22When I change that, you'll notice that quite a bit more red showed up.
18:25Some of the green moved away, and I got much more yellow.
18:28What this is showing that on the roads at five p.m. on Tuesday, it's traditionally slower on these roads.
18:37So let's go and turn our route layer back on and actually go and ask the network dataset to route considering that information.
18:49Now when I do that, the two things that hopefully you'll notice is that...
18:52...I'm no longer traveling through the surface streets near the downtown core to get to the interstate.
18:57Instead what it's doing is routing me on O'Shaughnessy Boulevard to get to the second stop.
19:02It's also exiting Interstate 80 much sooner than it did before...
19:06...before it encountered the stop-and-go traffic on the way outside of San Francisco.
19:11So these are just a few examples of some of the work that the Network Analyst team has done to augment the network dataset at 10.0.
19:18Okay. Okay, now those were some pretty interesting demos to really show you some of the greatly enhanced functionality...
19:29...that's been brought forward with the network dataset, our transportation model at 8.0.
19:34I think you'll agree that that's much more useful and interesting.
19:37I like seeing the time-of-day routing and whatnot.
19:40Now, finishing up the information model portion discussion, with topology, we ended up...
19:47...based upon a lot of feedback from you guys, the users, we ended up implementing six additional topology rules.
19:56You remember, the topology rules are how you essentially define in a...
20:02...essentially define what constitutes valid topology and good data for your organization.
20:08Remember, unlike coverages, it's a very flexible model that lets you specify whatever collection of rules you want across your data.
20:16So these are six of the new ones that we brought forward at 10.
20:21Now, next interesting topic.
20:27Mentioned this a little bit before, you know, trying to make our replication...
20:32...our data-sharing technology even better and more useful to our community.
20:37Now, just a quick refresher on replication.
20:42Basically, replication is a...it's a pretty cool technology that allows you guys...
20:47...to be able to share data across a heterogeneous collection of geodatabases.
20:54By heterogeneous I mean it's sort of like, unlike basic relational database replication where, you know...
21:02...maybe you can go from SQL Server instance to SQL Server instance...
21:05...or from Oracle instance to Oracle instance.
21:09This is from geodatabase to geodatabase.
21:13One geodatabase could be sitting on SQL Server, another could be on Oracle, another could be on DB2, or Informix, or Postgres.
21:23So it's pretty neat, and it also works in an asynchronous manner...
21:26...meaning you determine when you want the synchronization process to occur. Okay?
21:32So with this, as of 9.3, there were sort of four main models or use cases or topologies that replication worked really well with.
21:47Here's kind of a cheesy diagram but I think it explains to you what the differences are.
21:52You know, we often see people with regional offices.
21:55We do see the hierarchical level model where you're sharing between, say, federal, state, and local agencies.
22:02A very common one actually with, say, a lot of utility customers and whatnot...
22:07...now as we get more ubiquitous computing is the mobile user thing...
22:11...where you have your headquarters geodatabase and you have people replicating or sharing data out to their field units...
22:18...their mobile units.
22:20Another one that's very, very common, think about like government organizations.
22:25We saw some of this on the big stage yesterday where, you know...
22:28...you'll have your GIS professionals and whatnot working behind the scenes on their own geodatabase...
22:34...but then periodically, that's considered the production system...
22:38...periodically they're going to want to export that out to a publication system...
22:42...such that the end user community can then have access to at least read that data. Okay?
22:47And you want to separate the two.
22:49You don't want your live hot stuff that's being edited in the back offices to also be edited necessarily by your end user community.
22:58So that's the production-publication model.
23:00Now, the new one that was brought forward by the replication team inside the geodatabase was to support centralized data centers.
23:09Here's another cheesy graphic trying to show that.
23:12You know, you do have some organizations where the data is not flowing from...
23:19...sort of the central headquarters down to the end users in the field offices and then back again.
23:26Sometimes you only want it to be a one-way flow of information.
23:30And in this case, now we have three states, Idaho, Montana, and Colorado, all replicating...
23:35...pushing their data up to the big centralized mountain server or whatnot. Okay?
23:41Now, other things that were...so that's what we term one-way replication, child to parent.
23:48Some other interesting things, there's some schema mapping across replicas, meaning, you know...
23:53...table names don't have to be the same at both ends of the replication.
24:00Workflow also, we want to be able to support more replication or workflows...
24:06...when we're dealing with nonversioned database or the file geodatabases.
24:10We don't want to constrain you to just operating on enterprise geodatabases.
24:19Another good demo.
24:26Those are my cheesy graphics, by the way.
24:30Now see here. [Laughter]
24:31I don't appreciate that, sir.
24:33I certainly don't appreciate your tone.
24:35It was just disparaging and insulting.
24:39Okay, so I'm going to go through a quick demo, just highlighting some of the work the replication team has done at 10...
24:44...and highlight what Eric just walked through.
24:48The demo that I'm going to show here is highlighting that centralized data center workflow.
24:52Like Eric mentioned, a lot of our replication technology is centered on satisfying these user workflows.
24:57This was one that we got specifically from several different users out in the field who essentially asked the following from us.
25:05During different disasters, both natural and man-made, it's extremely important for the data to be in one central location.
25:15Think of a city or a county government, for example, that controls their own data...
25:21...but they would like to push it up to a central authority who can house the data in a read-only way...
25:28...and keep it online for disaster management scenarios.
25:32There's been several examples in the last couple years where you can think that...
25:35...that would have been extremely useful to have smaller agencies on the city, county, and even state level governments...
25:43...pushing their data up to a central authority who could manage it and keep it up and online for when a disaster hits.
25:50That's really what this workflow is meant to show.
25:53What I'm going to show here is really a feature function demo.
25:56It's not really showing a solution, but it's showing you the actual technology that the replication team has built...
26:01...and hopefully you can get some ideas of how to apply this in your own workflows.
26:06So, what I have here is my central geodatabase that's already been set up.
26:13I've got three different counties [sic] here.
26:15I've got Oregon, Idaho, and Montana.
26:17You can see I have the different county information colored through yellow to blue.
26:22I've also got a collection of points on it.
26:24I've got hospitals and schools.
26:26Typically in disaster scenarios, hospitals, obviously very important.
26:30But a lot of times schools are the central location where a lot of the disaster supplies are kept...
26:36...and where a lot of people actually congregate at in order to manage the disaster when it's happened.
26:43So, right now what I have is this one central database with all of the information from those three states...
26:48...and I'd like to set up this replication scenario out to the state of Idaho to be able to pull their data as they update it...
26:56...and keep it in this one central location for disaster scenarios.
27:00So, what I've already done here is I've got my data loaded in this central geodatabase.
27:05Idaho already has their data loaded as well.
27:07What I need to do is simply notify the two geodatabases that each other exists and create the relationship between them.
27:15So let me walk you through that.
27:19I'm going to go and create a one-way replica.
27:22And this is the thing that's different at 10.
27:24Previously, the one-way replica like Eric mentioned was satisfying the production-publication workflow.
27:30This one is a little different though, because the edits are actually going from the child up to the parent.
27:37So it's a slight twist on that previous workflow.
27:40In this example, the child geodatabases are the different state geodatabases, and the parent is this larger geodatabase...
27:49...the central one where all the information is collected.
27:53I'm going to go out and navigate to the geodatabase that I'd like to connect to.
27:56In this case, it'll be the Idaho geodatabase.
28:01I'm going to choose to register the existing data only because, again, my data's already loaded on both geodatabases.
28:07I'm just simply notifying them that the other exists.
28:14I'm also going to use another piece of functionality the replication team has added at 10.
28:18And that's the ability to use archiving to support one-way replication.
28:22What this does is it minimizes the footprint of replication on the server.
28:26It means less versions get created when you're using replication.
28:31So it can mean a smaller footprint running on your ArcSDE server for using replication in this example.
28:40Now I can go and actually pair up what I want to associate.
28:44Now the important thing here that I'd like to point out is a nice change that the team has made at 10.
28:48You can see that in my central geodatabase, my feature classes are named things like hospitals, schools, and counties.
28:55But in my target geodatabase, they actually have very different names, Idaho hospitals, and so forth.
29:01This is a new change at 10 where you can actually create replicas...
29:04...between geodatabases where the feature classes have different names.
29:08Previously that was a constraint or a rule that we imposed that the names had to be the same.
29:12We're now giving this flexibility to better model your data and add the names that are meaningful within the geodatabase...
29:19...but still set up the scenario.
29:24And lastly, I'll click Finish.
29:26And this will go through pretty quickly because, again, all it's doing is just notifying the two geodatabases of each other's existence.
29:32So now I can go out and actually open up that Idaho map document.
29:45And if we take a look at the version manager, you'll see that there's a replica out to my geodatabase, the central one that I just created.
29:55So now I'm going to start editing.
30:03I'm going to start editing my geodatabase and I'm going to make some very simple changes here...
30:06...just to demonstrate in a very visible way that we're pushing these edits up.
30:10So I'm going to go and select this one county and actually change its population to be 17,000...
30:19...and you'll see that when I do that the color of it changes.
30:23I'll also insert a new hospital and a couple new schools.
30:31Now obviously that's not how Idaho and the state of Idaho manages their hospitals and schools.
30:38Makes for good demo, though.
30:39That's how we do it in parts of Canada, eh?
30:42I'm going to put a school there.
30:45Really what I meant to show here is, this data would have been input over a series of time, right?
30:50Decisions would have been made, analyses run to determine where it is.
30:53But the key thing is that now I have the data in my Idaho geodatabase, I want to push it up to that central geodatabase...
30:59...so that it's stored in this centralized data center, backed up, if you will, at a different location.
31:05So now that that's done, I can go and synchronize my changes, and I'll go from the Idaho geodatabase that I have...
31:11...to the central geodatabase and push those changes up.
31:15And again, this is going to go pretty quickly 'cause there weren't a lot of changes.
31:19And as I go back to my central geodatabase, I want you to focus on this center area here that I was editing.
31:24I'll go and refresh the changes.
31:33I'll open up my Versioning toolbar actually to refresh the changes.
31:41Sometimes this is what happens when you practice the demo too many times.
31:46I'll just pull up the Refresh Version command quickly here.
31:54Then I'll go refresh my changes and you can see that the new color for that county appeared because of the population change...
31:59...and also the hospital and schools that I input.
32:03So that's just a quick example of one of the big things that the replication team did at 10.
32:07Really this demo's meant to highlight those three things, the new centralized data center's workflow...
32:13...the ability to use archiving to manage certain types of replicas, and the ability to have different feature class names across replicas.
32:25Very smoothly done. That was classic.
32:33Let's see. Good demo. Let's see.
32:37Let's get on to the next of the four major areas.
32:41We're actually going to spend a good amount of time talking about this topic.
32:46Let's see. It's all about more open and direct access to the data stored in your geodatabases.
32:53Now, one of the interesting things when you look back at the geodatabase when we released it at 8.0.
33:03It did have an ArcObjects API.
33:07Very nice ArcObjects API, but you couldn't really get at your data unless you were going through ArcObjects for our desktop application.
33:17The schema information, basically the metadata that we maintained to organize your data, was stored using binary representations...
33:26...and the data itself, the geometry was stored in binary representations.
33:30So basically the story at 8.0 was you had to go through ArcObjects.
33:35At 9.2, we started opening up the geodatabase a little bit more.
33:41Remember at 9.2 one of the big things was the new SQL types.
33:45What this allowed you to do was, again, you could use ArcObjects to get at either the schema or data inside your geodatabase.
33:53But then, if you actually want to get at the data itself, if it was stored in a SQL type, you could get at it through raw SQL. Okay?
34:03So that opened up the system a little bit and sort of like the first big step.
34:08Now, at 10.0, we're opening up the schema to try and complete the opening of the geodatabase.
34:18Now the schema is no longer stored in a binary representation.
34:23I don't know how many of you have gone and opened up a geodatabase and looked at the GDB underscore tables.
34:28You know that simple collection of 35 normalized, crazy tables and stuff with binary content.
34:35At 10, we changed that, okay?
34:38Now the schema essentially all the schema information about your datasets is stored using geodatabase XML. Okay?
34:49Now, talk a little bit more about this.
34:52So instead of having a constellation of 35 or so GDB system tables, you're going to have four tables.
34:59And the interesting aspect here is that in one of those tables, the items table, for each dataset there'll be one row in this table...
35:06...and its definition will be stored in XML that you can freely and readily get access to. Okay?
35:13Now we don't suggest that you go about, you know, modifying XML...
35:17...unless you really know what you're doing, but it's really, what's in my geodatabase?
35:21Now it's opened up.
35:24So, this is providing more open access.
35:28Also one of the things that we wanted to do was make it faster when you're running your applications...
35:33...to browse and search through the collection of datasets inside your information models, your data models.
35:40The geodatabase has always been scalable in terms of data volume. Okay?
35:46We have users out there with tens if not hundreds of millions of features in a table or a feature class.
35:53I mean we can handle that very well, always have.
35:57The difficulty was with end users that had very complex schemas...
36:02...very complex data models where maybe they had a thousand different tables inside their geodatabase...
36:10...2,000 tables inside their geodatabase.
36:13This new way, this new representation that we have for storing schema is much simpler...
36:18...and it allows us to be much more scalable in terms of being performant [sic] with large complex schemas that you may have. Okay?
36:27And this also is serving as a foundation for us moving forward...
36:30...such that we can introduce possibly new abstractions into the geodatabase in the future.
36:36So a big simplification effort there.
36:38Now, one thing to keep in mind, you will only get that new schema, simplified schema, if you upgrade your geodatabase. Okay?
36:49This is a geodatabase upgrade.
36:52You can upgrade either your enterprise geodatabases, your file geodatabases, or your personal geodatabases...
37:00...using a GP tool or a command that is fired off of the workspace property sheet.
37:07So all you have to do, go to either the connection or go to the workspace itself, right-click on it, brings open the property sheet...
37:17...and then hit the Upgrade button; it brings up a GP upgrade tool.
37:20You can also call it through Python if you want because it is a GP tool.
37:25In any event, it'll take care of everything.
37:28Now first, back up your data before you do that. Okay?
37:33Good idea, back up that data.
37:35Also, keep in mind, we are not touching your data. Okay?
37:40We're not going into your tables and your features and rows and things like that and mucking with those at all.
37:46No. We do not touch your data.
37:49What we are touching, or what we are changing, is how we internally organize the geodatabase.
37:56Those GDB system tables are turning from a collection of 35 down to 4...
38:01...but your data is not going to be touched at all in this process. Okay?
38:05But do back up your data.
38:09Now, one of the cool things here, and this was talked about on the big stage during the plenary yesterday...
38:16...is that because we've been able to simplify the schema of the geodatabase, how we organize the information...
38:23...it's now made it possible for us to create an open API to file geodatabases. Okay?
38:32This is basically it's going to be a C++ API, basically a DLL that you'll be able to download freely.
38:41It doesn't require any licensing.
38:43And it will allow you to get access to your file geodatabases, basically the simple feature model.
38:49It's not intended for updating topologies or network datasets or the more complex models...
38:56...but for your simple feature data, this would be a way that you would be able to both read it as well as update it. Okay?
39:03Through a public API. Okay?
39:06That does not require any licensing.
39:10No ArcObjects whatsoever.
39:12This is just a single DLL.
39:15Now, of course you can read all the data in a geodatabase including your more sophisticated dataset types...
39:22...but you'll only be able to update simple feature data. Okay?
39:26So here's a little pictorial representation where if you have a C or C++ client application...
39:33...using this new open API in conjunction with knowledge of the geodatabase XML spec...
39:39...which is another thing you can just download from our Web site.
39:43It's a standard thing, just go grab it.
39:45You'll be able to write applications that'll be able to read everything in a geodatabase...
39:49...or actually update simple feature inside your file geodatabases.
39:55Now, this is actually one of the neatest things, I think, that's happened with the 10 release in terms of the geodatabase functionality.
40:04Oh, also, I should go back and let you know the open API, the file geodatabase API, is not in the 10 box.
40:15We're still finishing it up. Okay?
40:17This will not be going out with the next major release of the software.
40:22It's a single DLL.
40:23It will be able to go out off cycle, what we term off cycle.
40:26That means basically when it's ready, it will go out. Okay?
40:30It's a single DLL. It's not a big process to install it.
40:34So expect that this fall. Okay?
40:36We're trying to finish up that work right now.
40:39We expect it to be out this fall.
40:41So stay tuned on that one.
40:44Now, getting back to accessing data, opening up the system.
40:48Well, you know, we have all kinds of different ways to get at your geodatabase spatial data.
40:53But then, as we're seeing these days with all these spatial types and whatnot, you know, look at what Microsoft's doing with SQL Server...
40:59...and their geometry storage type or their geographic storage types...
41:04...there's a lot more spatial data out there that's actually not inside a geodatabase.
41:11We have users that don't want to, you know, do what we've done in the past where we said...
41:15...well, if you want to be able to access that data, you have to register that data with the geodatabase.
41:20We have to be able to lay down our system tables and whatnot. No.
41:24This is basically a technology called query layers that will allow you to define layers that have full layer functionality...
41:32...to be able to access data stored in spatial databases. Okay?
41:39Now, this is pretty interesting stuff here.
41:42It's really opening up your modeling possibilities.
41:47Everything in your application, everything in your ArcGIS Desktop...
41:51...that you're manipulating does not have to exist inside a geodatabase. Okay?
41:56Now Brent's put together a pretty cool and completely new demo that he's going to take us through. Brent?
42:05It's a lot better than looking at PowerPoints.
42:08Or would you rather see PowerPoints and a demo?
42:13More demos always good.
42:17More demos, more cowbell, yeah.
42:20So in ArcMap here what I have is I'm consuming a basemap from ArcGIS Online, so just the ArcGIS imagery basemap...
42:31...and I've already gone ahead and added two query layers to map...
42:34...and these query layers are coming directly out of a SQL Server 2008 database instance that's on my laptop here.
42:39So really, as Eric mentioned, a query layer is really just a feature layer in ArcGIS that's defined by a SQL query.
42:46So what it lets you do is get access to data natively in a spatial database.
42:50No geodatabase involved at all.
42:52So these really act just like any other layer in ArcGIS.
42:56There's one subtle difference.
42:57When I open the layer properties for my oil spill layer, I can actually look at the query that was used to define that layer...
43:04...and I can change it, I can make modifications to this query.
43:07So I'm doing a very simple query here.
43:08For an example, I'm doing select star, so selecting all the rows in the trajectory oil table.
43:14And what this table is, the data it contains is just the daily snapshot of the oil spill information.
43:22And because that information has a time component to it, I can use it with the time slider and make it time aware in ArcGIS.
43:29So I've done that, and I've added the time slider to map.
43:32So it means I can just jump through and see what the oil spill looked like each day.
43:37This is really powerful functionality, and this is pulling this directly...
43:40...querying my database tables directly, and integrating it with ArcGIS.
43:44So you could use this to integrate with your feature classes, other layers and map, and work with this data natively in the database.
43:53So I'll go ahead and create a couple more query layers.
43:55I'll first create a very simple query layer.
43:57And how you do that is just go to the Add Data pull-right and you open the New Query Layer dialog.
44:02The first thing I'll do here is make a connection to my spatial database.
44:07It's the database instance I have running on my machine.
44:09It's the same database that I took data from the two previous query layers.
44:14And when I make the connection, I get to see all the tables and views that I have access to...
44:19...as the user I've connected from in this table list here.
44:22So I can use this to build up my query and as I go through, when I select any of these tables, I get to see the columns in those tables.
44:30I could see this is the date value that I was using to use with the time slider and map.
44:35And I can use this column information and pull this into the query layer which eventually helps me building up that query.
44:41But if I just want to do a very simple query and see all the rows in the table, I can just double-click on the table...
44:46...and it adds a simple select star query to the query, and I can go and modify this if I wanted to constrain it.
44:53But I really just want to see the total path, everywhere the oil has went over time...
44:59...so that total cumulative polygon, and this has been captured in a table called the uncertainty zone.
45:06So I'm just going to do a simple select star query from that.
45:08I'm going to paste in the uncertainty zone name to name the query.
45:12And then the next step, I have to validate the results of that query.
45:16And what this does is just check to make sure that the results that returned from the query conform to ArcGIS's data modeling standards.
45:22So it checks stuff like, is there only one shape field in this result set? Does it only have one spatial reference?
45:28And if it doesn't, the dialog gives you an opportunity to change that query so it does conform.
45:33So since it validated okay, I can select Finish, and I can add that layer to map.
45:38So here I could see the cumulative polygon of everywhere the oil might have touched over time.
45:46Now this is an example of a very simple query but I can do more complex queries...
45:49...and I can really do anything that is supported through SQL in the database.
45:55So I'll follow the same workflow, connecting my SQL Server instance...
45:59...but I've actually gone ahead and created a query here which is a little bit more complicated.
46:03So what I want to do is find the intersection of that uncertainty zone with oyster lease polygons I have in another table...
46:10...a SQL Server table in my database, finding all the oyster beds or oyster leases that have possibly come in contact with the oil spill.
46:18So here I'm using the SQL Server ST Intersects function, and this really does a spatial intersection down in the database...
46:25...to give me the result of the intersection between uncertainty polygons with the oyster leases.
46:31So I can go ahead and copy and paste this directly into the query window, and I will just name this Oyster Leases.
46:43Here, following the same workflow, I'll validate the query and then selecting Finish, it'll add those results to map.
46:50So you could see here anything that's supported...anything I could do via SQL in the database I could also do through query layers...
46:57...as long as the result set conforms to ArcGIS's data modeling standards.
47:02So I'm going to go ahead and zoom to that layer, ArcGIS, and you can see this is the result as it pulls up and renders here.
47:11This is a result of all the thousands upon thousands of oyster leases and oyster beds that have been affected to date by the oil spill.
47:20So say I want to share this information out with other stakeholders, other users within my organization.
47:25Well, at ArcGIS 10 through ArcGIS Online, I can quickly and easily share that information out...
47:31...with other people who might be very concerned about this information.
47:35So the first thing I'm going to do is open the properties on the layer and just give it a quick description.
47:40So I'll call this Affected Oyster Beds.
47:48Just to give it a little bit of metadata so people understand what they're working with and then right-clicking on the layer...
47:53...I can create a layer package and upload that layer package to ArcGIS Online.
47:58So I'm already logged in to ArcGIS Online, so I'll call this layer package Oyster Leases.
48:05I'll first validate it and then I'll share it out with ArcGIS.
48:10To save a little bit of time, I'll just copy and paste the description into the summary here.
48:17That's true, I'm big on metadata here.
48:19And then I'll just add a couple tags to this so it can be searched on.
48:24And I'll call it Oil Spill and maybe Oysters.
48:30So I've added some tags, and now do I want to share this with everyone who has access to ArcGIS Online...
48:35...or do I want to do it just for specific members of groups?
48:38So I've actually gone ahead, and I'm a member of the Gulf Oyster Fisheries Special Interest Group...
48:43...so I really just want to share it with members of that group.
48:45So I can do so easily through this and selecting Okay, we'll go ahead and package up this layer.
48:51So not only the layer properties, how I've visualized it and symbolized it in ArcGIS, but also the data that went into creating that layer.
48:59So what this means is I'm now uploading that layer package to ArcGIS Online...
49:04...so anyone that goes ahead and consumes this layer doesn't really need to know any SQL.
49:08They don't really need to know anything about databases at all.
49:10All they need to know is what the layer is and then they can bring that down into their map documents and work with that information.
49:17So it's gone ahead and uploaded that map package or that layer package rather to ArcGIS Online.
49:21So I'm going to close down the map doc...
49:23... 'cause I'm done with it, and I'm already connected to ArcGIS here in the Web browser.
49:30So I'm signed in as myself.
49:32But say I was signed in as a different user in my organization who also was a member of that group.
49:37They can go to the Groups tab here and they can see all groups they're a member of.
49:42So they'll go to the Gulf Oyster Fisheries Special Interest Group and they'll now see that layer that I've uploaded.
49:49So selecting the layer, they can see more information about that layer, if anyone had comments on what the layer was about...
49:55...what was in that layer, they could see these comments here.
49:59And if they want to use it within ArcGIS, all they have to do is select Open and that'll download that layer package locally to the disk.
50:09Opening that layer package will then open that layer within ArcGIS.
50:16So this is a quick way to share information with other stakeholders and display that information.
50:21You could see this user doesn't need to know anything about the SQL, the complex SQL that went into generating this layer.
50:27They just need to be a member of a group and have access to ArcGIS Online.
50:33So that's a very desktop-focused user story, but I want to show a second part to this demo...
50:37...that's a very quick explanation of how query layers can also work in ArcGIS Server.
50:42So if I was to publish any of these map documents as a map service, you could consume these within web applications.
50:49And I've gone ahead and done this with some other data...
50:53...and this is the map document that I've been using to seed the map service.
50:58So it doesn't really look like much.
50:59Really I'm just using it for symbology, but this is also a query layer here.
51:03And I'll just open the properties on this layer.
51:06And what I'm doing is I'm using some native SQL Server time functions to the DATEDIFF function...
51:14...and actually what this query is doing is just giving me all results within the last 180 seconds.
51:20So this is time-aware layer of all the 911 calls in the New York City area.
51:26And it's logging this to a database table, and I'm refining this massive amount of information just showing me the last three minutes...
51:33...or a sliding time window of the last three minutes of the most recent calls.
51:37So I've gone ahead and already created this map document.
51:39I published it to ArcGIS Server, and I've consumed it through a web application...
51:45...and this is a web application that's actually consuming this service which is showing me the sliding time window.
51:50And it'll periodically refresh to show me the latest calls.
51:55So this really demonstrates how you can consume native database information and quickly and easily bring it into both...
52:01...ArcGIS Desktop and ArcGIS Server through use of query layers which we added at ArcGIS 10.
52:15That's pretty cool stuff, Brent. Thanks. Good demo.
52:20I've never seen that one before.
52:21That's a nice demo.
52:23Also I got very hungry with you talking about steaks and oysters all that time.
52:28And I'm sure everyone else in here who's missed their lunch is hungry as well.
52:38OK. Well let's see.
52:41I think we have enough time for this PowerPoint slide and then we're done.
52:47Basically, you've seen some of the functionality we've been releasing with the geodatabase at 10.
52:55You know, we were, in the 9 time frame, we were trying to complete the initial information model of the geodatabase.
53:01Now we're just trying to refine it and extend it and will be continuing to do so in the future.
53:07Now you saw the enhancements we did to the information model for our...
53:11...network technologies that are used for both utilities and natural resources, the geometric network.
53:18You saw the work that's being done for our transportation network model.
53:22Craig gave a great demo there with time-aware traffic routing and all kinds of interesting things.
53:27Very cool and flexible barrier support.
53:30And then you saw the new topology rules that we've been introducing based upon all your feedback.
53:37Now in the domain of data sharing and whatnot, our replication...
53:40...we're supporting that other key replication scenario that Craig also demoed.
53:47Where from those three different states, Montana, Idaho, and I don't know where the other one was.
53:52I'm not very good with geography.
53:55Yeah. New Jersey?
53:58No, this is funny.
54:00Basically a statement has been made in our group by one of the other developers.
54:03I can't take credit for it.
54:04But, a lot of computer science guys like me said, everything I've learned about geography I've learned while fixing bugs.
54:15In any event, so we have that new workflow in the replication domain or data sharing technology.
54:20We're trying to open up the geodatabase to make it even more open and flexible for you.
54:25You know, first we simplified the schema.
54:27Okay. That's cool.
54:28It's all now in XML.
54:30Joined that bandwagon.
54:32The other thing is we've gone and we're finishing up right now a new freely distributable API to file geodatabases...
54:42...that'll give you read/write access to simple data and read access to more complex information models stored in your file geodatabases.
54:50And then finally, Brent gave us some really cool demos that brought together a lot of key pieces of technology...
54:56...that we've been introducing at 10 involving query layers at the heart of it all.
55:01And then taking it up to ArcGIS Online and sharing this stuff.
55:05This is one thing we really look forward to to see what you guys, what you gals will do when you get back to your offices next week...
55:12...and you start using ArcGIS 10 nonstop in everything that you do.
55:17We want to see people start sharing data and see what comes up because it is fun.
55:20You saw yesterday how you can click on buttons and you can see, oh, what's the most recent stuff people have been throwing up there.
55:26I think it should be quite interesting.
55:28Plus, you know, some of those downloadable apps for, you know, your smartphones and whatnot, iPods, Androids, flip phones, stuff like that.
55:40I love my phone. I actually downgraded my Android last weekend.
55:44Quite seriously, the people at Verizon could not understand anyone turning in an Android and wanting an old-school flip phone.
55:50But, it was fun. It was a good conversation.
55:53You should have gotten a pager.
55:58In any event, you've seen all this stuff.
56:01I hope you'll find it useful, and we're going to be continuing to move forward very aggressively in the future...
56:07...with enhancing the geodatabase to make it more open, faster, and flexible and useful.
56:12So, anyway, thanks a lot for coming.
56:14Please fill out those surveys.
56:16And if you have questions, we're kind of out of time, but come on up.
56:19And other members of the geodatabase team, there are a couple other criminals in here...
56:23...if they'd come forward, too, and help flag questions, that'd be great. Thank you.
What’s New in the Geodatabase at ArcGIS 10
This session will discuss and demonstrate the key geodatabase functionality in the upcoming ArcGIS 10 release, the next major release of ArcGIS. This session will focus on the new improvements in the geodatabase at 10, including improved data access, replication enhancements, additional topology rules, schema changes, and much more.
- Recorded: Jul 1st, 2010
- Runtime: 56:30
- Views: 86951
- Published: Aug 25th, 2010
- Night Mode (Off)Automatically dim the web site while the video is playing. A few seconds after you start watching the video and stop moving your mouse, your screen will dim. You can auto save this option if you login.
- HTML5 Video (Off) Play videos using HTML5 Video instead of flash. A modern web browser is required to view videos using HTML5.