Leveraging the Cloud for Data Sharing Between Remote Offices

Gary MacDougall describes how to synchronize data between a local office geodatabase and a geodatabase running in the cloud.

Jul 14th, 2011

Start From:
Player Color:

Right-click on these links to download and save this video.


00:01Okay, so I'm going to talk about leveraging the cloud for data sharing between remote offices.

00:05My name is Gary MacDougall.

00:08So, when you're going to the cloud, and by cloud, I mean the Amazon cloud…

00:15…you have ArcGIS Server running up in the cloud, and then you've got…

00:19…you want to publish services up there, but you've got your data in your office.

00:23So how do you publish services on the cloud? Well, you need to get your data up to the cloud.

00:28Not only that, you want to do it in such a way that when you make edits to your data in your office…

00:34…you can synchronize those edits up to the cloud. And also, if you have editors hitting that data and making changes…

00:41…they can synchronize back to the local office. Now, to do that, you can use geodatabase replication.

00:46That's what we're going to talk about it the session. So what is replication?

00:51Well, replication involves synchronizing data.

00:54You have copies of geodatabases, you make edits to them, and then you run processes later to synchronize the data.

01:03We have several ways of copying data, or several configurations where you want to synchronize your data.

01:10One configuration is the, from an editable geodatabase to a read-only geodatabase.

01:18I make edits here, and I want to push them out to a read-only copy.

01:21Example would be, I've got a department within my company, and someone is maintaining like basemap data.

01:28And they want to give that to all the other departments.

01:30But they don't want everybody from all the other departments connecting to their database.

01:34They don't want to take on that load. So what they can do is they can create a geodatabase on the side…

01:41…create a replica to it, and then periodically sync changes to that replica geodatabase.

01:47And then that copy can be accessed by all the other departments, and they have the most up-to-date information.

01:53Another case would be, you want to send changes in both directions. You have editors on both ends.

01:59And the cloud, which is the focus of this session, is a great example of that.

02:03I've got web editors making edits. I want to pull those changes into the office. I've got editors in the office making changes.

02:10I want to push them up to the cloud, and I want to have them available to those people that are doing web editing.

02:14I want those web clients to see those changes that I made in the office.

02:22So, again, the word cloud is pretty overloaded. Hear it in lots of different places…

02:26…arcgis.com, SQL Azure, Amazon. The one I'm talking about here is the Amazon cloud.

02:32So what you can do at 10.0 is you can purchase machine images that are preconfigured with ArcGIS Server 10 from Esri.

02:40So you buy it, you get the machine, it's running with Server. And it's hosted on the Amazon cloud.

02:46And once you get that machine, you can just remote Desktop into it and work with it like you do any other machine.

02:52Why go to the cloud? Well, the cloud gives you some really nice things. First of all, it gives you high availability.

03:01So if your geodatabase or your whole office system were to melt down, you still have it on the cloud.

03:07It's always there, always accessible, always running. It's also scalable. It's a big thing with the cloud; it's elastic.

03:14So you can set it up such that, if your site suddenly gets hammered by lots of different people, it'll scale right up and handle the load.

03:24Just as important, in slow times, scale right down so that you don't have to spend the money to, you know…

03:32…use this huge machine when it's not being utilized. And then, of course, it saves on hard-earned resources.

03:38You don't have to buy a bunch of machines, you don't have to train your staff to, you know, learn how to, you know…

03:44…manage web servers and all of that stuff. It takes care of a lot of that for you.

03:50Okay, synchronizing with the cloud. That's what the focus of this session is going to be about. What are we talking about there?

03:57Well, the way that's going to work is you're going to, is, the way it works is that, first…

04:01…you need to go to the cloud and create a geodatabase. So you go up and you create your geodatabase.

04:07And then you create your replica, and the replica creation process will copy the data from your local office up to that cloud geodatabase.

04:17Now in addition to that, it'll also register replicas on each end. So what that means is that if you make edits to each geodatabase…

04:25…you can then run a synchronization process, like I show in that double arrow…

04:31…to keep the local office and the cloud in sync with each other.

04:36Now, on top of that, you have your services. So if you look at the cloud on the right side…

04:43…I have my data in the cloud, and then I have a feature service and a geodata service.

04:47Those are the services I'm going to be creating for this session, but of course, you can create any type of service.

04:49Connect to that web browser, go in through the feature service, getting at the geo…the database and the geodatabase on the cloud.

04:53Those services are what your web editors see. So in the top diagram, I show a web browser.

05:04And then, as synchronizations happen, I just get updates.

05:13Okay, with that, I'm going to do a demo. We'll show this workflow in the…in a real case.

05:30Okay, so what I have here is a map document that shows some data. It's the local government data.

05:37You may have seen this throughout the week. It was shown in the Plenary Session.

05:43And on top of this data, I have some service requests, and this is also from a template that we provide on the Resource page.

05:51Service requests are items that citizens in the city report. For example, let me identify in one of them.

06:03Here we have somebody reporting a fire hydrant leak, and the attached photograph of it, and, you know…

06:12…they can let me know that that's happening and I should get out there and fix it.

06:15And then, when that information comes in to the city, the city can evaluate all of these unassigned service requests…

06:22…that the users provided, and then they can go in and they can assign a crew to work on it…

06:29…change its status to Assigned, have the crew go out and work, and when they're done, come back and mark it as closed.

06:35Now, what I want to do is, I want to take this; I want to put it out on the cloud.

06:38I want to publish services from this so I can have my citizens report these incidents, and at the same time…

06:45…as I'm updating the service requests, they'll see how progress is being made.

06:51Okay, so step one is to go to the cloud machine. So I had a…this is actually an Amazon cloud machine running…

06:57…I think in Virginia, and this is the…this is what I was talking about. It's a machine image of a remote desktop into it.

07:06And it has ArcGIS Server running on it. It also has ArcGIS Desktop.

07:10Now, on this machine, I have Workgroup…Workgroup geodatabase with SQL Express.

07:17So if I go to my database servers, you'll see that I have some databases here now.

07:23Before this session, I just came in and I created a new database.

07:27Now, what's really important to know is that with this type of connection, I'm using OS authentication…

07:31…operating system authentication.

07:34So what I have to do is, for my geodatabase that I plan to…to expose through the, through the web…

07:41…I need to go in and I need to set permissions. And I need to give the ArcGIS SOC user…

07:46…this is the user that the ArcGIS Server uses to access the data…I need to give him read-write permissions.

07:52Step one is to do that. Okay, so here's the connection file in my, in the other folder.

08:03Same connection. I just move it to a location, again, that the server can access, but it's the same connection.

08:09And now what I need to do is, I need to take this new geodatabase I created and make it into a geodata service on the server.

08:16And that'll expose it over the Internet. And that's what's going to allow me to create a replica to the cloud machine.

08:23So to do that, I just right-click on the connection, and I say, Publish to ArcGIS Server.

08:28It's going to put it in my Cloud Sync folder. This server that's listed is the built-in server that…

08:34…that comes with the, with the Amazon image, so that's just running for me and preconfigured.

08:40Hit Next, hit Next, hit Finish. And now this is going in publishing the service.

08:46Okay, so the service is now published. I'm going to my admin connection for ArcGIS Server.

08:51And I see it listed. So this is my geodata service, and now that geodatabase is available on the web.

08:59Now, one thing I need to do is, I need to stop the service and go into service properties and turn on replication.

09:07By default, we don't turn it on. We don't want to expose the ability for anyone out there to…

09:12…to make changes to your geodatabase by default. So we make you come in and explicitly turn it on.

09:18But in this case, that's what I want. I want to replicate, so I'm going to turn it on and then start the service.

09:25Okay, so now, that's going. Leaving the cloud, I'm moving back to the local office.

09:33Okay, so here's my map that, here's my map with the data I want to use.

09:36Now this is not the data that I want to replicate. All I want to replicate are the service requests.

09:42So I'm going to create a new map, and I'm just going to drop in the data that I want to replicate.

09:53And I do that by dropping in my service requests. Now the other thing I want to do is I want to set an extent.

10:00How much…Where do I want to replicate? That's the second thing I need to define.

10:04To do that, I'm just going to go in and maybe drop in the county boundaries, and all I'm going to do there…

10:11…is just going to zoom that layer's extent. So you see, there's the county boundaries.

10:17This'll give me all of the extent of the city, which is all I need to replicate, so I'm going to use that as my extent.

10:22And then, I'm just going to remove it. I'm not going to replicate that part. So an important thing to know here…

10:27…when you're replicating data, you don't necessarily want to use the map that you'd, you do to work with…

10:34…to view data or to edit the data. You want to create a quick map that represents the data you want to replicate.

10:39Once it's replicated, you can then build the maps afterwards. Okay, so now I'm ready to create the…

10:46I'll click Create Replica. Choose two ways since I want to sync in both directions.

10:53And now I'm going to browse to my cloud machine.

10:59So I've added a connection to the cloud, just like any other server, cloud server, just like any other server.

11:04Here it's listed. Here is that Cloud Sync directory. Here's that cloud sync geodata service.

11:10So I'm specifying it. I'm going to call the replica Two-Way.

11:19Hit Next and hit Finish. So this is going in, creating a replica - taking this data, putting it into a transport file…

11:27…uploading it to the cloud machine, then it's going to insert that data into that geodatabase on the cloud…

11:34…and then it's going to register replicas on each end. So it's, yeah, that's done.

11:40Now let's go back to the cloud machine and see what happened. Here's my Cloud Sync geodatabase.

11:45I'm going to hit Refresh on this guy. Let me connect to him. Sorry, that's my geodata service.

11:53Here's my geodatabase. I'll refresh this guy. Here's all the data. And you'll also see that there's a replica created.

12:04Now what I think I can do is start creating my services. So I have a map document here. I'm just going to throw it in…

12:12…to Map. This actually points to my Cloud Sync database. So now I'm ready to publish a service.

12:18Now in this case, I want to publish a feature service, because I want people to be able to make edits.

12:23So to do that, I just go to the Publishing toolbar and click Publish ArcGIS Server, pick again the directory I want to put it in.

12:31I'll leave it Service Request cloud, is the name. Hit Next, and then I hit this check box that says Feature Access.

12:38This is what makes it a feature service. And then I hit Finish.

12:45So now that's publishing the service. If we go and look at the service, the server connection, and I refresh this guy.

12:54You'll see that it added my feature service. So now, I can access that anywhere over the web.

13:00So let's go back to my office machine. I'm sitting in the office, I'm…that's running over in Virginia…

13:07…and I'm sitting here with my laptop, and I'm connecting to it. Now when you publish a service…

13:12…the first thing you want to do is you want to clear the cache in REST, when you use REST.

13:18'Cause what that does is that resets the list of services that are available that I can access.

13:25Okay, so I just got a…it timed out on me, so I just need to log in again and then…

13:28…here are the cache. Should be good. Now I go to my REST page. Here's my REST…

13:35…my services listed in REST. Now let me go over to that Cloud Sync directory, and then you're going to see…

13:41…you're going to see my feature service listed. Now you see Map and Feature listed.

13:45That's because a feature service is really an extension of the map service, but once you get to REST…

13:50…you can treat it as an independent thing. So I can click on the feature service, and then I can just go…

13:58…view in arcgis.com map, and there's my data, sitting on the web.

14:06So I can now go ahead and make some edits. So if you'll notice, there's a quick arcgis.com map, see…

14:14…has a base, a basemap with my feature service on top.

14:17'Cause it's a feature service, this Edit button has become enabled, and there's my templates.

14:22So now I can come in and I can start adding new service requests. Say I add one here, I'm going to make it a…

14:31…debris on the road service request. So citizens can connect, start doing this work. At the same time…

14:40…I can come into this map, and I can start editing, and I can process some of these things.

14:53So you'll click on this guy, and I'm going to go and I'm going to change the status from Unassigned to Assigned.

15:05And that just changes the color from red to yellow. I'll stop editing and save edits.

15:14Okay, so at this point, I can continue to make edits, and also, people accessing the services on the web…

15:23…can continue to add information. Now, to get these two in sync, I'll just have to run a geodatabase replica synchronize.

15:33So when I hit Synchronize, this change to the status of this particular incident will be reflected in this map.

15:41And this new request will come into my office map, so I hit the Synchronize button, choose Two-Way…

15:51…a sec…hit the Synchronize…we see, this guy is not refreshing. Give me one second and I'll reopen this map.

16:12Bear with me for a couple secs. So the replica was created but it didn't refresh my, my map.

16:22Okay, hang on one second. Still not seeing it. I'll go to my local office connection and…see…

16:44Refresh that guy. There's the replica listed. Now it's updated on my map. Just a little bit of a refresh issue there.

16:55Okay, so now I'm ready to synchronize. So let me hit the Synchronize button. Two-way…

17:00…going to my cloud sync, I want to go in both directions, hit Next, and then hit Finish.

17:09So this is going to collect those changes from the office, put them in the, in the cloud geodatabase.

17:16Then when I refresh my service, that'll be reflected. At the same time, the changes that were added by…

17:23…in the cloud geodatabase by citizens will come back into my office geodatabase.

17:28So you see that new incident? Shows up in my office now. I refresh my web map, and you will see…

17:38…that the status of that incident got updated, as well. Now this is a very, very basic example, but the point is…

17:46…you can get this data synchronized between the two sides. So you can build really nice, really good web maps…

17:54…to do really great work on the cloud end, and at the same time, do all of your management in your office…

18:02…and keep those two things in sync. Okay, I've got just a couple of final notes to make.

18:13When you go to do this, and you run the Create Replica command, one thing that's important to know…

18:17…is that there is a, that IAS, which is what I was using as my web server, has a size limit.

18:23The default is 4 megabytes. Anything bigger than 4 megabytes is going to time out. So you've got two options to deal with that.

18:30The first one is, you want to up that limit, and we have a technical article I've listed here that talks about how to do that.

18:39The other thing is, you can do it disconnected. That means instead of connecting directly to the geodata service…

18:44…you can export it through a transport file, upload it, copy and paste it in, and go that way.

18:50And to make that easier, we've created an add-in at 10.0 that's referenced at this link.

19:02[unintelligible audience question]

19:04Question with schema changes. So, I talked with this, actually, in the full replication session. What you can do is…

19:11…the next thing beyond this is synchronization and scheduling the synchronization.

19:15So you can create a Python model to schedule your synchronization. You can also, in that model…

19:20…do a schema synchronization to make schema changes like add/delete field, get synchronized…

19:26…followed by a data synchronization. Important thing to know…

19:29When services are running, by default they're going to lock, have some schema locks. So you're going to have to…

19:35…either stop your services, probably stop your services before you do that schema synchronization.

19:41Now there is some documentation in the help that talks about working with schema changes and services…

19:47…with locking in services, where you may have some options there to relax that rule. I would recommend…

19:53…you know, looking into that. But the technology is there to do it, that you can do schema and data synchronizations.

20:00Okay. Thank you.

Copyright 2016 Esri
Auto Scroll (on)Enable or disable the automatic scrolling of the transcript text when the video is playing. You can save this option if you login


No comments. Be the first to write one below.

Comment on this Video