00:01Okay, so I'm going to talk about leveraging the cloud for data sharing between remote offices.

00:05My name is Gary MacDougall.

00:08So, when you're going to the cloud, and by cloud, I mean the Amazon cloud…

00:15…you have ArcGIS Server running up in the cloud, and then you've got…

00:19…you want to publish services up there, but you've got your data in your office.

00:23So how do you publish services on the cloud? Well, you need to get your data up to the cloud.

00:28Not only that, you want to do it in such a way that when you make edits to your data in your office…

00:34…you can synchronize those edits up to the cloud. And also, if you have editors hitting that data and making changes…

00:41…they can synchronize back to the local office. Now, to do that, you can use geodatabase replication.

00:46That's what we're going to talk about it the session. So what is replication?

00:51Well, replication involves synchronizing data.

00:54You have copies of geodatabases, you make edits to them, and then you run processes later to synchronize the data.

01:03We have several ways of copying data, or several configurations where you want to synchronize your data.

01:10One configuration is the, from an editable geodatabase to a read-only geodatabase.

01:18I make edits here, and I want to push them out to a read-only copy.

01:21Example would be, I've got a department within my company, and someone is maintaining like basemap data.

01:28And they want to give that to all the other departments.

01:30But they don't want everybody from all the other departments connecting to their database.

01:34They don't want to take on that load. So what they can do is they can create a geodatabase on the side…

01:41…create a replica to it, and then periodically sync changes to that replica geodatabase.

01:47And then that copy can be accessed by all the other departments, and they have the most up-to-date information.

01:53Another case would be, you want to send changes in both directions. You have editors on both ends.

01:59And the cloud, which is the focus of this session, is a great example of that.

02:03I've got web editors making edits. I want to pull those changes into the office. I've got editors in the office making changes.

02:10I want to push them up to the cloud, and I want to have them available to those people that are doing web editing.

02:14I want those web clients to see those changes that I made in the office.

02:22So, again, the word cloud is pretty overloaded. Hear it in lots of different places…

02:26…arcgis.com, SQL Azure, Amazon. The one I'm talking about here is the Amazon cloud.

02:32So what you can do at 10.0 is you can purchase machine images that are preconfigured with ArcGIS Server 10 from Esri.

02:40So you buy it, you get the machine, it's running with Server. And it's hosted on the Amazon cloud.

02:46And once you get that machine, you can just remote Desktop into it and work with it like you do any other machine.

02:52Why go to the cloud? Well, the cloud gives you some really nice things. First of all, it gives you high availability.

03:01So if your geodatabase or your whole office system were to melt down, you still have it on the cloud.

03:07It's always there, always accessible, always running. It's also scalable. It's a big thing with the cloud; it's elastic.

03:14So you can set it up such that, if your site suddenly gets hammered by lots of different people, it'll scale right up and handle the load.

03:24Just as important, in slow times, scale right down so that you don't have to spend the money to, you know…

03:32…use this huge machine when it's not being utilized. And then, of course, it saves on hard-earned resources.

03:38You don't have to buy a bunch of machines, you don't have to train your staff to, you know, learn how to, you know…

03:44…manage web servers and all of that stuff. It takes care of a lot of that for you.

03:50Okay, synchronizing with the cloud. That's what the focus of this session is going to be about. What are we talking about there?

03:57Well, the way that's going to work is you're going to, is, the way it works is that, first…

04:01…you need to go to the cloud and create a geodatabase. So you go up and you create your geodatabase.

04:07And then you create your replica, and the replica creation process will copy the data from your local office up to that cloud geodatabase.

04:17Now in addition to that, it'll also register replicas on each end. So what that means is that if you make edits to each geodatabase…

04:25…you can then run a synchronization process, like I show in that double arrow…

04:31…to keep the local office and the cloud in sync with each other.

04:36Now, on top of that, you have your services. So if you look at the cloud on the right side…

04:43…I have my data in the cloud, and then I have a feature service and a geodata service.

04:47Those are the services I'm going to be creating for this session, but of course, you can create any type of service.

04:49Connect to that web browser, go in through the feature service, getting at the geo…the database and the geodatabase on the cloud.

04:53Those services are what your web editors see. So in the top diagram, I show a web browser.

05:04And then, as synchronizations happen, I just get updates.

05:13Okay, with that, I'm going to do a demo. We'll show this workflow in the…in a real case.

05:30Okay, so what I have here is a map document that shows some data. It's the local government data.

05:37You may have seen this throughout the week. It was shown in the Plenary Session.

05:43And on top of this data, I have some service requests, and this is also from a template that we provide on the Resource page.

05:51Service requests are items that citizens in the city report. For example, let me identify in one of them.

06:03Here we have somebody reporting a fire hydrant leak, and the attached photograph of it, and, you know…

06:12…they can let me know that that's happening and I should get out there and fix it.

06:15And then, when that information comes in to the city, the city can evaluate all of these unassigned service requests…

06:22…that the users provided, and then they can go in and they can assign a crew to work on it…

06:29…change its status to Assigned, have the crew go out and work, and when they're done, come back and mark it as closed.

06:35Now, what I want to do is, I want to take this; I want to put it out on the cloud.

06:38I want to publish services from this so I can have my citizens report these incidents, and at the same time…

06:45…as I'm updating the service requests, they'll see how progress is being made.

06:51Okay, so step one is to go to the cloud machine. So I had a…this is actually an Amazon cloud machine running…

06:57…I think in Virginia, and this is the…this is what I was talking about. It's a machine image of a remote desktop into it.

07:06And it has ArcGIS Server running on it. It also has ArcGIS Desktop.

07:10Now, on this machine, I have Workgroup…Workgroup geodatabase with SQL Express.

07:17So if I go to my database servers, you'll see that I have some databases here now.

07:23Before this session, I just came in and I created a new database.

07:27Now, what's really important to know is that with this type of connection, I'm using OS authentication…

07:31…operating system authentication.

07:34So what I have to do is, for my geodatabase that I plan to…to expose through the, through the web…

07:41…I need to go in and I need to set permissions. And I need to give the ArcGIS SOC user…

07:46…this is the user that the ArcGIS Server uses to access the data…I need to give him read-write permissions.

07:52Step one is to do that. Okay, so here's the connection file in my, in the other folder.

08:03Same connection. I just move it to a location, again, that the server can access, but it's the same connection.

08:09And now what I need to do is, I need to take this new geodatabase I created and make it into a geodata service on the server.

08:16And that'll expose it over the Internet. And that's what's going to allow me to create a replica to the cloud machine.

08:23So to do that, I just right-click on the connection, and I say, Publish to ArcGIS Server.

08:28It's going to put it in my Cloud Sync folder. This server that's listed is the built-in server that…

08:34…that comes with the, with the Amazon image, so that's just running for me and preconfigured.

08:40Hit Next, hit Next, hit Finish. And now this is going in publishing the service.

08:46Okay, so the service is now published. I'm going to my admin connection for ArcGIS Server.

08:51And I see it listed. So this is my geodata service, and now that geodatabase is available on the web.

08:59Now, one thing I need to do is, I need to stop the service and go into service properties and turn on replication.

09:07By default, we don't turn it on. We don't want to expose the ability for anyone out there to…

09:12…to make changes to your geodatabase by default. So we make you come in and explicitly turn it on.

09:18But in this case, that's what I want. I want to replicate, so I'm going to turn it on and then start the service.

09:25Okay, so now, that's going. Leaving the cloud, I'm moving back to the local office.

09:33Okay, so here's my map that, here's my map with the data I want to use.

09:36Now this is not the data that I want to replicate. All I want to replicate are the service requests.

09:42So I'm going to create a new map, and I'm just going to drop in the data that I want to replicate.

09:53And I do that by dropping in my service requests. Now the other thing I want to do is I want to set an extent.

10:00How much…Where do I want to replicate? That's the second thing I need to define.

10:04To do that, I'm just going to go in and maybe drop in the county boundaries, and all I'm going to do there…

10:11…is just going to zoom that layer's extent. So you see, there's the county boundaries.

10:17This'll give me all of the extent of the city, which is all I need to replicate, so I'm going to use that as my extent.

10:22And then, I'm just going to remove it. I'm not going to replicate that part. So an important thing to know here…

10:27…when you're replicating data, you don't necessarily want to use the map that you'd, you do to work with…

10:34…to view data or to edit the data. You want to create a quick map that represents the data you want to replicate.

10:39Once it's replicated, you can then build the maps afterwards. Okay, so now I'm ready to create the…

10:46I'll click Create Replica. Choose two ways since I want to sync in both directions.

10:53And now I'm going to browse to my cloud machine.

10:59So I've added a connection to the cloud, just like any other server, cloud server, just like any other server.

11:04Here it's listed. Here is that Cloud Sync directory. Here's that cloud sync geodata service.

11:10So I'm specifying it. I'm going to call the replica Two-Way.

11:19Hit Next and hit Finish. So this is going in, creating a replica - taking this data, putting it into a transport file…

11:27…uploading it to the cloud machine, then it's going to insert that data into that geodatabase on the cloud…

11:34…and then it's going to register replicas on each end. So it's, yeah, that's done.

11:40Now let's go back to the cloud machine and see what happened. Here's my Cloud Sync geodatabase.

11:45I'm going to hit Refresh on this guy. Let me connect to him. Sorry, that's my geodata service.

11:53Here's my geodatabase. I'll refresh this guy. Here's all the data. And you'll also see that there's a replica created.

12:04Now what I think I can do is start creating my services. So I have a map document here. I'm just going to throw it in…

12:12…to Map. This actually points to my Cloud Sync database. So now I'm ready to publish a service.

12:18Now in this case, I want to publish a feature service, because I want people to be able to make edits.

12:23So to do that, I just go to the Publishing toolbar and click Publish ArcGIS Server, pick again the directory I want to put it in.

12:31I'll leave it Service Request cloud, is the name. Hit Next, and then I hit this check box that says Feature Access.

12:38This is what makes it a feature service. And then I hit Finish.

12:45So now that's publishing the service. If we go and look at the service, the server connection, and I refresh this guy.

12:54You'll see that it added my feature service. So now, I can access that anywhere over the web.

13:00So let's go back to my office machine. I'm sitting in the office, I'm…that's running over in Virginia…

13:07…and I'm sitting here with my laptop, and I'm connecting to it. Now when you publish a service…

13:12…the first thing you want to do is you want to clear the cache in REST, when you use REST.

13:18'Cause what that does is that resets the list of services that are available that I can access.

13:25Okay, so I just got a…it timed out on me, so I just need to log in again and then…

13:28…here are the cache. Should be good. Now I go to my REST page. Here's my REST…

13:35…my services listed in REST. Now let me go over to that Cloud Sync directory, and then you're going to see…

13:41…you're going to see my feature service listed. Now you see Map and Feature listed.

13:45That's because a feature service is really an extension of the map service, but once you get to REST…

13:50…you can treat it as an independent thing. So I can click on the feature service, and then I can just go…

13:58…view in arcgis.com map, and there's my data, sitting on the web.

14:06So I can now go ahead and make some edits. So if you'll notice, there's a quick arcgis.com map, see…

14:14…has a base, a basemap with my feature service on top.

14:17'Cause it's a feature service, this Edit button has become enabled, and there's my templates.

14:22So now I can come in and I can start adding new service requests. Say I add one here, I'm going to make it a…

14:31…debris on the road service request. So citizens can connect, start doing this work. At the same time…

14:40…I can come into this map, and I can start editing, and I can process some of these things.

14:53So you'll click on this guy, and I'm going to go and I'm going to change the status from Unassigned to Assigned.

15:05And that just changes the color from red to yellow. I'll stop editing and save edits.

15:14Okay, so at this point, I can continue to make edits, and also, people accessing the services on the web…

15:23…can continue to add information. Now, to get these two in sync, I'll just have to run a geodatabase replica synchronize.

15:33So when I hit Synchronize, this change to the status of this particular incident will be reflected in this map.

15:41And this new request will come into my office map, so I hit the Synchronize button, choose Two-Way…

15:51…a sec…hit the Synchronize…we see, this guy is not refreshing. Give me one second and I'll reopen this map.

16:12Bear with me for a couple secs. So the replica was created but it didn't refresh my, my map.

16:22Okay, hang on one second. Still not seeing it. I'll go to my local office connection and…see…

16:44Refresh that guy. There's the replica listed. Now it's updated on my map. Just a little bit of a refresh issue there.

16:55Okay, so now I'm ready to synchronize. So let me hit the Synchronize button. Two-way…

17:00…going to my cloud sync, I want to go in both directions, hit Next, and then hit Finish.

17:09So this is going to collect those changes from the office, put them in the, in the cloud geodatabase.

17:16Then when I refresh my service, that'll be reflected. At the same time, the changes that were added by…

17:23…in the cloud geodatabase by citizens will come back into my office geodatabase.

17:28So you see that new incident? Shows up in my office now. I refresh my web map, and you will see…

17:38…that the status of that incident got updated, as well. Now this is a very, very basic example, but the point is…

17:46…you can get this data synchronized between the two sides. So you can build really nice, really good web maps…

17:54…to do really great work on the cloud end, and at the same time, do all of your management in your office…

18:02…and keep those two things in sync. Okay, I've got just a couple of final notes to make.

18:13When you go to do this, and you run the Create Replica command, one thing that's important to know…

18:17…is that there is a, that IAS, which is what I was using as my web server, has a size limit.

18:23The default is 4 megabytes. Anything bigger than 4 megabytes is going to time out. So you've got two options to deal with that.

18:30The first one is, you want to up that limit, and we have a technical article I've listed here that talks about how to do that.

18:39The other thing is, you can do it disconnected. That means instead of connecting directly to the geodata service…

18:44…you can export it through a transport file, upload it, copy and paste it in, and go that way.

18:50And to make that easier, we've created an add-in at 10.0 that's referenced at this link.

19:02[unintelligible audience question]

19:04Question with schema changes. So, I talked with this, actually, in the full replication session. What you can do is…

19:11…the next thing beyond this is synchronization and scheduling the synchronization.

19:15So you can create a Python model to schedule your synchronization. You can also, in that model…

19:20…do a schema synchronization to make schema changes like add/delete field, get synchronized…

19:26…followed by a data synchronization. Important thing to know…

19:29When services are running, by default they're going to lock, have some schema locks. So you're going to have to…

19:35…either stop your services, probably stop your services before you do that schema synchronization.

19:41Now there is some documentation in the help that talks about working with schema changes and services…

19:47…with locking in services, where you may have some options there to relax that rule. I would recommend…

19:53…you know, looking into that. But the technology is there to do it, that you can do schema and data synchronizations.

20:00Okay. Thank you.