00:01My name is Ismael Chivite. I'm the ArcGIS Server product manager...
00:03...and with me is Marwa Mabrouk who works in the Professional Services Division in Esri.
00:08And she has extensive experience deploying ArcGIS Server on Amazon EC2.
00:13So Marwa's going to present at the end kind of some experiences that she had with this...
00:18...and I will kind of cover the product presentation, explaining what the product is and how...how you use it.
00:23So we will have kind of a step-by-step process so you can see how this thing really works.
00:29I want to point out that there is another session on ArcGIS Server on Amazon EC2, which covers the business aspects of it.
00:37Like why should I run ArcGIS Server on Amazon? Or why I shouldn't.
00:43This session is more about, okay, I think I...I know why I want to run on Amazon, so just tell me how this thing really works. Okay?
00:51So this is kind of a technical aspect of it, and then we have the business aspect of it.
00:56Although there is a little bit of overlap between the two, as you will see.
01:00Also I want to remind you that we have these surveys...
01:03...that you should fill out and put in the boxes outside if you remember that at the end of the session.
01:10Please, if you have one of these, probably you don't because it means it's very old, but if you have one of these, you know what to do with it.
01:17Right? Just turn it off, and I'll do that as well. That's kind of embarrassing when...okay. Let's go into it.
01:29First, a brief introduction to ArcGIS on Amazon EC2, then basic concepts. This is the step-by-step process to get started.
01:39We will also touch on some of the key areas where people have problems with ArcGIS Server on Amazon EC2...
01:46...specifically managing your GIS data; how do you get your data into the cloud and off the cloud as well.
01:52And also we will explain how you scale out and how you shrink ArcGIS Server deployments in the Amazon cloud.
02:01We'll go and look at the real-world examples and then hopefully have the Q&A.
02:06Let's just start with the introduction, and this will be very basic, just in case that you are not familiar with Amazon Web Services.
02:14Amazon Web Services is really a cloud infrastructure that is...
02:19Basically, these guys have data centers around the world, and they rent these data centers.
02:26You can rent computing, that is, servers. You can rent storage. You can also rent kind of network services for improved bandwidth.
02:39All these capabilities - computing, storage, network - you can rent, you can purchase through Web services.
02:47It's kind of a self-service approach to building an IT infrastructure.
02:54They have different Web services to access these capabilities; for example, EC2 is Elastic Cloud Computing...
03:01...is the service that allows you to rent virtual servers.
03:05So you can say on...you can go and say, I want one server with two cores, or I want two servers with four cores. That's EC2.
03:13They have another one, which is CloudWatch. Is a service that allows you to monitor the services that you rented.
03:21Other services like CloudFront allow you to speed up download of files from the cloud.
03:28So you are storing your map tiles in the cloud...
03:30...and you can enable CloudFront to kind of use edge caching to accelerate download of these map tiles into the clients.
03:38They have actually many, many different services, including other network services like virtual private network, storage, and so on.
03:47The unique characteristic of these data centers is that not only they offer you the self-service experience...
03:54...you don't need to call anybody to get these resources; you just go to a website and you...you rent these things.
03:59But the unique characteristic is that you pay as you go, so storage, you pay...
04:05Well, you are storing 100 gigabytes, then every month, you will pay for this storage.
04:11And with computing, it's the same way. Are you using two computers or one computer?
04:16Oh, you stopped using one computer, then you can stop paying for that. You get the point?
04:20It's just pay as you go.
04:24We will sometimes refer to this as the Amazon cloud or Amazon EC2.
04:32For ArcGIS Server users, the advantage of running ArcGIS Server in this infrastructure are...simplified management is one of them.
04:40It's very easy to set up ArcGIS Server in this Amazon cloud infrastructure because we have preconfigured machine images...
04:51We'll talk about that. We call them AMIs.
04:53....that are basically virtual images that we give you so you can pretty much select a size of the hardware you want to run ArcGIS Server on...
05:02...and literally activate it. Say, Just make it happen; install ArcGIS Server. It will be preconfigured.
05:08And also, simple management because once the server is up and running, they have these very advanced tools to create backups...
05:17...to monitor the CPU usage and disk I/O of your servers...
05:21...to add more machines to a cluster of servers as needed, as the load increases, and also to shrink the deployment.
05:30And you do all of these doing this self-service experience.
05:34It's also interesting because Amazon gives you...I wouldn't call it real time, but almost real-time access to these resources.
05:42So if you need computing power, you don't need to wait for a machine to be delivered to your organization...
05:49...and then you plug it in and then IT kind of goes and plugs all the things.
05:55I know of some organizations that it takes them six months, literally six months...
06:00...to just get a machine where they can deploy their applications.
06:04And then in Amazon, once you have this self-service experience, it's kind of, you go there, you select...
06:09...and in a few minutes, you have a server up and running on the Web.
06:13And that's a big, big reason why people move to the Amazon cloud, because they can go from idea to application very quickly.
06:20And often, it's not just about the technical aspects of these things, but just the red tape that gets you to the final solution.
06:29The same with storage and so on.
06:31And finally, as we mentioned before, you don't make a long-term investment into your infrastructure.
06:37People refer to this as pay-as-you-go model; you will also hear it...hear it as elastic...elastic computing in the context of servers.
06:46And the classic example is, I have an application and I honestly don't know how many people are going to use it.
06:52I'm going to try with one server. And that server is going and then more people use the application, right?
06:59As the load increases, response time decreases.
07:00Now you buy a second server for that increased load, and you buy a third server.
07:06What happens when people stop using your application?
07:08You wish you could go down to one server, but you already purchased two servers.
07:14In the Amazon cloud, you say now I want two additional servers; now I don't want them.
07:19And when you shut them down, you stop paying for that.
07:21And that is interesting from a financial point of view in some cases.
07:26It's not like going to the Amazon cloud is cheaper than deploying on premise, but in these scenarios where they have...
07:32...you have these peaks of load, this can save you a lot of money.
07:39Here are some typical scenarios where people want to run ArcGIS Server on Amazon. One is development.
07:46For developers, EDN subscribers, starting with ArcGIS Server 10, have access to our virtual images in the Amazon cloud.
07:54And it's interesting from a development point of view because it saves you time.
07:58You can set up an ArcGIS Server 10 instance in just a few minutes as a developer.
08:04And you don't need to allocate a machine within your organization to run that server.
08:08Just go to the Amazon cloud, you pay $50 for running 25 hours of your server, and then once you are done, you terminate the machine.
08:18So it kind of saves money in that regard.
08:22The other aspect is that you may want to run, on the Amazon cloud, your ArcGIS Server just for development purposes.
08:28Maybe you are developing an application that you will deploy on your own premises servers.
08:34Right? Develop in the cloud; once you have done your work, you terminate the machines...
08:39...and then you bring down your applications, and then you deploy them on premise. Or you deploy them on the cloud; that's your choice.
08:46The other one is [unintelligible] ad hoc GIS analysis. As you know, many GIS operations are very high CPU intensive.
08:55You need a lot of computing power if you want to get your job done in a small amount of time.
09:00So the easiest thing to do is to add more iron to these tasks; you add more servers.
09:06In the Amazon cloud, the advantage is...is that you have almost unlimited access to this computing power.
09:12So you can say, Give me 10 servers, and then boom.
09:16Ten servers are there, with ArcGIS Server, your services, your data...
09:19...and you can split or break down your task across these different machines.
09:25We published a white paper some time ago, it's called ArcGIS Server in Practice, large batch geocoding...
09:31...where we explain a technique by which you can take a cluster of peers, of machines that are running ArcGIS Server...
09:39...and kind of break down your long tasks across all these machines.
09:43This technique works well on premise, and it works well, as well, in the Amazon cloud.
09:52The other one is I just want to have a Web mapping site up quick, so I go to the Amazon cloud...
09:57...because I don't have to worry about having this machine 24/7 running here on premise.
10:02I don't need to worry about people using bandwidth in my network.
10:06It's all running in the Amazon cloud.
10:08And this links again with this concept of elasticity where I cannot afford to be down.
10:14This is a screen shot from an application that British Petroleum has for the oil spill thing.
10:18Probably the last thing they want at this point is to have their website down after what happened...
10:22...so they are deploying their...this website in Amazon because in case that many people hit...
10:27...they can almost immediately react to that and add more machines or drop them. Okay.
10:35Let's talk about some basic concepts of how you deploy ArcGIS Server in the Amazon cloud.
10:43And I will start with us giving you some terminology that we are going to use.
10:49The AWS, Amazon Web Services, management console. It's a web application.
10:55You open a web browser, you go to this URL, aws.amazon.com, and then boom. You can get into this console.
11:02It's basically a dashboard and a management tool. A dashboard because allows you to monitor the servers that are running for you...
11:10...and a management tool because allows you to launch new servers...
11:16...stop them, start them, terminate them, create backups, upload data, monitor the activity.
11:22And it also allows you to do that across any of the data centers that Amazon has around the world.
11:30That's what gives you this self-service experience that we were talking about.
11:36Another concept is this concept of AMI.
11:39AMI stands for Amazon Machine Image, and it's nothing but a virtual image that works in the Amazon cloud.
11:46It's kind of a cookie-cutter approach to putting an operating system, middleware or, in this case, ArcGIS Server...
11:55...as well as the data that ArcGIS Server needs, into a server.
12:00So you get a server, you rent one, and then you say, I want to use this AMI.
12:05This AMI has ArcGIS Server, it has the data for world population and my services.
12:11And when you put that AMI within your server, your server is literally up and running; just can go and it works.
12:19We have two AMIs, we, Esri. We provide two AMIs to you.
12:24Basically, once you have licensed ArcGIS Server, you can call customer service and then tell them...
12:30...Look, this is my Amazon account ID, and I want you to share the AMIs with me. So we share these AMIs.
12:37So when you go to this management console that you saw in the previous slide...
12:40...you can select the ArcGIS Server AMI or the Enterprise Geodatabase AMI.
12:45The ArcGIS Server AMI, think of it as the GIS services server.
12:51We put...basically the...the GIS services install in there...
12:55...map services, geoprocessing, geolocators, ArcGIS Server Manager, the services directory.
13:02All of these things are preconfigured in this AMI.
13:04It also has a SQL Server Express database.
13:08We initially put it there because it's kind of nice to have a...as a user store so you can secure your services.
13:14And we wanted to kind of simplify that experience, so SQL Server Express is there.
13:19But then, we also enabled ArcSDE. So if you have an ArcGIS Server Standard Workgroup license, for example, we'll give you this.
13:28And that gives you SQL Server with ArcSDE, ArcGIS Server.
13:31It also has ArcGIS Desktop installed in the machine, which you should use to administer your services.
13:38It's not for you to sit and start an edit session or do cartographic production because you will interact...
13:44...as you will see, with the servers through Remote Desktop.
13:46So it's kind of a clanky experience when you are using ArcMap over Remote Desktop, but it's there for you to do things like...
13:54...fixing the path of your map documents or your geoprocessing services...geoprocessing models before you create these services.
14:02We also attach 100 gigabytes of additional storage to these machines, then you can actually get rid of it or add additional storage if you like.
14:09But it's...it's there. So that's the GIS services AMI.
14:13And then we have the Enterprise Geodatabase AMI, which is basically a Postgres database with ArcSDE enabled.
14:21And the idea is that you launch the ArcGIS AMI and you have ArcGIS Server running...
14:26...and then you launch another server where you put this Enterprise Geodatabase AMI.
14:31And you really don't need to log in to this machine unless you want to get into the table spaces management and so on.
14:38You can literally connect from your ArcGIS Server using ArcCatalog into this Enterprise Geodatabase AMI.
14:45So this again, completely preconfigured.
14:47Both AMIs run only on Windows 2008 64 bit. Only Windows 2008 64 bit for now.
14:59Another concept is EC2 instances. Amazon Web Services refers to servers as EC2 instances.
15:07An EC2 instance is a virtual server. Okay? Just think of it as just a server, and they have different sizes of them.
15:15They have the standard sizes, dual-core, four cores, and then they have the high-memory and high-CPU groups of EC2 instances.
15:26So this means that, well, you are really working with virtual images all the time, 'cause data centers have physical servers...
15:31...but you only can instantiate virtual servers within those.
15:35And depending on how much of these physical servers you want to use, you will start a big EC2 instance or a small EC2 instance.
15:43Do you got the point?
15:45So now you can make the link between the AMIs and EC2 instances.
15:50If EC2 instance is my, let's call it the hardware, I put the AMI, that's the software.
15:55And together, it's a fully functional machine, right, with the software running.
16:01As you can imagine, these guys are making money somehow, right? So the bigger the machine, the more you are going to pay for it.
16:08But it's always an hourly price for these machines. It ranges from I think it's like 98 cents or so to about two-point-something dollars per hour.
16:20And you can...those are the prices on demand when you make a commitment for one hour.
16:24And you can actually rent these servers for one or three years. They call that reserved instances.
16:30And then the hourly prices drop, although you have to pay an up-front fee.
16:34This is all very well explained in their website. They are pretty transparent about pricing for their EC2 instances.
16:41Also, the prices vary with size, but vary also with the location.
16:46The European data center is slightly more expensive than the ones that we have in the U.S. and so on.
16:57Okay. Another concept. I'm going to throw some concepts, and then we...you'll start all the pieces getting together.
17:03Elastic IPs. So what happens when you start a new EC2 instance? The clock starts ticking, right? They start charging you per hour.
17:13At some point, say you are developing an application, you went...you may want to say, I want to stop my instance...
17:18...because I want to go home to sleep, and I don't want to pay for this...this instance while it's running overnight.
17:23So you stop the instance. Now the next morning, you come and you start the instance again...
17:28...and it will just have your applications as you left...as you left them the day before.
17:33But the thing is that because these are virtual machines that they hydrate and rehydrate when you start and stop...
17:39...your virtual machine may show up in actually a physical server that is different from the one you ran yesterday. Right?
17:47It will still be a small machine or a big machine as you decided, but physically might be a different machine.
17:52And I'm oversimplifying things here, but essentially the...the IP of your machine may change. It actually changes quite often.
18:00So what you do for your applications, your applications are going to point to your IP, the IP of your machine...
18:07...so if the IP changes, the applications will stop working.
18:10So these elastic IPs are basically IPs that you rent and you say, I want to reserve an IP, and they give you an IP that you can associate with an instance.
18:20So your applications will always go through this fixed IP regardless of the host name of the virtual machine. Okay?
18:28This is important for you to keep in mind, because our AMIs are not just, oh, we put ArcGIS Server in a virtual machine and there you go.
18:36No. These AMIs that we built and that you can get have special logic that makes that, on start-up, we check for these host name...
18:45...and we do all the necessary changes in ArcGIS Server Services Directory, the cfg files, and so on, so ArcGIS Server runs.
18:53So when you start, ArcGIS Server is ticking, right there. Okay?
18:59Now, elastic IPs are good for this use case where you want to have a well-known IP for your applications...
19:05...but there is also another interesting case for elastic IPs.
19:08And it's one where I'm going to start developing my application on a small instance because it's cheap.
19:15But...and then I put elastic IP on top.
19:19But when I go into production, I don't want to use the small machine; I want to use a bigger machine.
19:23So you may want to start a bigger machine and then associate your elastic IP with the bigger machine.
19:30But the software, the AMI that runs this bigger machine, is the same as the small one.
19:35So you're actually, under the covers, you are switching your server from a small machine to a big machine...
19:41...and the applications don't even notice. Do you...do you follow the idea?
19:52Of course, they charge you for elastic IPs.
19:55Regions and availability zones. I said that Amazon Web Services has several data centers around the world. They call them regions.
20:02They have two in the U.S., one in Virginia, the other one in...is it California or Oregon? California.
20:10And then they have one in Singapore and one in Ireland.
20:14And that is kind of interesting because allows you to actually deploy your application across different data centers if you like...
20:20...but it's also interesting because of legal regulations. Some countries will not allow you to get your data outside of your country.
20:28Like in the European Union, if you have confidential information, the data shall be hosted in the European Union.
20:35So you will go to that data center, right?
20:39Canadians don't like to...to store data in the U.S. You know that, right? Don't you know that? Yeah. It's by law in some cases.
20:47So it's good to be able to play with different data centers for those reasons.
20:51Another aspect is, aside from the regions, within each data center, which in...within each region, they have availability zones.
20:59These are sections of the data center that have different Internet access point, isolated Internet access points...
21:09...as well as isolated...how do you call that? Like energy.
21:15Like if the power goes out, each of these availability zones has its own source of electricity.
21:25They are...okay. Within one region, they have independent data centers, and they call them these availability zones, right?
21:32So you will work with these availability zones if you want to build high-availability applications, right?
21:38You deploy everything in Ireland, but some of the machines in my cluster are in availability zone A; others are in B.
21:45And then you can kind of do network load balancing between these two availability zones.
21:54The AMIs that we provide to you can work in any region, any availability zone.
22:00So how to get started. I kind of explained this, but let's go through the workflow again.
22:04First, you...you make a commitment with Amazon. You create an Amazon account...
22:11...because to us, Amazon is kind of another delivery mechanism for ArcGIS Server.
22:16You want to run your ArcGIS Server license in Dell, fine. Go to Dell and buy a machine. IBM? Go to IBM and buy the machine.
22:23You want to run on Amazon, go to Amazon and buy a machine.
22:26So you go to Amazon, and you create your Amazon account ID. At that point, you can start servers.
22:31But you cannot start servers with ArcGIS Server. You need to call Esri and tell them, This is my Amazon ID; it's a 12-digit number.
22:39And if you are licensed, we will share the AMIs with you.
22:42So the next time you launch a server, you'll see an option to launch ArcGIS Server.
22:46And at that point, you are off and going; you will have ArcGIS Server.
22:50So you just authorize normally. You authorize ArcGIS Server so it works, and that's pretty much it.
22:57We'll see that in another demonstration right now.
23:12Okay. So the first thing is I'm going to go to this aws.amazon.com. This is the website.
23:21Here you can learn about pricing, all the services they have, and so on.
23:25Today, we are going to click on the AWS Management Console at the top, and because Firefox is remembering my Amazon...
23:34...I...I just logged in a few minutes ago. But basically, unless you have an Amazon account, you cannot get into this application. Okay?
23:41You will be prompted to type your region and password.
23:45This is not the typical Amazon account you use to buy books.
23:47This is an Amazon account from Amazon Web Services that has...is enabled to use EC2, Elastic Cloud Computing.
23:55It's fairly easy to create. I mean, not more difficult than...not harder than creating your account to buy books.
24:02And as you can see here, you can operate in different regions.
24:07Here are the different data centers. U.S. East, U.S. West, European Union, and Singapore. In this case, I'm going to work in U.S. East.
24:16And just to get things off the ground very quickly, I'm going to launch an instance.
24:22Now you remember, this EC2 instance concept we talked about, I'm going to launch one. Okay?
24:27And the first question that they ask is what type of software did you want to run on the server you are about to launch?
24:37These are public AMIs. So you can see Windows and Fedora virtual images in there.
24:45So let's say you want to start a server that runs Windows SQL Server 2008 and Windows Server 2008.
24:51Well, you come here, you say select. And that's the software that they will put into your machine.
24:56The ArcGIS Server AMIs won't show up here because our AMIs are not public; they are private...
25:03...and we only share them with ArcGIS Server licensed users. Okay?
25:07So you will go to this tab, the Community AMIs tab, and then there are many community AMIs out there.
25:14You filter by either Esri or ArcGIS, and then you will see two.
25:19Actually, you see here a bunch because these are kind of the daily builds that we work...it's kind of a...this account is kind of a mess.
25:25But in your account, you will only see two AMIs, the ArcGIS Server AMI and the Enterprise Geodatabase AMI.
25:33And the process to start them up is very similar.
25:35You basically pick the software; in this case, ArcGIS Server 10. You select that and then, well, that's the software.
25:43But now tell me, how many machines do you want to start? Do you want to start 1, or do you want to start 5, or do you want to start 10?
25:51In this case, at this point, it doesn't really make sense to start 10; it makes sense to start only 1.
25:58Because this is an AMI that has ArcGIS Server, but it's not authorized, it doesn't have your data, and so on.
26:03But we'll see later, when you create your own custom AMIs, that that option is pretty powerful.
26:08Anyways, let's go with one.
26:10We are working on U.S. East, but which availability zone? So in this case, I have no preference; it's just for a demo.
26:18But in some cases, if you want to build high availability, you may want to start one here and the other one there.
26:25And this is a good one. What is the instance type? What is the size of the virtual machine you want to create?
26:31Is it a dual core? Four core? Eight core? You...you pick, right? You pick there.
26:39Amazon has way many EC2 instances that you see here, but because our AMI works on 64-bit, this is kind of a subset of their choices.
26:48So for development, when you are just starting, you can just go with a large...large, which is actually the smallest.
26:54This is about $1 per hour, I think, in...in U.S.
27:02The next step is...shows some advanced instance options. And I just want to highlight one, which is this one, monitoring.
27:09This is an additional service that, when activated...
27:11And then I click Continue.
27:13...will allow you to ping your servers and find out things like what is the CPU usage at that point, disk I/O, and so on.
27:21And that's key information that you may want to use when you are doing out scaling of your application...
27:27...is to add more machines or fewer machines.
27:30It's a for-fee service; it's additional. In this case, I'm going to turn it off.
27:37They always give you choices to buy more, buy more, buy more.
27:43Okay. The next one is important. This step is important. This is a step where you define the firewall settings of your instance.
27:53And this is very interesting because when you start machines in the Amazon cloud, you have pretty much control over your machine.
27:59It's a virtual machine, but you can lock any port and open any port that you like...
28:04...and you can restrict access through those ports to your machine from pretty much any IP.
28:24[Unintelligible] Continue... Okay. So actually I...I...I went too quick.
28:31Before the firewall settings, this is the...the key pair so the idea is when you launch an instance from scratch...
28:38...from the AMIs we provide, we create a random password so nobody can get in your...in your server.
28:45And for you to decrypt this password, you need to create one of these key pairs.
28:50So...well, if you want to create one, just click here and then you download it. This...this just takes two clicks.
28:57But I'm going to use an existing one, and when you download the key pair, you will download this file.
29:02It's a PEM file, and it has this cryptic information in it. Okay? You will...we will use this later, as you will see.
29:12Anyways, you select that key pair and then this is the firewall settings.
29:18So in the firewall settings, let's create a new security group for you.
29:22You create security groups because normally the firewall settings are common to all your instances.
29:27So you create a security group and all these rules are preconfigured so you can reuse them in the future.
29:32So I'm going to say, okay, this is my security group for ArcGIS Server services, public-facing applications.
29:41Okay, so rules. First of all, I want HTTP access over port 80.
29:47And I want every machine in the world to be able to access this machine through HTTP 80.
29:54So I'm not going to restrict the IPs for access.
29:58And then I also want to open Remote Desktop Access, but I don't want anyone to remote desktop to my machine...
30:06...so here you can type the range of IPs that can actually remote desktop to your machine for administrative purposes.
30:13So you would add the rule, and then you would continue. And actually, we can just continue, right? Those are my rules.
30:20And then you say, Launch. At this point, something is happening in U.S. East, and a machine, a virtual machine is created with me...
30:31...is created for me with all these characteristics.
30:33So I can go to the instances page and see my instances. And actually, it went pretty quick.
30:41It goes from...oh, here it is. Pending. That machine is being built for me.
30:46This flag may turn red...I mean green, saying running, pretty quickly...
30:50...but normally it takes between 10, 15 minutes to really have a machine up and running. And the times vary.
30:58I have seen them start in 4 minutes, in 20, in 25 minutes. So that's kind of a...an unknown. Sometimes take longer than others.
31:06It also depends on the size of the machine that you use.
31:10Bigger machines start faster, not because they are big but...but because there are more available; people tend to get the smaller machines.
31:18Anyways, if I go on, right-click here, I go...I can look at the system log.
31:24And the system log is empty; this is telling me that the machine is not ready to go.
31:29So, as I said, this process may take more time or less time, so before the presentation, I actually launched an instance...
31:36...I gave it 15 minutes to start, and hopefully, it's up and running. Okay.
31:41So that's what you should be looking for. Ready to use. It says, Windows is ready to use.
31:45At this point, I can go into the machine and use it.
31:50Okay. So to get into this machine, the first thing I need is to get the Windows password, right?
31:55So I click here, and this dialog shows up.
31:59So now is when I come here, I take my key pair, I paste it here, and then I decrypt the password.
32:10That's the administrator password. And only I will know this password, so I just did copy it in my clipboard.
32:19And now I'm going to access to...the machine. You can see this option, Connect.
32:25Let me double-check that this is in fact the machine I want to use. Yes.
32:29And I say Connect. And there are two gateways into this machine.
32:33This machine is running right now with ArcGIS Server, but ArcGIS Server is not authorized yet.
32:37The gateway on the right, okay, this is the public DNS. This is the URL of my machine.
32:43If people want to access it with a Web browser, they will go there.
32:46And then the other gateway is this Remote Desktop file.
32:50So I click here, and this basically downloads a desktop file, Remote Desktop file, which I can use to connect to my machine in the Amazon cloud...
33:00...and then I paste my Windows password, and I open a Remote Desktop connection to it.
33:24Okay. Okay. So this is...this is pretty impressive. I mean, this is the machine we just launched, and it's running somewhere in Virginia...
33:35...and we are accessing it through Remote Desktop to control it.
33:39So now the machine is kind of starting and it shows up this home page.
33:44And you can see in the home page that it says that ArcGIS Server is not authorized.
33:50So now is where you need to authorize the software. And I'll do that very quickly.
33:54So first thing is, I need to get my authorization files, which I have here in my laptop.
34:01And I'm going to simply copy and then go to my remote desktop machine, and I'll paste on my desktop. Okay? Desktop.
34:15And at that point, I can go and say, okay, I have my authorization file, desktop and server, and this will authorize ArcGIS Server.
34:35That's it. Okay. In fact, if I now go back to my home page, you'll see that the message for the license went away...
34:44...because ArcGIS Server is authorized. So this means that ArcGIS Server is running.
34:49So just to kind of have something that you...you would see that it's running, what we did was we created in these AMIs two sample services.
35:00So if I go back to Firefox, let's go with the other gateway we talked about, the public DNS. Okay?
35:06I'm going to use that and I'm going to copy that into my Web browser, Firefox, and I'm going to navigate to the services directory.
35:21It's the first hit, and the first hit, it always takes a bit more of time. Did that right, right?
36:00Okay. Who knows. Anyways, that's my services directory, and it has two services up and running already, okay?
36:07It's just you normally will, when you take ownership of your machine, you normally get rid of these services, right...
36:14...and you put your own, but they are actually up and running.
36:18The other thing you may want to do is have a look at the...at the service. Okay, that's the service. That's the machine we just started.
36:30Okay? So how much time we needed to do this? Around 20 minutes to get [ArcGIS] Server up and running on the web.
36:36Because if now...you saw the plenary, right? You know you can go with an iPhone now and look all this up.
36:41Yes, in fact, you could...you just connect to that, and you connect to it.
36:44Because my firewall settings allow you to do that.
36:49Okay. So I just want to finalize the authorization because of course [ArcGIS] Desktop is installed, but it's not authorized.
36:58So I'm going to authorize it. I open the administrator, desktop, authorize, same process, ArcInfo.
37:15In customer service, always they ask me why I authorize [ArcGIS] Desktop twice a day. Okay. That's it.
37:25So now [ArcGIS] Desktop will authorize here. Before I leave, I will change the password.
37:32Actually, I'm not going to change the password, but this...that's one of the typical things that you would do.
37:37You would authorize ArcGIS Server and then you change the administrator password that...to something that you will...you will remember.
37:45Okay. So that showed the process of starting ArcGIS Server in the Amazon cloud. Is that something interesting? Yeah?
37:57How much time do we have? How much time do we have left?
38:03Okay. So now we are going to talk about some of the...well, this is not...not nice.
38:09Uploading data to the Amazon cloud is probably one of the most difficult things. That's why we have a few slides here.
38:16There are two types of storage in Amazon Web Services, EBS and S3.
38:22EBS is called Elastic Block Storage, and it's ideal for fast access from your EC2 instances.
38:27That's where you put your GIS data so it can be accessed from your GIS services and your databases.
38:33And then they have S3, which is Simple Storage Service, and that's storage for the Internet.
38:39It's used as a backup, so if you have GIS data in EBS, you can create a snapshot and put it into S3...
38:46...and that's very, very, very reliable. Okay? That's where you make your backups.
38:51Also, it's kind of a staging place where you can upload data to Amazon S3, and then once the data's in S3...
38:58...you can download it to your EBS drives, which are really the drives that are attached to your EC2 instance.
39:04And there are some people who are also using S3 to store their map tiles.
39:11Just the map tiles that you create with [ArcGIS] Server, you put them into S3 because S3 has some additional services...
39:16...like this CloudFront we talked about, that accelerate download of map tiles to clients.
39:23So let's talk about these EBS drives in your EC2 instance.
39:26When you launch the instance, you have two drives. You have the C drive and the D drive.
39:30The C drive only has 30 gigabytes, and you should not use it for storing your data.
39:36Two reasons. One, it's too small; it's only 30 gigabytes...
39:39...and second, it's a space that the operating system itself uses for temp, for virtual memory, for many things. So don't compromise that space.
39:49The other reason is that it's not a safe place because if you terminate your instance, that drive will go away; you will lose it.
39:57However, if you go to your D drive, which is called GIS Data, that's a safe place.
40:03It's also an EBS drive attached to the machine, but it's bigger. It has 100 gigabytes.
40:10And as I said, it's ideal for your GIS data. The machine will access the data very quickly.
40:15Think of it like a virtual or a portable...portable hard drive for these data centers.
40:21You can attach this EBS drive somehow to different machines. And it's easy to back up.
40:28You can literally take an EBS volume and say, Create a snapshot, and that will take that, store it in...in S3 so you can reuse it in the future.
40:36If you need additional storage attached to your EC2 instance, you can create EBS drives that are up to 1 terabyte...
40:43...and you can attach up to 12 drives to an EC2 instance on Windows.
40:51But by default, as I said, it ships with 100 gigabytes on D:\GIS data.
40:58Now that tells you where to put the data for your GIS applications. It doesn't tell you how you get the data into your machine.
41:06And we are going to discuss four different options.
41:09One you already saw. It works for very small files like this copy and paste or drag and drop with Remote Desktop.
41:17It's not a safe way to move sensitive data, but it's kind...it's...it's handy.
41:22I mean, when, you know, we have 30 megabytes or 50 megabytes, I just do it that way. It's...it works fairly well.
41:29The only problem is that you cannot interrupt this copying of files, like the data's in your clipboard...
41:34...and it's being moved to your machine, if you use Ctrl+C again, it will...the process will be interrupted.
41:40So you need to leave the machine alone.
41:43So we say around 100 megabytes. I mean, if you go beyond that, who knows? You may need to try a couple of times before you succeed.
41:51It's quite convenient.
41:52And you can see that if this option is not enabled, when you open the Remote Desktop connection...
41:57...if you go to the Advanced options, you can actually make sure that in fact the Clipboard option is enabled.
42:05The other one is FTP. FTP works with much larger files. And there are two strategies.
42:09One is to start your ArcGIS Server instance in the Amazon machine and have it access an FTP server that you may have somewhere else.
42:17That's pretty...you know, pretty standard. You already have an FTP site, and you just pull the data.
42:22Or if you want to do a backup copy or whatever from the Amazon cloud, you can push to the FTP site.
42:27The other...other strategy is to set up FTP server in your own Amazon machine, but I want to give you a warning.
42:35It's not really safe to enable FTP sites on production systems, so you should do that in an isolated machine.
42:42You start a new machine, a small one, you enable FTP, and you put all your data on that FTP site.
42:50And then ArcGIS Server is running on a separate EC2 instance. So if people want to attack this instance, it's fine; it's just a staging area.
42:57How do you get your data, once it's in this FTP instance, to your ArcGIS Server instance?
43:03Well, because the data is in an EBS drive, in your FTP instance you can detach it and then attach it to your ArcGIS Server instance.
43:12Or you could simply create a snapshot, S3 snapshot, and then create as many volumes, EBS volumes, as you like...
43:18...to attach them to the...to your other ArcGIS Server instances. Is this making sense? Yeah?
43:24So just that warning there; be careful with FTP servers, right, running on your production systems, and that's how you get around the issue.
43:32S3 is another strategy to get data into the cloud. So you can use different tools, like S3 Organizer is a Firefox plug-in. Right?
43:42And it allows you to pretty much drag and drop data from your local drive into this thing called S3...
43:47...which is this storage for the Internet.
43:50And this works well also, like FTP, for very large files. Like if you have hundreds of gigabytes, you could actually go this way.
44:01And, well, it always takes time; it's the Internet, right?
44:04There are some third-party business partners of Amazon that actually accelerate this upload of data to S3...
44:10...and they are, you know, quite efficient, but of course, this is kind of an additional service.
44:17So you get your data into S3 and then with that same S3 Organizer plug-in for Firefox, you can download from S3 into your EBS drive.
44:27Okay? So S3 is kind of a staging area for the data that you move to the cloud.
44:36Import/Export is an interesting service. Basically, you take your drive, you mail it Amazon...
44:43...to the data center where you have your EC2 instances, and then they plug it, and they...
44:47...they move that information into a S3 so you can download it. That's probably the fastest way to do it.
44:53If you are going to do a bulk data loading, kind of I have 1 terabyte of data - Import/Export.
44:59Send it, takes a couple of days to get there; one day if you rush, and then you have it in S3 pretty much there.
45:10And it works the other way around as well. You can tell them, Why don't you give me all this data I have in the Amazon cloud...
45:15...you send the drive to Amazon, and they put the data into your drive, and they send it back to you for recovery, whatever.
45:24Just a few tips. Organize your data for data transfer.
45:28Make sure that you are using relative paths to your map documents, to your geoprocessing services, and so on.
45:33Eliminate unnecessary data; that's obvious.
45:35Compress whenever possible. Compress for the purpose of transfer the less amount of information possible.
45:42Break down your compressed files into smaller chunks so if something goes wrong, you can try again but not start all over.
45:49Be aware of S3 bucket restrictions. This is well documented in Amazon.
45:54You cannot use just any name for your compressed files. You need to be careful with that.
45:59If ...if you are transferring terabytes, either you have a lot of time and patience, or you should go import and export.
46:05And then create a data transfer strategy. This is something where you will need to deal with moving data to the cloud.
46:12Not only with the first initial load but also incremental updates.
46:16So just think about how you are doing...going to do the incremental updates.
46:21Think about geodatabase replication, for example, as another mechanism to push data from geodatabase to geodatabase...
46:26...straight through geodata services.
46:34Yep. Let's talk about elastic deployments of ArcGIS Server. Let me introduce the concept of a custom AMI.
46:42You saw how I launched that AMI that Esri provides, it has ArcGIS Server.
46:46Now the great thing is that once you have your AMI running, let's say if I go back to my...
46:56If I go back here to my ArcGIS Server instance, I could now start ArcGIS Desktop and I could download data to this machine.
46:59I could create my own services. I could put my own applications in this machine in the Amazon cloud.
47:11And at some point, you have a server that is functional, meaning that it has your own applications, your own services, and so on.
47:19And that's the point where you want to save your work, and you do that by creating a custom AMI.
47:25So you move all the data here, and then from the management console, you can select your instance and say...
47:35...save or create an EBS AMI. This will save your AMI.
47:41You should do this with the instance stopped, so nobody's monkeying around with the instance.
47:46You stop it, then you create the instance.
47:48And your AMI will display here. Okay? These are AMIs I created for all the demos for the Users Conference.
47:55And the great thing is that now that you have your custom AMI, you can...you can come here and you say...
48:01...Launch the instance, and the same wizard that we used before shows up.
48:06And I can say, Why don't you create 10 instances of this project A AMI. Right?
48:18Once you have these custom AMIs, you can launch as many servers with your services as you like...
48:24...and then you can do things like horizontal scaling.
48:29You add 10, 15 servers running in parallel, all running the same services because all...they come from the same AMI...
48:36...and then you pull...you put this thing called Elastic Load Balancer on top.
48:41So when a request comes, gets to the Elastic Load Balancer, and the request gets directed to different EC2 instances.
48:48That's how you scale horizontally.
48:51Custom AMIs can be also used to scale vertically, meaning that once I have my AMI running...
48:57...I can start a small machine and have people hit that.
49:01But I can launch a bigger machine with the same AMI and then swap the EC2 instance and say, okay, get rid of this small one; put a big one.
49:11Right? It's the same application, but you make the machine bigger as opposed to adding more machines.
49:20Okay. So this slide is important 'cause it tells you how you scale up ArcGIS Server in the Amazon cloud.
49:27Traditionally, you add more SOC machines. In the Amazon cloud, you don't do that.
49:33You don't want DCOM stuff going across the data center.
49:37You want self-contained machines that have everything they need to run for the Web, the SOM, the SOC, and the data.
49:43Actually, the data might be living in a separate EC2 instance running ArcSDE with the geodatabase AMI.
49:49But your map services will point to that.
49:52And then once you have that, you have these clones, this cluster of peers, where all the machines have the same services...
49:59...and then you put Elastic Load Balancer on top.
50:02Yep. That's how you scale in the Amazon cloud. And it's fairly efficient if you work well with these custom AMIs.
50:11A machine goes down; don't even remote desktop to that machine.
50:14In fact, in production, when you set the security groups, you probably don't even want to enable Remote Desktop.
50:19You just open 80P...HTTP 80. Just don't remote desktop. Just kill the machine, start a new one, and put it under the ELB.
50:26That's how you fix things, right, in the Amazon cloud, because that AMI is like, yeah. Ten minutes, boom.
50:32Your machine is up and ready, and then you put it into the cluster.
50:37So this leads to the concept of Elastic Load Balancer.
50:40As you can imagine, it's just a load balancing that works in the elastic cloud.
50:44It supports sticky and nonsticky sessions, so you can imagine in this configuration, if you have an asynchronous geoprocessing service...
50:51...unless you're client is always going to the same machine, you will lose the job, right?
50:56So for that, you need to enable sticky sessions.
51:00Unfortunately, sticky sessions don't work with ArcMap yet because the stickiness works with browser cookies...
51:07...and ArcMap doesn't use browser cookies.
51:08So we'll try to fix that.
51:10But if you are using Web browsers to run asynchronous processing, it will work fine as long as you set sticky sessions.
51:17Let me give you a quick demonstration with...I think we're running out of time, right? Yes. No demonstration. No demo. Okay.
51:29CloudWatch is that service I talked about which allows you to monitor.
51:33Very important for this environment because you may want to add and remove instances from your ELB as the load goes up and down. Okay?
51:43And things to remember. I already talked about these issues with sticky sessions and...and so on.
51:51No demo. Maybe at the end if we have time.
51:55Anyways, before...before I...we jump into the real scenarios thing, we kind of followed a workflow through the presentation.
52:04This is what we call the Amazon EC2 workflow. First you launch the ArcGIS Server instance, then you take ownership of it...
52:11...you authorize it, you change the password, you add your services, you add your data.
52:16Then you do your functional testing on that AMI; you test how the application works.
52:21And if it works, you save that work as a new AMI.
52:24And then you launch that AMI in a small machine, in a big machine, and you do nonfunctional testing, performance testing.
52:32That will give you an idea of how many concurrent users a machine of type A or type B can hold.
52:38That's information you need when you need to scale up. That's what we mean by measure the throughput.
52:44And then you just put your machine in production; you have an estimate of the load, you know how much load a machine can hold.
52:52And then if things go wrong because you have a lot of users, you can enable CloudWatch and add more machines to that deployment.
53:01Or remove them. Almost as important as add is to be able to...to remove.
53:05So, many people actually...we are releasing this with ArcGIS 10 as you know...
53:10...but we have been working with ArcGIS Server on Amazon for quite a while.
53:14So I'm going to let Marwa talk about some of these experiences from real clients.
53:21Thank you, Ismael. [Unintelligible] the other part.
53:30So I hope everybody's familiar now with these concepts for Amazon.
53:35What I'm going to talk about next is a few examples of real projects that are hosted right now in the cloud environment.
53:44They're out there happening today. I always have this problem, so I'll just hold it.
53:52So what I'm going to talk about next is a number of projects and applications that are running in the Amazon cloud.
54:00They're hosted there today based on the AMIs that Ismael has been talking about...
54:05...using the same kind of services and concepts he's explained already.
54:11So I want to...I...I just have a few examples.
54:14We actually have a few more running today, but I wanted to use these because each of them has a unique story.
54:21This application, UDS Mapper, was developed by a partner of ours; their name is Blue Raster.
54:27They've built a very nice application, and they've set up the data behind it.
54:32And what they wanted from us to help them with was hosting the ArcGIS Server part.
54:38The application had a unique life span. It was intended to be running for...actively running for a period of 90 days.
54:47It was going to be public facing. They expected there would be heavy load during the first part of the hosting period and the end of it...
54:55...but not really very heavy during the middle of that period. And due to this variation, and being a public site...
55:05...it was very hard to predict how many people were really going to come and use it at a certain time.
55:10So from that perspective it wasn't really clear.
55:12Should we buy very large machines or very small machines, and how are we going to handle that?
55:17So the cloud seemed perfect due to the limited duration as well as due to nature of it being public facing...
55:24...and we weren't really sure how this is going to scale.
55:28So for this application, what we decided to do was host both the application and the tile data content on Esri.
55:39And then we started ArcGIS Server instances based on a custom AMI that we created for that specific project.
55:45So we just launched a couple of...an instance from the original ArcGIS Server AMI, set it up with the data...
55:53...as well as the service, and then created a custom AMI for that.
55:57And from there, we launched two instances, set them up with the Elastic Load Balancer, and that was it.
56:04The application was a Flex application that is using both the S3 tiles as well as the services that are being published by ArcGIS Server...
56:12...and it's running on the Internet. It's available for everybody to use.
56:17Now, because of the nature of being public facing, what we decided to do was to start with smaller-sized instances...
56:25...a couple of them, because we wanted it to be high availability, so if one fails, it kind of goes to the other one.
56:31So we started with two standard large, but what we did was we used the CloudWatch to kind of monitor the utilization of these instances...
56:39...and if we saw that the utilization was going beyond a certain extent, we were prepared to go and start more instances.
56:46So we decided to go with the horizontal scaling, and that kind of worked really well for that project.
56:55Another example is another project we worked on with USDA.
57:01And in this case, USDA was actually hosting a number of applications already themselves, so they felt pretty confident about how things ran.
57:09However, the problem they was after press releases, especially big ones like ones that go on the White House website...
57:17...or something that goes on CNN or other news channels as a big application, they found that for a very short duration of time...
57:25...they would get very high loads, and that caused a lot of issues and problems.
57:31Then after that, the loads go back into very low, normal ones, maybe not even a lot of usage.
57:38So this seemed to be like the perfect opportunity to leverage the cloud.
57:42What they wanted to do, rather than invest in a major infrastructure, to meet public demand at these peaks...
57:48...was to have a cloud setup that will go high when we need that load and then when we don't need it, we shut down all these extra machines.
57:57So what we did in this case, again, we started an instance from the AMI, set it up with the data...
58:02...and this time put the application as well on this instance, and then saved the custom AMI.
58:08And from the custom AMI, we launched two instances that were registered with the Elastic Load Balancer.
58:14The application is using ArcGIS Online for the tiles, the basemaps and between the application being hosted...
58:22...and the ArcGIS Services being hosted in the cloud and using the tiles from ArcGIS Online, they have the application running in the Internet.
58:30And whenever we expect a certain load, they notify us there's going to be a press release.
58:35So we keep an eye on it, and if we see that utilization exceeds a certain limit...
58:39...that's when we go and prepare ourselves to launch more instances based on these custom AMIs.
58:44So the beauty of it is that in a matter of 15 to 20 minutes, if there is a load, we have the custom AMI...
58:51...we can go and add more instances, two or three machines depending on how much we need.
58:57And when we don't need it, we just shut it down.
59:02Also, I thought this would be a good example. This is the public-facing PB [sic] application that's hosted by PB on their website.
59:10The story for this application, I'm sure everybody's pretty familiar with what happened for this pretty sad disaster.
59:18But what we did was we were on the phone with them around three p.m. one day...
59:22...and they said that they really needed to go and put an interactive application on the website that the public can get access to.
59:31And they wanted to make sure this information was publicly available at any...in an application where users can do some analysis themselves.
59:40They've already had this data on their website as PDF files, so what they wanted to do, rather than just keep publishing the PDF files...
59:45...they also wanted in addition to that an application to show the content.
59:53And it was, of course, pretty quick turnaround. They wanted to see as soon as possible a prototype application up and running.
1:00:02And what was really great was that within that same night... So we kind of finished the call around three-thirty.
1:00:10In a matter of a couple of hours or three hours later, we actually had that server up and running with the prototype application.
1:00:17And they were hitting on it and kind of thinking, oh, you know, we need to change a few things here and there...
1:00:24....and they started modifying a few things.
1:00:26And within less than a week, we had this up and running and it was available for public.
1:00:32There was a few challenges related to this application.
1:00:37One of them was the data was constantly being updated.
1:00:42What we kind of opted for to start things really quickly was to set up an FTP server on a staging machine...
1:00:49...where we go and upload the data updates to that server, and from there disseminate it to all the production machines.
1:00:57They also wanted to make sure this application is going to be public facing so we can scale it depending on demand.
1:01:04So we started pretty big instances, so this is where we went to start with vertical load matching.
1:01:14So we started pretty...pretty big instances, about a couple of them, and then added a few more...
1:01:21...and then we keep them monitored as well.
1:01:23When we see utilization is going to be needing more instances, we can add more, depending on the custom AMIs.
1:01:29In this case, the data is not part of the custom AMIs due to its nature of being updated all the time...
1:01:36...so the custom AMI was mainly targeting setting up ArcGIS Server with the number of services in addition to the application.
1:01:43Then after we launched these instances, we go and get the latest update and make sure they are up and running...
1:01:49...before we make them active behind the load balancer.
1:01:53And that's one of the nice uses of the load balancer is you can have your instances up and running...
1:01:57...but that doesn't mean they're accessible.
1:01:59You can detach them and attach them to the load balancer depending on when you think it is the right time to make them available for end users.
1:02:07So while we're doing the updates, we detach them from the load balancer, and then after we're done, we register them again.
1:02:15This was kind of done in a very big hurry.
1:02:17What we're moving to right now is we're starting to think about doing replication using Enterprise Geodatabase AMI...
1:02:24...so what we're going to do is we're going to replicate with the BP database...
1:02:29...get updates, a more real-time nature, to that staging server, and from there disseminate it to the production machines.
1:02:38So that's pretty much the number of examples I wanted to highlight quickly and next we want to do the QA.
1:02:46You want to do that, Ismael?
1:02:47Actually, just throw any questions that you may have. Yes?
1:02:53[Audience question] [Inaudible] support [inaudible] enterprise for SQL Server Express on the AMIs?
1:03:01Okay. So the question is if we're going to provide AMIs with SQL Server Enterprise and ArcSDE?
1:03:07No. The issue we had was...is not a technical issue; we could do that, but we...
1:03:12We are working with Microsoft and with other database providers so we can create AMIs for you.
1:03:17The issue is that we cannot right now redistribute anything but SQL Server Express and Postgres in our AMIs.
1:03:25However, you can...you can install ArcSDE yourself from the cloud and create your custom AMI.
1:03:33But one note there. When you run ArcGIS Server in the Amazon cloud with the AMIs we provide, you get full tech support...
1:03:39...meaning that you can call Tech Support, they will take your calls, and if you have any issues, we will be able to repro.
1:03:44Because we have the environment you are running on.
1:03:46But if you install ArcSDE on your own, then we don't have that environment set; we have not certified that AMI.
1:03:53So you can still call and we'll try to help you...
1:03:55...but if we cannot repro on one of the certified environments, we won't be able to address the issue.
1:04:02So we are working on that, but right now, only Postgres database with the AMIs. Questions? Yes?
1:04:12[Audience question] Yeah. That [unintelligible] process of starting up your instances [unintelligible] can be automated in any way?
1:04:20Okay. Can you automate the process of launching and dropping AMIs? Yes.
1:04:26As we said in the first slide, all of this is accessible through Web services.
1:04:30It just happened that we were using a Web application to do things manually, but there are toolkits for Java, for .NET.
1:04:38They have the SOAP web services, and you can automatically, you know, monitor your machines, attached drives...
1:04:45Scaling service. There's also the Auto Scaling Service, so this is a service that comes with no extra charge.
1:04:52They only charge you for starting the instances and the monitoring.
1:04:55And it will use the CloudWatch to monitor the utilization on the instance and automatically start more instances as needed depending on a criteria.
1:05:03Like for example, the processor is 70 percent utilized, so go and launch another instance based on this AMI.
1:05:10It also can...allows you to limit the number of instances you will need to start so you don't end up with a bill that's completely unexpected.
1:05:20[Audience question] Can you tell us something about the licensing on the Esri side?
1:05:24Okay. The licensing on the ArcGIS side. Okay. Good question.
1:05:28So people say, oh, this idea of elasticity is great because if I don't use the server, I don't have to pay, and that is true for hardware.
1:05:38You have to license ArcGIS Server. Okay? So if you start 10 servers with four cores, you need 10 licenses of ArcGIS Server.
1:05:48Now, even if you eliminate the machines, you have already paid for the software.
1:05:54So what we have done is, right now, you can...you can buy a subscription...you can license ArcGIS Server for one, three...
1:06:05Sorry. ...for 3, 6, and 12 months. So it's not an hourly rate; it's a monthly rate for now.
1:06:12I have to say that this ... the elasticity with ArcGIS Server on the Amazon cloud realistically only comes true...
1:06:21...when you have an enterprise license agreement where you have unlimited access to ArcGIS Server licenses.
1:06:27If you have just an ArcGIS Server license, you may want to run in Amazon because it's quick to set up...
1:06:32...because you don't need to fight...you don't need to fight your IT department, but not because of elasticity.
1:06:40Because you will have one license.
1:06:41So you can have ArcGIS Server running, and it's pretty affordable and it's pretty good, but the elasticity...
1:06:47...either ELA or this monthly termed licenses of ArcGIS Server. Good question.
1:06:54[Audience question] Well, if you throw money at it, you could scale up too.
1:06:56Oh, of course. Let me...let me honest...let me be honest with you. Forget about the ELA; just purchase the ArcGIS Server licenses.
1:07:03So if you start 50, just 50 licenses. I'm just kidding.
1:07:09[Audience question] So you talked about the benefits. Besides loading data, what are the downsides of using Amazon?
1:07:17The downsides of using Amazon. Well, I guess one you just highlighted. It's like, moving data to the cloud happens frequently over the Internet.
1:07:26So if you...this bulk...this bulk update of data can take time and effort.
1:07:33And then for incremental updates, if you don't...if you don't have big incremental updates, this geodatabase replication works well.
1:07:40The FTP thing that you described.
1:07:42I mean, you can get around, but if you have, you know 100 gigabytes of data changing every day is...is really a big challenge for you to do that.
1:07:51I'm not sure if you want to add.
1:07:53There's a few more. I mean, for example, folks that work with Image Server, that is, we're talking about terabytes and terabytes of data...
1:08:01...and management and updates, it is...it is kind of a challenging process.
1:08:05We're trying to figure out what is the best way to deal with it now so we can come up with a few best practices.
1:08:09[Inaudible audience question]
1:08:10Yeah, and kind of how to move the data around, how to manage the updates and things like that.
1:08:15Also worth mentioning, because of the nature of the cloud and it being a shared environment, we've seen sometimes the performance...
1:08:24...you can benchmark it to be a certain performance, and then when you run it on another day, you can see that vary a little bit.
1:08:32So these type of variations in the performance, the capacity of these resources has been something to keep in mind when you're planning.
1:08:42They do have different models between the on-demand, reserved, and the spot and the different regions...
1:08:48...so there are variations between the different regions in terms of their capacity, in terms of their performance, and things like that.
1:08:55So that kind of comes with some experience.
1:08:58And, you know, a lot of people share their experiences online. so you can always learn about that as well.
1:09:05So just to give you an example, we uploaded, right before the User Conference, 1 terabyte of imagery to Amazon.
1:09:12It took, I think, four days to get the 1 terabyte of data from...from my machine into the EBS drive.
1:09:19[Audience question] With a DSL connection?
1:09:21I don't know exactly what we...what we used.
1:09:23Yeah. It's one of the tools [unintelligible].
1:09:27Okay. We can give you...we should connect with the image service guys to just give you the details.
1:09:31Another example is the same machine, the same data, the same testing, 30 percent variation in performance.
1:09:38Why? It wouldn't...it wasn't the Internet, because we were testing from inside the Amazon cloud; it was the machine.
1:09:43The machine itself was not as fast as the day before. And then the day later, it was faster. Thirty percent variation.
1:09:50[Inaudible audience question]
1:09:51We know...Amazon told us that if you have an eight-core machine, and eight-core virtual image, you are going to own one physical machine.
1:09:59So the machine is yours, and the performance is going to be pretty consistent.
1:10:03If you purchase or you rent one of these smaller machines, you are sharing the machine with others and...
1:10:09Well, it depends on how busy the other guys are and also...
1:10:13Although these virtual machines are all sharing kind of the same port to communicate, and that's where the bottleneck happens.
1:10:19So when you are renting small machines, you may observe these variations in...in performance.
1:10:24And between regions depending on the [inaudible].
1:10:26Of course, between regions, yeah. Between...yeah, real...yeah.
1:10:29So always try to do performance testing and measure to what extent...what each EC2 instance can give you. Right.
1:10:39[Inaudible audience question]
1:10:46Well, actually, no. We...we published a white paper not long ago; it's called Building High-Capacity Map Services.
1:10:54It's in the white paper section of esri.com.
1:10:57And it explains how to use network...network load balancers with ArcGIS Server following this peer architecture...
1:11:03...not using the Web SOM/SOC kind of traditional thing.
1:11:06And it has many advantages to it. So this is for the Amazon cloud, but it also works perfectly on premise.
1:11:12In fact, we have people using VMware on premise with virtualization in VMware.
1:11:17Kind of the same technique. You know, I create my image, I start it, put ELB on top, and I just forget about SOMs and things like that.
1:11:26[Inaudible audience question]
1:11:27Well, read the white paper. There are good things and bad things about them.
1:11:31But it's definitely a good pattern for some use cases.
1:11:35[Inaudible audience question]
1:11:44Okay. So extensions. When you get the AMI, we have installed pretty much everything in there...
1:11:49...and then when you use your license file, you unlock one feature or the other. Right?
1:11:54So all the extensions work, including Image [Server] extension, which as you know is part of ArcGIS Server in 10.
1:12:01There are some...exceptions. JTX is not installed and Geoportal toolkit is not installed.
1:12:10[Inaudible audience question]
1:12:18Yeah, we use Amazon ourselves. If you look at the effort...relief effort kind of websites for the earthquake in Chile and the Haiti thing...
1:12:28...everything was running on Amazon for some of the reasons we explained.
1:12:32And also, ArcGIS Online itself, our service is running Amazon machines using this technology that you saw. Yes?
1:12:44[Audience question] Amazon charges by the hour for these machines to be up. How do you address the situation...
1:12:49...where you have a service that needs to be available but doesn't need to be available all the time?
1:12:54[Unintelligible] pull licenses so that they can share [unintelligible]?
1:13:01Do you understand the question? We don't understand the question. Sorry.
1:13:09I think...Is it a licensing question regarding if you want to run multiple services on the same server?
1:13:16[Audience question] No, but... You don't want to run the server 24 hours a day for a user to be able to...
1:13:22...you know, randomly throughout the day be able to [unintelligible] service.
1:13:25But could you pool other users and start up one machine and use it to run your own services on [inaudible]?
1:13:37Once you start the machine, you have a machine running with ArcGIS Server for just four hours a day.
1:13:42You will pay for four hours of hardware and the storage that you need and all of that.
1:13:46From a software point of view, you have to license an ArcGIS Server license for four cores...
1:13:50...regardless whether you use it 24/7 or just four hours a day. That's how it stands right now, okay?
1:13:58The granularity is at this...you know, we...we mentioned this 1-month or 3-month, 6-month, 12-month subscriptions.
1:14:06That's the level of granularity we can give you right now.
1:14:09Hourly rate for ArcGIS Server, right now we...we are not there.
1:14:16If we didn't get your question right, you can just come over at the end. Any other questions? Yes.
1:14:24[Inaudible audience question]
1:14:32Yeah. I mean, not only 8, but 24. If you have a machine with 24 cores ArcGIS Server, it's going to be beauty in there.
1:14:40[Inaudible audience question]
1:14:47Yes. The question is about licensing.
1:14:49As we...we mentioned before, if you do outdoor scaling, you need to license every ArcGIS Server that you are running.
1:14:56So if you, at some point, start 10 servers with four cores each, you need 10 licenses of ArcGIS Server.
1:15:02That's where we mentioned the idea of the enterprise license agreement. That's the way to go for elastic.
1:15:15Ok. Thank you very much. If you have additional questions, just come over. Thank you very much for your time.
Using ArcGIS for Server in the Amazon Cloud
Join this session to learn how to leverage the Amazon cloud with ArcGIS for Server. You will learn how to setup ArcGIS for Server in the Amazon Elastic Computing Cloud, typical deployment patterns and workflows. We will also discuss how to estimate costs for your ArcGIS for Server deployment in Amazon and present several projects running ArcGIS for Server in the Amazon cloud.
- Recorded: Jul 1st, 2010
- Runtime: 1:15:20
- Views: 104089
- Published: Aug 25th, 2010
- Night Mode (Off)Automatically dim the web site while the video is playing. A few seconds after you start watching the video and stop moving your mouse, your screen will dim. You can auto save this option if you login.
- HTML5 Video (Off) Play videos using HTML5 Video instead of flash. A modern web browser is required to view videos using HTML5.