00:01I'm Sepehr. I'm going to talk about the geoportal project that we've been working on for the past couple of years.

00:08To give you a little bit of background, our organization mostly does services that are used by all university libraries across our province.

00:19So, for example, one of the oldest projects was electronic journals...

00:23...and they centralized all the electronic journals in one place for all the university libraries to access.

00:30The next one was books and came along this project for centralizing geospatial data for all universities again...

00:40...and to provide a platform for data delivery and visualization.

00:48I was taking a few courses when I was undergrad in geography, and I remember what we did.

00:54What we had to do was go to libraries, bring a USB [unintelligible] and get the data and then bring it back to our lab...

01:02...put it on ArcMap Desktop, and in some instances was realized that this was not even the data that we wanted.

01:10So this was a pretty good project for me personally to work on, and I knew how useful it is to all the students.

01:20So we started development, I would say, the major software development less than two years ago...

01:26...and we officially launched this on beginning of March.

01:31So talking a little bit about the architecture, there are several components involved in the portal.

01:39There's a metadata and search component.

01:42There's authentication, entitlement, user accounts, and, obviously, ArcGIS Server and the web application itself...

01:50...and I'll talk about each one of them in a little bit of detail...

01:54...and during the demo I'll try to show you how each one of them plays out.

02:02For metadata and search, we use MarkLogic Server.

02:07If you're not familiar with that, it's a repository for unstructured data, but it's mostly XML...

02:15...and it's really powerful and fast, and our organization has invested a lot in it...

02:21...so a lot of developers and a lot of generally licenses so that was a de facto option for us to use...

02:32...but you could, for example, replace it with [unintelligible] if you like to.

02:37This MarkLogic Server provides editing, storage, and takes [unintelligible] search and also geospatial search...

02:46...but it's a custom built module but it uses the functionality of MarkLogic spatial.

02:53So what we do is, we take the original metadata that comes from data provider, and we convert it to a format called...

03:00...North American Profile.

03:02This being a library, they're really serious about metadata and the standards.

03:07And then we enhance that metadata by adding our own custom fields to that...

03:12...which are used later to connect everything between our services.

03:17So we've kind of, in the beginning, we decided on a convention to use for naming everything so our, for example...

03:25...our data sources in SDE, our service URL as well, we'll publish our metadata URIs and all of that...

03:34...and these are what we put in custom fields along with, you know, what the service type is, if it's raster or vector or tiled, all of that...

03:44...what the coverage area is, for example.

03:49And authentication, I have to say that mostly our users access our services from campus...

03:58...and authentication happens when users are coming from outside campus, so, for example, from home.

04:07For this module, we used a project called Shibboleth, which is an open-source project for federated identity-based authentication.

04:17So essentially users authenticate with their own university, and we verify their identity with the university.

04:25They don't have to do anything...we don't have to do anything on our own and pretty much.

04:32So for the entitlements, last year when I was in DevSummit, I spent quite a lot of time with a few Esri staff in development...

04:44...to figure out a model that would address our data access level and how we interact with our users and, eventually...

04:53...we couldn't come up with a good solution that would utilize the security model that ArcGIS Server provides.

05:01So what we did was we built our own system, and the way it works is, basically...

05:09...we put all the data or services in collection names, and then we have university collections...

05:17...and we assign, for each university, we assign what data collection they have access to...

05:23...and then based on where the users are coming from, which we decide either based on their IP or that authentication process...

05:33...that I talked about before, we grant or reject access to certain datasets.

05:40And user accounts are mostly just for keeping track of users' activities, and you'll see that on the portal...

05:49...when I show you what their setting are, for example, they can save searches, they can save maps, they can save what they're doing...

05:59...and they can annotate services if they want to, and these are stored in cached DB.

06:10For web application, on the front end, we use ArcGIS JavaScript API, which comes with Dojo...

06:19...and we use a little bit of background GIS mostly for those user accounts that I mentioned before.

06:27In the back end, we have a Java Spring web app that takes care of mostly database operations.

06:36It does the proxying, logging, and communicating with other components.

06:42And proxy is actually a big part of the whole web application...

06:47...because we pretty much handle everything through proxy to secure all the services.

06:53So our ArcGIS Server instances are close to everything, well, between themselves, they're open and they can talk to each other...

07:01...but other than that, they are only open to our proxy, and nothing else can go through.

07:06So proxy manages all the security issues, and it takes care of logging and all of that.

07:15In ArcGIS Server, we have three parallel servers.

07:20We have two full vector services, and these two are identical, so each service we publish, we publish to both of them...

07:30...and we have one for raster-based services, and we have our SC in the active-passive SQL Server cluster...

07:41...and we approximately have 70 vector and 20 raster services, and now that we're done with development on the portal...

07:51...we're going to be publishing a lot of datasets, and we're almost doubling that soon.

07:57In our test server, we actually have 130 vector, as far as I remember.

08:02And one thing that I have to mention, though, is that, in the beginning we decided to publish one data layer per service.

08:11So our services are not serving more than one data layer.

08:15They don't have multiple layers in them, and this was a decision we made in the beginning for a variety of reasons...

08:24...which is out of the extent of this talk.

08:27But, we also cache the vector services.

08:32So in the beginning, we published them as dynamic, and then we started caching process on the side...

08:40...and when the caching is done, we switched them to the tile services...

08:45...and we store all the caches on each one of the servers to avoid any kind of network overhead and latency.

08:56We have two major geoprocessing services that we use.

09:00One of them is print, and you might have seen in the presentations that there's a new print service coming in 10.1...

09:10...and we have to wait to see that, but at the time, printing...high-quality printing was impossible.

09:18So we developed our own print service, which the way it works is, it pretty much gathers a lot of information.

09:27When users print, it gathers all the information about the state of the app, so what layers they have on the map...

09:36...what is the extent, what is basemap, and it sends it to the print service.

09:41The service goes and finds, excuse me, corresponding layer files.

09:47So for each mapping service, we also create a layer file so that we can preserve the symbology for later...

09:55...and we access that to, you know, print whatever the users are actually seeing.

10:01Then, we use ArcPy mapping module to create a proper map document, you know, with the correct orientation...

10:10...portrait, landscape, and size, and then we add the data, all the map components, you know, like scale, data source...

10:22...title, all of that, legend, and then format the elements properly, and then we save that and return the URL back.

10:32The other geoprocessing service with it was for download, which is a customized version of the clip and ship...

10:40...if you're familiar with that from Esri.

10:46But the clip and ship didn't work...it wasn't flexible enough, I would say, for our needs, so we changed it a little...

10:54...but our download handles both vector and raster at the same time...

11:01...although the raster download is done through the REST API pretty much, but Python does that through the REST API.

11:09There's no dependency on map documents, which is the case for clip and ship.

11:13So in the beginning when we were using clip and ship, we had to maintain one map document with all the service data...

11:24...that we published in it to be able to access those data, but now, the download just goes directly to SDE and gets the data from there...

11:34...and we also include additional files in download results so the layer files that I mentioned before and our custom metadata.

11:47I don't know if you can see this, but this is kind of an overview of how the architecture is.

11:53There's a load balancer in the beginning which redirects user to one of our services and one of our servers in the beginning...

12:01...and after that, the only, for example, if they end up on one, they only talk to the first ArcGIS Server at that point...

12:10...and, but both of them talk to the only image server that we have...

12:14...and SC in the back end, and some file server that hosts all the required files that all these services need to access...

12:25...and we're going to make a few changes to this.

12:29One of them is the AV, one additional image server, so that we'll have the parallel and redundant system for image services as well...

12:40...and make our SQL Server cluster active access, so there's no one point of failure in the system...

12:48...and this is how, what I mentioned about all the components in the web application...

12:55...you can see the login talks to authentication, search from metadata and MarkLogic...

13:02...and everything goes through proxy to ArcGIS Server, logs are stored in SQL...

13:08...and also the web app and search both use entitlement module.

13:16So I'm going to switch here to show you a little bit of demo here.

13:32Is that good enough or...I think the resolution is a little bit...

13:39[Inaudible audience comment]

13:41Yeah.

13:43[Inaudible audience comment]

13:49Can't see where...

14:00Just going to try to [unintelligible]. How's that? Okay, we'll try to just make use of this.

14:12So this is interface. So, you can't see some of the functions on the left side, but, okay, let me just try this.

14:21So you come here and search and you see that, first of all, if you hover over the results, you can see the data coverage...

14:31...and you notice that this Add button is disabled, which is because accessing the portal from outside campus...

14:38...and they can't verify where I'm coming from, and these are open datasets.

14:42That's why these are...I'm able to add these to the map.

14:49So I'm going to log in using my university accounts, and you can see this is University of Toronto logging in...

15:03...and sending me back to the portal again.

15:10Now, if I do the same search, I can add this layer now.

15:15So let me just quickly add a couple of layers, agreeing to license, can add a soil layer.

15:27This is pretty, beautiful, and one image layer.

15:42So I'll just close this for a second.

15:51Here are the data. This is a soil layer, image layer, can see that all year...

15:58...and these are all regular Dojo and ArcGIS JavaScript API functionalities.

16:05So, you know, capacity, moving up and down, zooming, things like that.

16:11We have a legend here coming automatically, and we have annotations.

16:18If users like to, they can add things to the map.

16:24There's history, which preserves what I'm doing.

16:27At any point, I can share that as a link with, for example, my colleagues if I want to.

16:37And in my account, I have things like extent.

16:41These are all the stuff that I can search.

16:47Selections still [unintelligible]...oops. Fortunately I can't see...okay...here.

17:13So this tape that I'm in, I can get a link for that, and if I want to I can share that, and this will load the exact same thing...

17:31...and with the same layers and extent and all of that unloads it.

17:36And let me show you the print.

17:41So I just need to provide a few things if I want to, file name, output format, and then I say Export...

17:54...and this is going to take a while, so I'm just going to switch to this tab and show you some other stuff while that's being done.

18:05Another thing is accessing the data table...and I'm not sure how much of that is visible there...

18:22...but this is all coming from the service.

18:28You can page through the data table.

18:30You can export that if you want to as Excel file.

18:34You can construct your own query if you want to, and one more thing I'm going to show you...ah, this resolution...is download.

18:58So for download, you have the option of downloading entire datasets.

19:02So we create three packages of files for our vector data that users can download or you can download an area of interest.

19:11You can select from predefined areas, which we provide for like provinces, census tracts, and all of that...

19:19...or you can draw your own region.

19:24Just going to remove this soil just because that will take some time for completing, and I'm going to zoom in here...

19:35...and I'm going to draw my own extent.

19:38I'm going to cover a little bit of the raster along with the vector data.

19:44You can choose vector format what you want.

19:49There are lots of formats to choose from.

19:51You can choose your own projection if you like to.

19:55I'm going to go with the default and say Download, and at the bottom you can see that this is working.

20:03The vector download is already done.

20:05I choose [unintelligible[ a zip.

20:07The raster download is still working.

20:09You can see the ties that are in this region.

20:16This printer is still working.

20:18The soil layer is pretty heavy.

20:24And here, all the ties for the raster are back, and users can download that.

20:31I downloaded...I did the print before, so I can show you how it looks like.

20:47So this is the result of that print when it completes.

20:58Okay. At the end, I just want to thank my team.

21:01There's a lot of people working on this project, and it was impossible without this great of a team.

21:06You can check our human stacks if you like to know more about them, and thank you.

21:14Thanks a lot for being here, and you can contact me if you have any questions with my e-mail or send me a message on Twitter. Thanks.