00:01Good morning, everyone. My name is Marten Hogeweg.
00:03I thought I'd try starting early so we can actually release early, for once.
00:09I'm product manager for the Esri Geoportal Server…
00:12…and I'll be talking about this open source product from Esri for the next sort of hour or so.
00:19And there's some time for questions after that, so if you have a question in between, please interrupt me. I don't mind that.
00:28You might actually ask yourself, What is this about Esri and open source?
00:34I think if you were at the Plenary Session on Tuesday, you heard people talk about a number of things…
00:40…so I think it's time to actually get used to the fact that Esri and open source are not two opposites…
00:45…are not conflicting terms or contradictions anymore.
00:51I'm going to talk about the Geoportal Server. My team also develops the ArcGIS Desktop Editor for OpenStreetMap…
00:57…which is an open source extension to Desktop that allows you to contribute to OpenStreetMap.
01:03If you followed the Flex Viewer developments over the last couple of months…
01:06…you've noticed that it has been released under Apache 2.0 as open source.
01:11The Silverlight SDK has been available as open source for quite a while, and last Tuesday…
01:17…a number of the Runtime SDKs and applications that were built on that were mentioned and also to become open source.
01:26However, if you've used ArcScripts up to 14 years ago or so, you've actually been able to get open source software from Esri.
01:36The license text that was on the previous slide actually came from one of these scripts that people had shared…
01:41…an Avenue script or an AML script or such.
01:45And so there's been a community or practice of sharing code, maybe not under a formal open source license…
01:51…but a practice of sharing code, helping each other.
01:55Think about the developer samples that you can find for all the web APIs online that you can use.
02:00Think about the template applications that you can download from ArcGIS Online…
02:04…and then start building your own templates based on those.
02:07And so there's this practice.
02:10With the Esri Geoportal Server, we've started to make this more of a formal step…
02:15…and actually released a product under an Apache license.
02:20So what is this Geoportal Server? The Geoportal Server is really basically a metadata catalog tool and discovery tool…
02:28…that allows you to describe, find, and access resources, particularly in heterogeneous environments.
02:36And with that, I mean environments where Esri technology is not the only technology being used.
02:42Users may serve data and services using the OGC specifications or other specifications.
02:50They want to describe those resources using standardized metadata…
02:55…and the consumers in those environments are also using various technologies other than just Esri technology.
03:05A step to make the Geoportal Server open source happened in October 2010.
03:09It was actually not a very sort of big step.
03:12We started off nine years ago, as a result of Geospatial One-Stop, which was the Bush administration's website…
03:20…for access to geospatial data in the US from federal, state, and local government and academia and so on.
03:29And for that we built a technology using ArcIMS metadata server actually, and once we were done with that…
03:36…we had our distributors and other users asking, We want that same website now for our organization.
03:41So we turned that project - I work in Professional Services within Esri.
03:45We turned that project into a product called the GIS Portal Toolkit.
03:49And actually, from day one, we allowed or we gave users access to the source code of the GIS Portal Toolkit if they asked for it.
03:57We didn't provide it just for everyone.
04:02If you don't use it, don't need it, then just stick to the compiled applications and we will help you if you want to.
04:10Right? So it was very much sort of a project solution, if you will, a services solution that we implemented.
04:16So over time, as we sort of productized it more and moved away from ArcIMS to the technology that we're using now…
04:25…it became sort of more and more sort of independent of ArcGIS technology…
04:29…to a point where now we're not relying on anything ArcGIS if you don't want to.
04:34Right? So if you want to use a third-party service as the basemap, you can use WMS, or you can use OpenStreetMap as a layer.
04:42If you want to - you're not bound or using any SDE licensing or anything like that.
04:48If you want to use MySQL as your database, right, that's fine.
04:53So that's sort of an evolution, and in 2010, as I said, we decided to actually just release it under Apache 2.0…
05:01…as a developer- and user-friendly open source license with very limited restrictions.
05:08Just say thank you or just say reference that you used the geoportal, and that's about it.
05:12You can create a commercial product out of it if you want to without having to reshare what you add and do with it.
05:20Being within Professional Services, we do provide support for it, and as you can see, there's almost 20,000 downloads.
05:27When I created this slide, or updated this slide, last night…
05:30…I thought about maybe I should download it a couple hundred times or stuff, but that's cheating, right?
05:36And also what you can see is that we have people downloading the geoportal from 150 countries, right?
05:41So it's international being used.
05:45It's only 30 percent, or actually less than 30 percent is actually within the US; 27 percent or so.
05:52That sort of illustrates that it's international focused.
05:55And the Geoportal Server, for those who are from Europe, and if you are sort of familiar with INSPIRE…
06:01…Geoportal Server is one of the components in ArcGIS for INSPIRE…
06:04…which is a solution that my team builds for the European market.
06:08So what can you really do with the Geoportal Server?
06:10Well, there's a couple of use cases that I typically go through, and I'll go through these sort of, hopefully, quickly enough…
06:15…and then I'll actually do a little bit of a demo because showing is much more interesting than telling.
06:21So we do support searching based on simple keywords, but we do spatial searching, we do spatial ranking.
06:29The index of our catalog is highly configurable. You can add your own fields to the index, and then you can search on those fields.
06:36And that brings up an interesting concept of what is called faceted search, and I'll show an example of that.
06:43If you want to, you can sort of include a thesaurus, which means that if you search for the word river in your text search…
06:52…it may also find stream and lake and water body, right?
06:55But river - those items that have river somewhere in the description rank a little higher than the ones that have water body…
07:03…because they're more closer to the word that you typed in.
07:06That thesaurus is part of the Geoportal Server.
07:09You can tailor it; you can configure it, and so that's one thing to sort of get more results, get more sort of chances of success there.
07:18Another thing that you can do, we're using the Lucene indexing technology.
07:23Lucene, if you know Lucene, works with analyzers and parsers, and you can use the one that we configure out of the box…
07:32…but you can also use ones that understand stemming, right?
07:36If you're from some - I'm from the Netherlands, right?
07:39We like to concatenate words, so we have long words, 20, 30 characters, if you want to.
07:46And I type only in a portion of that word, and with some of the advanced analyzers and stemming capabilities of Lucene…
07:54…you can actually find those things even though you only typed in a partial word…
07:59…or a word that is almost the same as one of those partial words.
08:03Right? So there's a lot of configuration that you can do without actually getting into too much coding.
08:10Sometimes, searching is not the right solution, right? So if I go to Lowe's, which is one of the department stores here…
08:18…and I have one of these electric lawn mowers and had a little wire on it, if I want to sort of figure out which part is that again…
08:25…right, I may go to the website, then I go to the garden section, then I go to the power tools…
08:30…and then I go to the lawn mowers and then the electric lawn mowers and then I go to the parts and pieces, right?
08:35I don't just type in lawn mower wire, because I have no idea what part number or name that they use for that.
08:43So sometimes this sort of navigation through a hierarchy is actually better.
08:48So you can configure those kinds of hierarchies and navigation structures in the Geoportal Server as well.
08:54And they can actually be based on sort of some elements of the metadata…
08:59…but they can also include relationships between items, right?
09:03So maybe I'll show some of that as well if I have time for that.
09:07So, for example, if you have a server with 50 services on it, then if you find one of those services, through those relationships…
09:18…through the fact that the other 49 came from the same server, you can actually find those other 49 as well.
09:23So you can navigate not just based on sort of a formal description of something, but you can navigate through relationships.
09:29And those relationships can actually be determined by Geoportal at run time, so to say.
09:33They don't have to be defined prior to navigating there.
09:39Obviously, you want to sort of see what you get, right, so I find a service; is this the service really that I want to use?
09:47Similar to YouTube or Flickr, you don't just read the description.
09:52You actually play the video and after half a minute, you say, oh, no, that's nothing. Similar with maps.
09:57We have a little preview function. We have a sort of richer viewer that's actually built using the Flex Viewer.
10:03There's also one for Silverlight, and we have some widgets that work in those viewers…
10:07…search widgets that allow you to start up a viewer and then from the viewer, search for some content…
10:13…and then if it's a service, add that service to the map and build some map using multiple services.
10:19And if you look through some of the description already approaches that we have taken…
10:23…you can actually add your own viewer if you wanted to.
10:27As long as there is a little handshake between the geoportal and your viewer, your viewer can actually be anything you want to.
10:35So as we're moving to a services sort of infrastructure, right?
10:39Everyone is sort of starting to build services, and with the ArcGIS Online subscription…
10:42…it becomes very easy to host services somewhere.
10:44You don't need your own server anymore; you just put it in the cloud somewhere.
10:48But there are still very many use cases, situations where people just want the data.
10:52Right? "Just give me the data; I want to do some analysis locally and combine it with some other thing…
10:58…or integrate it in my organization's database."
11:01So we include a clip, zip, and ship function that is based on an ArcGIS Server geoprocessing service.
11:07It's an optional configuration that you do, right? So we don't depend on it, we don't rely on it.
11:11If you have Server, you want to expose your data for clip, zip, and ship, then you can do that.
11:17And it allows sort of you to configure the layers that people can download.
11:20Users can then, in a very simple client, sort of draw an area; select an output format, projection if you wanted to…
11:26…and then sort of send me an e-mail when the data is done.
11:30This is a normal geoprocessing service written in Python, ModelBuilder, so you can also change that if you wanted to.
11:36Right? Include some equal [unintelligible] step in the process or something like that.
11:42People may want to sort of find resources, but it actually means that your catalog needs to be populated, right?
11:49So if no one populates something in eBay, then I can go to eBay to find something but there's nothing there, right?
11:53So what we tried to do is make it very easy for those who actually share their content to make that process of sharing very easy.
12:03So we have some capabilities of harvesting, as we call it, that can automatically, on a regular basis…
12:09…maybe daily or weekly depending on how often your information changes, harvest your services…
12:14…or harvest your catalogs or harvest your folder of XML files.
12:18I mean, there's various different formats and mechanisms for that kind of process.
12:25Your descriptions could be following one of these standards for metadata.
12:31You can actually create your own standard for metadata, right?
12:34So we've seen a couple of situations already where people say, well, we're in Europe and we use INSPIRE…
12:40…but our country has its own profile of INSPIRE metadata.
12:43Right? So we have extra fields or extra domain values in the - or we have…
12:49…we want this particular field always to be populated when someone submits something.
12:53So you can configure all of that, and so that means that that process of registering something becomes very easy.
13:04And I mentioned Server, right? So one of the things we can do is if you have an ArcGIS Server instance…
13:09…you basically registered the ArcGIS Server instance; not an individual service but the instance…
13:14…and then the Geoportal will essentially crawl that server and fetch all the end points, right?
13:19The REST end points and the WMS and the other kinds of OGC end points.
13:23If you have an individual WMS service, you register the individual WMS service…
13:28…and we will sort of go and visit that service however often you configure that to do.
13:33And what is interesting about those WMS or OGC services themselves is that they actually have some metadata in them.
13:40Right? There is a metadata element, part of the WMS specification, that allows you to…
13:46…have a fairly complete description of that service - who is it, what can you do with it, are there fees associated with it, et cetera.
13:52And through a mechanism of understanding those different specifications, we can index those…
13:59…and we can sort of harmonize the view of the different metadata, right?
14:03So a title in Dublin core, a title in FGDC, a title in INSPIRE, a title in WMS - they're all in different locations, right?
14:10Different XPaths, so to say. And we sort of abstract that into the title, and that's what we call a meaning.
14:17So we assign a meaning to a field or an element in XML, and you can add more meanings, right?
14:22So that's what I meant when I talked about adding your own fields to the index and building things like faceted search with that.
14:31Another thing that we have seen over these number of years that we have used and supported the Geospatial One-Stop portal…
14:39…for example, is that people go to a site; they do a search…
14:44…"I want geology data in this particular area," and they don't find anything, right? Darn.
14:49So they go away. They try to go somewhere else and they won't come back because they say your site doesn't have it.
14:55Well, what we have done is we've implemented some very simple RSS capabilities that, if you do a search…
15:01…every search you do, you can subscribe to through an RSS feed…
15:05…whether it includes a spatial extent or just some keywords or topics, and so on.
15:11And that means that you can get notified once the site gets new content that matches your very specific criteria.
15:18You get notified of that, and you get a little alert in your - maybe in Outlook, right?
15:23You can register RSS feeds in Outlook or so, if that's your e-mail program…
15:26…and then sort of go back to the catalog and see what's new.
15:35And then last, but not least, as a thing that is very frustrating for very many people is that I find this service…
15:41…it looks promising based on the description, I click Preview, and it doesn't work.
15:45Right, so to address that we have sort of connecting to a third-party service, a service that FGDC…
15:52…Federal Geographic Data Committee here in the US, is building, and that…
15:56…and the purpose of that service is to check the availability and performance and conformance of your services.
16:04And at the moment they're focusing on OGC service, WMS - actually, ArcIMS is included also, a few other things…
16:10…but they're extending that capability, as we speak, to also include the ArcGIS Server or ArcGIS REST-based services…
16:17…which, by the way, is on its way to become an OGC spec also, right? So that's also good.
16:23So the idea there is I can get a little report, scorecard if you will, is the service up and running?
16:28And if the service is not up and running, then no, maybe I should go look for some other services…
16:35…that might have the same kind of information.
16:39Alright, so that's sort of an introduction to what is Geoportal Server, what is sort of the role that we play…
16:46…and the context that we play it in. Let me do a little sort of bit of a demo.
16:54Let me see if I can view, little more view of that.
17:05Alright. So we're looking at the Geoportal sort of home page here.
17:08This is my local geoportal; it's slightly customized, I add a nice picture to it, and I added a few other things to it here.
17:16So what's interesting about our index is that it's not just available for searching, but I can do other things with it as well.
17:24So, for example, I can see how many types of things of different kinds I have, right…
17:29…so applications or downloadable data or live map services, and we have some information.
17:34We have basically some statistics about the index that are available through a little REST interface as well.
17:39So I can do little graphs like this, sort of what are my different kinds of content or what's new, right?
17:48So like I said, you can subscribe to a news feed.
17:50Well, that news feed is not just available to you in an RSS reader, but you can also get that in a simple sort of web page format…
17:58…that you can then style and include in a little web page like this here.
18:05So I did a search for some interesting services.
18:08This is one of sort of services that I've been starting to rely on when I do my demos.
18:14It's actually a service from the State of Idaho, and as you can see, I get a few sort of items.
18:21For every search that I do, I get some items, and let me just zoom to that.
18:28Alright, this particular one is over there. That's how every item that we have has some sort of spatial extent, right?
18:34That's typical for services and datasets and so on.
18:38And what you also see is that this same service is available…
18:41…not just as an ArcGIS Server REST service but also as WFS and WMS.
18:47And what you immediately see with that is that, given different kinds of services, I can do different things, right?
18:53So with a REST-based service, I can add it to ArcMap using a layer file…
18:57…or ArcGIS Explorer Online or Desktop using the NMF and so on.
19:02With the WMS service, there's a few less things I can do. I can still add it to a map, right?
19:06So our Flex Viewer supports WMS just like the ArcGIS Online viewer does.
19:12And let me just sort of preview this.
19:15Right? So very simple previewer, not a lot of fancy functionality.
19:28So that's easy enough.
19:31The same with the WMS service. Right? So just wait a little for WMS.
19:36We add a little basemap because sometimes those WMS services have very sparse sort of content, if you will…
19:43…and so in this case, we're adding a basemap. I can turn it off if I wanted to.
19:47Similar, similar experience, right?
19:49So for the end user, it doesn't really matter which kind of service end point or interface they choose.
19:55The experience is more or less similar.
19:58We have a little embed, and it allows you to add the little preview window to your website if you wanted to.
20:06What you also saw with the WMS service here is this little Test button, right? So that's the Test button I spoke about here.
20:15That Test button shows up with all WMS services.
20:17So let me just do a little text search for WMS because this was an interesting sample as well.
20:24So if I go to Test button for this biomass service, you can see I get a link, right?
20:30The service URL is good; it checks some things. However, there's some issue here, right? I get some error messages.
20:43And when I went to this service before - let me see if I can do that - hmm, right?
20:59So that's the experience that you may get and that the service checker is trying to avoid.
21:04Let me see if I can go to the REST end point itself. ArcGIS...REST...
21:18What we see a lot in ArcGIS Server configurations that people put out on the web is that…
21:24…the REST services are all working fine and everything is good, right?
21:28And the publishers, they think that their WMS service also works because they checked the little box…
21:33…they filled in some metadata, and it all seems to be fine.
21:36However, we see a lot of cases where instead of - this doesn't respond right now; that's not very nice.
21:44…what happens a lot is that for WMS end points to actually work properly…
21:50…you have to do some configuration in your ArcGIS Server.
21:52You have to make sure that the ArcGIS Server name that is used somewhere is actually the external server name…
21:58…and not the actual, the machine name, which shows up by default.
22:02Right? So that's a fairly, fairly common sort of configuration thing that's not very difficult to fix…
22:07…but it's sort of overlooked many times.
22:11Alright. So that's not happy right now, so let's not do that.
22:16So what you see across the bottom here is a couple of these links that allow you to…
22:19…actually get the search results in different formats, right?
22:23So GeoRSS, like I described, or just a little HTML snippet.
22:28This is actually what I used on my home page, right? And I apply a little style sheet to it.
22:33You can put that on the URL, so that's fairly easy.
22:36And then you can render this in a nicer way than what you see here, but it's just a little HTML fragment that you get.
22:46And obviously, there's you can see KML and JSON, and you can do other things like that.
22:51Okay. So now let me show you this here.
22:54So one of the things that we have done in the geoportal is enable federated search.
22:59And this allows me to not only search my own catalog or the things that I have in my own catalog…
23:04…I have a couple thousand to maybe 20,000 or so things in my catalog - but I can also reach out to other sites.
23:11And what's interesting about this, what you see is that I can actually search ArcGIS.com for maybe geology…
23:20…right? and I get some results from my own site, and I also get some results from ArcGIS Online.
23:27And those things from ArcGIS Online I can then preview, open up in a viewer, right?
23:31So I can, we can sort of interact with ArcGIS Online through this federated search.
23:36Of course, that's easy, because that's ours, right, so we know that language.
23:42But we can also connect to, for example, YouTube. You might say, Why would you do that?
23:49Well, there might actually be situations where finding a little video clip about geology on YouTube might be interesting, right?
23:56So in this case I find some YouTube videos and I can play them, right?
24:00So the thing that you do with video is play, right, so that's what we do.
24:06So different kinds of things that you find will result in different things that you can do with that.
24:12And what we have done here is basically built sort of an extensible framework of the geoportal…
24:18…interacting with other search services and search catalogs, and they might speak Esri language…
24:24…the ArcGIS Online, the ArcGIS Portal API; they might speak OGC language, CSW for example…
24:31…or they might speak something completely different.
24:34Typically, open search is what we like because it's a fairly well-formed language or dialect of searching…
24:42…but it doesn't have to be. If it's XML, we know what to send, we know what to get…
24:47…and through some sort of transformations that we would apply, we can configure this.
24:52And you can add to this, right, so you can add to this on your own.
24:57So what we see here is a geoportal configuration file. When you scroll down, there's a bunch of things to configure.
25:03So let me find the YouTube one. So here, what you see, right, this is a YouTube repository.
25:12At the moment, this is how we configure it.
25:13You go to the geoportal configuration file, you add some XML to it, and then you have an additional search capability.
25:19We're going to move that into an environment where you can just do it through the web and configure that through the web.
25:25Something entirely trivial; I'll come back to that.
25:28So we basically have some classes that send the request that I entered, right, my search phrase…
25:34…and it translates that into something that YouTube understands.
25:38And, for example, what is important here is this particular link here, right?
25:42So YouTube wants to find your videos using a particular URL…
25:48…and they have a parameter called q, and that's where your searchTerms goes.
25:54And then you can do paging, of course, right…
25:56…so important thing to support if want to develop or add your own search provider, so to say, to this.
26:03Does it support paging? No one is benefiting from a 20,000 hit in one single page, right, so do it in chunks. People like that.
26:13And so this pattern, so to say, can be applied in many different ways…
26:17…and what happens when we get the results from that, we may actually have little sort of a profile.
26:23Right? So there's a little sort of back-and-forth that you need to do. Let's see here. Maybe looking at the different…
26:36There you go. There's the YouTube profile.
26:39Right? So this profile is sort of our understanding of how YouTube works.
26:44There's some transformation to transform the search criteria that you fill out on the page to this URL that I showed.
26:52Then once we get a result back, YouTube may come back in actually XML, right?
26:56So it's - YouTube comes back with an XML structure, maybe an RSS feed or something like that…
27:01…and then we read that using a transformation and we transform that to a structure that the geoportal understands, right?
27:08So we're sort of abstracting from the individual languages and dialects of XML or JSON or something from the search providers…
27:17…into sort of a common language that the Geoportal Server understands so that we can present search results.
27:22That's sort of a standard way of doing this, and you can extend it, your own.
27:26You can create those XSLTs on your own, and of course you can use any of the samples that are included there already…
27:32…and you can configure your own search interaction.
27:38So that's interesting, right? You can search sort of other catalogs like the World Bank projects…
27:45…and see if they have anything related to geology.
27:50Right? So sometimes what happens is the search provider doesn't actually tell you how many hits there were, right…
27:56…so that's fine; at least something came back.
27:59So here we get World Bank projects, and if I go there and I open this…
28:04What do you do with a project file? You actually open it.
28:07So in this case, I get to a page on the World Bank website where I just get a view…
28:11…of that particular project that had something to do with geology. So that's the idea.
28:15So the geoportal can really reach out to all sort of different catalogs in the back end to do that kind of searching.
28:22What you see actually right below the search box is not only a little feature layer.
28:26I've used the information that we have about the index, and that feature sort of tells me how many items I have…
28:34…in my catalog with a specific keyword.
28:36So in this case, geology, apparently I have 276 items in my catalog about geology, alright?
28:45So that sort of helps me understand a little bit, and as I type here - so maybe weather or so…
28:51…you can see that the list of things actually updates.
28:55So every time I type a character, it goes to sort of a different REST end point in the geoportal…
29:01…and it says, Tell me how many items I have in my catalog with a keyword that starts with that particular phrase.
29:10And then I can just click on that, and I get apparently three NOAA weather space program items.
29:17Right? So that sort of helps the user also to not just type in any word without any luck.
29:22They get a little hint, and they can use that.
29:28Alright, so that's great.
29:31So here we have the Flex Viewer. Let me just refresh that, because you never know; they sometimes go to sleep.
29:39And that same sort of Biota service that we had before, I can find that in my Flex Viewer.
29:46Right? So here's that Biota item; zoom to that, add to my map, voilà, right? Easy as pie.
29:54So the idea is that it's very simple to add multiple services.
29:57I can add, do another search, right? I can add more services to this; maybe Biota in other states surrounding Idaho…
30:05…and build up a map using multiple services and so on, right?
30:11So this other thing that I talked about was data download, and here I'm actually using the Oregon Spatial Data Library website.
30:18They make some interesting data available in the state of Oregon.
30:23And what they've done is they've organized that in a number of layers, so for example, this geology data.
30:30So I'll type in my e-mail address, right?
30:41And I choose the format that I may want to download this in and then choose the projection I want this in…
30:47…and using this little tool here, I can just draw an area - I want this layer.
30:54Send it off, get a little message; it won't actually get anywhere because that's not my e-mail address.
31:01And sort of that's the experience for an end user using the data download feature, right?
31:06And there's a couple sites out there that have implemented this; they've sort of tweaked this page.
31:10The North Carolina uses this to allow you to download imagery…
31:14…and they require you to zoom in to a smaller extent before you can actually download something.
31:19Right? So the idea is that you can sort of tweak these pages as you want to.
31:25Now the other thing that we have started to look at comes from the open government data sort of movement, right?
31:32If you're familiar with sort of the semantic web, semantic web like tries to sort of really understand sort of…
31:40…have machine-readable and understanding sort of the meaning of the datasets…
31:44…and the meaning of the descriptions across different systems, right?
31:47So it's a very sort of interesting idea here.
31:51And as part of our work in that area, Data.gov has a geospatial catalog that's built using the Geoportal Server…
31:59…and serves almost a million datasets, right?
32:02So that is the kind of catalog, well, that's great, but now I find this million, it becomes a Google, right?
32:07I have too many. I want to find the ones that are related and came from the same provider…
32:12…and that's where sort of linked data is starting to become interesting.
32:17So here we have built a simple SPARQL interface. SPARQL is the query language used in the semantic web very much.
32:24And that query language, that interface, works on my geoportal index.
32:28Right? So we can support things like SPARQL, and as with anything geospatial, it's still a little special, right?
32:35We can filter on the coordinates - and let me see if I can get more items, get a little more items.
32:43So what I've done here is basically configured a SPARQL query that is - well, you can see…
32:48…it's almost like a SQL query but not entirely, right?
32:50So I look for resources that have a title...
32:54...I look for resources that have a description, and I look for resources that have these coordinates.
33:01It's not really - this is not really sort of the WHERE clause. It really is assertions, right?
33:07So find me a resource that has this particular relationship with this particular value, right?
33:16And then find a resource that has a description relationship with a description, right?
33:21So it breaks down the index into things and relationships.
33:26Well, this looks like a spatial query, but it's not really, right?
33:29So what OGC is doing is they're extending SPARQL to include geospatial operations.
33:36So in this case I have modified my query a little bit to use the GeoSPARQL GIS within operation.
33:46So instead of just filtering on sort of individual these attributes now, we can also include these operations.
33:52So this is very much work in progress, and the whole sort of semantic web is, of course, still very much a work in progress…
34:00…but it's, I think, an interesting new sort of addition to this…
34:04…and it opens up the catalog for other sorts of integration with other systems.
34:10Another example that I wanted to show is this PivotViewerSample.
34:14So I talked about faceted search a little bit. I'll just reload that.
34:19And Microsoft has an interesting tool to support faceted search. It's this Pivot Viewer tool.
34:26And if you design your - and it works on a collection.
34:30And in this case, my collection is a set of a couple thousand metadata records that I exported using one of those end points, right?
34:38The export to CSV, and then from using Excel, there's a little tool you can generate a collection description…
34:42…and you can generate these images, and then I can use it in this Pivot Viewer.
34:47This Pivot Viewer allows me to look at my catalog in a kind of different way, right?
34:52So it's not searching; I'm not typing in keyword searches, although I could, but I won't.
34:57And it's not really browsing in terms of what I described at the department store, right?
35:02You don't start at the top and then you go down. You can sort of browse in multiple different ways.
35:07I can look at this collection in different ways.
35:11I can organize the things that I have by type, right?
35:15So as you can see, I have a lot of downloadable data.
35:18If I select that, I zoom in to that downloadable data piece, right, and then I can sort of navigate in different ways.
35:25I can say, Let's see where I have some data here, right?
35:29So apparently, looking at the west bounding coordinate in this case of my metadata, that is downloadable data.
35:37And as you can see, apparently I have a lot of data in the western hemisphere, right? The US; not surprising.
35:43And then there's a few data items here on the east side.
35:47I can sort of zoom to those by adjusting that little slider, right?
35:51So that's the idea of doing this kind of sort of interaction.
35:56And as soon as I make a change in one of these facets, the other facets also get updated.
36:05So let's limit a little further, right? So I can do things and then I can look at other categories…
36:13…and then, if I find something that I think is what I want, I can choose that, and from there, it's going to launch my geoportal.
36:21And I found one that's no longer there, so that's too bad…
36:24…but the idea is that you can actually connect back to a page that shows you details about that particular item, right?
36:31So it's sort of a new way of sort of looking into a catalog of things.
36:38Alright, so that's sort of a what can you do with the geoportal search, without a lot of coding, without changing it…
36:45…just using and thinking about the approaches that we have taken.
36:50Now another thing that I mentioned is open search.
36:53And Geoportal Server supports open search, and that brings up sort of a couple interesting options.
36:59So one of the things that you can do with open search is actually add it to your browser, right?
37:04So I can choose the Geo.Data.gov search here, and I'll search again for geology…
37:12…and right from my browser, I go and find content in Geo.Data.gov, right?
37:18So that's sort of an interesting sort of new sort of concept similar to Wikipedia and Google and others, right?
37:23You can do that with any Geoportal Server, and you can actually configure what happens when you do that search.
37:27In this case, I do that search and it shows me the search results page.
37:32If you want it to show up an HTML page with the items that you then style, you can do that too, right?
37:36So you have a lot of control about that.
37:39But this actually is even more interesting, this open search thing.
37:43My machine is a Windows 7 machine, although apparently, I need to replace it with a Mac…
37:49…and one of the things I can do with this is actually open the openSearchDescription for my Windows environment.
37:59Right? So I open this Windows - this OpenSearch file, and Microsoft Windows knows what to do with it.
38:05And it says, Do you really want to add this as a connector? Yeah, I do want that.
38:09So what it does is, actually, it adds geoportal search to my desktop, to my Windows machine.
38:16We will search for geology again; it finds geology items.
38:20In my local catalog, in this case, but like I said, any geoportal server supports this…
38:25…so you can search any Geoportal Server directly from your desktop without going to the website.
38:29And then when you do want to find something you get, in this case, to the metadata, again, that is configurable.
38:35You want to sort of go to a nicely styled page of this or you want to go to some other thing about this item…
38:41…you can do that using sort of configuration capabilities.
38:47Alright, so that's sort of the idea of a couple of the things that you can do with Geoportal, right?
38:52So open-ended on sort of connecting to sort of other source catalogs or other kinds of search providers…
38:59…and we're trying to make it open-ended on the front end also so that you can choose your favorite client applications…
39:04…or client development environments using the JSON flavor of the outputs and so on…
39:09…or just embedding a simple HTML piece or so.
39:14Alright, that's sort of what the Geoportal is and what you can do with that.
39:20What you've also heard a lot about over the last couple of days is ArcGIS Online and the ArcGIS Portal, right?
39:26And what I told you here might actually sort of create some confusion about that…
39:32…and we get a lot of questions from people on Give us some direction, right? So tell us where we need to go.
39:38Is it ArcGIS Portal or is it Geoportal? And if you have a little smork about it, then you can…
39:45This is actually a real…it's a real sign.
39:50After an 18-hour flight to Narita airport, nothing is better than a good smork, right?
39:58Well, the answer to this really is there's a time and place for everything, right…
40:02…whether you're Tweeting or whether you talk about ISO 1939 [sic] metadata…
40:09…with profiles and XML and schemas and Schematron and all that good stuff, right?
40:13So what this slide tries to explain is that there are sort of these different views of items, right?
40:20So one is the very simple view, the YouTube/Flickr view, right?
40:25That's I just put in some words, specification in Japan or wherever you went, and that's it…
40:32…and look at the picture and lets people sort of see the picture and they're interacting with that.
40:37Large user community; it's everyone basically.
40:41It's the non sort of specialist.
40:42It's everyone uses YouTube and everyone knows how to do it and how to find a video clip that's interesting.
40:48So at the other end of this spectrum is the people who really describe their content in large amounts of detail.
40:58And there's a very good reason for this.
41:00If you are a climate scientist and you create data and you manage to create a dataset now…
41:05…that dataset may be used in 10 or 20 years from now, right? Because your timeline is very long.
41:10It's not the 20-minute time span that people have on YouTube or so.
41:15So you really have to describe it very well.
41:17Or if you are an engineer and you use some sort of report that someone created…
41:23…you really want to make sure that this is an authoritative report with the proper credentials and so on, right?
41:30Or if you are a commercial entity and you want to build some commercial service around content that you find somewhere else…
41:36…you really want to make sure that you're not sort of getting into all kinds of lawsuits…
41:40…because you violated some license agreement or patent or something.
41:44Right? So there's a reason for good, verbose metadata. There is a reason for very brief metadata.
41:52I think one of the things to look at this is that ArcGIS Online sort of focuses on that simple stuff for now, and we'll come to that…
42:00…and Geoportal focuses on that sort of verbose, if you will, standardized.
42:06If you're in Europe, you have to create INSPIRE metadata. Sorry.
42:10There is sort of a legislation aspect to this as well, and a lot of the federal agencies here in the US…
42:15…they have to use FGDC metadata. There's no coming around that.
42:21So what we're working on, one of the things we're working on in Geoportal Server is really sort of bridge that gap…
42:26…sort of create the glue between ArcGIS Portal and its use cases and the Geoportal Server and its use cases, right?
42:35So how can we combine those two?
42:38Well, one of the things that I actually showed you already is searching ArcGIS portals from the Geoportal Server, right?
42:45ArcGIS Online is essentially an instance of ArcGIS Portal.
42:48I can search ArcGIS Online from my geoportal, and if you paid attention to that one sort of moment in my demo…
42:54…you saw the NOAA GeoPlatform and the EPA GeoPlatform; those are ArcGIS Portal implementations.
43:00I can search those as well from my geoportal. And so that's very simple integration here.
43:05What's, I think, more interesting is that ArcGIS Online, ArcGIS Portal, and ArcGIS Server 10.1…
43:10…will allow you to create and have verbose metadata, or full metadata, with any item in those systems.
43:20And what we do within Geoportal, what we have done sort of over the last couple years…
43:24…is create the editors that do standardized metadata.
43:28So you'll put one and one together, you get standardized metadata for ArcGIS Online items…
43:33…for ArcGIS Portal items, and for ArcGIS Server 10.1 items, whichever standard is your standard.
43:38And one of the things that we are trying to do is figure out…
43:40…how can the Geoportal Server contribute to that, using, for example, its editor?
43:45And then you can think of other things, right?
43:47So Geoportal supports OGC interfaces. Well, we can add that to ArcGIS Online or ArcGIS Portal.
43:54You can register a resource with the geoportal, and it'll harvest that over time and do that automatically…
43:59…weekly or monthly or daily or so, right?
44:02Well, we can actually push that content into ArcGIS Portal or ArcGIS Online so that you get a very simple sort of workflow…
44:08…in ArcGIS Online or ArcGIS Portal to update the content in those systems, right?
44:16So I think there is a place for both.
44:19Let me show you what I'm talking about here.
44:24So I showed you searching ArcGIS portals and ArcGIS Online from a geoportal, right, so…
44:31Let me just clear this here.
44:37Let me go to NOAA in ArcGIS.com, see what they have. Right?
44:44So apparently, NOAA has four items about geology, right?
44:50In this case, they don't really have anything specific about geology.
44:54It's the standard content that comes with ArcGIS Portal when they implement it.
44:58ArcGIS Online has a little more, and my geoportal has something, right?
45:01So I'm searching ArcGIS portals and ArcGIS Online from my geoportal…
45:06…without the user actually sort of needing to know how that works behind the scenes.
45:12Let me do the other, the opposite way, right?
45:15So what I have here is an ArcGIS Portal implementation on my machine here locally, and I changed it a bit.
45:22Like I said, I work in Professional Services; I do a project with the government of Indonesia.
45:28And let me do that geology search again.
45:36They'll come back with some results or not…
45:38…but what you see on the left-hand side there is for me an ability to search other catalogs.
45:45Right? So what we have done here is integrate my Geoportal federated search capability into this ArcGIS Online site.
45:55We'll not wait for that.
45:58I can get those results of geology, right? The 3 out of 25? Did anyone pay attention? Same number, right?
46:04Or find them in ArcGIS Online or find them in Data.gov, right?
46:12So the idea is that, using the Geoportal search capability, we can federate searches…
46:17…and integrate that into an ArcGIS Portal instance.
46:26So the other thing that I did here - let me sign in - what I talked about is adding metadata to items in ArcGIS portals, right?
46:38Standards-based metadata. So I'm connected here and I have some items.
46:47Let me see. Maybe I need to restart something here. Just to be sure.
47:16That's not nice. One second.
47:27So the idea is that we can actually add metadata.
47:29So while this is going with this ArcGIS Portal, let me show you what we do with metadata.
47:34And if we are good to go, then we can actually show that in ArcGIS Online.
47:45So I'm connecting to my Geoportal as a user, and in the Administration section…
47:50…we have some capabilities to upload and register content, like I've mentioned.
47:55So I can add a resource. I can do that using the URL to that resource, right, if I want to register a WMS service, for example…
48:02…or I can, if I have an XML file, I can upload that; or I can use an editor online and use that.
48:11So let me show the FGDC editor for here. As you can see, I have a number of different profiles and schemas configured.
48:16You can change that to only show the ones that you want to show.
48:21And so this is an editor that we have created over the last year or so that allows you to…
48:25…edit metadata in a standards form, extensible, and do that all online.
48:33So the idea is that, if you are familiar with any of these specifications, right, you can have sort of repeated elements…
48:38…and repeated sections of the metadata; they can be multilingual, and so on.
48:42You can configure all of that using these editors.
48:46And in FGDC, they have some particular values, like the Unknown value, and other things.
48:53And once I'm done with this, I can actually try to save this.
48:57What you see is that Geoportal simply validates the form that you tried to submit, and it does that here…
49:02…based on some mandatory elements that I've configured.
49:05So, for example, I do need to put in a title, and as soon as I type in a title, it's right below my screen here…
49:13…it says it's good to go, right?
49:16When I type in the abstract, and so I can save that.
49:19Well, I want to sort of move on with my demonstration, so I can save it as a draft…
49:24…which means that it's available for me as a user, but it's not discoverable, right?
49:29So that's sort of the idea.
49:32Now let's see if my portal here has woken up again.
49:55There you go.
49:57So I have a few items, and let me open up this BeeSpread item here.
50:03So this is a normal ArcGIS Portal item page, so sort of standard stuff here…
50:08…and the only thing that was added here is this Metadata button.
50:13And what this does, it opens up the metadata view, in this case, of this item.
50:19And if you want to view the XML, you can see this is actually FGDC metadata, right?
50:23Well, that's great; I can view metadata.
50:25This metadata is not stored in my geoportal, right?
50:27This metadata is stored in ArcGIS Portal with the item that I'm looking at.
50:33Want to edit this? Familiar editor, right?
50:36So we're accessing the geoportal editor from ArcGIS Portal, and I can update this title with more text, right?
50:47So and then I can then save this.
50:51As you can see, I can also create a new one. I can upload an XML file by importing it.
50:56I can actually download this XML file and do something else with this, right?
51:00I'll save that, and now go here. Right? So the idea is that, doing this…
51:10…I can edit standards-based metadata for any item in ArcGIS Online or ArcGIS Portal by fairly limited integration.
51:20So that's just sort of integration with Geoportal and ArcGIS Online or ArcGIS Portal…
51:27…searching ArcGIS Online from a geoportal, searching geoportals from ArcGIS Online, editing standardized metadata.
51:33And then last but not least here, let me go back to that Biota service…
51:38…because I also mentioned that we can connect to any viewer as long as we have a little handshake.
51:45And so what I've done here is, instead of using the Flex Viewer…
51:53Right? Simple handshake, configure, update, save, et cetera. There you go, right?
51:59So imagine, this could be the ArcGIS Online viewer itself, which would allow me to share…
52:05…a map service that I found in my local geoportal; I open it up in ArcGIS Online, its viewer…
52:10…and then from there, I can share it with others through ArcGIS Online.
52:14So sort of a fairly sort of simple way to share content from your local geoportal with ArcGIS Online users…
52:21…using this integrated viewer capability.
52:25Alright. Let's see, a little time check. I think we're good.
52:30So then what I have here is a little page that we created to sort of start prototyping…
52:38…the editing of metadata with ArcGIS Server services in 10.1.
52:43Right? So these are some URLs to a service. Let me sort of make sure that this service is available.
52:59Right? So it is a simple map service.
53:0110.1 running in Redlands in the office, and it has a number of new interfaces, if you will.
53:10Right? So for example, there is this info interface, and from there, I can get to the metadata.
53:16And in this case, we actually see it's FGDC metadata again. Right?
53:19So the ArcGIS Server item has FGDC metadata…
53:25…and using this little sort of tool, we can actually edit that metadata using the same editor as we have seen before.
53:34Right? View it in a styled way, edit it using the editor.
53:40So integration of Geoportal Server with ArcGIS Server at 10.1 allows you to create standards-based editing capability…
53:48…editing and standard-based metadata online, right? There is no…
53:53And obviously, in ArcGIS, you would typically edit your metadata in the desktop…
53:57…but there may be a reason to do this online as well or maybe change some elements online once the service is published.
54:04Well, this would allow you to do that.
54:15Alright. So some last words. I think I'm ahead of schedule, so I don't hear any questions…
54:22…so if there is questions afterwards, that's great.
54:25Last words, because you never know. You might get abducted.
54:32That's actually the more important feature on this page from July 4th.
54:37So what are we trying to do this year, right?
54:39So I told about some existing capabilities.
54:41I told a little bit about sort of SPARQL and the editing of ArcGIS metadata, and so on…
54:46…because those are obviously things we're still working on.
54:50One of the things that I'm trying to do with Geoportal is really involve the user community more.
54:54Right? We have a fairly active forum where people ask questions and other people answer those…
54:59…and sometimes my team answers those, and so on.
55:01We have people who submit patches, right.
55:04So for example, our distributor in Italy deserves a little bit of credit because he creates a lot of little enhancements…
55:11…little tweaks, and little things and then submits those as a patch to SourceForge where we have the Geoportal.
55:18And we can then sort of integrate those or users can download those and update their Geoportal…
55:26…like I did with that little Test link, right?
55:28So including a Test link like I did is a very simple update to some pages, and so on.
55:36Geoportal sort of has been in its current state for a couple years now; user interface hasn't really changed much and so on.
55:42I mean, we'd like to sort of get some feedback from users, and that's one of the reasons we set up this e-mail alias also…
55:49…to get a dialog going around that.
55:51And really sort of what we're trying to do is follow the Apache way of opening up the code base…
55:58…and allowing others, non-Esri people, to commit code to that.
56:03It's very scary for us, right? But we're learning. We're getting there.
56:07And rather than us just sort of dumping a code base over the fence, which is not really what we want to do.
56:16So that's hopefully something that you might be interested in supporting.
56:21The other way to sort of support or collaborate is, instead of coding, is just documenting, right?
56:27Create a sample or create some documentation.
56:31We have a wiki as part of our Geoportal project in SourceForge that we can allow you to edit and so on…
56:37…and we have some external editors for that also.
56:40And obviously, the forum. We see users who are experienced users…
56:45…they answer questions of people who are not so experienced, and they help each other, right?
56:49So that's great. That's actually what we want to get to.
56:54Alright. In the road map, version 1.2 is what we released in December.
57:00Before that last year, we had a few versions that…
57:03I mean, the editor like that I showed was sort of built up over a couple of versions last year.
57:09And in December, we released 1.2, which adds the ability to have a WMS service in OpenStreetMap…
57:15…as a basemap in the search window, right?
57:18That little window of the search map that I showed.
57:21We added some capabilities to have collections of content to it.
57:25So I want to have everything related to geology in an easily accessible collection…
57:32…maybe with its own REST end point, maybe with its own CSW end point…
57:35…so I can search just those geology-related items, regardless of where they came from, right?
57:40Regardless of individual user or catalog they were harvested from, and so on.
57:45This is actually being used in Geo.Data.gov at the moment…
57:49…to distinguish the formally blessed, Data.gov-approved content from everything else, right?
57:55So if you go to Geo.Data.gov and you do a search…
57:58…you will find maybe half a million items or so out of the one million that's there…
58:03…and that is because the other half has not gone through the official open government, GSA approval process…
58:10…of being vetted and being verified for release, and so on.
58:14Right, so there's…they use the collection feature to support that, to implement that.
58:21MySQL support, right? So ArcGIS Portal, if you've implemented that, it runs in MySQL.
58:26If you want to have Geoportal running with it, you could actually do the geoportal database in MySQL also…
58:31…or you can use Postgres or some other database if you want to.
58:35We added Schematron validation, right?
58:38So the validation that I showed in the editor is fairly simple; this field is mandatory, and this field is optional, right?
58:45So that is the distinction there.
58:47We also do schema, schema validation itself, so if the schema says that something is mandatory…
58:52…then you need to have that.
58:54And then there is Schematron validation as sort of the next level up that is more sophisticated validation.
59:00If this element is filled out with this value, then this needs to be such and so on.
59:04So there's a sort of three levels of validation there that you could do if you wanted to.
59:08We added THREDDS harvesting.
59:10THREDDS is a real-time data, sort of database, that has large amounts of very…
59:16…sort of multidimensional, scientific data typically, right?
59:19But all the datasets are accessible through ISO metadata, and they have their own sort of particular interface.
59:26Well, we can harvest those catalogs now also and give you some view into this scientific content.
59:35So that was available now; it's available since December, and as you can see…
59:40…the ArcGIS Online integration, the ArcGIS Server integration are really sort of at the top of my list, of our list…
59:47…because we see many projects and users who implement both Geoportal…
59:52For example, NOAA. NOAA has a number of data centers, and each of those data centers uses Geoportal Server…
59:58…and then they have this higher-level, NOAA-wide platform implementation of ArcGIS Portal, and now they're sort of…
1:00:06…we're figuring out with them, using Geoportal Server capabilities of harvesting and pushing things…
1:00:11…into Catalog and other catalogs, and so on…
1:00:13…how can we streamline that process of getting content from these individual nodes…
1:00:17…if you will, into this master catalog and from the master catalog out to GeoPlatform, GEO, WMO, et cetera, right?
1:00:27So there's - on the consuming side, there are a lot of different sort of communities and organizations…
1:00:32…that want to get access to the content, and that's where I think Geoportal plays a key role.
1:00:37We're looking at sort of enhancing the structure of the catalog if you want to, right?
1:00:41So it's almost going a little bit deeper even in that pyramid, right?
1:00:46More structure, more structured content.
1:00:48And sometimes, people want that.
1:00:50There's a catalog information model approach from OASIS, ebRIM, and OGC has adopted that…
1:01:00…to sort of add some geospatial components to that. It's one flavor of OGC catalog that we want to support.
1:01:07But then there's also things like Feature Data Dictionaries, where you can be very precise and very verbose…
1:01:12…in the describing of individual feature types and their definition and where did the definition come from, and so on, right?
1:01:18In some situations, that is very, very important.
1:01:22Last, but not least, WMTS support, SOS support, WPS support, and so on.
1:01:27It's different kinds or additional OGC specs that come out over time, we'll just keep up with that.
1:01:33And then the faceted search like I showed…
1:01:34…I want to sort of figure out something that we can do with that in the Geoportal…
1:01:38…whether it's using Microsoft Pivot or whether it's using Apache Solr, which is a fairly popular faceted search tool.
1:01:46So look for that, right? In addition to the standard search and the browsing capabilities.
1:01:53Alright, questions? We have some time for questions. Question?
1:02:00[Inaudible audience question]
1:02:07Oh, very good question. So if you search things like YouTube or Flickr or other sites, what about the map?
1:02:14Well, actually more and more, those sites have some sort of spatial information, right?
1:02:20Your photos are geotagged or maybe your video's even geotagged, and more and more…
1:02:24…the interfaces that those sites make available expose that information, right, coming back.
1:02:29Doing a spatial search, as in a spatial constraint on the search, is not there yet in many of those sites…
1:02:35…but at least, when you get a response back, you see where the photo was taken or where the video was taken. Yeah.
1:02:44And for configuring that in the Geoportal, it's essentially just a matter of updating that…
1:02:50…typically, that XSLT that parses the response and says, oh, now there is a spatial extent, right?
1:02:56So add it to your XSLT and then the Geoportal will use it.
1:02:59Question there, sorry.
1:03:00[Inaudible audience question]
1:03:06You can find it here at the moment. Yeah.
1:03:13Any more questions here? So let me show you where you want to go.
1:03:19If you want to get more information in general, go to geoportal.sourceforge.net or go to…
1:03:44Let me see if that is the right link.
1:04:02Go to esriurl.com/geoportalserver, and that'll take you to the SourceForge page also.
Creating Geoportals using Esri's Open Source Geoportal Server
Marten Hogeweg describes how Esri Geoportal Server supports both standards-based and Web 2.0-style discovery and access to geospatial resources for SDIs.
- Recorded: Mar 29th, 2012
- Runtime: 1:04:17
- Views: 1463
- Published: Apr 25th, 2012
- Night Mode (Off)Automatically dim the web site while the video is playing. A few seconds after you start watching the video and stop moving your mouse, your screen will dim. You can auto save this option if you login.
- HTML5 Video (Off) Play videos using HTML5 Video instead of flash. A modern web browser is required to view videos using HTML5.