00:01Good morning, everyone. My name is Marten Hogeweg.

00:03I thought I'd try starting early so we can actually release early, for once.

00:09I'm product manager for the Esri Geoportal Server…

00:12…and I'll be talking about this open source product from Esri for the next sort of hour or so.

00:19And there's some time for questions after that, so if you have a question in between, please interrupt me. I don't mind that.

00:28You might actually ask yourself, What is this about Esri and open source?

00:34I think if you were at the Plenary Session on Tuesday, you heard people talk about a number of things…

00:40…so I think it's time to actually get used to the fact that Esri and open source are not two opposites…

00:45…are not conflicting terms or contradictions anymore.

00:51I'm going to talk about the Geoportal Server. My team also develops the ArcGIS Desktop Editor for OpenStreetMap…

00:57…which is an open source extension to Desktop that allows you to contribute to OpenStreetMap.

01:03If you followed the Flex Viewer developments over the last couple of months…

01:06…you've noticed that it has been released under Apache 2.0 as open source.

01:11The Silverlight SDK has been available as open source for quite a while, and last Tuesday…

01:17…a number of the Runtime SDKs and applications that were built on that were mentioned and also to become open source.

01:26However, if you've used ArcScripts up to 14 years ago or so, you've actually been able to get open source software from Esri.

01:36The license text that was on the previous slide actually came from one of these scripts that people had shared…

01:41…an Avenue script or an AML script or such.

01:45And so there's been a community or practice of sharing code, maybe not under a formal open source license…

01:51…but a practice of sharing code, helping each other.

01:55Think about the developer samples that you can find for all the web APIs online that you can use.

02:00Think about the template applications that you can download from ArcGIS Online…

02:04…and then start building your own templates based on those.

02:07And so there's this practice.

02:10With the Esri Geoportal Server, we've started to make this more of a formal step…

02:15…and actually released a product under an Apache license.

02:20So what is this Geoportal Server? The Geoportal Server is really basically a metadata catalog tool and discovery tool…

02:28…that allows you to describe, find, and access resources, particularly in heterogeneous environments.

02:36And with that, I mean environments where Esri technology is not the only technology being used.

02:42Users may serve data and services using the OGC specifications or other specifications.

02:50They want to describe those resources using standardized metadata…

02:55…and the consumers in those environments are also using various technologies other than just Esri technology.

03:05A step to make the Geoportal Server open source happened in October 2010.

03:09It was actually not a very sort of big step.

03:12We started off nine years ago, as a result of Geospatial One-Stop, which was the Bush administration's website…

03:20…for access to geospatial data in the US from federal, state, and local government and academia and so on.

03:29And for that we built a technology using ArcIMS metadata server actually, and once we were done with that…

03:36…we had our distributors and other users asking, We want that same website now for our organization.

03:41So we turned that project - I work in Professional Services within Esri.

03:45We turned that project into a product called the GIS Portal Toolkit.

03:49And actually, from day one, we allowed or we gave users access to the source code of the GIS Portal Toolkit if they asked for it.

03:57We didn't provide it just for everyone.

04:02If you don't use it, don't need it, then just stick to the compiled applications and we will help you if you want to.

04:10Right? So it was very much sort of a project solution, if you will, a services solution that we implemented.

04:16So over time, as we sort of productized it more and moved away from ArcIMS to the technology that we're using now…

04:25…it became sort of more and more sort of independent of ArcGIS technology…

04:29…to a point where now we're not relying on anything ArcGIS if you don't want to.

04:34Right? So if you want to use a third-party service as the basemap, you can use WMS, or you can use OpenStreetMap as a layer.

04:42If you want to - you're not bound or using any SDE licensing or anything like that.

04:48If you want to use MySQL as your database, right, that's fine.

04:53So that's sort of an evolution, and in 2010, as I said, we decided to actually just release it under Apache 2.0…

05:01…as a developer- and user-friendly open source license with very limited restrictions.

05:08Just say thank you or just say reference that you used the geoportal, and that's about it.

05:12You can create a commercial product out of it if you want to without having to reshare what you add and do with it.

05:20Being within Professional Services, we do provide support for it, and as you can see, there's almost 20,000 downloads.

05:27When I created this slide, or updated this slide, last night…

05:30…I thought about maybe I should download it a couple hundred times or stuff, but that's cheating, right?

05:36And also what you can see is that we have people downloading the geoportal from 150 countries, right?

05:41So it's international being used.

05:45It's only 30 percent, or actually less than 30 percent is actually within the US; 27 percent or so.

05:52That sort of illustrates that it's international focused.

05:55And the Geoportal Server, for those who are from Europe, and if you are sort of familiar with INSPIRE…

06:01…Geoportal Server is one of the components in ArcGIS for INSPIRE…

06:04…which is a solution that my team builds for the European market.

06:08So what can you really do with the Geoportal Server?

06:10Well, there's a couple of use cases that I typically go through, and I'll go through these sort of, hopefully, quickly enough…

06:15…and then I'll actually do a little bit of a demo because showing is much more interesting than telling.

06:21So we do support searching based on simple keywords, but we do spatial searching, we do spatial ranking.

06:29The index of our catalog is highly configurable. You can add your own fields to the index, and then you can search on those fields.

06:36And that brings up an interesting concept of what is called faceted search, and I'll show an example of that.

06:43If you want to, you can sort of include a thesaurus, which means that if you search for the word river in your text search…

06:52…it may also find stream and lake and water body, right?

06:55But river - those items that have river somewhere in the description rank a little higher than the ones that have water body…

07:03…because they're more closer to the word that you typed in.

07:06That thesaurus is part of the Geoportal Server.

07:09You can tailor it; you can configure it, and so that's one thing to sort of get more results, get more sort of chances of success there.

07:18Another thing that you can do, we're using the Lucene indexing technology.

07:23Lucene, if you know Lucene, works with analyzers and parsers, and you can use the one that we configure out of the box…

07:32…but you can also use ones that understand stemming, right?

07:36If you're from some - I'm from the Netherlands, right?

07:39We like to concatenate words, so we have long words, 20, 30 characters, if you want to.

07:46And I type only in a portion of that word, and with some of the advanced analyzers and stemming capabilities of Lucene…

07:54…you can actually find those things even though you only typed in a partial word…

07:59…or a word that is almost the same as one of those partial words.

08:03Right? So there's a lot of configuration that you can do without actually getting into too much coding.

08:10Sometimes, searching is not the right solution, right? So if I go to Lowe's, which is one of the department stores here…

08:18…and I have one of these electric lawn mowers and had a little wire on it, if I want to sort of figure out which part is that again…

08:25…right, I may go to the website, then I go to the garden section, then I go to the power tools…

08:30…and then I go to the lawn mowers and then the electric lawn mowers and then I go to the parts and pieces, right?

08:35I don't just type in lawn mower wire, because I have no idea what part number or name that they use for that.

08:43So sometimes this sort of navigation through a hierarchy is actually better.

08:48So you can configure those kinds of hierarchies and navigation structures in the Geoportal Server as well.

08:54And they can actually be based on sort of some elements of the metadata…

08:59…but they can also include relationships between items, right?

09:03So maybe I'll show some of that as well if I have time for that.

09:07So, for example, if you have a server with 50 services on it, then if you find one of those services, through those relationships…

09:18…through the fact that the other 49 came from the same server, you can actually find those other 49 as well.

09:23So you can navigate not just based on sort of a formal description of something, but you can navigate through relationships.

09:29And those relationships can actually be determined by Geoportal at run time, so to say.

09:33They don't have to be defined prior to navigating there.

09:39Obviously, you want to sort of see what you get, right, so I find a service; is this the service really that I want to use?

09:47Similar to YouTube or Flickr, you don't just read the description.

09:52You actually play the video and after half a minute, you say, oh, no, that's nothing. Similar with maps.

09:57We have a little preview function. We have a sort of richer viewer that's actually built using the Flex Viewer.

10:03There's also one for Silverlight, and we have some widgets that work in those viewers…

10:07…search widgets that allow you to start up a viewer and then from the viewer, search for some content…

10:13…and then if it's a service, add that service to the map and build some map using multiple services.

10:19And if you look through some of the description already approaches that we have taken…

10:23…you can actually add your own viewer if you wanted to.

10:27As long as there is a little handshake between the geoportal and your viewer, your viewer can actually be anything you want to.

10:35So as we're moving to a services sort of infrastructure, right?

10:39Everyone is sort of starting to build services, and with the ArcGIS Online subscription…

10:42…it becomes very easy to host services somewhere.

10:44You don't need your own server anymore; you just put it in the cloud somewhere.

10:48But there are still very many use cases, situations where people just want the data.

10:52Right? "Just give me the data; I want to do some analysis locally and combine it with some other thing…

10:58…or integrate it in my organization's database."

11:01So we include a clip, zip, and ship function that is based on an ArcGIS Server geoprocessing service.

11:07It's an optional configuration that you do, right? So we don't depend on it, we don't rely on it.

11:11If you have Server, you want to expose your data for clip, zip, and ship, then you can do that.

11:17And it allows sort of you to configure the layers that people can download.

11:20Users can then, in a very simple client, sort of draw an area; select an output format, projection if you wanted to…

11:26…and then sort of send me an e-mail when the data is done.

11:30This is a normal geoprocessing service written in Python, ModelBuilder, so you can also change that if you wanted to.

11:36Right? Include some equal [unintelligible] step in the process or something like that.

11:42People may want to sort of find resources, but it actually means that your catalog needs to be populated, right?

11:49So if no one populates something in eBay, then I can go to eBay to find something but there's nothing there, right?

11:53So what we tried to do is make it very easy for those who actually share their content to make that process of sharing very easy.

12:03So we have some capabilities of harvesting, as we call it, that can automatically, on a regular basis…

12:09…maybe daily or weekly depending on how often your information changes, harvest your services…

12:14…or harvest your catalogs or harvest your folder of XML files.

12:18I mean, there's various different formats and mechanisms for that kind of process.

12:25Your descriptions could be following one of these standards for metadata.

12:31You can actually create your own standard for metadata, right?

12:34So we've seen a couple of situations already where people say, well, we're in Europe and we use INSPIRE…

12:40…but our country has its own profile of INSPIRE metadata.

12:43Right? So we have extra fields or extra domain values in the - or we have…

12:49…we want this particular field always to be populated when someone submits something.

12:53So you can configure all of that, and so that means that that process of registering something becomes very easy.

13:04And I mentioned Server, right? So one of the things we can do is if you have an ArcGIS Server instance…

13:09…you basically registered the ArcGIS Server instance; not an individual service but the instance…

13:14…and then the Geoportal will essentially crawl that server and fetch all the end points, right?

13:19The REST end points and the WMS and the other kinds of OGC end points.

13:23If you have an individual WMS service, you register the individual WMS service…

13:28…and we will sort of go and visit that service however often you configure that to do.

13:33And what is interesting about those WMS or OGC services themselves is that they actually have some metadata in them.

13:40Right? There is a metadata element, part of the WMS specification, that allows you to…

13:46…have a fairly complete description of that service - who is it, what can you do with it, are there fees associated with it, et cetera.

13:52And through a mechanism of understanding those different specifications, we can index those…

13:59…and we can sort of harmonize the view of the different metadata, right?

14:03So a title in Dublin core, a title in FGDC, a title in INSPIRE, a title in WMS - they're all in different locations, right?

14:10Different XPaths, so to say. And we sort of abstract that into the title, and that's what we call a meaning.

14:17So we assign a meaning to a field or an element in XML, and you can add more meanings, right?

14:22So that's what I meant when I talked about adding your own fields to the index and building things like faceted search with that.

14:31Another thing that we have seen over these number of years that we have used and supported the Geospatial One-Stop portal…

14:39…for example, is that people go to a site; they do a search…

14:44…"I want geology data in this particular area," and they don't find anything, right? Darn.

14:49So they go away. They try to go somewhere else and they won't come back because they say your site doesn't have it.

14:55Well, what we have done is we've implemented some very simple RSS capabilities that, if you do a search…

15:01…every search you do, you can subscribe to through an RSS feed…

15:05…whether it includes a spatial extent or just some keywords or topics, and so on.

15:11And that means that you can get notified once the site gets new content that matches your very specific criteria.

15:18You get notified of that, and you get a little alert in your - maybe in Outlook, right?

15:23You can register RSS feeds in Outlook or so, if that's your e-mail program…

15:26…and then sort of go back to the catalog and see what's new.

15:35And then last, but not least, as a thing that is very frustrating for very many people is that I find this service…

15:41…it looks promising based on the description, I click Preview, and it doesn't work.

15:45Right, so to address that we have sort of connecting to a third-party service, a service that FGDC…

15:52…Federal Geographic Data Committee here in the US, is building, and that…

15:56…and the purpose of that service is to check the availability and performance and conformance of your services.

16:04And at the moment they're focusing on OGC service, WMS - actually, ArcIMS is included also, a few other things…

16:10…but they're extending that capability, as we speak, to also include the ArcGIS Server or ArcGIS REST-based services…

16:17…which, by the way, is on its way to become an OGC spec also, right? So that's also good.

16:23So the idea there is I can get a little report, scorecard if you will, is the service up and running?

16:28And if the service is not up and running, then no, maybe I should go look for some other services…

16:35…that might have the same kind of information.

16:39Alright, so that's sort of an introduction to what is Geoportal Server, what is sort of the role that we play…

16:46…and the context that we play it in. Let me do a little sort of bit of a demo.

16:54Let me see if I can view, little more view of that.

17:05Alright. So we're looking at the Geoportal sort of home page here.

17:08This is my local geoportal; it's slightly customized, I add a nice picture to it, and I added a few other things to it here.

17:16So what's interesting about our index is that it's not just available for searching, but I can do other things with it as well.

17:24So, for example, I can see how many types of things of different kinds I have, right…

17:29…so applications or downloadable data or live map services, and we have some information.

17:34We have basically some statistics about the index that are available through a little REST interface as well.

17:39So I can do little graphs like this, sort of what are my different kinds of content or what's new, right?

17:48So like I said, you can subscribe to a news feed.

17:50Well, that news feed is not just available to you in an RSS reader, but you can also get that in a simple sort of web page format…

17:58…that you can then style and include in a little web page like this here.

18:05So I did a search for some interesting services.

18:08This is one of sort of services that I've been starting to rely on when I do my demos.

18:14It's actually a service from the State of Idaho, and as you can see, I get a few sort of items.

18:21For every search that I do, I get some items, and let me just zoom to that.

18:28Alright, this particular one is over there. That's how every item that we have has some sort of spatial extent, right?

18:34That's typical for services and datasets and so on.

18:38And what you also see is that this same service is available…

18:41…not just as an ArcGIS Server REST service but also as WFS and WMS.

18:47And what you immediately see with that is that, given different kinds of services, I can do different things, right?

18:53So with a REST-based service, I can add it to ArcMap using a layer file…

18:57…or ArcGIS Explorer Online or Desktop using the NMF and so on.

19:02With the WMS service, there's a few less things I can do. I can still add it to a map, right?

19:06So our Flex Viewer supports WMS just like the ArcGIS Online viewer does.

19:12And let me just sort of preview this.

19:15Right? So very simple previewer, not a lot of fancy functionality.

19:19If you want to, you can change that - simple JavaScript API viewer - and I can interact with this service here, right?

19:28So that's easy enough.

19:31The same with the WMS service. Right? So just wait a little for WMS.

19:36We add a little basemap because sometimes those WMS services have very sparse sort of content, if you will…

19:43…and so in this case, we're adding a basemap. I can turn it off if I wanted to.

19:47Similar, similar experience, right?

19:49So for the end user, it doesn't really matter which kind of service end point or interface they choose.

19:55The experience is more or less similar.

19:58We have a little embed, and it allows you to add the little preview window to your website if you wanted to.

20:06What you also saw with the WMS service here is this little Test button, right? So that's the Test button I spoke about here.

20:15That Test button shows up with all WMS services.

20:17So let me just do a little text search for WMS because this was an interesting sample as well.

20:24So if I go to Test button for this biomass service, you can see I get a link, right?

20:30The service URL is good; it checks some things. However, there's some issue here, right? I get some error messages.

20:43And when I went to this service before - let me see if I can do that - hmm, right?

20:59So that's the experience that you may get and that the service checker is trying to avoid.

21:04Let me see if I can go to the REST end point itself. ArcGIS...REST...

21:18What we see a lot in ArcGIS Server configurations that people put out on the web is that…

21:24…the REST services are all working fine and everything is good, right?

21:28And the publishers, they think that their WMS service also works because they checked the little box…

21:33…they filled in some metadata, and it all seems to be fine.

21:36However, we see a lot of cases where instead of - this doesn't respond right now; that's not very nice.

21:44…what happens a lot is that for WMS end points to actually work properly…

21:50…you have to do some configuration in your ArcGIS Server.

21:52You have to make sure that the ArcGIS Server name that is used somewhere is actually the external server name…

21:58…and not the actual, the machine name, which shows up by default.

22:02Right? So that's a fairly, fairly common sort of configuration thing that's not very difficult to fix…

22:07…but it's sort of overlooked many times.

22:11Alright. So that's not happy right now, so let's not do that.

22:16So what you see across the bottom here is a couple of these links that allow you to…

22:19…actually get the search results in different formats, right?

22:23So GeoRSS, like I described, or just a little HTML snippet.

22:28This is actually what I used on my home page, right? And I apply a little style sheet to it.

22:33You can put that on the URL, so that's fairly easy.

22:36And then you can render this in a nicer way than what you see here, but it's just a little HTML fragment that you get.

22:46And obviously, there's you can see KML and JSON, and you can do other things like that.

22:51Okay. So now let me show you this here.

22:54So one of the things that we have done in the geoportal is enable federated search.

22:59And this allows me to not only search my own catalog or the things that I have in my own catalog…

23:04…I have a couple thousand to maybe 20,000 or so things in my catalog - but I can also reach out to other sites.

23:11And what's interesting about this, what you see is that I can actually search ArcGIS.com for maybe geology…

23:20…right? and I get some results from my own site, and I also get some results from ArcGIS Online.

23:27And those things from ArcGIS Online I can then preview, open up in a viewer, right?

23:31So I can, we can sort of interact with ArcGIS Online through this federated search.

23:36Of course, that's easy, because that's ours, right, so we know that language.

23:42But we can also connect to, for example, YouTube. You might say, Why would you do that?

23:49Well, there might actually be situations where finding a little video clip about geology on YouTube might be interesting, right?

23:56So in this case I find some YouTube videos and I can play them, right?

24:00So the thing that you do with video is play, right, so that's what we do.

24:06So different kinds of things that you find will result in different things that you can do with that.

24:12And what we have done here is basically built sort of an extensible framework of the geoportal…

24:18…interacting with other search services and search catalogs, and they might speak Esri language…

24:24…the ArcGIS Online, the ArcGIS Portal API; they might speak OGC language, CSW for example…

24:31…or they might speak something completely different.

24:34Typically, open search is what we like because it's a fairly well-formed language or dialect of searching…

24:42…but it doesn't have to be. If it's XML, we know what to send, we know what to get…

24:47…and through some sort of transformations that we would apply, we can configure this.

24:52And you can add to this, right, so you can add to this on your own.

24:57So what we see here is a geoportal configuration file. When you scroll down, there's a bunch of things to configure.

25:03So let me find the YouTube one. So here, what you see, right, this is a YouTube repository.

25:12At the moment, this is how we configure it.

25:13You go to the geoportal configuration file, you add some XML to it, and then you have an additional search capability.

25:19We're going to move that into an environment where you can just do it through the web and configure that through the web.

25:25Something entirely trivial; I'll come back to that.

25:28So we basically have some classes that send the request that I entered, right, my search phrase…

25:34…and it translates that into something that YouTube understands.

25:38And, for example, what is important here is this particular link here, right?

25:42So YouTube wants to find your videos using a particular URL…

25:48…and they have a parameter called q, and that's where your searchTerms goes.

25:54And then you can do paging, of course, right…

25:56…so important thing to support if want to develop or add your own search provider, so to say, to this.

26:03Does it support paging? No one is benefiting from a 20,000 hit in one single page, right, so do it in chunks. People like that.

26:13And so this pattern, so to say, can be applied in many different ways…

26:17…and what happens when we get the results from that, we may actually have little sort of a profile.

26:23Right? So there's a little sort of back-and-forth that you need to do. Let's see here. Maybe looking at the different…

26:36There you go. There's the YouTube profile.

26:39Right? So this profile is sort of our understanding of how YouTube works.

26:44There's some transformation to transform the search criteria that you fill out on the page to this URL that I showed.

26:52Then once we get a result back, YouTube may come back in actually XML, right?

26:56So it's - YouTube comes back with an XML structure, maybe an RSS feed or something like that…

27:01…and then we read that using a transformation and we transform that to a structure that the geoportal understands, right?

27:08So we're sort of abstracting from the individual languages and dialects of XML or JSON or something from the search providers…

27:17…into sort of a common language that the Geoportal Server understands so that we can present search results.

27:22That's sort of a standard way of doing this, and you can extend it, your own.

27:26You can create those XSLTs on your own, and of course you can use any of the samples that are included there already…

27:32…and you can configure your own search interaction.

27:38So that's interesting, right? You can search sort of other catalogs like the World Bank projects…

27:45…and see if they have anything related to geology.

27:50Right? So sometimes what happens is the search provider doesn't actually tell you how many hits there were, right…

27:56…so that's fine; at least something came back.

27:59So here we get World Bank projects, and if I go there and I open this…

28:04What do you do with a project file? You actually open it.

28:07So in this case, I get to a page on the World Bank website where I just get a view…

28:11…of that particular project that had something to do with geology. So that's the idea.

28:15So the geoportal can really reach out to all sort of different catalogs in the back end to do that kind of searching.

28:22What you see actually right below the search box is not only a little feature layer.

28:26I've used the information that we have about the index, and that feature sort of tells me how many items I have…

28:34…in my catalog with a specific keyword.

28:36So in this case, geology, apparently I have 276 items in my catalog about geology, alright?

28:45So that sort of helps me understand a little bit, and as I type here - so maybe weather or so…

28:51…you can see that the list of things actually updates.

28:55So every time I type a character, it goes to sort of a different REST end point in the geoportal…

29:01…and it says, Tell me how many items I have in my catalog with a keyword that starts with that particular phrase.

29:10And then I can just click on that, and I get apparently three NOAA weather space program items.

29:17Right? So that sort of helps the user also to not just type in any word without any luck.

29:22They get a little hint, and they can use that.

29:28Alright, so that's great.

29:31So here we have the Flex Viewer. Let me just refresh that, because you never know; they sometimes go to sleep.

29:39And that same sort of Biota service that we had before, I can find that in my Flex Viewer.

29:46Right? So here's that Biota item; zoom to that, add to my map, voilà, right? Easy as pie.

29:54So the idea is that it's very simple to add multiple services.

29:57I can add, do another search, right? I can add more services to this; maybe Biota in other states surrounding Idaho…

30:05…and build up a map using multiple services and so on, right?

30:11So this other thing that I talked about was data download, and here I'm actually using the Oregon Spatial Data Library website.

30:18They make some interesting data available in the state of Oregon.

30:23And what they've done is they've organized that in a number of layers, so for example, this geology data.

30:30So I'll type in my e-mail address, right?

30:41And I choose the format that I may want to download this in and then choose the projection I want this in…

30:47…and using this little tool here, I can just draw an area - I want this layer.

30:54Send it off, get a little message; it won't actually get anywhere because that's not my e-mail address.

31:01And sort of that's the experience for an end user using the data download feature, right?

31:06And there's a couple sites out there that have implemented this; they've sort of tweaked this page.

31:10The North Carolina uses this to allow you to download imagery…

31:14…and they require you to zoom in to a smaller extent before you can actually download something.

31:19Right? So the idea is that you can sort of tweak these pages as you want to.

31:25Now the other thing that we have started to look at comes from the open government data sort of movement, right?

31:32If you're familiar with sort of the semantic web, semantic web like tries to sort of really understand sort of…

31:40…have machine-readable and understanding sort of the meaning of the datasets…

31:44…and the meaning of the descriptions across different systems, right?

31:47So it's a very sort of interesting idea here.

31:51And as part of our work in that area, Data.gov has a geospatial catalog that's built using the Geoportal Server…

31:59…and serves almost a million datasets, right?

32:02So that is the kind of catalog, well, that's great, but now I find this million, it becomes a Google, right?

32:07I have too many. I want to find the ones that are related and came from the same provider…

32:12…and that's where sort of linked data is starting to become interesting.

32:17So here we have built a simple SPARQL interface. SPARQL is the query language used in the semantic web very much.

32:24And that query language, that interface, works on my geoportal index.

32:28Right? So we can support things like SPARQL, and as with anything geospatial, it's still a little special, right?

32:35We can filter on the coordinates - and let me see if I can get more items, get a little more items.

32:43So what I've done here is basically configured a SPARQL query that is - well, you can see…

32:48…it's almost like a SQL query but not entirely, right?

32:50So I look for resources that have a title...

32:54...I look for resources that have a description, and I look for resources that have these coordinates.

33:01It's not really - this is not really sort of the WHERE clause. It really is assertions, right?

33:07So find me a resource that has this particular relationship with this particular value, right?

33:16And then find a resource that has a description relationship with a description, right?

33:21So it breaks down the index into things and relationships.

33:26Well, this looks like a spatial query, but it's not really, right?

33:29So what OGC is doing is they're extending SPARQL to include geospatial operations.

33:36So in this case I have modified my query a little bit to use the GeoSPARQL GIS within operation.

33:46So instead of just filtering on sort of individual these attributes now, we can also include these operations.

33:52So this is very much work in progress, and the whole sort of semantic web is, of course, still very much a work in progress…

34:00…but it's, I think, an interesting new sort of addition to this…

34:04…and it opens up the catalog for other sorts of integration with other systems.

34:10Another example that I wanted to show is this PivotViewerSample.

34:14So I talked about faceted search a little bit. I'll just reload that.

34:19And Microsoft has an interesting tool to support faceted search. It's this Pivot Viewer tool.

34:26And if you design your - and it works on a collection.

34:30And in this case, my collection is a set of a couple thousand metadata records that I exported using one of those end points, right?

34:38The export to CSV, and then from using Excel, there's a little tool you can generate a collection description…

34:42…and you can generate these images, and then I can use it in this Pivot Viewer.

34:47This Pivot Viewer allows me to look at my catalog in a kind of different way, right?

34:52So it's not searching; I'm not typing in keyword searches, although I could, but I won't.

34:57And it's not really browsing in terms of what I described at the department store, right?

35:02You don't start at the top and then you go down. You can sort of browse in multiple different ways.

35:07I can look at this collection in different ways.

35:11I can organize the things that I have by type, right?

35:15So as you can see, I have a lot of downloadable data.

35:18If I select that, I zoom in to that downloadable data piece, right, and then I can sort of navigate in different ways.

35:25I can say, Let's see where I have some data here, right?

35:29So apparently, looking at the west bounding coordinate in this case of my metadata, that is downloadable data.

35:37And as you can see, apparently I have a lot of data in the western hemisphere, right? The US; not surprising.

35:43And then there's a few data items here on the east side.

35:47I can sort of zoom to those by adjusting that little slider, right?

35:51So that's the idea of doing this kind of sort of interaction.

35:56And as soon as I make a change in one of these facets, the other facets also get updated.

36:05So let's limit a little further, right? So I can do things and then I can look at other categories…

36:13…and then, if I find something that I think is what I want, I can choose that, and from there, it's going to launch my geoportal.

36:21And I found one that's no longer there, so that's too bad…

36:24…but the idea is that you can actually connect back to a page that shows you details about that particular item, right?

36:31So it's sort of a new way of sort of looking into a catalog of things.

36:38Alright, so that's sort of a what can you do with the geoportal search, without a lot of coding, without changing it…

36:45…just using and thinking about the approaches that we have taken.

36:50Now another thing that I mentioned is open search.

36:53And Geoportal Server supports open search, and that brings up sort of a couple interesting options.

36:59So one of the things that you can do with open search is actually add it to your browser, right?

37:04So I can choose the Geo.Data.gov search here, and I'll search again for geology…

37:12…and right from my browser, I go and find content in Geo.Data.gov, right?

37:18So that's sort of an interesting sort of new sort of concept similar to Wikipedia and Google and others, right?

37:23You can do that with any Geoportal Server, and you can actually configure what happens when you do that search.

37:27In this case, I do that search and it shows me the search results page.

37:32If you want it to show up an HTML page with the items that you then style, you can do that too, right?

37:36So you have a lot of control about that.

37:39But this actually is even more interesting, this open search thing.

37:43My machine is a Windows 7 machine, although apparently, I need to replace it with a Mac…

37:49…and one of the things I can do with this is actually open the openSearchDescription for my Windows environment.

37:59Right? So I open this Windows - this OpenSearch file, and Microsoft Windows knows what to do with it.

38:05And it says, Do you really want to add this as a connector? Yeah, I do want that.

38:09So what it does is, actually, it adds geoportal search to my desktop, to my Windows machine.

38:16We will search for geology again; it finds geology items.

38:20In my local catalog, in this case, but like I said, any geoportal server supports this…

38:25…so you can search any Geoportal Server directly from your desktop without going to the website.

38:29And then when you do want to find something you get, in this case, to the metadata, again, that is configurable.

38:35You want to sort of go to a nicely styled page of this or you want to go to some other thing about this item…

38:41…you can do that using sort of configuration capabilities.

38:47Alright, so that's sort of the idea of a couple of the things that you can do with Geoportal, right?

38:52So open-ended on sort of connecting to sort of other source catalogs or other kinds of search providers…

38:59…and we're trying to make it open-ended on the front end also so that you can choose your favorite client applications…

39:04…or client development environments using the JSON flavor of the outputs and so on…

39:09…or just embedding a simple HTML piece or so.

39:14Alright, that's sort of what the Geoportal is and what you can do with that.

39:20What you've also heard a lot about over the last couple of days is ArcGIS Online and the ArcGIS Portal, right?

39:26And what I told you here might actually sort of create some confusion about that…

39:32…and we get a lot of questions from people on Give us some direction, right? So tell us where we need to go.

39:38Is it ArcGIS Portal or is it Geoportal? And if you have a little smork about it, then you can…

39:45This is actually a real…it's a real sign.

39:50After an 18-hour flight to Narita airport, nothing is better than a good smork, right?

39:58Well, the answer to this really is there's a time and place for everything, right…

40:02…whether you're Tweeting or whether you talk about ISO 1939 [sic] metadata…

40:09…with profiles and XML and schemas and Schematron and all that good stuff, right?

40:13So what this slide tries to explain is that there are sort of these different views of items, right?

40:20So one is the very simple view, the YouTube/Flickr view, right?

40:25That's I just put in some words, specification in Japan or wherever you went, and that's it…

40:32…and look at the picture and lets people sort of see the picture and they're interacting with that.

40:37Large user community; it's everyone basically.

40:41It's the non sort of specialist.

40:42It's everyone uses YouTube and everyone knows how to do it and how to find a video clip that's interesting.

40:48So at the other end of this spectrum is the people who really describe their content in large amounts of detail.

40:58And there's a very good reason for this.

41:00If you are a climate scientist and you create data and you manage to create a dataset now…

41:05…that dataset may be used in 10 or 20 years from now, right? Because your timeline is very long.

41:10It's not the 20-minute time span that people have on YouTube or so.

41:15So you really have to describe it very well.

41:17Or if you are an engineer and you use some sort of report that someone created…

41:23…you really want to make sure that this is an authoritative report with the proper credentials and so on, right?

41:30Or if you are a commercial entity and you want to build some commercial service around content that you find somewhere else…

41:36…you really want to make sure that you're not sort of getting into all kinds of lawsuits…

41:40…because you violated some license agreement or patent or something.

41:44Right? So there's a reason for good, verbose metadata. There is a reason for very brief metadata.

41:52I think one of the things to look at this is that ArcGIS Online sort of focuses on that simple stuff for now, and we'll come to that…

42:00…and Geoportal focuses on that sort of verbose, if you will, standardized.

42:06If you're in Europe, you have to create INSPIRE metadata. Sorry.

42:10There is sort of a legislation aspect to this as well, and a lot of the federal agencies here in the US…

42:15…they have to use FGDC metadata. There's no coming around that.

42:21So what we're working on, one of the things we're working on in Geoportal Server is really sort of bridge that gap…

42:26…sort of create the glue between ArcGIS Portal and its use cases and the Geoportal Server and its use cases, right?

42:35So how can we combine those two?

42:38Well, one of the things that I actually showed you already is searching ArcGIS portals from the Geoportal Server, right?

42:45ArcGIS Online is essentially an instance of ArcGIS Portal.

42:48I can search ArcGIS Online from my geoportal, and if you paid attention to that one sort of moment in my demo…

42:54…you saw the NOAA GeoPlatform and the EPA GeoPlatform; those are ArcGIS Portal implementations.

43:00I can search those as well from my geoportal. And so that's very simple integration here.

43:05What's, I think, more interesting is that ArcGIS Online, ArcGIS Portal, and ArcGIS Server 10.1…

43:10…will allow you to create and have verbose metadata, or full metadata, with any item in those systems.

43:20And what we do within Geoportal, what we have done sort of over the last couple years…

43:24…is create the editors that do standardized metadata.

43:28So you'll put one and one together, you get standardized metadata for ArcGIS Online items…

43:33…for ArcGIS Portal items, and for ArcGIS Server 10.1 items, whichever standard is your standard.

43:38And one of the things that we are trying to do is figure out…

43:40…how can the Geoportal Server contribute to that, using, for example, its editor?

43:45And then you can think of other things, right?

43:47So Geoportal supports OGC interfaces. Well, we can add that to ArcGIS Online or ArcGIS Portal.

43:54You can register a resource with the geoportal, and it'll harvest that over time and do that automatically…

43:59…weekly or monthly or daily or so, right?

44:02Well, we can actually push that content into ArcGIS Portal or ArcGIS Online so that you get a very simple sort of workflow…

44:08…in ArcGIS Online or ArcGIS Portal to update the content in those systems, right?

44:16So I think there is a place for both.

44:19Let me show you what I'm talking about here.

44:24So I showed you searching ArcGIS portals and ArcGIS Online from a geoportal, right, so…

44:31Let me just clear this here.

44:37Let me go to NOAA in ArcGIS.com, see what they have. Right?

44:44So apparently, NOAA has four items about geology, right?

44:50In this case, they don't really have anything specific about geology.

44:54It's the standard content that comes with ArcGIS Portal when they implement it.

44:58ArcGIS Online has a little more, and my geoportal has something, right?

45:01So I'm searching ArcGIS portals and ArcGIS Online from my geoportal…

45:06…without the user actually sort of needing to know how that works behind the scenes.

45:12Let me do the other, the opposite way, right?

45:15So what I have here is an ArcGIS Portal implementation on my machine here locally, and I changed it a bit.

45:22Like I said, I work in Professional Services; I do a project with the government of Indonesia.

45:28And let me do that geology search again.

45:36They'll come back with some results or not…

45:38…but what you see on the left-hand side there is for me an ability to search other catalogs.

45:45Right? So what we have done here is integrate my Geoportal federated search capability into this ArcGIS Online site.

45:55We'll not wait for that.

45:58I can get those results of geology, right? The 3 out of 25? Did anyone pay attention? Same number, right?

46:04Or find them in ArcGIS Online or find them in Data.gov, right?

46:12So the idea is that, using the Geoportal search capability, we can federate searches…

46:17…and integrate that into an ArcGIS Portal instance.

46:26So the other thing that I did here - let me sign in - what I talked about is adding metadata to items in ArcGIS portals, right?

46:38Standards-based metadata. So I'm connected here and I have some items.

46:47Let me see. Maybe I need to restart something here. Just to be sure.

47:16That's not nice. One second.

47:27So the idea is that we can actually add metadata.

47:29So while this is going with this ArcGIS Portal, let me show you what we do with metadata.

47:34And if we are good to go, then we can actually show that in ArcGIS Online.

47:45So I'm connecting to my Geoportal as a user, and in the Administration section…

47:50…we have some capabilities to upload and register content, like I've mentioned.

47:55So I can add a resource. I can do that using the URL to that resource, right, if I want to register a WMS service, for example…

48:02…or I can, if I have an XML file, I can upload that; or I can use an editor online and use that.

48:11So let me show the FGDC editor for here. As you can see, I have a number of different profiles and schemas configured.

48:16You can change that to only show the ones that you want to show.

48:21And so this is an editor that we have created over the last year or so that allows you to…

48:25…edit metadata in a standards form, extensible, and do that all online.

48:33So the idea is that, if you are familiar with any of these specifications, right, you can have sort of repeated elements…

48:38…and repeated sections of the metadata; they can be multilingual, and so on.

48:42You can configure all of that using these editors.

48:46And in FGDC, they have some particular values, like the Unknown value, and other things.

48:53And once I'm done with this, I can actually try to save this.

48:57What you see is that Geoportal simply validates the form that you tried to submit, and it does that here…

49:02…based on some mandatory elements that I've configured.

49:05So, for example, I do need to put in a title, and as soon as I type in a title, it's right below my screen here…

49:13…it says it's good to go, right?

49:16When I type in the abstract, and so I can save that.

49:19Well, I want to sort of move on with my demonstration, so I can save it as a draft…

49:24…which means that it's available for me as a user, but it's not discoverable, right?

49:29So that's sort of the idea.

49:32Now let's see if my portal here has woken up again.

49:55There you go.

49:57So I have a few items, and let me open up this BeeSpread item here.

50:03So this is a normal ArcGIS Portal item page, so sort of standard stuff here…

50:08…and the only thing that was added here is this Metadata button.

50:13And what this does, it opens up the metadata view, in this case, of this item.

50:19And if you want to view the XML, you can see this is actually FGDC metadata, right?

50:23Well, that's great; I can view metadata.

50:25This metadata is not stored in my geoportal, right?

50:27This metadata is stored in ArcGIS Portal with the item that I'm looking at.

50:33Want to edit this? Familiar editor, right?

50:36So we're accessing the geoportal editor from ArcGIS Portal, and I can update this title with more text, right?

50:47So and then I can then save this.

50:51As you can see, I can also create a new one. I can upload an XML file by importing it.

50:56I can actually download this XML file and do something else with this, right?

51:00I'll save that, and now go here. Right? So the idea is that, doing this…

51:10…I can edit standards-based metadata for any item in ArcGIS Online or ArcGIS Portal by fairly limited integration.

51:20So that's just sort of integration with Geoportal and ArcGIS Online or ArcGIS Portal…

51:27…searching ArcGIS Online from a geoportal, searching geoportals from ArcGIS Online, editing standardized metadata.

51:33And then last but not least here, let me go back to that Biota service…

51:38…because I also mentioned that we can connect to any viewer as long as we have a little handshake.

51:45And so what I've done here is, instead of using the Flex Viewer…

51:48…I configured my geoportal to use the ArcGIS Portal JavaScript-based viewer.

51:53Right? Simple handshake, configure, update, save, et cetera. There you go, right?

51:59So imagine, this could be the ArcGIS Online viewer itself, which would allow me to share…

52:05…a map service that I found in my local geoportal; I open it up in ArcGIS Online, its viewer…

52:10…and then from there, I can share it with others through ArcGIS Online.

52:14So sort of a fairly sort of simple way to share content from your local geoportal with ArcGIS Online users…

52:21…using this integrated viewer capability.

52:25Alright. Let's see, a little time check. I think we're good.

52:30So then what I have here is a little page that we created to sort of start prototyping…

52:38…the editing of metadata with ArcGIS Server services in 10.1.

52:43Right? So these are some URLs to a service. Let me sort of make sure that this service is available.

52:59Right? So it is a simple map service.

53:0110.1 running in Redlands in the office, and it has a number of new interfaces, if you will.

53:10Right? So for example, there is this info interface, and from there, I can get to the metadata.

53:16And in this case, we actually see it's FGDC metadata again. Right?

53:19So the ArcGIS Server item has FGDC metadata…

53:25…and using this little sort of tool, we can actually edit that metadata using the same editor as we have seen before.

53:34Right? View it in a styled way, edit it using the editor.

53:40So integration of Geoportal Server with ArcGIS Server at 10.1 allows you to create standards-based editing capability…

53:48…editing and standard-based metadata online, right? There is no…

53:53And obviously, in ArcGIS, you would typically edit your metadata in the desktop…

53:57…but there may be a reason to do this online as well or maybe change some elements online once the service is published.

54:04Well, this would allow you to do that.

54:15Alright. So some last words. I think I'm ahead of schedule, so I don't hear any questions…

54:22…so if there is questions afterwards, that's great.

54:25Last words, because you never know. You might get abducted.

54:32That's actually the more important feature on this page from July 4th.

54:37So what are we trying to do this year, right?

54:39So I told about some existing capabilities.

54:41I told a little bit about sort of SPARQL and the editing of ArcGIS metadata, and so on…

54:46…because those are obviously things we're still working on.

54:50One of the things that I'm trying to do with Geoportal is really involve the user community more.

54:54Right? We have a fairly active forum where people ask questions and other people answer those…

54:59…and sometimes my team answers those, and so on.

55:01We have people who submit patches, right.

55:04So for example, our distributor in Italy deserves a little bit of credit because he creates a lot of little enhancements…

55:11…little tweaks, and little things and then submits those as a patch to SourceForge where we have the Geoportal.

55:18And we can then sort of integrate those or users can download those and update their Geoportal…

55:26…like I did with that little Test link, right?

55:28So including a Test link like I did is a very simple update to some pages, and so on.

55:36Geoportal sort of has been in its current state for a couple years now; user interface hasn't really changed much and so on.

55:42I mean, we'd like to sort of get some feedback from users, and that's one of the reasons we set up this e-mail alias also…

55:49…to get a dialog going around that.

55:51And really sort of what we're trying to do is follow the Apache way of opening up the code base…

55:58…and allowing others, non-Esri people, to commit code to that.

56:03It's very scary for us, right? But we're learning. We're getting there.

56:07And rather than us just sort of dumping a code base over the fence, which is not really what we want to do.

56:16So that's hopefully something that you might be interested in supporting.

56:21The other way to sort of support or collaborate is, instead of coding, is just documenting, right?

56:27Create a sample or create some documentation.

56:31We have a wiki as part of our Geoportal project in SourceForge that we can allow you to edit and so on…

56:37…and we have some external editors for that also.

56:40And obviously, the forum. We see users who are experienced users…

56:45…they answer questions of people who are not so experienced, and they help each other, right?

56:49So that's great. That's actually what we want to get to.

56:54Alright. In the road map, version 1.2 is what we released in December.

57:00Before that last year, we had a few versions that…

57:03I mean, the editor like that I showed was sort of built up over a couple of versions last year.

57:09And in December, we released 1.2, which adds the ability to have a WMS service in OpenStreetMap…

57:15…as a basemap in the search window, right?

57:18That little window of the search map that I showed.

57:21We added some capabilities to have collections of content to it.

57:25So I want to have everything related to geology in an easily accessible collection…

57:32…maybe with its own REST end point, maybe with its own CSW end point…

57:35…so I can search just those geology-related items, regardless of where they came from, right?

57:40Regardless of individual user or catalog they were harvested from, and so on.

57:45This is actually being used in Geo.Data.gov at the moment…

57:49…to distinguish the formally blessed, Data.gov-approved content from everything else, right?

57:55So if you go to Geo.Data.gov and you do a search…

57:58…you will find maybe half a million items or so out of the one million that's there…

58:03…and that is because the other half has not gone through the official open government, GSA approval process…

58:10…of being vetted and being verified for release, and so on.

58:14Right, so there's…they use the collection feature to support that, to implement that.

58:21MySQL support, right? So ArcGIS Portal, if you've implemented that, it runs in MySQL.

58:26If you want to have Geoportal running with it, you could actually do the geoportal database in MySQL also…

58:31…or you can use Postgres or some other database if you want to.

58:35We added Schematron validation, right?

58:38So the validation that I showed in the editor is fairly simple; this field is mandatory, and this field is optional, right?

58:45So that is the distinction there.

58:47We also do schema, schema validation itself, so if the schema says that something is mandatory…

58:52…then you need to have that.

58:54And then there is Schematron validation as sort of the next level up that is more sophisticated validation.

59:00If this element is filled out with this value, then this needs to be such and so on.

59:04So there's a sort of three levels of validation there that you could do if you wanted to.

59:08We added THREDDS harvesting.

59:10THREDDS is a real-time data, sort of database, that has large amounts of very…

59:16…sort of multidimensional, scientific data typically, right?

59:19But all the datasets are accessible through ISO metadata, and they have their own sort of particular interface.

59:26Well, we can harvest those catalogs now also and give you some view into this scientific content.

59:35So that was available now; it's available since December, and as you can see…

59:40…the ArcGIS Online integration, the ArcGIS Server integration are really sort of at the top of my list, of our list…

59:47…because we see many projects and users who implement both Geoportal…

59:52For example, NOAA. NOAA has a number of data centers, and each of those data centers uses Geoportal Server…

59:58…and then they have this higher-level, NOAA-wide platform implementation of ArcGIS Portal, and now they're sort of…

1:00:06…we're figuring out with them, using Geoportal Server capabilities of harvesting and pushing things…

1:00:11…into Catalog and other catalogs, and so on…

1:00:13…how can we streamline that process of getting content from these individual nodes…

1:00:17…if you will, into this master catalog and from the master catalog out to GeoPlatform, GEO, WMO, et cetera, right?

1:00:27So there's - on the consuming side, there are a lot of different sort of communities and organizations…

1:00:32…that want to get access to the content, and that's where I think Geoportal plays a key role.

1:00:37We're looking at sort of enhancing the structure of the catalog if you want to, right?

1:00:41So it's almost going a little bit deeper even in that pyramid, right?

1:00:46More structure, more structured content.

1:00:48And sometimes, people want that.

1:00:50There's a catalog information model approach from OASIS, ebRIM, and OGC has adopted that…

1:01:00…to sort of add some geospatial components to that. It's one flavor of OGC catalog that we want to support.

1:01:07But then there's also things like Feature Data Dictionaries, where you can be very precise and very verbose…

1:01:12…in the describing of individual feature types and their definition and where did the definition come from, and so on, right?

1:01:18In some situations, that is very, very important.

1:01:22Last, but not least, WMTS support, SOS support, WPS support, and so on.

1:01:27It's different kinds or additional OGC specs that come out over time, we'll just keep up with that.

1:01:33And then the faceted search like I showed…

1:01:34…I want to sort of figure out something that we can do with that in the Geoportal…

1:01:38…whether it's using Microsoft Pivot or whether it's using Apache Solr, which is a fairly popular faceted search tool.

1:01:46So look for that, right? In addition to the standard search and the browsing capabilities.

1:01:53Alright, questions? We have some time for questions. Question?

1:02:00[Inaudible audience question]

1:02:07Oh, very good question. So if you search things like YouTube or Flickr or other sites, what about the map?

1:02:14Well, actually more and more, those sites have some sort of spatial information, right?

1:02:20Your photos are geotagged or maybe your video's even geotagged, and more and more…

1:02:24…the interfaces that those sites make available expose that information, right, coming back.

1:02:29Doing a spatial search, as in a spatial constraint on the search, is not there yet in many of those sites…

1:02:35…but at least, when you get a response back, you see where the photo was taken or where the video was taken. Yeah.

1:02:44And for configuring that in the Geoportal, it's essentially just a matter of updating that…

1:02:50…typically, that XSLT that parses the response and says, oh, now there is a spatial extent, right?

1:02:56So add it to your XSLT and then the Geoportal will use it.

1:02:59Question there, sorry.

1:03:00[Inaudible audience question]

1:03:06You can find it here at the moment. Yeah.

1:03:13Any more questions here? So let me show you where you want to go.

1:03:19If you want to get more information in general, go to geoportal.sourceforge.net or go to…

1:03:44Let me see if that is the right link.

1:04:02Go to esriurl.com/geoportalserver, and that'll take you to the SourceForge page also.