00:01Welcome to this presentation, "Back to the Future."

00:05We like to tell you something about our adventures traveling in time using ArcGIS instead of a DeLorean.

00:16My name is Marc Hoogerwerf. I work with Royal Haskoning Engineering and Consultancy Company, and this is...

00:23...Maarten Tromp, my copresenter, and he works for the Dutch Ministry of Economic Affairs, Agriculture, and Innovation.

00:32It's quite a mixed bag of civil servants.

00:38Okay, an outline. I will be doing some of the - well, providing you with some of the background information...

00:46...on our project and on time travel, and then later on, Maarten will take over and do some demonstrations...

00:54...and maybe we see some codes and see what we did and then have some time for questions and answers.

01:07So to provide you with some background on the crop parcel registry in the Netherlands, well, first of all...

01:17...the Netherlands in Europe, if we transpose ourselves to this area, this is about the size of our whole country.

01:28And there is about 70 million of people of us living there, and most of us are dwelling in urban areas...

01:39...so a large part of our country is still agricultural land, about 70 percent or so.

01:48And as you can see here, in the dark areas, that's where we live, and the lighter areas is where we grow our crops and stuff.

01:58We have a crop parcel registry in place since 2001, and we use it, well, in order to comply to some European directive...

02:08...and also as a fundament for the payment of European agricultural subsidies.

02:21In this system, there are about 800,000 current parcels, and we have an archive of about 11 million parcels.

02:32And our farmers use an online system to enter their parcels and enter the crops they grow on those parcels...

02:41...and they have to do that every year.

02:45In a six-week period starting from April, they start filing their applications, and as you can see here...

02:53...towards the deadline, we get more load on our system, about 5,000 farmers a day doing their applications online.

03:08Another interesting thing, in this chart is also the dips in the graph; these represent Sundays.

03:17Farmers don't like to fill in applications on Sundays, but as you see, when the deadline approaches, more of them start doing that.

03:28So what are the challenges in a system like this?

03:32Well, first of all, accountability.

03:36Because the system is being used to pay subsidies, it is also being audited, so an auditor will come in and ask...

03:46...Okay, you paid this farmer this much of subsidy; well, how did you come to this amount?

03:58And we have to do it now but also in the past because auditors can also audit stuff that happened in the past.

04:07Then there's the question of consistency in space and time.

04:12Parcels should not overlap the parcels of the neighbor, and they should be consistent in time...

04:17...that when the farmer stops growing one crop and plants another, they should be consistent as well.

04:26And of course, we have an issue with the quality control of our input data, because we have 70,000...

04:35...well, non-GIS editors entering sometimes complex geometries into that system.

04:47So our proof of concepts. The system has been in place since 2001, but now it's up for a redesign.

05:02And in preparing this redesign, we are conducting a series of proof of concept studies...

05:12...to see if we can meet those challenges I just mentioned before with out-of-the-box functionality...

05:19...or at least a minimum of customized tools and using the ArcGIS frameworks to do that.

05:29Because over the past 11 years, we've been building lots of custom tools, mixing and matching...

05:39...all the technologies available at that time, so we used some Esri tools but also from other technologies and platforms...

05:49...for validation, workflow management, and editing...

05:52...and doing that with different solutions on the desktop, on the web, and in the database environment.

06:04And, well, what we found, that we have been reinventing the square wheel several times...

06:11...coming up with less-than-optimal solutions, and one of the things in the proof of concepts here...

06:17...is that we try to think about this again and see if we can use, well, the round wheels...

06:24...that are available in ArcGIS at the moment and the frameworks that support.

06:34A bit about time travel. We have a formal information model in the Netherlands that defines two types of time...

06:46...material time and formal time, and material time being the time that records stuff and, well, what's out there.

06:57What was on this piece of land between April 2004 and October 2004?

07:04And the formal time being what did the geodatabase know at a certain point of time of what happened at that time?

07:15So two different time concepts that are in this information model.

07:22And as you can see here, in order to apply some changes in your database, recording those changes...

07:32...takes quite a few steps because you have to record those changes in crop growing from winter to summer corn...

07:43...to potatoes to barley to onions in several steps, stopping and ending the formal and material time.

07:57So what we did is, well, we looked at what kind of time support there is in ArcGIS 10.1, and there are several mechanisms.

08:09One, editor tracking, where we can record who created an object or feature and when it was last edited and by whom.

08:20Geodatabase archiving, a mechanism to record this notion of formal time in an archive...

08:29...and time-enabled layers that will allow you to travel through material time and through formal time.

08:38And of course, in other extensions like the Workflow Manager, there are even more concepts of time and recording time.

08:47But we'll focus on these ones.

08:52Then if you want to implement this time in a geodatabase, we found out, well, we have two options.

09:00One, I just call it traditional option, and it's traditional in the sense of, you know, when we all used Workstation ArcInfo...

09:11...and an occurrence at a certain point in time, we had one dataset with nonoverlapping features.

09:20And the other option is the heap option, and this is something that is now in place in this crop parcel registry.

09:29We just keep adding parcels one on top of the other...

09:34...and meanwhile, keeping the history of those parcels in our geodatabase archive.

09:49So to help us in our proof of concept, we developed this user story about two farmers...

09:55...Wallace and Gromit, and the type of changes they recorded in the crop parcel registry.

10:03So first, we started out with nothing, then there's this Farmer Wallace who grows some corn in 2004.

10:13Then he decides to build himself a new farmhouse, so the crop parcel gets changed.

10:21He's still growing corn then switches over to summer corn and decides to sell part of his parcel to Gromit.

10:35Gromit starts a new crop, and at a later point in time, Wallace decides to buy this parcel back from Gromit...

10:48...and a year later, changes it into one-crop parcel with potatoes and then, in the end, decides to have some onions.

10:59So this is fairly straightforward, but our system also has to support another notion, is that, well...

11:07...what happens when we find out that, instead of winter corn in 2004, it was actually barley that was grown on this crop parcel?

11:23And as you can imagine, in this option 1, where we have only one parcel, you have to go out to the archive...

11:33...get your data, change it, and then store it again.

11:37And in the heap option, you just go to the correct parcel in time and then change whatever was there.

11:46And another one is, well, Wallace made a mistake or someone made a mistake and there was no application in 2005...

11:58...so we also have to be able to delete some stuff that happened in the past.

12:05And these were the bases for what we did with time.

12:11So, and with that, I'll turn it to Maarten.

12:15Thank you. I will show you some things about what you can do when you have this heap of parcels sitting in your database...

12:30...and you want to visualize that. That's my first demo.

12:35So I just show the basic functionality in 10.1; actually, this was there in 10.0 already, so I'll start with that.

12:45If you have any questions, you can just ask, and we can have a discussion. I think we have some time.

12:54So I will open up ArcMap. I have my feature class sitting there, and the feature class has certain fields...

13:05...and this material time is modeled in two columns. It's validfrom and validto.

13:15So what you can do, if you have those two columns, you can go into the Time tab, which is there since, like I said, 10.0...

13:26...and you can enable time on that specific layer and fill out the start and end times, or the validfrom and the validto fields.

13:41And ArcMap, ArcGIS for Desktop comes with a time slider, and with that time slider, you can look at those two columns...

13:55...and it's basically a query definition that takes place in the background.

14:00And right now, I'm looking at the year of 2000.

14:03Like Marc said, we started recording all the parcels starting in about the year 2000...

14:12...so that's why, basically, we see those three parcels in that first year of recording.

14:20If I switch to 2001, we get the parcels that were there in 2001, well, et cetera. You get it.

14:29So you can go through time and look at the parcels valid in that specific period of time.

14:38And the nice thing is the services of ArcGIS Server are also time enabled.

14:45So if you take this map, this MXD, and you publish it to your ArcGIS Server...

14:52...this notion of time is also implemented in the services.

14:58So that's what I did here, so I published the MXD as a map service.

15:07I switched on the capability also of WMS, the OpenGIS standard, and the service that's now there...

15:17...because the MXD was time enabled, has also time-enabled functionality.

15:25The next thing I did was I went into the samples of the ArcGIS API for JavaScript, and there's a dynamic layer example...

15:40...and underneath here you have this slider bar, and I think this is about gas fields. Whoops.

15:49Well, I took that sample and used that to visualize my crop in using the time steps.

16:01So we look at the same area of the Netherlands; it's the island of Texel.

16:06And again, under here, you can see that we're in the year 2000, and if you look very well...

16:13...you can see those three parcels that we just saw in ArcGIS Desktop.

16:19And the API also provides you with a slider, and you can go through time.

16:26And we're now in the year of 2001, 2002, and with every step...

16:34...it shows you the parcels that we have in our database for that specific year.

16:43Like I said, I also switched on the WMS capabilities, and what I just did during the plenary...

16:56It was a good plenary, but I was also a bit thinking about this presentation.

17:02And we have in the Netherlands quite a large group that uses OpenGIS and open source software.

17:12So I looked at a sample of WMS-T, so time-enabled WMS, and I thought, Well, I have 20 minutes...

17:20...so maybe I can turn this into a nice sample I can use.

17:26And again, I zoom in to the island of Texel.

17:32This, on top, it says the year 2000, and again, we see a few of those crops that were there.

17:39And it doesn't come with a nice slider bar, so I have to type it in. I go to the year 2001, 2002.

17:49Well, basically also, this OpenGIS-enabled WMS layer is capable of requesting the time from our ArcGIS Server.

18:06Any questions about this part? Okay.

18:12Then I switch over to the coding part. This is more about how to build the formal and material time into the database.

18:28This [unintelligible] 3616, how do you - how can you capture that formal and material time in the geodatabase?

18:40And there are some challenges to record that time, especially the material time.

18:50The formal time with the editor tracking, you can basically track when a parcel is formally changed in your database...

18:59...so that helps tracking the formal time, but with the material time, there's a challenge how to build that history.

19:11So what you can do is customize your application, your JavaScript API application, your Flex application...

19:21...and build custom logic to build the material time in the correct way.

19:29But that's not a very nice solution because that's for - you build a solution for that one client.

19:37If another client comes in and is starting editing your data, your data gets corrupted...

19:43...because you don't have the code running that's building your material time correctly.

19:51So what we did was, looking at the possibilities in ArcObjects to interfere with what is happening while storing the data.

20:08In Desktop, you can build an Editor extension, but then, again, it's solved in your ArcGIS for Desktop.

20:17What we did was looking at the feature class extension possibilities...

20:22...and with 10.0, there was a TimeStamper class extension sample...

20:36...which is basically what is now implemented in 10.1 as being the editor tracking capability.

20:46So we took that sample and looked at ways to correct the material time correctly.

20:59So it basically does - let me see, how can I...

21:10I implemented two things in this extension, which is I want to track in which version of the database the edit has taken place...

21:21...so in that way, you can, like, separate web editors, mobile editors...

21:26...and in the changed feature, it will capture the type of editor that did that specific change.

21:36That's one feature I implemented.

21:39And the other one is related to building the material history, and what we want to do there...

21:53We can dive into the code if we have some time, but I will show what is happening in the desktop version.

22:08So what we have here are three parcels, which those parcels have no end date on the validto date...

22:20...which actually means that the crop that is actually listed here is there in present time.

22:28What we now want to do is to create a new parcel on top of that starting on a certain validfrom date...

22:38...and what should happen is the parcels that are sitting underneath the new parcel should have a validto date, should be ended.

22:49So the validto date should be the same as the validfrom date of the new parcel.

22:55And that's basically the functionality that we programmed in this feature class extension.

23:01So I can start editing and wait a little bit. It's all sitting on my laptop, so it sometimes needs some time to spin up.

23:30Okay. So I pick a new type of crop and I start tracing, just use the edit functionality of Desktop...

23:48...and make myself a nice new parcel. (I didn't see that one. Ah! Bummer!)

24:13We did screw up last night, I guess. We tried to fix something.

24:41Do we have time? Well, not too much.

24:57Just try to run it from Visual Studio, see if that works.

25:11I'll continue, I'll wait...no. I'll show you the other demo.

25:23Basically, if this feature class extension is sitting also on the server and you are editing through the REST API...

25:33...the same behavior should happen when editing via another API.

25:44And that's the strength of this solution.

25:46You build it in a place, and you just keep using your normal APIs to do the editing...

25:53...and the building of the material time is happening automatically.

25:58So hopefully this one works fine.

26:04I go into the study area. I think my computer is about to die. Okay.

26:35So I'm looking at parcels. Because we have this stack of parcels sitting on top of each other...

26:43...we have a lot of parcels that have been valid in the past.

26:49So I click through them, and in the end, I will find parcels that have a validto date which is not filled in.

26:59So when I'm going to draw a parcel on top of this one, this specific one, the maize one should be closed.

27:09The validto date should be filled in, so hopefully - let's cross fingers. Yep.

27:17And I draw myself a new parcel on top, I fill in the validfrom date, so this new parcel is valid from this date...

27:36...so the parcels underneath there - I should have drawn a better parcel, really, on top of the previous one, but this will work.

27:48I'll zoom out and in again a little bit because the experience was that the caching kicks in and we won't see any changes.

28:03So I do the identify again on the same area, go through the parcels that sit at that location, and I will find myself right here.

28:21This was the parcel I just showed which had a validto date, which was null.

28:27Because there's a new parcel sitting on top, the feature class extension kicked in at the moment I filled in a new validfrom date.

28:37And that date is taken and put into the validto date of the underlying parcels.

28:45This is another one where we see the new validto date.

28:52And this is the new parcel, and we have a little bug because it says 1970.

29:00And that's behavior we see in the web application and not in the ArcGIS Desktop application...

29:08...so we have to figure out where the null and the zero are confused with each other and where that error sits.

29:19But I think the message is clear.

29:23Draw a new parcel, put a validfrom date; the underlying parcels that have an open validto date are changed.

29:35So in conclusion, I think we can say that in our proof of concept, we've shown that we can use standard functionality from ArcGIS...

29:45...and we do have to do some customization, but use the existing frameworks to get things done...

29:54...instead of, well, reinventing our own square wheels again. And, well, that was it, and thank you for your attention.