00:01Source code and build management is a lot of what software configuration management is, but there's really more to it than that.

00:09And I'm going to cover a lot more than just source code and build management in this talk.

00:16First thing I want to talk about is the fact that there's a lot of developers and a lot of people that are responsible...

00:25...for creating products simply look at development and creating products.

00:32I should also say that all the PowerPoints are going to end up being online anyway...

00:36...so we upload them to a site and they create a Postscript file out of it that anybody can download...

00:42...so don't worry about keeping track of everything. But again, so, developers in creating product usually is this line...

00:53...and the reason I've created this graphic this year is that there's a lot more teams and a lot more things...

01:01...that you need to think about when you create products, and things that you need to factor into your processing in order...

01:07...to make sure that all the different powers that be and all the required teams have a say in how the product gets created.

01:17So for example, when a research project starts, often they don't worry about anything but just getting research done.

01:24And so they're not worried about localization; they're not worried about setups.

01:28They're not worried about what the legal department's going to think about; the third-party dependency that they got...

01:34...which is going to cost $50,000 to ship to each user. I mean, they just don't think that way.

01:38They think, I've got a function that I need to create, and I don't worry about the rest of it.

01:42So one of the things that we've done over time at Esri is that we make sure that people who are starting research...

01:48...are thinking about more than just creating a function, because there's a lot other things that go into making it fit within the product than just creating a function.

02:01So again, this slide is there as an idea to keep in mind as we're going through the presentation that...

02:07...just don't think about development and creating a product. There's all kinds of other things that go into doing that.

02:13And as a source code manager and as a build manager...

02:17...you need to think about as you're ramping up projects and getting them into your mainstream.

02:26Esri development--I've been at Esri for 27 years, and when I first got to Esri, there were six programmers in a room...

02:34...and the only thing that we really normalized was we all went to coffee at the same time.

02:39And the only thing that automated, really, was that we all had to go to the bathroom at the same time...

02:42...about 20 minutes later. Other than that, we didn't do normalization and automation of just about anything.

02:50So over time, you know, 27 years, we've gone through, you know, PIOS; we've gone through ArcInfo.

02:55We've gone through ArcView; and we've gone through PC ARC/INFO; we've gone through ArcGIS...

02:59...in all of its different flavors of ArcGIS, 'cause it's this big piece now that has all kinds of stuff.

03:04But at this point, there's about 700 PEs and devs that are working within our code base on the ArcGIS product.

03:13And again, I'm going to stress ArcGIS, I'm going to stress Windows a little bit in this presentation...

03:17...but I'm going to try to cover as many of the other platforms and other products as I can.

03:23On a daily basis within the ArcGIS code base, 300 people are logged in to access the code base.

03:29So it's a big thing, and a lot of people are working on it. So, you know, 8,600 files are used within the build process.

03:40We build 1,100 projects every day through 150 solution files. And the process from labeling the code...

03:50...the code base, until publishing a setup for everybody to use takes about five hours.

03:56So when you think about how big it is, five hours isn't really that slow, but it takes some time to get a lot of work done.

04:05The other thing to keep in mind about where I come from and what's going on is that we released 9.0 in 2004...

04:14...actually, about a week and a half ago in 2004, we had our last build.

04:20We have more than doubled in size in the code base over that time, but we have kept the same kind of structures...

04:27...and a lot of the same paradigms. We've improved some things, but how we organize our source code...

04:32...and how we go about doing a lot of the stuff really hasn't changed from back then.

04:38So the idea here is that as you scale, you know, source code's going to grow, and you've got to plan for it.

04:48So that's sort of background. What I want to talk about in this presentation is organizing source code...

04:55...how we break it up and how we make sure that it's legible or navigatable.

05:01I'm going to talk about some coding ideas and some standards and how to come up with some coding standards.

05:08I'll talk about how we manage version control, both in the main build and the mainline and how we think about...

05:15...creating branches that do development to releases and things like that.

05:20I'll also talk about build management, and build management sort of has two different phases.

05:24One is build scripting, which is the--each different, you know, project getting built and copied to a certain location...

05:31...so that we can create setups. And the other's build automation, which is, how do we fire a build...

05:35...when do we fire a build, why do we fire a build--why is it important to have a daily build, et cetera. And finally...

05:42...I hope to spend quite a bit of time--I don't know how much--on how we manage and maintain our information...

05:50...'cause that's one of the most important things is, as new developers come into your team or...

05:55...as new functionality gets added into the code base, how do you train people and teach people in your organization...

06:04...how to code and how to do their daily jobs.

06:07So a lot of that is in our Managing and Maintaining Information website.

06:14So 86,000 files in our build process, so we have to break it up.

06:22What we've done within our code base is created a number of high-level--I like to call them team folders...

06:28...so the map team has a folder and the geoprocessing team has a folder, the 3D team has a folder...

06:35...the geodatabase team has a folder. And each one of those high-level folders contains a number of subfunctional folders.

06:44The geodatabase might have an ArcSDE database, so it has its own folder, or a shapefile would have its own folder.

06:51Map has different folders for the different facets within the map.

06:57The key thing about one of those folders is that folder contains a solution, and that folder contains project folders.

07:04And the project folders are where the VCproj files are, the [unintelligible] files, and all the project-specific source code.

07:12If a functional folder needs to reference source code which is in another functional folder...

07:20... we automatically place that within a common resource source folder, so we have a shared ArcGIS source folder.

07:28The reason why that's important is that by having these folder structures...

07:33...a team can simply check out its high-level folder, or a subfunctional folder team can...

07:39...check out a folder which has a solution, and then the shared source can get checked out.

07:43And then a developer can do all their work without checking out the other 80,000 files within the code base.

07:49So they have 6,000 files instead of 80,000. So it makes developers--it makes it easier for developers to get the job done.

07:57So the stuff that's in the shared is cross-functional dependencies, header files...

08:01...things that allow the build to be maintained. I'll talk about some of those in a minute.

08:05And then any external dependencies, things that we don't build within the build...

08:09...but we have the libs and the header files and things like that for available--for when the build occurs.

08:15Another very important thing that we've sort of mandated with the code base...

08:19...although there's a couple of rules that have been broken--is this.

08:22When you create files and folders and high-level folders, make sure you use the real names for what those things are.

08:30Call it Map. Call it Geodatabase. Call it Geoprocessing.

08:33Don't call it Moe, Curly, and Larry, because eight or nine years from now, you're going to get some new guy...

08:39...coming in, and he's going to go, "What's Moe?" And you've got to show him this map. It's like, no, it's called Map.

08:46The other thing about that is that when you create functional folders and project folders, they have a name.

08:52Make sure that whatever that name is is the same name that you put on the VC project.

08:58And make sure that whatever the VC project builds has the same name as the folder.

09:03So now you've got all these things that are maintained the same way.

09:06Years ago, we'd have a VC project that was called one thing and then the output would be something else...

09:11...and then we'd have--debug would be different than release, and it would just get--it just gets all messy.

09:17So the idea again here is that you want to divide it; you want to normalize it and--as things get added--own it.

09:25Don't let things get out of whack. The other thing to think about besides the source code...

09:30...is that there a lot of nonbuild resources that you need to maintain.

09:35Make sure, as you're putting things together, you think about those things. Within ArcGIS...

09:40...there's projection files and simple files and printer files and all kinds of support files that go into the product.

09:48Think about that folder structure, how that gets laid out when the setup occurs, and it'll make your life a lot easier...

09:56...down the road when you're actually putting your projects together.

10:02So here is that high-level team folder structure along with the functional folders.

10:08This actually is last year's slide, so there are some folders that no longer exist on here, 'cause we've cleaned up some stuff.

10:15And there are some new folders. But the idea here is that, you can see, for example, a 3D Analyst folder is the one...

10:20...in the upper left-hand corner; there's a 3D Analyst functional folder, which contains all of our low-level 3D code...

10:27...and a 3D Analyst GUI folder, which is our UI portion, because we actually break up the UI from the Engine portion...

10:34...because the Engine portion of our source code has to run on multiple platforms, not just Windows...

10:39...whereas the UI just needs to run on Windows. So we do that for a good reason.

10:44And then ArcScene is part of the 3D team's sort of area of expertise...

10:48...but ArcScene is an independent application that runs.

10:53You'll also notice, I have put the very obvious red circle around Shared ArcGIS. So if I'm on the 3D team...

10:58...I can pull the 3D Analyst folder, get it out of the code base, and I can pull the shared ArcGIS folder and I can work...

11:05...and I don't have to pull any other folders within the code base.

11:10So my analogy right now is going to be LEGOs. This a big box full of LEGOs that are all sitting in there.

11:20They don't really have a purpose or a place within the system.

11:25And this is the LEGOs built as the Millennium Falcon X-Wing Fighter.

11:35This is the diagram about how all of our source code fits together. It's our dependency diagram.

11:43It shows each one of the boxes that you see in this diagram are going to--it's hard to see 'cause there's so many boxes...

11:48...and I'll zoom in in a second on a subsequent slide. But each one of these boxes is a solution...

11:54...and in there, it lists the binaries that that solution builds. The lines designate dependencies within the system.

12:02So in order to have a lower--a box that has a line connected to it, in order for it to build, it needs some libs or some OLBs...

12:11...or something for the box below it. So there's definitely a build order within our system.

12:18There's two diagrams. One is that Engine diagram that I talked about that we built on multiple platforms...

12:23...and the other one is the apps--the applications folder; that's sort of the framework UI and part of the system.

12:30The other thing that you'll notice on here is there's a number of green boxes and yellow boxes.

12:35Those green boxes and yellow boxes--the green boxes are external dependencies that we have...

12:40...like Visual Studio runtime, SQL Server--things like that. Things in the yellow boxes are items...

12:48...that aren't part of the build but need to be in the setup so that the product will run at runtime.

12:55So those are like protection files and--like I said before--nonbuild resources.

13:02So if we zoom in, we can see now some more information about what's in those boxes.

13:08We can see what's in it. So the top one you see is the team folder map, and specifically, it's the ArcMap folder...

13:16...so that builds arcmap.exe. We build an OLB in that.

13:19We build a context sensitive help file in that, and we build some .NET applications within that.

13:26And we also depend on some external dependencies.

13:29We also see that it's the twelfth solution that builds within the build order...

13:34...and there are things below it that need to build first.

13:38We also note in the upper left-hand corner, there are some certain people that own this functional area.

13:46The name outside the box is the development lead for that, so if anything ever goes wrong or there's an error...

13:52...a build error or something, that's the person that you potentially would go to.

13:55The name on the inside of the box is the lead PE, product engineer. So those two people own that...

14:02...and you can get almost any information out of that. So what this really gives us is an idea about how we're organized...

14:09...how we're structured; if we need to make any changes to our system, if we need to add any functionality in our system...

14:15...we go to these diagrams and look at them and we say, you know, Here's a new feature...

14:19...does it go into an existing functional area, and if it does...

14:23...does it need any additional dependencies that currently aren't in the existing dependency diagram?

14:31Or it just needs so much other stuff that we want to make it its own, new functional area?

14:37So we use the diagrams quite a bit as we're adding new features and functions to the system.

14:55Okay, now we're getting into some build things. I wanted to talk about VS prop files...

15:00...and this was the best place to put it within the presentation. So it doesn't fit with the previous one...

15:04...but it's very good information for normalizing how you build software. So for those of you that don't know...

15:12...VS prop files are kind of INIT files that Visual Studio project can reference, that can set specific settings...

15:21...and then multiple projects can use that same VS prop file in order to set global settings.

15:28So what ends up happening is that if you ever need to change the output locations...

15:33...you don't need to go into every single Visual Studio project--you know, 1,100, 1,500, or whatever they were of them.

15:38You can go just to the VS prop files and make the changes there.

15:42So we do things like the output location for the DLLs and the EXEs; where the libs go; interops...

15:49...they go in the .NET folder. All the intermediate output goes into a top-level OBJ folder...

15:56...and why that is kind of important is that if we ever want to rebuild a system, we don't have to run clean, rebuild...

16:01...clean, rebuild, clean, rebuild. We simply go up to the top level and delete the OBJ folder...

16:06...and it deletes all the intermediate files in any [unintelligible] build.

16:13Preprocessor definitions are in there. Levels of optimization are in there.

16:18Some other key things to know about the VS prop files and how we create them is...

16:23...we have an independent one for debug and release.

16:26We have one for static libs. We have one for EXEs. We have one for DLLs. So again...

16:33...each type has some specific settings, and you create one VS prop file for each kind of project that you have in your system.

16:42We also run a number of JavaScripts, and I've built some Java projects, and we use Ant and Maven for that.

16:50And Ant has something similar to a VS prop file. It's called a common.properties file...

16:55...that you can have your Ant scripts basically get the same kind of information that normalized all of your output...

17:01...and build locations by doing that instead of having to go into every single Ant script...

17:06...and change output locations and input locations.

17:14So, rules for source code--this is actually how to make it easy to train devs and maintain your source code.

17:22One thing that Esri is definitely guilty of is coming up with a coding standard. I'm firmly against that in this day and age.

17:32There are so many coding style guides and coding conventions out in the public now, you can find one that matches...

17:39...and almost maps exactly onto how you want your system to be coded--you know...

17:45...how you invent different variable names, all that kind of stuff--has already been documented and is available.

17:52You don't have to write it yourself. The other great thing about that is...

17:55...there's somebody behind that website whose job it is to keep that document up-to-date. So if new things come out...

18:01...usually the person that owns that web page or style guide will also be updating it. We have quite a bit...

18:09...of internal documentation and we've had programmers spend hours and hours and hours and hours...

18:14...updating documentation at the time to try to define what our coding standard is. Find one out there that's available.

18:21The other thing is--and we don't do this enough because our source code has gotten kind of big...

18:27...and we'd actually be spending a lot of time reformatting lots of our files, but if you're starting off...

18:36...it's a very good idea to run a style cop, which is a Windows plug-in, a Visual Studio plug-in...

18:42...or one of those 50 Lint-ish applications out there to have a consistency within your coding style.

18:51I also suggest running Doxygen. It's something that takes maybe 10 minutes to run the first time you ever run it...

18:59...and once you run it, your eyes kind of open, and it's like, oh, it's really easy to document...

19:05...and it's easy to add a header that adds some information.

19:08And Doxygen is something that if you start running it and building it daily, over time...

19:13...people get interested in it or if they have some available cycles, they can add information headers.

19:18And you eventually have almost a training guide for your devs on your software system.

19:24So I highly recommend using Doxygen. And finally, this isn't so much real resource code...

19:30...but it's very much how do you train new people coming to your system.

19:36And that is, have a document available that defines how to set up the machine...

19:40...and everybody should set up the machine the same way or as close to the same way as possible.

19:44What version of the operating system do you install? What version of the compiler do you install?

19:49What version of Doxygen do you install if you're running Doxygen? So all that kind of stuff...

19:57...so, step by step, how do you get your machine set up so that it's runnable?

20:01Potentially, it's run to set up--the actual product setup is part of that--hopefully that's part of it.

20:07That's what we do. One of our steps for, say, a developer system is actually running a product setup.

20:16So the heart of product development I've kind of named the source code management system.

20:22So the code base is the heart, and that's the thing that everything sort of points to, and that's how you get your product.

20:31I'm not going to spend a lot of time talking about all the different types of version control systems that are available.

20:37We use a product called StarTeam. It costs hundreds of thousands of dollars.

20:42I don't recommend you spending lots and lots of money unless you need to, unless you've got 700 people that log in...

20:48...and change code all the time. What I will say is that--actually, we also use TFS; I shouldn't leave that.

20:55We have projects and products that are building daily within the TFS framework.

21:01I will say, even if you find a free version control system, find something that manages your work items...

21:09...your CRs, along with the source code, something that dovetails in.

21:15Don't find a CR system and then use the source code system that matches it.

21:19Find the source code system and then find a CR system that matches it. 'Cause you're--again, you're developing.

21:27The other thing is that a good merge ability within the--hopefully, you know...

21:31...most of the good merge systems are more expensive...

21:35...although there are some version control systems that quote-unquote don't require you to merge.

21:42I've tried it and I've had to merge. So the point here is, though, is that as much as you don't need branches...

21:51...and you don't need alternate living code bases--I mean, you end up creating those things...

21:56...and you eventually have to merge source code. So make sure that you know how to do that with the tool that you get.

22:03And that's all I'll really say about--you know, I'm not up here waving a flag about a specific version control system.

22:10I would like to say that it's very good to manage all of your source code for your product in a single code line.

22:18A number of teams historically have said, Okay, this team's got this code base and this team's got this code base.

22:23And then you pull this and pull this, and then you build this product out of it. What ends up happening is...

22:28...that this team is working on one schedule and this team's working on another schedule, but the product has to ship.

22:34And all the stakeholders are saying, Oh, you know, here's this, here's this; here's the status of this...

22:39...here's the status of that. It becomes very difficult to ship a product when you've got all these different code lines...

22:43...that people have to look at. If everything's in one code base, you're labeling it once per day...

22:48...you're building it once per day, all the change work items are in one place and everybody can look it....

22:53...and review it. It becomes large, but it's a lot easier to ship a product when you have your code bases set up that way.

23:05That's something else that I want to sort of mention at the beginning that I didn't...

23:08...was that I've been at Esri for 27 years, and there have been a lot of other companies...

23:15...that do similar things to us, and actually have had some pretty cool technology.

23:21Many of those companies never shipped a product; they did lots of demos, they did a lot of development...

23:26...they did a lot of research. Esri, over time, has been the company that I've seen continue to research and develop...

23:38...and productize and ship software. So, I know, I don't want to say Esri knows how to do everything exactly right...

23:47...but by hook or by crook, by using the methodologies that we've created over time, we've been able to ship products.

23:58So the other thing that we do within our large code base is that we have separate code lines for nonbuild resources.

24:09Runtime support--file's I already talked about. Test is something we manage in an independent code base.

24:16In some of our code bases, we actually test as part of our build process, and so we have it in the same code base...

24:22...and we label it and pull it the same way. The problem that we've had over time is that there's currently 86,000 files...

24:30...in the code base. If you add Runtime support files and you add test and you add all the third-party source files...

24:36...that aren't part of the actual build in your code base, you have developers checking out 200,000 files...

24:42...and two hours later, when they get all the files checked out on disk, they can start working.

24:46So it's much easier to manage these things independently. Run a setup to get Runtime support files.

24:53Check out test. If you're going to run Test, check out from the test code base just what you need...

24:58...instead of worrying about checking out so many files as code bases get large.

25:03Another thing, and it's not so much the--actually, sort of shows why the code management system is the heart...

25:11...of product development, is that we historically have used our code versioning system...

25:19...as a legal document in court cases to prove that we have created--historically--technology.

25:28So it's important to make sure it's backed up; make sure that your legal department is involved.

25:35Make sure all your source code is copyrighted. Because at some point in time in the future, someone's going to say...

25:42...Open up that folder and look at the history of that source file and who did it on what day, why...

25:49...and you'll see the check-in comment, because it could potentially be very important farther down the road...

25:55...when you're sued or someone's protesting you shipping something...

25:59...technology within your source code that they think own the rights to.

26:08Branching--before I get into branching, the first thing I'd like to talk about is the fact that...

26:14...I don't want to say branching is evil, but there are alternatives to branching. If you have to create a branch...

26:21...and I'll talk about this in the next slide, but if you have to create a branch...

26:24...that's fine, and it's fine to manage it that way.

26:27What I find, because I am in charge of a lot of the source code control goings on, I get weekly...

26:34...monthly, multiple people coming up to me saying, I need a branch, I need a branch, I need a branch...

26:38...I need a branch. And they haven't talked to their dev lead, they haven't talked to Scott Morehouse...

26:42...they haven't talked to people that are in charge of figuring out what we work on--they just want a branch to put code in.

26:48Like, you know, settle down--you know, let's figure out what you're trying to do.

26:51So I like to talk to them about alternatives to creating a branch initially.

26:56So developers, source code on a disk is technically a branch. You've got to back it up yourself.

27:02You've got to worry about being out of sync with the main code line. But for all intents and purposes...

27:08...it is a branch of a source line; it's an alternate view of the source line. So people can develop that way.

27:16If you're doing new development, we have had a tendency to add new team folders and...

27:22...new functional folders for new features and simply not add the solution into our product build.

27:28And that allows teams to sort of sync up with the existing product...

27:32...and add new technology without affecting other people on the team. And again, this is all about...

27:38...I don't want to have to worry about merging this code in four months from now.

27:42And the other thing that's really neat about that is that potentially, that team can go to a team that they will depend on...

27:48...and say, "I'm developing this thing and I'm in the code base--can you look at my code? 'Cause I depend on this...

27:52...and it needs to do this little thing." But it's in the same code base...

27:56...so that developer that's working over here can look at whatever that code is and say...

27:59..."Oh, I know exactly what you need--it's right here. You can use that; I don't need to change anything."

28:04So it creates some communication as well.

28:09The other thing is that although it's another code line, we generate, research code bases that aren't branches.

28:18Research really is going out and figuring out how something's going to work with the intent of proposing it...

28:24...as development at some point in time. When it becomes development, potentially we'll create a branch or...

28:30...figure out how to get it into the code base; but as a research project, I really don't want to see another branch...

28:37...underneath a main code line just to allow that team to do whatever it is that might get thrown away eventually.

28:44When Scott Morehouse and all the dev leads say, "Okay, it's going in," then we start thinking about creating branches...

28:49...and putting in a timeline and making sure that all the I's are dotted and T's are crossed.

28:57So now we're going to talk about--okay, we're going to create a branch.

28:59What does the developer need or what does the team need in order to get a branch, and when do we create branches?

29:06The very first and probably most important thing is that I want to see and the dev leads and Scott want to see...

29:13...a well-defined plan and time line for getting whatever this development is into the code base.

29:18It shows the team's invested, they've got energy, they know what they're doing.

29:23Here's where we're starting; here's where we're ending. We have these milestones--we're thinking about it...

29:27...we're planning on it. Historically, we've had a lot of devs just say, "Okay, I've got this idea"...

29:35...and then just start going and kind of going and doing development, and then two years later, they're...

29:39..."Oh, I need to rearchitect it," and...try to stay away from that. Get those teams involved and make sure that...

29:46...the localization team and the setup team and the legal team and all the other teams are involved in making sure that...

29:51...that this team can be successful at what they do.

29:55As we do development, there's really--and this is very obvious--there's two reasons to create a branch.

30:01One is to protect the main code line from the actions of the development project. So they're making changes to code...

30:08...that we're building within the product, but we can't see it in the product yet. We don't want it in the product yet.

30:12So you want to make sure that development happens independently so we don't have the build break every other day.

30:21The other thing we do is--development projects don't want to worry about...

30:26...okay, I'm branching from the code base on March 1st.

30:29On March 5th, someone makes some global change to the code base...

30:33...and now I've got to merge that whole change in the development project. When we create development projects...

30:38...we very, very seldom have the development project try to keep current with the main code line.

30:46They're off doing development. At a certain point in time, they're going to have to merge back in.

30:52Instead of doing all this work on a daily basis just to try to keep current with the main code line...

30:56...we worry about that at the tail end. Sometimes that means it takes two days to merge.

31:00Sometimes that means it takes two weeks to merge. But what it really means is that, on a daily basis...

31:06...the development project wasn't broken while they were trying to figure out what was going on in the main line.

31:10So it actually saves time in the long run.

31:14We also have a tendency to branch to ship, and when I say that, I don't mean ArcGIS...

31:21...the thing that comes out every 16, 18, 24 months. I mean things like Explorer and other products that are...

31:29...built from ArcGIS source code but have a more frequent delivery cycle. So the Explorer team will create a branch...

31:40...and it's usually at a beta or prerelease or something, and then they'll work on...

31:44...their part of the software and make sure that it's shippable and make changes as they need and sometimes look...

31:49...check stuff out and merge it into their code line in order to make sure it's completely stable before it ships.

31:57Again, the idea is that we do create branches to ship, and that's a branch paradigm.

32:03And we always, always, always create branches for service packs.

32:07As soon as we have the final label within our code base, we create something we call a QFE branch...

32:13...which is a quick fix branch. And that's where, you know, as soon as we ship, we already know...

32:19...there's 55 known issues that we want to at least start investigating, looking at...

32:23...and that QFE team has already got those on their radar and they're starting to look at.

32:28It's just part of shipping software. You do all your high-priority items and you just have a number of low-priority items...

32:35...and 500 contracts and 100,000 people are waiting for your software to ship...

32:41...and so we're always ready to do QFEs as soon as that final field is completed and then verified...

32:49The other thing is sort of important about how we do QFEs, and then I'll mention here...

32:52... is that when we don't fix our QFEs normally within the QFE branch.

32:58What we do is that we have the next version of the product or main branch--fix whatever the issue is in the main branch...

33:06...because that's what the developers and the product engineers have on their desktop.

33:09So constantly, we're trying to replicate whatever the QFE issue is in the main branch...

33:15...because it just becomes easier for teams to address issues that way. They get the daily build...

33:21...and they can see that it got fixed in the daily build and not have to go to the QFE branch...

33:24...and set some special source line up on their machines.

33:27The QFE team, then, takes that fix, extracts it, and merges it into the QFE--eventually merges it into...

33:33...the old version of the source code. Early on, that's really easy because almost all the fixes--you know...

33:41...most of the source code hasn't changed very much and it's easy to pick things out and get them into QFEs.

33:46Farther down the road as software's been rearchitected and edited a lot, it becomes a little bit more difficult.

33:51At that point, the teams have to spend more time with the QFE team...

33:55...to make sure that a fix that needs to get into the QFE gets it.

34:01If the code base is the heart of the development at Esri, the build management, without a doubt, is the heartbeat.

34:10There again as I talked about, there's two points to build management. One is automating the build...

34:16...when do you fire it, how do you fire it. And the other is scripting the build process...

34:23...what source do you check out, where do you check it out to, what do you build...

34:26...where does that output go--all that kind of stuff.

34:29The important thing at the bottom of the slide is that we expose our build process to the consumers of our builds.

34:38So because our build takes five hours, we have web pages for all the products and all the build machines...

34:43...and we allow people to see their real-time updates of how the projects...

34:49...and solutions as they're going through are being built. If there are errors, they can see it immediately.

34:55As soon as there's an error, the build team gets notified and potentially...

34:58...we're looking at that architecture diagram and looking for a dev lead or the programmer that checked in...

35:05...a change to address whatever that build issue is.

35:08So we don't wait till the build finishes to start fixing it. This is because the length of our current build.

35:15The other thing that's important is to maintain build results and history.

35:20Over time, it's important to make sure that as your builds get longer and longer, you can see what's taking more time.

35:28Potentially, there's an optimization that needs to be changed, or a build machine is having an issue...

35:34...or some project file is referencing way more stuff than it needed to and it just had some included that created...

35:42...all kinds of stuff that needed to create. Keeping track of how long things take to build...

35:47...can address some issues farther down the line.

35:53So first, build automation. The number one thing that if you don't do this now...

36:02...the number one thing you should take from this presentation is have a daily build.

36:07There was no source code checked in yesterday. Have a daily build.

36:10There was five million files checked in yesterday. Have a daily build.

36:14The daily build is the heartbeat. It proves that the build process, the build source code, everything is working.

36:23You know, you don't want to have to get the pads out and stop it and start it up again.

36:31We build every day at noon, so the first bullet there is start at a specific time every day.

36:37It's very well known that we build at noon. The code base is closed at noon.

36:41We keep the code base closed from noon to one o'clock in case someone e-mails us and says...

36:46..."Oh, I'm kind of late on a check-in; I need to get it in. It's almost noon. I don't think I'm going to make it."

36:51So we say between noon and one, no one can check in code.

36:53The other 23 hours of the day, the code base is open for people to install for the daily build.

37:01The first thing the build does is create a build label, and the reason why that's important is...

37:06...that label gets attached to every work item within the code base. That label gets attached to every file in the code base.

37:14That build label is a build number, so it actually--we modify a build--that H file, its CS file...

37:21...and all the other build type files for different languages. And that number is embedded with it there...

37:26...so that when binaries and libs and OLBs get generated, the build number is in the DLL.

37:33So the same code label in the code base is the same label in the DLL. When we copy that file to have the setup generated...

37:44...the setup generates it with that same build label. So now we have a setup, we have a DLL...

37:50...and we have a code label in the system. Everything can be tied together all the way from soup to nuts.

37:57If a build error happens or the build breaks, if we can fix it while the build's occurring...

38:04...it's kind of like the tree falling in the woods and nobody heard it...

38:08...although we usually point blame to whoever actually did break the build.

38:13But if the build finishes and it's still broken, we tend to not want to start another build until the build is fixed.

38:21You don't want to start a build knowing it's going to fail. I said, have a build every day.

38:28This is sort of the exception to that rule.

38:32One of the exceptions; there's another exception to it I'll talk about later.

38:37Now that's the product build. We also, for some products, have development builds...

38:42...where we're iteratively building and generating builds throughout the day.

38:48Those products have a noon product build, as well, as their iterative daily build.

38:53Because our ArcGIS product is five hours long, we don't want to run more than one...

38:58...potentially two builds a day, but really, we just do one build of ArcGIS a day when we create setups.

39:07So there's two kind of theories of thought. One is that every two hours or every hour or every three hours...

39:13...you run a build. The other theory of thought is that source code got checked in--you run a build.

39:20Source code got checked in--you run a build. That second thing you can do when you don't have a big build process.

39:30The problem, though, with that second thing is that we have found over time at Esri that a developer will check in...

39:38...a source code, and then he'll check in another source code and he'll check another source code...

39:42...and it's not until the third source file got checked in that he's done checking in.

39:47So some developers will check everything in at once and everything's fine.

39:51Other developers will do check-in once--oh, here's the header file; oh, I needed to add this file. Boom.

39:56So if you've got a build process that's potentially triggering on build changes, that can be an issue.

40:02The way to get around that, again, is work items, making sure if a work item gets verified or checked off as checked in...

40:09...then you run a build. There are some things with that. But what we do normally is that we have interval builds.

40:16Everybody knows that every two hours, a build's released, every odd two hours it runs debug--something like that.

40:25For build automation--you know, how do we start the build in both of these instances?

40:30We use Jenkin's, Hudson--you know, there's lots of different things available.

40:35Jenkin's is the publicly available version of the current Hudson. We use TFS to fire off development builds.

40:45We have in the past used the Platform Scheduler.

40:49One thing to note here is that it's really good to have a web-based automation tool to fire your builds.

41:00You don't want to have to go in, log in on the machine, and hit Build.

41:04You want to go to some HTML website, find the build you want, and press Build; and you can go home...

41:11...and remote desktop in and bring up the HTML page, and hit Build, and you can manage it independently.

41:18Jenkin's is free; the TFS costs some money. Platform Scheduler is free but, again...

41:24...you want to try to make sure that it's easy to manage and schedule builds and fire them off as you need to.

41:31So like if a build ever happens and something needs to get rebuilt, you want to be able to just fire it.

41:38Build scripting. Build scripting is the thing that makes the build repeatable.

41:47The important note here is that when you create a script or a set of scripts that run sequentially...

41:55...you need to make sure that you script your entire build process. Like I said, initially we label the code base...

42:02...at the very beginning of the build. The very last thing that we do in the build, when you think about the build...

42:06...is we publish the setups for internal use. So no one has to--if someone's sick, it's not a problem.

42:18The source code gets labeled every day at noon, and sometime after five o'clock, around five o'clock...

42:23...there's public setups available. So make sure you do everything.

42:30We pull source code, we label source code pull and build it.

42:34We potentially run some tests. We stage those resources. We stage them in two ways.

42:39We stage them for the teams to get them immediately, so the built resources are actually on disk, available in a location...

42:46...I'll talk about that in a minute. And we also--that red line really denotes the difference between a product build...

42:51...and a development build. The development build will go up to that red line; the product build will actually...

42:57...when it's done, copy the files to a location for the setup build to start; initiate the setup build process to start.

43:03Those things get built, they get tested, and they get deployed.

43:11So what we use on Windows is FinalBuilder. I'm not sure how many non-Windows people are in here...

43:18...but if you ever saw FinalBuilder and you're on a Windows machine, you kind of lust after this.

43:25It's the best scripting environment that I've known. We use Ant and Maven and Lua and things like that...

43:31...for non-Windows platforms and for some of those things on Windows platforms as well.

43:37This is a kind of a snapshot of what FinalBuilder looks like. Again, it's a Windows-specific product.

43:46What I'm sort of saying in a nutshell is what Visual Basic did for Basic, FinalBuilder does for scripting.

43:55It's a GUI that knows about every action, everything you can think about. It knows about signing DLLs...

44:03...it knows about merge products, it knows about documentation products, it knows about all kinds of stuff.

44:09So if you create an action and you put it within your list and you have a script that runs through when you run it...

44:15...you can see on the right-hand side, it has the looping, it has variables.

44:19It's got wizards. It's got history. It's got logging. It's just a cool thing.

44:34I talked about copying, staging the resources for team members to get access to--that's our archiving process.

44:43So what we have is we have a disk server on the network that everybody has access to--and when I say everybody...

44:50...I mean everybody, even marketing and anybody that has a login has access to get at the daily builds.

44:58We've maintained the last 30 daily builds on the log server.

45:03Incidentally, the folder name of each build is the build number.

45:09So remember I talked about the setups and the source code and the labeling and everything else.

45:14Resource files are staged under the build number. We put the build binaries--you put the PDBs...

45:22...because we use a simple server quite extensively for debugging release crash dumps both internally and externally.

45:31So you make sure that the most current ones are available so when PEs have issues with the daily setup...

45:37...it's really easier for them to give a dev a crash dump and have the dev...

45:43...and point the simple server at the built location and get that information.

45:47All the build log files are there. If something seemed to have worked yesterday but it really didn't...

45:56...you always want to be able to go back through and look at the log file of how that thing got built.

46:00So you want to maintain those for your most recent builds.

46:03Again, we also maintain all the setups. As soon as setups are generated, they're put on the archive location, as well.

46:11Every final build that we deploy outside of Esri is maintained permanently.

46:17So we have the 9.0, 9.1, 9.2 builds on this archive server available if anybody needs to go back and get those binaries...

46:26...look at the source code, do whatever they need to do. And again, it has all the same things...

46:30...it has the PDBs, it has the simple server, it has the setups. And actually, as we create the final staging locations...

46:38...we don't just copy the build log files and the output binaries and things like that.

46:41We also copy every single file under the built location to there.

46:47So if a developer ever needs to figure out why something's broken, they can copy that big, huge, monstrous thing...

46:54...over to their machine, but they don't have to rebuild anything. All the intermediate files are there--everything built.

47:01They just have to go back to the document, figure out how to set up their 9.0 dev machine...

47:06...and then they can go from there.

47:09[Inaudible] Oops. [Inaudible]

47:32Managing information--managing and maintaining information. So we have something at Esri called Dev Central.

47:39It's where developers go to figure out what's up. It's where developers go to figure out how to use the source code.

47:47It's where developers go to figure out how to use a tool that we use within development.

47:53It's got links to the built status page. We index all of our source code, and so it has a link to our source search page.

48:06We do Agile projects and we have a Scrum Works instance, and there's a link there for Scrum Works.

48:16We run a product called Prevent from Coverity, which is a code analysis tool that looks for coding errors.

48:24We have links for that at this one website. A link to coding guidelines is there--I'm just going down the list...

48:33The list of projects that are currently branched is available.

48:38We also keep track of branched views that have been checked in and merged back in the main code line at this website.

48:47All the third-party resources are documented at this website...

48:51...so that our legal department knows that we know what we're doing.

48:56This website manages discussions on different topics. It also points people to where different resources are...

49:05...so where is the latest build, where is the service pack build, where can I get the symbols for X, Y, Z build...

49:12...that's available there. And also important--e-mail aliases. Because developers--the builds...

49:18...if they're trying to check in and they're not going to make it in by the noon deadline, who do they e-mail...

49:23...who do they talk to--all that information's there.

49:27I'm going to show a little bit of that now. What time is this over? I'm sorry.

49:34[Inaudible audience response]

49:35Five-forty, okay, so I've got a little bit. Uh-oh. Oh, look.

49:58We use [unintelligible] for Windows to build the MSIs and the MSMs and all that kind of stuff. It's kind of an older product...

50:06...but it's sort of embedded within our setup team. We haven't moved off it to anything else.

50:16What's the other product that was...?

50:17[Inaudible audience response]

50:18Say it again?

50:19[Inaudible audience response]

50:20Yeah, it's InstallShield. I mean, we had InstallShield in its couple of groups many, many, many years ago...

50:27...but it never really caught on to our main setup team. So they continue to use...

50:34[Inaudible audience response]

50:36Yeah, yeah. Exactly. It is.

50:38[Inaudible audience response]

50:39You're actually--and they still use it and they have that end-of-life product and it's still doing everything...

50:43...and Microsoft comes down and talks with us and helps us when we have issues and...

50:49Actually, they have a really neat thing, and it probably doesn't apply to ArcGIS and Windows installer so much...

50:55...but you saw that interesting demo that you and Cameron did with the dongle where...

51:01...the app was running on one machine and you ported it over and put it on another machine and have it run.

51:05I'm waiting for that. That's what I want everything to work like. And then I want a dongle...

51:10...that's like 900 gigabytes and runs really fast--USB 95 or whatever it's going to be.

51:17So, here--oops--here, live, is our daily build. Hopefully, it's okay--yeah, it's okay.

51:26The release 32-bit Windows build is still running, and all the other builds have completed...

51:30...except for the service pack you see down at the bottom.

51:35 And let's see if I can scrunch this over. So here's the whole--these are all those 115 projects building...

51:43...the [inaudible] system and Beagle Graphic [inaudible].

51:45Beagle Graphics is an example of a code name that made it into the code base.

51:49Okay, so Beagle Graphics is there, and I've tried to stamp it out but I can't. It's a team name.

51:55It was the beagle team, and they did some of the graphics development.

51:59That's another example of a team that likes to develop for a long period of time as well.

52:04But we go through all the UI stuff and down at the bottom, we're actually running...

52:09...what the build is doing now is running Passolo. Passolo is our localization product.

52:14It takes all of our resource strings, and it puts it in the database, and the database...

52:19...I won't explain, but it's our localization build--part of the build.

52:27But all the other builds are completed successfully, so that's a good sign.

52:31What else do we got here? Developer page...

52:35Source search. This is like the most simple source search--most simple thing about this site.

52:43We could implement a better source search, but this is very fast and is very free. It's the Windows search server...

52:51...and the really cool thing about it is I can say void...which there should be a couple.

53:00There's 38,400 instances of the word void within our code base, and that took less than a second to figure that out.

53:09So that's one of the reasons why I--we've looked at some indexing things that do more with source code...

53:16...but a lot of times, what ends up happening is that the time it takes to index takes hours and hours and hours...

53:21...which is longer than I have to wait and worry about if something breaks.

53:26And the time it takes in order to find things many times is a lot longer. This is free, and it's mega fast.

53:36And we also like [unintelligible], so like our SDE code base is an independent code base that we...have in there too.

53:45Coding guidelines. Like I said before, we'd be much better off if we just said...

53:52...this coding guideline at this website externally was our coding guideline.

53:57There's a number of things in the ArcGIS code specifically to deal with--OLBs and IDL and things like that...

54:03...that require us to sort of go outside the box, and that was sort of the opening to do a lot of coding style guides outside the box.

54:13We have a coding guideline page, but I'd like to have it at the very top say, Link to this global website...

54:19...'cause that's the right way to code. Scrum Works...

54:23...that it would take me to the Scrum Works web page to log in and manage my PBIs and all that kind of stuff.

54:30Source code validation--I won't link on that, either. That'll bring up another database.

54:35That's where we have automatically found coding errors...

54:40...and it actually is fairly clean compared to how it was when we first starting running it.

54:46We cleaned up a lot using this product. Again, it's called Prevent from Coverity.

54:52Every port site--this is the site where, when we get crash dumps internally from our PEs or potentially...

54:59...get a crash dump from someone out in the field, we can type it in, enter the crash dump in here...

55:08...and it will look up and see if that crash dump has occurred prior.

55:12How many people have reported that crash dump? Potentially, what part of the code the crash dump is in...

55:19...who's responsible for it--there's all kinds of metrics that we have in order to figure out...

55:24...this is one way that we figure out what's the next thing on my list that I want to fix?

55:29It sets priority; it sets severity.

55:33It's a really cool thing. It's--again, if you don't use simple servers and you don't manage crash dumps...

55:39...and you've got issues out in the field that you can't reproduce, this is the only way to do it.

55:46War room procedures--I won't get into war room; that's sort of a whole different topic.

55:50But as we get closer and closer to releasing a final release, the war room...

55:57...the group that determines what actually gets into the code base...

56:00...sort of tightens down the hatches and makes it harder...

56:04...to get things checked in and requires more testing and more verification, all that kind of stuff.

56:08But anyway, we have a website that talks about the release schedule, what it takes to get something in...

56:13...what mode we're in currently, how hard is it to get something in...

56:20And really what that ends up meaning to me is, when I go down to war room, is it going to take five minutes...

56:24...or is it going to take an hour? And that's sort of the difference in the different lengths of the war room...

56:30...although usually you can get through pretty quickly.

56:35The daily setup status--I showed you the build status and that was for the actual building of the applications.

56:39The setup team has their own page for when the setups get initiated and what's been output...

56:44...what products have been outputted. They build like 115 setups, so that's a much bigger page.

56:51The QFE team has their own website for that information. I'll show the branch view site.

56:59This is the list of development projects that people are working on, what their names are, who's responsible for them...

57:06...when they said they were going to check it in but they haven't, what they depend on...

57:12...what branch they want to be branched off of. So this helps us keep track of the branches that exist.

57:18We also have development projects that have already been checked in and merged and who did it and when they did it.

57:29We're going to change requests, but we have a link to all the assembly, the diagrams that I showed earlier...

57:35...how things are put together. That's a web page unto itself.

57:41We have a page for--this is in the left-hand side, the list of developer tools that we use internally--Dependency Walker.

57:48Get Build is how we get the build. Again, the idea here is that this is the one page where I can point anybody to and say...

57:57...Go here. Don't e-mail me--go here. And then after you've gone here and if you can't find it and it is there...

58:04...then I want to know, because I want to make it easier for you to find it. But if you don't look...

58:08...then I won't know that it needed to be improved.

58:12Third-party dependency is something that is very important. We list all of our third-party dependencies on this page...

58:21...what version we're currently using, who owns them, what teams, what license type is required...

58:27...where you get it if you need to download it online, what view it's currently in...

58:33...what folders do we take these contents and put them into so if we get a lib, it goes in the Lib file; if we get...

58:38...a header file, it goes in Include folders, under common files, under the shared common files...

58:44...what projects depend on it. If that third-party dependency actually is a binary that gets deployed with a product...

58:51...we need to keep track of that. If it depends on anything else, et cetera.

58:57Again, this is where our legal department comes in, and they want to know everything that's going on...

59:00...and make sure that we know what's going on within a product.

59:04The other thing that's here in the left-hand column that's really important is the discussions.

59:09Again, I was talking about--if you can't find the information, then let me know about it.

59:14Even before the e-mail, look at the discussions and make sure I haven't answered the question already there...

59:20...or someone hasn't answered the question already there. And many times if someone e-mails me...

59:25...I will not e-mail them back the response. If it doesn't already exist in the discussion, what I will do is...

59:31...I'll create a new discussion, answer their question there, and send them a link to it.

59:34And then the next person that asks the question, they just go to the discussion.

59:38And we can have a rolling dialog within discussions as well. But we have Visual Studio discussions, code base...

59:44...star team code base discussions, discussions about the build--what do I do if it's new and I'm about to check in...

59:51...that's one that always gets hit. Et cetera, et cetera.

1:00:00I can't see...[unintelligible]. E-mail aliases and who we contact. Developer system setup...

1:00:09...these are the documents that say how to set up a 9.2 developer system. It's archived there...

1:00:14...so if anybody ever needs to do it, they can go back there and do it.

1:00:17Ten-one server, 10.1 mobile--you know, all the information's there. It's the document of how to set up a machine.

1:00:24And over here on the right-hand column is the disk location. So all of the current 10.1 ArcGIS resources...

1:00:31...that's all the builds, the daily builds. ArcGIS Runtime, which is something you and Cameron talked about...

1:00:38...up onstage yesterday morning--that's an independent build, and we archive those resources independently.

1:00:46Debug release, service packs; also the developer-only locations. Not everybody at Esri can get to them.

1:00:56Only developers can get to them, 'cause they contain source code...

1:00:59...the actual source code for those final builds that are available.

1:01:07Okay, I'm going to go through these quick because they were in case I couldn't get online.

1:01:17And finalize with--this is a book that I picked up a year and a half ago, maybe two years ago.

1:01:23It was written in 2010: Software Configuration Management Patterns.

1:01:29This writer goes through and talks about a number of software development tasks, tests...

1:01:38...you know, like a testing track, branching track, releasing software track. Very clear and concise ideas about...

1:01:48...the kinds of things that you, as software source code managers...

1:01:55...and build engineers and software configuration managers...

1:01:59...should be thinking about. The thing that I found out, what was really cool about it is that they had like 22 or 24...

1:02:06...different types of things to do. We did all but 2 of them within Esri. So like, build the ship was one of the things...

1:02:14...I'm sorry, branch the ship was one of them, and we do that.

1:02:18Create branches with QFEs--we do that. It's all that kind of stuff. So this is a very good sort of overview, it's an easy read and if you're interested in software configuration management, its a sort of little how-to guide; what did I miss in the organization. So, fill out the survey and ask away if you have questions.