Performance when building large projects

Topics: User Forum
Jan 17, 2007 at 9:02 PM
I´m building what I think is a pretty large project with the December CTP that takes around 10 hours to complete. I know Microsoft boasts that it generated the help for the entire .NET framework in 30 minutes (as opposed to 10 hours before this CTP). Has anyone else seen times around that same order of magnitude? Is this a limitation of Sandcastle, or is SHFB doing something fancy that may cause this? What sort of things can I do to speed this up, other than the obvious options of not including private members, setting SdkLinkType to None, etc.? Should I be approaching this differently, as in making one project per assembly and then combining the output in some manner?

For specifics on my project/environment:
- 131 assemblies documented
- 250 dependency assemblies
- 194 namespaces
- 2597 types
- 40176 members

Note that I´m running this on a virtual server that is normally pretty fast (though it is still a virtual server .. maybe MSFT is using the latest 8-way box to get that 30 minute number).

Jan 18, 2007 at 3:15 PM

I noticed a big performance hit when going from the November release to the December release. With the November release, it would take about 20 minutes to build a chm file. After switching to the December release, it would take about 3 hours to build the exact same project. I also have a fairly large project with 105 assemblies being documented in the chm.
Jan 18, 2007 at 3:24 PM
It could be because you are using a virtual server. The BuilderAssembler step is extremely memory intensive. One change I did notice is that all of the individual files in the Sandcastle CPref folder are now one huge 200MB+ XML file. This consumes a large amount of memory (600MB+) twice as it's used by two components. A lot of it doesn't get garbage collected so it still tends to have about 400MB allocated during the rest of the run. I did some testing and by splitting the file back out individually by running MRefBuilder on the .NET assemblies, it did greatly reduce memory usage which may help with the runtime on a system with less memory. This is probably worth bringing up in the MSDN documentation forum as it seems to have been a change for the worse.

Jan 19, 2007 at 3:14 PM
I don't know if it was either of you, but somebody posted a message in the MSDN documentation forms about the slowness. They said that the ApplyVisibilityProperties and AddingMissingDocumentationTags steps take the longest. Those have been there since about the October CTP though so I'd of thought they'd have been an issue earlier as well. However, in order to see if there is a problem, could either of you do a build and when it hits the ApplyVisibiltyProperties step abort the build. If you could then send me a copy of your project and the content of the .\Working folder I'll take a look at it and see what's happening. You don't need to include the assemblies. My e-mail address is in the help file builder's About box and in the footer of the pages in its help file.

Jan 19, 2007 at 6:47 PM
Just sent you a ten meg zip file. Hopefully it will help you glean some insight.
Jan 23, 2007 at 2:34 AM
To anyone else experiencing this problem:

I've resolved most of the slow build issues. The last one remaining is adding missing documentation tags. Most of the issues were related to the size of the reflection information file and performing XPath queries on it. I've reworked the code and those steps now all run in under a minute. Adding missing documentation tags still takes a while (probably about an hour or so to add the auto-documented constructors). I'm going to take a look at that later. There isn't a quick fix for that one and I may end up moving it to a build component so that the queries have less to look at as they would then deal with each topic as it gets generated. I need to make one other fix to the PostTransformComponent as well so that it doesn't load the whole reflection information file too as it only needs the assembly information to do the version number lookups.

I've released a special build which you can find in the Releases section of A test release labeled " Special" can be found under the Planned Releases option. Give that a try and it should run much faster. You can set the AutoDocumentConstructors option to false to skip that step and save more time as well.