Tasks in the context of SCM96

Contents

• General
• Related Items
• Official, agreed upon tasks
• Own choices made
• Results
• Comments
• Plans

Related Items

• Installation related issues
• Makefile related issues
• Terminology
• Critique of the current build system

General

These have been tasks, in the context of assisting the SCM96 project to switch -at first for evaluation purposes- to using ClearCase, as the configuration management tool for NMS.

Official tasks

These have been agreed upon.

• Build under Clearcase the program blocks in the p1 VOB.
  • Start with werlib and werwin, and the blocks depended upon, among which wwllib
  • Leave qmclib for a start.
• Pay attention to performance, and especially, evaluate and try to minimize the penalty incurred for configuration lookup. Build optimization is one main asset we capitalize upon.
• Criticize the current process, and make suggestions to improve it. This is maintained apart.

Own choices

• Implement and use a staging VOB as a distribution area only, i.e. do not use elements or derived objects from there while building the platform. The rationale is explained in my notes on installation.
• Implement site specificities as makefile macros instead of as soft links in a main VOB. These are grouped in a specific paths.mak makefile component.
• Maintain a global Makefile, containing the information previously in build.lst.
• Make all directories derived objects, instead of view private ones:
  • integrate the dodirs.sh functionality into the makefiles
  • make dependencies upon the installation directories explicit, so that the install.sh script may be simplified.

Results

• Set up and documented a hierarchical makefile system, based on the versions labelled P1_DEVENV.
• Processed manually a few POBs, so as to validate this system, and tune tools for automatically processing the remaining makefiles
• Produced binaries for these VOBs, out of which two, werlib and werwin, were regression tested.
• Processed all the makefiles with the tools
• Run clearmake on the whole platform and corrected errors. There were residual problems in 29 POBs, some trivial (in cases I have not reached at all), some induced.
• I have only superficially tried parallel building with deceptive results. Some tuning is needed (but Delphi could help).
• The description of dependencies and the integration of the installation process are far from complete, so that the building is not yet free from some hysteresis.
• Some useless rebuilding still occurs, due to improper dependencies.

Comments

Concerning the performance

1. the reference duration of a build with the pac driven RCS based system
2. the duration of a full build under ClearCase, with no winkin (e.g. first build in a new replica, with a new staging VOB)
3. the duration of an unconditional full build under ClearCase
4. the duration of a null build under ClearCase
5. the duration of an optimized parallel build under ClearCase, over the appropriate number of hosts.

I expect that the second is longer than the first, due to the extra work performed by ClearCase in its configuration lookup (totally useless in the case I mention, where no benefit is expectable).
On the other hand, the RCS scheme performs a lot of checking out, with afferent copying, which remains unmatched...

I expect the third to be shorter than the second (and hopefully than the first), since ClearCase should then not spend time looking for reusable bits (not checked though...)

By "null" build, I mean a build following a succesfull full build. It should then do nothing but validating the existing derived objects.
The ratio between the two latter should give an idea of the ideal gain to be obtained by build optimization.
In normal use, only a marginal part of the work has to be done again.

The last time is a farther target.
Clearmake may distribute independent builds to separate processes on the same or different hosts. Due to waiting for I/O, it is reasonable to expect significant benefits from using several (e.g. 4, of course depending on the actual build characteristics) processes per node.

This requires however that the dependencies between the various makefile targets are described properly. This is a huge work, although one that may be performed progressively, e.g. first at the highest level (building the POBs in parallel, but every one of them as a sequential process).
A 4-fold increase in overall performance should be obtainable, with respect to the normal build duration, as mentioned here above.

Plans

• I shall of course be available for consulting and various explanation (e.g. imprecisions).