Any experience with parallel builds in clearcase environment ?

[Martin]

I am posting this query to inquire about if anyone has experimented
with doing parallel builds in clearcase and would like to share that
experience results, findings, issues, problems. By this I am not
referring to loadsharing; what I am thinking of arranging the sequence
of builds so that there is no contention on VOB access.

What are some of the options available to make this possible. I know
this could be totally impossible, but does not hurt to ask. Thanks.

Martin

[David Masterson]

>>>>> Martin writes:
I am posting this query to inquire about if anyone has experimented with doing parallel builds in clearcase and would like to share that experience results, findings, issues, problems. By this I am not referring to loadsharing; what I am thinking of arranging the sequence of builds so that there is no contention on VOB access.
What are some of the options available to make this possible. I know this could be totally impossible, but does not hurt to ask. Thanks.

It's been a long time since I looked into this, soo...

* Read up on the capabilities of clearmake -- particularly the (IIRC)
"-J" option. Configure your systems to suit the capability.

* Make sure your Makefiles are clear and complete about every step of
the build. Understand the build tree.

* Try your build with different mixes of "-J" and build hosts to see
what performs the best.

* Implement monitoring at key "targets" (time the target or perhaps
run "sar" in background).

* If you're using dynamic views, continually tune your Makefiles such
that, if you did a full build in one view and then an incremental
build in other view (same config-spec), the second build would
*ONLY* winkin the first build.

I think the better your build works with "winkin", the better
constructed the build tree will be and the better the "-J" option will
be able to divide up the work across your build hosts.

--
David Masterson David DOT Masterson AT synopsys DOT com
Sr. R&D Engineer Synopsys, Inc.
Software Engineering Sunnyvale, CA

[Koushik Banerjee]

"David Masterson" <David.Masterson@synopsys.com> wrote in message
news:ud6h1s8jn.fsf@synopsys.com...
: >>>>> Martin writes:
:
: > I am posting this query to inquire about if anyone has experimented
: > with doing parallel builds in clearcase and would like to share that
: > experience results, findings, issues, problems. By this I am not
: > referring to loadsharing; what I am thinking of arranging the
: > sequence of builds so that there is no contention on VOB access.
:
: > What are some of the options available to make this possible. I know
: > this could be totally impossible, but does not hurt to ask. Thanks.
:
: It's been a long time since I looked into this, soo...
:
: * Read up on the capabilities of clearmake -- particularly the (IIRC)
: "-J" option. Configure your systems to suit the capability.
:
: * Make sure your Makefiles are clear and complete about every step of
: the build. Understand the build tree.
:
: * Try your build with different mixes of "-J" and build hosts to see
: what performs the best.
:
: * Implement monitoring at key "targets" (time the target or perhaps
: run "sar" in background).
:
: * If you're using dynamic views, continually tune your Makefiles such
: that, if you did a full build in one view and then an incremental
: build in other view (same config-spec), the second build would
: *ONLY* winkin the first build.
:
: I think the better your build works with "winkin", the better
: constructed the build tree will be and the better the "-J" option will
: be able to divide up the work across your build hosts.
:

We had been till a few weeks back been using clearmake. We have makefiles
for each module/submodules but there was no top-level makefile. For the
top-level we have perl wrappers, which reads a dependency
file(hand-written/maintained), and accordingly calls each module;target. We
have used parallel makes in *nix boxes. Most of the time it looked good, but
at times the build used to fail by saying something like "interference from
other process"..
We stopped recently because we found that our VOBs always were above 90%
most of the time, build was (very) slow, and there was not much gain other
than the winkin -facility. Since we have lots of customization over base
clearcase, we thought we can achieve what winkin does through perl scripts.
We are still not thru with the transition but builds are quite fast and the
space requirement is going down. We estimate that it will come down to 20%
usage.

Parallel make can be achieved by simple makefiles also, some environment
variable needs to be set. So you can now think what you would achieve by
clearmake. For parallel builds there are always other options.

Koushik.

: --
: David Masterson David DOT Masterson AT synopsys DOT com
: Sr. R&D Engineer Synopsys, Inc.
: Software Engineering Sunnyvale, CA

[David Masterson]

>>>>> Koushik Banerjee writes:
We had been till a few weeks back been using clearmake. We have makefiles for each module/submodules but there was no top-level makefile. For the top-level we have perl wrappers, which reads a dependency file(hand-written/maintained), and accordingly calls each module;target. We have used parallel makes in *nix boxes. Most of the time it looked good, but at times the build used to fail by saying something like "interference from other process".

Hmmm. Did you ever try switching from clearmake to GNU make? I
believe GNU make also has a parallel build option that might've been
faster in your environment -- plus you wouldn't have had to come up
with Perl wrappers.
We stopped recently because we found that our VOBs always were above 90% most of the time, build was (very) slow, and there was not much gain other than the winkin -facility. Since we have lots of customization over base clearcase, we thought we can achieve what winkin does through perl scripts. We are still not thru with the transition but builds are quite fast and the space requirement is going down. We estimate that it will come down to 20% usage.

How do you measure that? Were all your VOBs at 90% usage or just a
key VOB (or two) where (say) include files were kept? Are you going
to use "clearaudit" on your Perl builds?
Parallel make can be achieved by simple makefiles also, some environment variable needs to be set. So you can now think what you would achieve by clearmake. For parallel builds there are always other options.

I still think that you need to have an extremely well organized set of
Makefiles to avoid the ClearCase "interference" issue as well as get
good speed out of parallel build in general. I also think it's best
to work with the tool (ClearCase in this case) as much as possible to
get the full facilities it has to offer. Optimizing winkin is just
one way of evaluating how good your Makefiles are.

I have seen an environment where we moved a product from Sun's
TeamWare to ClearCase. The original full (non-parallel) build of the
product took 4-5 hours and the incrementals (due to bad Makefiles)
were not much better because winkin was minimal. After rewriting the
Makefiles to optimize winkin (and, as it turned out, simplify the
Makefiles), the full build was still 4-5 hours, but the incrementals
(which all developers used) were mere minutes since everything winked
in from the nightly build view or other developer views. Also, after
a time, the nightly build view also became an incremental build
because, through ClearCase config-records, we could verify that the
build was done correctly and that only things that changed from last
night's build were rebuilt. Because of this, our need for parallel
build kind of went away (it was on the futures list and we thought we
were on the right track).

--
David Masterson David DOT Masterson AT synopsys DOT com
Sr. R&D Engineer Synopsys, Inc.
Software Engineering Sunnyvale, CA

[Marc Girod]

>>>>> "DM" == David Masterson <David.Masterson@synopsys.com> writes:

DM> I think the better your build works with "winkin", the better
DM> constructed the build tree will be and the better the "-J" option
DM> will be able to divide up the work across your build hosts.

Yes, which is IMHO the greatest use for parallel building: check that
your makefiles are in shape (that your dependencies are explicit).

The point is that parallel building is a plus only for builds from
scratch (e.g. "nightly builds"), but not for incremental builds with a
lot of winkin. If many people build in the same time on the network,
"parallel" building will only increase the overall load.

So, it is a tool for selfish administrators, not for good scm...

--
Marc Girod 313 Fairchild Drive desk: +1 650 864 6501
Nokia NBI CA 94043 Mountain View mobile: +1 817 703 8975
Bld C #314 USA fax: +1 650 691

[Jerome de Vivie]

Marc Girod wrote:
Yes, which is IMHO the greatest use for parallel building: check that your makefiles are in shape (that your dependencies are explicit).

Better than makefiles: a single makefile which could be split under
included subparts like defs.mk, rules.mk, targets.mk, ... A complete
global dependencies DAG is the key to success. Not using recursive
makefile could help. The following paper discuss this point:

"Miller, P.A. (1998) Recursive make considered harmful. AUUGN Journal of
AUUGN Inc, 19(1), pp, 14-25 "

Yes: explicit dependencies are goods even if clearmake can grab some
dependencies like ones with "#include" because if we need to deliver or
build without "clearmake", we still need thoses dependencies.

Generating thoses kind of dependencies only outside "clearmake" could be
achive by using "clearmake -C gnu" and conditions like 'ifeq
($(MAKE),clearmake)'.
The point is that parallel building is a plus only for builds from scratch (e.g. "nightly builds"), but not for incremental builds with a lot of winkin. If many people build in the same time on the network, "parallel" building will only increase the overall load.

Fully agree, it's only a plus.

--
Jérôme de Vivie

[Jerome de Vivie]

Martin wrote: I am posting this query to inquire about if anyone has experimented with doing parallel builds in clearcase and would like to share that experience results, findings, issues, problems. By this I am not referring to loadsharing; what I am thinking of arranging the sequence of builds so that there is no contention on VOB access.

I have found one bug with -J (this bug is not yet corrected by IBM):
When there are multiples targets on one rule, "clearmake" does not
extract any parallelism. For example, if we have this Makefile:

all: tg1.o tg2.o

%.o %.ali:
@echo start $@
sleep 3
@echo end $@

$clearmake -J -C gnu all
start tg1.o
end tg1.o
start tg2.o
end tg2.o

whereas

$gmake -j all
start tg1.o
start tg2.o
end tg1.o
end tg2.o

What are some of the options available to make this possible. I know this could be totally impossible, but does not hurt to ask. Thanks. Martin

Try "-J [nr]" for parallel jobs on a same host, you can set nr to the
number of cpus + 1 and also "-B build_host_file" for building on remote
hosts.

regards,

--
Jérôme de Vivie