Versioning Binaries

[Dan Ayers]

Is there an industry standard for calculating the verison number of a
compiled binary from the version numbers of of all its components?

[William]

"Dan Ayers" <Daniel.Ayers@ir.com> wrote in message
news:ItsKa.8$bW.307@news.optus.net.au... Is there an industry standard for calculating the verison number of a compiled binary from the version numbers of of all its components?

I think you're thinking of a build number or a release version, something
like that? Build numbers are pretty arbitrary, though sequential, and
versions are often a marketing decision. -Wm

[Jerome de Vivie]

David Masterson wrote:
The version number of a compiled binary is given when it is checked into the repository like any other file.

Why using SCM tools to store compiled binaries ? That is the role of
backup tools.

regards,

j.

--
Jérôme de Vivie

[Jerome de Vivie]

William wrote:
I think you're thinking of a build number or a release version, something like that? Build numbers are pretty arbitrary, though sequential, and versions are often a marketing decision. -Wm

Builds could be identified by configurations, ie: the set of
("file","version") where all "file" are in the "make" dependencies list
of the build.

j.

--
Jérôme de Vivie

[Dwain Wilder]

Jerome de Vivie wrote: David Masterson wrote:>>>>Jerome de Vivie writes:>>>Why using SCM tools to store compiled binaries ? That is the role ofbackup tools.Because, depending upon your SCM requirements, you may want to versioncontrol the release as well as the source. Compiled binaries should be regenerable from sources. Version control generated binaries is dangerous because you don't test further your build process. Putting a label or tag and storing binaries should be enough.

There was a time, not so long ago, when I supported this position. However, I've
come to think that there might be any number of reasons for storing binaries in
the same structure and the same methodology you use for the rest of your
development assets. And please explain the labelling and tagging storing methods
you refer to. If you mean storing in the filesystem, how do you label and tag
the binaries?
There is another issue with generated binaries. POSIX permissions are not honored with existing SCM tools: checked-in version are read-only. This could lead write access problems at build process.Plus, if you do itproperly under ClearCase (ie. don't break the config-rec), you canwink it in in future builds and, thus, save considerable timereconstituting your environment to bug-fix old releases. Winkin need not version control of binaries.

I think he was referring to the Configuration Record generated by the build
process. It describes the whole build environment, as well as the version
selected for each module used.

Dwain
regards, j.


--
Dwain Wilder
Bear Meadow Folk Instruments
http://www.bearmeadow.com
_______________________________
You have to leave the city of your comfort and go into the wilderness of your
intuition. You can't get there by bus, only by hard work and risk and by not
quite knowing what you're doing.

- Alan Alda

[Jerome de Vivie]

Dwain Wilder wrote:
There was a time, not so long ago, when I supported this position. However, I've come to think that there might be any number of reasons for storing binaries in the same structure and the same methodology you use for the rest of your development assets.

I think that only elements which are modified should be managed under
version control. I don't understand your reason to use version control
when a simple backup could satisfied the need.
And please explain the labelling and tagging storing methods you refer to. If you mean storing in the filesystem, how do you label and tag the binaries?

I just tag all sources and i backed up sources + binaries. I use
ident(1) to identify products outside configuration management. That's
clean and enough. No need to version control generated things...

regards,

j.

--
Jérôme de Vivie

[Dwain Wilder]

Jerome de Vivie wrote: Dwain Wilder wrote:There was a time, not so long ago, when I supported this position. However, I'vecome to think that there might be any number of reasons for storing binaries inthe same structure and the same methodology you use for the rest of yourdevelopment assets. I think that only elements which are modified should be managed under version control. I don't understand your reason to use version control when a simple backup could satisfied the need.

The issue is configuration management, not version control. The reason to get
binaries stored is to associate them unequivocally with their sources.
And please explain the labelling and tagging storing methodsyou refer to. If you mean storing in the filesystem, how do you label and tagthe binaries? I just tag all sources and i backed up sources + binaries. I use ident(1) to identify products outside configuration management. That's clean and enough. No need to version control generated things...

Here's your original statement: "Putting a label or tag and storing binaries
should be enough." I understood you to mean that you were labeling and tagging
your stored binaries. Sorry if I misinterpreted you. Again, though, version
control isn't the only reason to consider when choosing items to be Configured
Items.

All the best,
Dwain
regards, j.


--
Dwain Wilder
Bear Meadow Folk Instruments
http://www.bearmeadow.com
_______________________________
You have to leave the city of your comfort and go into the wilderness of your
intuition. You can't get there by bus, only by hard work and risk and by not
quite knowing what you're doing.

- Alan Alda

[William]

"Jerome de Vivie" <"jerome . devivie "@ free . fr> wrote in message
news:3f0b0e4c$0$26626$626a54ce@news.free.fr... David Masterson wrote: Compiled binaries should be regenerable from sources. Version control generated binaries is dangerous because you don't test further your build process. Putting a label or tag and storing binaries should be enough.

We version our images for two reasons: 1) It's a convenient way to
distribute
them - faster and more reliable then the shared file system. 2) It gives us
a
reference in case a subtle bug appears - we can quickly go back to a
previous version and test for the bug without going through several hours
of building.

We also version binary tools, libraries, etc.
There is another issue with generated binaries. POSIX permissions are not honored with existing SCM tools: checked-in version are read-only. This could lead write access problems at build process.

Our SCM tool does handle permissions for the most part. Execute bits
can be associated with the file and the file can be left writable after
check in. Of course, the build process itself can always check out the files
if we want them to stay read-only when not checked out.-Wm

[Greg A. Woods]

David Masterson <David.Masterson@synopsys.com> writes: David Masterson (?) wrote: Compiled binaries should be regenerable from sources. Remember that the system environment is part of this equation. That is, compiled binaries should be regenerable from sources *AND* build tools *AND* O/S patch level *AND* ...

If your product files have inputs that are not tracked in your source
code change control system then your source code is incomplete, by
definition. Whether or not this matters will depend a great deal on
exactly how precise your build reproducibitliy must be. If you need
byte-for-byte regeneration (except maybe for timestamps identifing build
times for intermediate objects) then you need to be a hell of a lot more
complete than if all you care is that the resulting product "run" within
defined parameters sufficient to satisfy your Q/A department.

e.g. if you've got a C program that can only be successfully compiled by
a specific version of a specific compiler (a bad thing, but bear with
me), then your build system/script/makefile should explicity check for
the correct compiler and compiler version before proceeding and should
give meaningful errors to the builder when the appropriate compiler
cannot be found in the build environment.
Where do you store the binaries? Why shouldn't they be stored in your CM system (which should be backed up regularly anyway)?

It depends entirely on your CM system. :-)

Keep in mind that many version/change control tools are only tiny
components in an entire SCM system (e.g. CVS alone is not a SCM system,
and make alone is not a build system).

Also keep in mind that many change/version control tols are extremely
bad at managing files that cannot be meaningfully compared and merged
with line-oriented (text) diff/patch algorithms.

Finally, the whole idea of trying to find one gargantuan feature-rich
tool that can do everything for everyone is a waste of time. Use the
right tools for the right parts of your SCM. A good SCM will indeed
keep binary products around, but it won't store them in the source code
control system used for the project sources. Perhaps it will simply
keep them in well labeled archive files (e.g. *.tar.gz files), or
perhaps a more sophisticated repository that uses a good binary delta
compression algorithm (e.g. xdelta) can be used.

--
Greg A. Woods

+1 416 218-0098 VE3TCP RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com> Secrets of the Weird <woods@weird.com>

[Ron Ruble]

"Greg A. Woods" <woods@building.weird.com> wrote in message
news:87fz7ths8h.fsf@building.weird.com... David Masterson <David.Masterson@synopsys.com> writes:
<snip> e.g. if you've got a C program that can only be successfully compiled by a specific version of a specific compiler (a bad thing, but bear with me), then your build system/script/makefile should explicity check for the correct compiler and compiler version before proceeding and should give meaningful errors to the builder when the appropriate compiler cannot be found in the build environment.

Not atypical. Many environments run into problems in
C and especially C++ if the source code goes back
before certain revisions or requires certain features
that may be more recent or inconsistently supported
among vendors.

Even worse when you need third party libraries.
Where do you store the binaries? Why shouldn't they be stored in your CM system (which should be backed up regularly anyway)? It depends entirely on your CM system. :-) Keep in mind that many version/change control tools are only tiny components in an entire SCM system (e.g. CVS alone is not a SCM system, and make alone is not a build system).

I wish more people mentioned that ;>
Also keep in mind that many change/version control tols are extremely bad at managing files that cannot be meaningfully compared and merged with line-oriented (text) diff/patch algorithms.

The better ones allow you to specify which tools to
use for differencing.
Finally, the whole idea of trying to find one gargantuan feature-rich tool that can do everything for everyone is a waste of time. Use the right tools for the right parts of your SCM. A good SCM will indeed keep binary products around, but it won't store them in the source code control system used for the project sources. Perhaps it will simply keep them in well labeled archive files (e.g. *.tar.gz files), or perhaps a more sophisticated repository that uses a good binary delta compression algorithm (e.g. xdelta) can be used.

Right; just because you can do something doesn't mean you should.

[Marc Girod]

>>>>> "Ron" == Ron Ruble <raffles2@att.net> writes:

Ron> The better ones allow you to specify which tools to use for
Ron> differencing.

IMHO, this is not the issue.
Saving binaries as versioned objects only shows lack of imagination
(if it is not for backup purposes, i.e. in order to optimize a
potential restoration). There should be better ways to manage the
information they convey.

What you actually want to compare is not their (byte) contents, but
their meaning. One good idea is to notice that they are "derived
objects", i.e. "higher level configuration items" (first level items
are opaque to dependency management, they are equal in a flat world).

So, what you may want to compare is their dependency tree, what they
build upon. This is what ClearCase provides with clearmake -- only for
the result of builds, and with restrictions.

I am surprised that this wonderful idea hasn't had more success during
the last ten years. Even ClearCase support for it has been weakening
steadily.

--
Marc Girod P.O. Box 323 Voice: +358-71 80 25581
Nokia BI 00045 NOKIA Group Mobile: +358-50 38 78415
Valimo 21 / B616 Finland Fax: +358-71 80 64474

[Ron Ruble]

"Marc Girod" <girod@shire.ntc.nokia.com> wrote in message
news:u0tkbacxwleg6.fsf@merleau.ntc.nokia.com...>>> "Ron" == Ron Ruble <raffles2@att.net> writes: Ron> The better ones allow you to specify which tools to use for Ron> differencing. IMHO, this is not the issue. Saving binaries as versioned objects only shows lack of imagination (if it is not for backup purposes, i.e. in order to optimize a potential restoration). There should be better ways to manage the information they convey.

Key word being "should". I do not argue with your
point, I'm merely saying that in practice, there are
often constraints that limit flexibility.

Your comment: Saving binaries as versioned objects only shows lack of imagination
is partly true, but there may be other factors than
"lack of imagination". On a project I was on where
this was needed, we had the imagination to resolve
the issue and solve the problem. The third-party
tools we were required to use, coupled with a
lcak of calendar time, prevented implementing
these alternatives.*
What you actually want to compare is not their (byte) contents, but their meaning.

Very true.
One good idea is to notice that they are "derived objects", i.e. "higher level configuration items" (first level items are opaque to dependency management, they are equal in a flat world). So, what you may want to compare is their dependency tree, what they build upon.

True. Unfortunately, in some environments, separating
all the items the binaries are dependent on is non-trivial.

To sum up, I agree wholeheartedly with your
statements. Just pointing out that real-world
situations other than "lack of CM knowledge/
lack of imagination" can result in these kinds of
poor compromises.

---------------------------------------------

* I'll fess up. Excel spreadsheets with VBA code. I
wrote some macros to export and check in/check
out the source, and others to do a range-by-range
diff on two spreadsheets. We -should- have put
together code to rebuild the spreadsheet from a
flat configuration file describing it, but we just did
not have the time. It was faster to check in the
base spreadsheet, and import/export the VBA
modules when needed.

This project was "challenged" prior to any of the
development staff being brought on. We were
-not- brought on to do the project in VBA; the
original architecture was quite different at the
start. Various decisions were made in the absence
of technical staff caused huge changes; we just
tried to cope. Getting the decisions reversed
was not possible in this organization; corporate
culture was such that all communication from
above was orders, and the only acceptable
response from below was "yes, sir."

[Marc Girod]

>>>>> "Ron" == Ron Ruble <raffles2@att.net> writes:

Ron> Your comment: Saving binaries as versioned objects only shows lack of imagination
Ron> is partly true, but there may be other factors than
Ron> "lack of imagination".

I apologize for the harshness of my wording.
My motivation in chosing it was to fight in favour of some "not in the
main stream" thinking.

Too often, people confuse version control and SCM. By too often, I
mean for instance ClearCase UCM and subversion (not speaking of CVS
and Synergy). Both focus exclusively on "first level configuration
items", i.e. sources (whether text or not).

--
Marc Girod P.O. Box 323 Voice: +358-71 80 25581
Nokia BI 00045 NOKIA Group Mobile: +358-50 38 78415
Valimo 21 / B616 Finland Fax: +358-71 80 64474

[William]

"Greg A. Woods" <woods@building.weird.com> wrote in message
news:87fz7ths8h.fsf@building.weird.com...
Also keep in mind that many change/version control tols are extremely bad at managing files that cannot be meaningfully compared and merged with line-oriented (text) diff/patch algorithms.

In the case of most binaries, I'm more interested in reproducing a state
as of a certain point in time rather than finding minute changes from one
state to another. (In the case of a 3rd parth compiler, for example, I
care about the specific release of it, but not whether a line changed in
a library include file.)
Finally, the whole idea of trying to find one gargantuan feature-rich tool that can do everything for everyone is a waste of time. Use the right tools for the right parts of your SCM. A good SCM will indeed keep binary products around, but it won't store them in the source code control system used for the project sources.

Why not? We've had great success doing just that. Our VCS handles
them just like any other file set as far as the user is concerned (except
for
diffing - although it can be made to diff certains types such as MS
Word files).
Perhaps it will simply keep them in well labeled archive files (e.g. *.tar.gz files), or perhaps a more sophisticated repository that uses a good binary delta compression algorithm (e.g. xdelta) can be used.

It stores them as a compressed archive, but that's transparent to the
user. It's just another versioned object to the outside world. -Wm

[Marc Girod]

>>>>> "Wil" == William <Reply@NewsGroup.Please> writes:

Wil> Why not? We've had great success doing just that. Our VCS handles
Wil> them just like any other file set as far as the user is concerned
Wil> (except for diffing - although it can be made to diff certains
Wil> types such as MS Word files).

And for any others as well: run zcat, nm, tar, gdb, a disasembler,
etc...

Only, this is not very _generic_ as "management". It soon becomes
"specific". It is not SCM anymore.

What it looses on the other hand, is the generic information: e.g. the
dependency tree.

Wil> It's just another versioned object to the outside world.

Well, this shouldn't be a goal in itself...

--
Marc Girod P.O. Box 323 Voice: +358-71 80 25581
Nokia BI 00045 NOKIA Group Mobile: +358-50 38 78415
Valimo 21 / B616 Finland Fax: +358-71 80 64474

[Marc Girod]

>>>>> "MG" == Marc Girod <girod@shire.ntc.nokia.com> writes:>> "Wil" == William <Reply@NewsGroup.Please> writes:

Wil> It's just another versioned object to the outside world.

MG> Well, this shouldn't be a goal in itself...

I mean that presenting indistinctly derived objects and versioned
elements as normal files, e.g. in a file system as in usual
environments, *is* a reasonable goal (amenable to tradeoffs, as
usual).

Users should not be forced to care for the distinction. They should be
free to set their priorities.

--
Marc Girod P.O. Box 323 Voice: +358-71 80 25581
Nokia BI 00045 NOKIA Group Mobile: +358-50 38 78415
Valimo 21 / B616 Finland Fax: +358-71 80 64474

[FredBear]

Marc Girod wrote:
So, what you may want to compare is their dependency tree, what they build upon. This is what ClearCase provides with clearmake -- only for the result of builds, and with restrictions. I am surprised that this wonderful idea hasn't had more success during the last ten years. Even ClearCase support for it has been weakening steadily.

Build specification files like "Makefile" should be managed under
configuration, so we can track dependency tree without clearmake.

[FredBear]

jerome . devivie at free . fr wrote: Marc Girod wrote: So, what you may want to compare is their dependency tree, what they build upon. This is what ClearCase provides with clearmake -- only for the result of builds, and with restrictions. I am surprised that this wonderful idea hasn't had more success during the last ten years. Even ClearCase support for it has been weakening steadily. Build specification files like "Makefile" should be managed under configuration, so we can track dependency tree without clearmake.

And for external items like library or environment variables, it's smart
to manage under configuration an environment file which can be sourced.

Then, we can wrote a "Software Product Specification" or better: include
those informations into a "Build" chapter of the "Version Description
Document".

[Marc Girod]

>>>>> "jd" == jerome devivie
Build specification files like "Makefile" should be managed under configuration, so we can track dependency tree without clearmake.

jd> And for external items like library or environment variables, it's
jd> smart to manage under configuration an environment file which can be
jd> sourced.

jd> Then, we can wrote a "Software Product Specification" or better:
jd> include those informations into a "Build" chapter of the "Version
jd> Description Document".

I am sorry, but I believe your completely miss the point. Quite a
common case, in fact.

Makefiles may be versioned, yes, and it is is good, but totally remote
to what I wrote. In the same vein, you can document whatever, and
again, it is not bad, but again with only shallow relationship to the
issue.

Think at walking. You can walk, or write a document describing how
some people walked. Which one relates to managing walking? Now, start
running, swimming, or get a horse to trot or amble. And keep on with
your document.

What I am saying is that eventually, the part of "first level
configuration items" is negligible, both quantitatively, and
qualitatively.
In fact, there are essentially no such things. Treating a resource as
a versioned element is only a compromise, a poor man's starting point.

The "higher level CI" is more general: a (sophisticated) software
configuration is a balanced system, not a tree.

--
Marc Girod P.O. Box 323 Voice: +358-71 80 25581
Nokia BI 00045 NOKIA Group Mobile: +358-50 38 78415
Valimo 21 / B616 Finland Fax: +358-71 80 64474

[Zenin]

Marc Girod <girod@shire.ntc.nokia.com> wrote:snip< In fact, there are essentially no such things. Treating a resource as a versioned element is only a compromise, a poor man's starting point. The "higher level CI" is more general: a (sophisticated) software configuration is a balanced system, not a tree.

I'm not following you. In reading this and your other posts you
obviously look down upon tracking tool dependencies as standard objects,
although what's unclear is what failing you think this has and more
importantly what you believe is a better, "more sophisticated", method?

-Zenin

[FredBear]

Marc Girod wrote: I am sorry, but I believe your completely miss the point. Quite a common case, in fact. Makefiles may be versioned, yes, and it is is good, but totally remote to what I wrote. In the same vein, you can document whatever, and again, it is not bad, but again with only shallow relationship to the issue.

clearmake generate 3 things:
-the cmd (which could be documented in the SPS).
-a part of the environment (which could be managed under configuration
on a file)
-a list {(source_file,versions), ... } on which a derivate objects
depends on.

The last list is describe in the Makefile. So, by stamping those 3
things + the Makefile, i am able to do the relationship between sources
and objetcs without clearmake. They are both physicaly identified. For
me, the issue is fulfiled without clearmake or another crap tool.
Think at walking. You can walk, or write a document describing how some people walked. Which one relates to managing walking? Now, start running, swimming, or get a horse to trot or amble. And keep on with your document. What I am saying is that eventually, the part of "first level configuration items" is negligible, both quantitatively, and qualitatively. In fact, there are essentially no such things. Treating a resource as a versioned element is only a compromise, a poor man's starting point. The "higher level CI" is more general: a (sophisticated) software configuration is a balanced system, not a tree.

IMO, a "sophisticated" software configuration is a bad software
configuration. SCM should be as simpliest as possible and that's really
not the case of clearcase with it's concept of "derivated/versionned
objects".

[Marc Girod]

>>>>> "Zen" == Zenin <zenin@rhps.org> writes:

Zen> I'm not following you. In reading this and your other posts you
Zen> obviously look down upon tracking tool dependencies as standard
Zen> objects, although what's unclear is what failing you think this
Zen> has and more importantly what you believe is a better, "more
Zen> sophisticated", method?

What you call "standard objects" here is "versioned elements".
I.e. you take the point of view of the SCM administrator, with the
filtering glasses brought by his tool.

From the point of view of the user, it should be at first sight
indifferent whether a resource is a versioned element or a derived
object (or something else). Correct?

SCM should provide "generic" support for resources which have value
for the user. Its main weapon has been so far grouping certain
resources found across various software configurations (SC) into bags,
so that only one instance of every bag is found at most in any SC.
This is called the "exclusion principle", and allows for a two-step
identification: (1) the bag; (2) the instance within the bag. Bags are
made so that various instances within a bag bear something in common,
which makes it sensible to compare them.

Elements and versions have only been one mean to achieve this goal. No
more.

Now, with the technology of the previous century (barring clearmake),
this was acceptable, including the obvious shortcomings:
A. This creates an artificial partition between resources, from the
point of view of the user, as explained above.
B. The comparaison between instances within a bag is *specific* to the
type of the objects in the bag, assuming all instances bear the
same type (which is ...an artificial restriction). It is thus left
out of the scope of SCM proper.

I'll break here for now.

--
Marc Girod P.O. Box 323 Voice: +358-71 80 25581
Nokia BI 00045 NOKIA Group Mobile: +358-50 38 78415
Valimo 21 / B616 Finland Fax: +358-71 80 64474

[Marc Girod]

>>>>> "jd" == jerome devivie

jd> The last list is describe in the Makefile. So, by stamping those 3
jd> things + the Makefile, i am able to do the relationship between
jd> sources and objetcs without clearmake.

By the same reasoning, you don't need *any* SCM tool: just save
successive versions of your directories, and document the names of the
files. Why this doesn't work is left as an exercise for the reader.
Hint: not a good idea to play the tool yourself.

jd> They are both physicaly identified. For me, the issue is fulfiled
jd> without clearmake or another crap tool.

Wrong attitude here, IMHO.

jd> IMO, a "sophisticated" software configuration is a bad software
jd> configuration.

That's because you buy the bells and whistles of marketing people.
I am not speaking of graphical user interfaces, but of fundamental
issues (well in SCM, there is often something fundamentally backwards
in GUIs, and there, I may be following you: people who use a GUI tend
not to "manage" anything anymore; the system itself is soon
unmanageable and opaque).

Walking is fine, but with a plane, you can go farther.

--
Marc Girod P.O. Box 323 Voice: +358-71 80 25581
Nokia BI 00045 NOKIA Group Mobile: +358-50 38 78415
Valimo 21 / B616 Finland Fax: +358-71 80 64474

[FredBear]

Marc Girod wrote:
I am not speaking of graphical user interfaces, but of fundamental issues (well in SCM, there is often something fundamentally backwards in GUIs, and there, I may be following you: people who use a GUI tend not to "manage" anything anymore; the system itself is soon unmanageable and opaque). Walking is fine, but with a plane, you can go farther.

Unfortunately, "sophisticated" tools have problems with *fundamental*
issue of SCM; that's not a GUI problem. Clearcase is a well known SCM
tool and good exemple which goes in this category.

I will start by the first error which is an "historical error" in how a
great majority of SCM tools have been designed. This problem came from
which elementary object should be identify. All historical "version
oriented tools" are starting to identify individual objects of the
filesystem (file, directory, ...) with versions (or branch/revision).
IMO, it is a bad approch because when we do SCM, the interesting level
is the combination of software components which is important, i.e.: a
configuration: {(file,version), ...}. To achieve this, every "version
oriented tools" needs a second identification system to identify a set
(call tags, mark or label). This second identification level is stacked
on top the first one. There is a need to have a link between those 2
systems.

The main purpose of SCM is to identify software component. In "version
oriented" tools, we need 2 identifications systems and, we are not able
to link each individual version tree with the global developpement tree.
But that's not all: we are also able to construct random configuration
(with the "configspec" or individual selection) which has never exist by
the past / which do not derive from an historical stat of the filesystem
! This may introduce incoherency problems due to the fact that each file
has it's own "version tree"/"referential" independently to the others.
Version oriented tools are all based upon this not robust identification
model.

Few tools (arch ?), which i will call "configuration oriented" are based
on only one identification system "the configuration" which is a "change
set"+ "a link to a parent/baseline configuration". This "configuration"
identify a snapshot of the filesystem and thus, can identify each file
or directory inside this filesystem. From only one "configuration
tree"/"referential", we are able to deduce the version tree of each
files or directory. Historical changes can be followed globally. I'm
convinced that working at "configuration" or "change set" level is
natural and right.


Clearcase have just followed that historical design error. It have also
introduce many others tricky things. For exemple, it doesn't manage
under configuration metadata (permission, and other attributes). So a
"chprotect" will modify every version of an element. So, with a simple
command, we are able to modify previous release. The integrity and the
reproducablility of previous release is not fulfiled: that's a big issue
for a SCM tools !


With clearcase, it's possibility to use dynamic label like "LATEST" or
others moving label. I'm conviced that it is also a design error. For
me, only frozen configuration should be used as baseline. Letting user
work with dynamic baseline is not serious. For the same reason, we can't
rely on "clearmake" when there are "CHECKEDOUT" files.


I have not buy the bells, i have just suffer management choice. The SCM
process should be safe but clearcase is not. To be deployed and adopted
by everybody, the SCM process should be simple but clearcase is complex.
Yes, marketing works and probably better than engineering...

regards,

j

[Zenin]

Marc Girod <girod@shire.ntc.nokia.com> wrote: Zenin <zenin@rhps.org> writes: I'm not following you. In reading this and your other posts you obviously look down upon tracking tool dependencies as standard objects, although what's unclear is what failing you think this has and more importantly what you believe is a better, "more sophisticated", method? What you call "standard objects" here is "versioned elements". I.e. you take the point of view of the SCM administrator, with the filtering glasses brought by his tool.

A "versioned element" handled with "standard" or typical methods for
handling such. Semantics, whatever. -I think we both understand each
others terms well enough to not need to bat such silliness around, yes?

And if you'd come down from your Tower for a moment, you'd clearly see I
ware no such "filtering glasses". I take offense at the implication you
make.
From the point of view of the user, it should be at first sight indifferent whether a resource is a versioned element or a derived object (or something else). Correct?

Huh? We're talking about tool dependencies, no? The user shouldn't
know or care if their cc.exe is a derived object? I can't answer that,
the question is nonsensical.

Please clarify as you again aren't making much sense. This part isn't
semantics, I'm pretty sure I know what a "versioned element" and a
"derived object" is. I just don't understand why you're even bringing
up the distinction when talking about tool dependencies, or the user for
that matter?

Again to simplify the question asked is two fold:

A) What is wrong with versioning dependent (binary) tools in the repo
for the sake of more easily managing the full dependency graph?
B) If A can be answered reasonably (or for that matter at all), then
what is your suggestion to solve the problem differently?
SCM should provide "generic" support for resources which have value for the user. Its main weapon has been so far grouping certain resources found across various software configurations (SC) into bags, so that only one instance of every bag is found at most in any SC. This is called the "exclusion principle", and allows for a two-step identification: (1) the bag; (2) the instance within the bag. Bags are made so that various instances within a bag bear something in common, which makes it sensible to compare them. Elements and versions have only been one mean to achieve this goal. No more.

Thanks for the refresher course in SCM ideology; but you're still not
presenting any alternative whatsoever to throwing dependent tools into
the SCM repo. Heck, you still have yet to make any case whatsoever for
why such is at all a bad idea?
Now, with the technology of the previous century (barring clearmake), this was acceptable, including the obvious shortcomings: A. This creates an artificial partition between resources, from the point of view of the user, as explained above.

You haven't explained anything, above or elsewhere? What "artificial
partition"...the SCM client tool? What in blazes does any of this have
to do with sanely tracking (very real) tool dependencies?

You refer to the "previous century"; are you actually not arguing a
current practical solution and instead talking about some Ivory Tower
future "the way it should be" solution that has not current practical
relevance? I'm all for new ideas, new tactics, and discussions about
what the "perfect SCM" would look like, but I'd prefer to separate my
practical discussions from my theoretical. So can you please clarify of
which you are trying to describe?
B. The comparaison between instances within a bag is *specific* to the type of the objects in the bag, assuming all instances bear the same type (which is ...an artificial restriction). It is thus left out of the scope of SCM proper.

Who cares? The *only* comparison that has any meaning in this instance
is the simple test, "is something different". What is different really
doesn't matter, it's a question no one will ever ask of the SCM tool
when talking about dependent tools.

Throwing dependent tools into the repo is about sanely (and fully)
managing the dependency graph. It isn't about comparing two versions
of a tool (that's what the provider's documentation is for).

If you have a better solution to sanely tracking this part of the
dependency graph I'm all ears.

-Zenin

[Zenin]

jerome . devivie at free . fr <"jerome . devivie at free . fr"> wrote:snip< I will start by the first error which is an "historical error" in how a great majority of SCM tools have been designed. This problem came from which elementary object should be identify. All historical "version oriented tools" are starting to identify individual objects of the filesystem (file, directory, ...) with versions (or branch/revision). IMO, it is a bad approch because when we do SCM, the interesting level is the combination of software components which is important, i.e.: a configuration: {(file,version), ...}. To achieve this, every "version oriented tools" needs a second identification system to identify a set (call tags, mark or label). This second identification level is stacked on top the first one. There is a need to have a link between those 2 systems.

I'd call this good software design, not a product flaw. It affects not
the use of the tool in any way other then to offer more flexibility
should it be needed.
The main purpose of SCM is to identify software component. In "version oriented" tools, we need 2 identifications systems and, we are not able to link each individual version tree with the global developpement tree. But that's not all: we are also able to construct random configuration (with the "configspec" or individual selection) which has never exist by the past / which do not derive from an historical stat of the filesystem ! This may introduce incoherency problems due to the fact that each file has it's own "version tree"/"referential" independently to the others. Version oriented tools are all based upon this not robust identification model.

So the only possibly configuration you'd ever want to possibly exist are
complete snap shots of a given filesystem? Why not just use a network
filer with snap shot ability?

I'd view the base of your complaint here again as a feature, not a flaw,
as it offers flexibility without forcing any additional complexity on
those that don't use it.
Few tools (arch ?), which i will call "configuration oriented" are based on only one identification system "the configuration" which is a "change set"+ "a link to a parent/baseline configuration". This "configuration" identify a snapshot of the filesystem and thus, can identify each file or directory inside this filesystem. From only one "configuration tree"/"referential", we are able to deduce the version tree of each files or directory. Historical changes can be followed globally. I'm convinced that working at "configuration" or "change set" level is natural and right.

Change sets are very nice, but file system snap shots they do not imply.
BTW, "random configurations" can also be had just as easily with change
sets as they can without.

"random confs" can also be avoided just as easily. Again, it's
flexibility that's there to use if you want/need, but which costs you
nothing if you don't.
I have not buy the bells, i have just suffer management choice. The SCM process should be safe but clearcase is not. To be deployed and adopted by everybody, the SCM process should be simple but clearcase is complex. Yes, marketing works and probably better than engineering...

Clearcase is complex largely because it is flexible. What you ask for
sounds great if you're managing the /etc of a number of servers, but for
a software development group would severely limit workflow options while
doing absolutely nothing to improve build reliability.

But to turn this around, from what I'm reading it looks to me like
you're putting the tool before and above the process. SCM is 90%
process, 10% tools. Treat it otherwise and you'll be working too hard.

-Zenin

[Marc Girod]

>>>>> "Zen" == Zenin <zenin@rhps.org> writes:

Zen> And if you'd come down from your Tower for a moment, you'd
Zen> clearly see I ware no such "filtering glasses". I take offense at
Zen> the implication you make.

I don't think you should. I didn't intend to offense you, and it is
absolutely normal to try and model our different in order to resolve
it.

Roughly, I must assume there is a certain amount of misunderstanding
or disagreement between us. If you misunderstand what I say, it would
be much more serious if I believed --what I do not do-- that it was
desperate. Otherwise, believing that I am right, I must assume that
you are somehow wrong. Again, there is nothing infamous there, and it
depends only on you to convince me of the contrary (that you are
right, and I was wrong).

Zen> Huh? We're talking about tool dependencies, no? The user
Zen> shouldn't know or care if their cc.exe is a derived object? I
Zen> can't answer that, the question is nonsensical.

At first sight, the user doesn't care much how the cc.exe has arrived
there. In fact, if it does what she expects, she will probably not
look at this detail. I don't see what is nonsensical there.

The same file could be compiled and linked from sources, or extracted
from a tar, installed from an rpm, patched with a binary debugger such
as adb, unzipped, whatever, even poked byte per byte with a binary
editor. For most users, this is irrelevant.

She will become interested in the differences only if she intends to
fix or to modify something, or to compare with something else, etc.

If we cannot agree on the basics and on terminology, there is no point
in going further. If we need to recap some obvious things, that's too
bad, but shouldn't matter either.

So, do we continue, or should we stop here?

--
Marc Girod P.O. Box 323 Voice: +358-71 80 25581
Nokia BI 00045 NOKIA Group Mobile: +358-50 38 78415
Valimo 21 / B616 Finland Fax: +358-71 80 64474

[FredBear]

Zenin wrote:
I'd call this good software design, not a product flaw. It affects not the use of the tool in any way other then to offer more flexibility should it be needed.

[...jdv...]
So the only possibly configuration you'd ever want to possibly exist are complete snap shots of a given filesystem? Why not just use a network filer with snap shot ability?

Yes. Configurations should be snapshot of a filesystem (not necessary
form / but from startting at a mountpoint) at a given time.

I'm not using a network filer with snapshot ability because it has
limitations:
-the snapshot is often done at periodical time (time constraint) and not
on users demand (when they want to mark a new configuration).
-the backup process need to pass throw the whole filesystem (space
constraint), even for incremental snaps.
-there are possibility for fork/merge branch.

Each SCM tools have 2 layers: the storage layer and the communication
layer. For me a network filer cover only a very small part of the
storage layer. So that's not enough.

I'd view the base of your complaint here again as a feature, not a flaw, as it offers flexibility without forcing any additional complexity on those that don't use it.

[...jdv...]
Change sets are very nice, but file system snap shots they do not imply. BTW, "random configurations" can also be had just as easily with change sets as they can without. "random confs" can also be avoided just as easily. Again, it's flexibility that's there to use if you want/need, but which costs you nothing if you don't.


Imagine that your software have those dependencies (build dependencies):

myexe: producer.o consumer.o
producer.o: producer.c common.h
consumer.o consumer.c common.h

Now as each file has it's own referntial, you are able to compile
producer.o with (common.h, v1) and consumer.o with (common.h, v2). And
after that, link myexe with those 2 objetcs. I will not detailed
possibles consequencies...

This kind of problem is always possible with a "version oriented" tool
or, a tool which permit to create new configurations with random
selection (and selecton that have not been historical/ versions that
have not exist together at a time). So it is not a feature, it's a
*real* and demonstrated bug.


Fundamentaly, i see 3 way to construct new configurations:
-direct inheritance from a baseline/parent configuration.
-merge from a parent/baseline and form another configuration (deltas
from those two configurations can be compared with the first common
ancestor).
-direct selection of individual software component.

I can understand the use of the third one but only when components are
not linked together and should have their own lifecycle (i.e: ok at CSCI
level but not between CSUs which are parts of a CSCI).

Clearcase is complex largely because it is flexible. What you ask for sounds great if you're managing the /etc of a number of servers, but for a software development group would severely limit workflow options while doing absolutely nothing to improve build reliability.

When i spoke about snapshot of a filesystem, it's not necessary for the
whole filesystem, but only a part of it: the directory which contain our
CSCI.

IMHO, robustess is more important than flexibility in SCM. Using a
"configuration oriented" SCM tool limit possibilies but that's really
not a misfeature.

But to turn this around, from what I'm reading it looks to me like you're putting the tool before and above the process. SCM is 90% process, 10% tools. Treat it otherwise and you'll be working too hard.


We can't ignore tools. It is seen by people which use it and we also
rely on it to identify our software. Clearcase experience has shown that
a "flexible" and "sophisticated" need a hard work and a full time SCM
administrator. This kind of tool choice error is in fact very expensive.
The process is always up to me, so there is no problem or limitation.
The point for me is to avoid tool choice error made by management.


IMHO, "sophisticated", "feature rich" or "complexity" often sounds evil
is software.


regards,

j.

[William]

"jerome . devivie at free . fr" <"jerome . devivie at free . fr"> wrote in
message news:41051024$0$24060$626a14ce@news.free.fr... Zenin wrote: So the only possibly configuration you'd ever want to possibly exist
are complete snap shots of a given filesystem? Why not just use a
network filer with snap shot ability? Yes. Configurations should be snapshot of a filesystem (not necessary form / but from startting at a mountpoint) at a given time.

What is it about your tool that makes this difficult to do? (I can certainly
version nearly any portion of my filesystems at any time I chose and
recreate
it with a command or two (assuming I wanted to devote the storage space
to it - I prefer a more triaged approach in reality).

Each SCM tools have 2 layers: the storage layer and the communication layer. For me a network filer cover only a very small part of the storage layer. So that's not enough.

Mine has 3, storage, communication and database. How that third one is
implemented and the features it implements are, I think, the key differences
in SCM tools. It's what lets me reproduce past configurations based on
several "markers" - some of which do not require a past action to place.
(specifically, a date and time or at the momemt a given change was
submitted).

-Wm

[FredBear]

William wrote:
What is it about your tool that makes this difficult to do? (I can certainly version nearly any portion of my filesystems at any time I chose and recreate it with a command or two (assuming I wanted to devote the storage space to it - I prefer a more triaged approach in reality).

In filesystems, 3dfs from AT&T or translucency FS from Sun, have some
approch which could do this.

Arch seems the more mature "change set" oriented SCM tools in free software.

Mine has 3, storage, communication and database. How that third one is implemented and the features it implements are, I think, the key differences in SCM tools. It's what lets me reproduce past configurations based on several "markers" - some of which do not require a past action to place. (specifically, a date and time or at the momemt a given change was submitted).

Storage & database layer seems near. If your tool is able to reproduce
*exact* past configuration based on markers, it fulfield completely
integrity & identification needs of SCM. Very few SCM tools does this
correctly...

By curiosity, which tool do you use ?

regards,

j.

[Zenin]

Marc Girod <girod@shire.ntc.nokia.com> wrote:>>> "Zen" == Zenin <zenin@rhps.org> writes: Zen> Huh? We're talking about tool dependencies, no? The user Zen> shouldn't know or care if their cc.exe is a derived object? I Zen> can't answer that, the question is nonsensical. At first sight, the user doesn't care much how the cc.exe has arrived there. In fact, if it does what she expects, she will probably not look at this detail. I don't see what is nonsensical there. The same file could be compiled and linked from sources, or extracted from a tar, installed from an rpm, patched with a binary debugger such as adb, unzipped, whatever, even poked byte per byte with a binary editor. For most users, this is irrelevant.

Yes and no. Yes, they don't care so much how it gets there, but they
do/will care that it gets updated (or downgraded) correctly, easily, and
reliably as required to match the dependency tree. Similarly it's in
the interests of *other* users that everyone be on the same known page
(environmental configuration).

The environment is the root of the configuration. Whenever technically
possible it makes sense to manage that part of the configuration with
the same mechanisms; Why reinvent the wheel? If it's a direct "install"
via the SCM tool or a set of Tool/Setup.EXE packages it seems to me the
easiest and most reliable method for a user to obtain the correct
environment for the project they are working on is to fetch that
environment from the same SCM tool that's already tracking everything
else.

Sanity checks of tool versions inside make files and such can be useful
for checking the local config, but by definition they don't do anything
to maintain the config. Versioned spreadsheets listing dependent tools
and their versions is a step up from nothing, but it's both easier and
far more reliable to put the tools themselves under the SCM tool.

Again, if there's a better (easier, more reliable, etc) method I'm all
ears. But so far you have yet to offer even the most vague of
alternatives? Saying something is bad without offering any alternative
whatsoever or even just explaining why it is bad isn't helpful.
She will become interested in the differences only if she intends to fix or to modify something, or to compare with something else, etc.

Or something won't build because a tool is of the wrong version, or her
unit testing will be for not because her environment doesn't match what
will actually be used, or someone else will care when she used features
of a newer version of a tool that no one else has upgraded to yet.

Tacking tool dependencies isn't about any need to ever modify or compare
them. I think perhaps you've created a red herring for yourself with
this issue?
If we cannot agree on the basics and on terminology, there is no point in going further. If we need to recap some obvious things, that's too bad, but shouldn't matter either. So, do we continue, or should we stop here?

Lets continue. I'm still interested in actually hearing why it's at all
a bad idea to track binary tools within the SCM system and if it
actually is bad what better methods you'd suggest?

How about you give us/me the benifit of the doubt and just make believe
for one post that any semantics you choose to use will be 100%
understood as you do. In this post please actually address the topic at
hand. We can haggle out semantics if we need to after you've actually
tried to make a point, but please note there is a reason glossaries are
placed at the *back* of the book.

Your terms are understood crystal clearly; there is no need to have a
long pre-thread on semantics before actually making your point. If you
have a point to make please make it. Thanks.

-Zenin

[Zenin]

jerome . devivie at free . fr <"jerome . devivie at free . fr"> wrote:snip< Imagine that your software have those dependencies (build dependencies): myexe: producer.o consumer.o producer.o: producer.c common.h consumer.o consumer.c common.h Now as each file has it's own referntial, you are able to compile producer.o with (common.h, v1) and consumer.o with (common.h, v2). And after that, link myexe with those 2 objetcs. I will not detailed possibles consequencies... This kind of problem is always possible with a "version oriented" tool or, a tool which permit to create new configurations with random selection (and selecton that have not been historical/ versions that have not exist together at a time). So it is not a feature, it's a *real* and demonstrated bug.

For such "random" selection to take place someone needs to actually ask
for it explicitly in most every tool I've ever seen. Such
"non-historical" configurations are not random, they are in fact very
deliberate.

It's very often useful to create a new configuration piecemeal. It's
also just as reliable as any "explicitly historical" version; In the end
it's going to have to be built, reviewed, and tested through the same
process. Little is lost. Add a tool with change sets and nothing is
lost.

What you are asking for is the default for probably every tool ever
made. In an ideal world branches would always be created exactly when
needed automatically and would have zero overhead or adverse impact of
any kind. Practical engineering however, especially in highly dynamic
settings such as corporate IT, often demand considerably more
flexibility.

The *ability* to do something different, demonstrable or not, is *not* a
bug. Simply because you do not find it useful doesn't mean that a great
many others haven't either. I'd go further to say that if it wasn't
such a wildly popular feature, demonstrated to be very reliable when
used correctly, that it wouldn't be included in nearly every SCM repo
tool ever made.
Fundamentaly, i see 3 way to construct new configurations: -direct inheritance from a baseline/parent configuration. -merge from a parent/baseline and form another configuration (deltas from those two configurations can be compared with the first common ancestor). -direct selection of individual software component. I can understand the use of the third one but only when components are not linked together and should have their own lifecycle (i.e: ok at CSCI level but not between CSUs which are parts of a CSCI).

In a group with long development lifecycles such as developing a
shrink-wrap product you can probably manage to do this. In a typical IT
setting however, with very fast lifecycles and very dynamic requirements
practicality trumps all. There is no one size fits all and the ability
to quickly (often within minutes) rearrange a configuration drastically
and reliably is of critical value to me and many others.

I should note here that shrinkwrap software groups are but a fraction of
the software engineering landscape; the typical case has a very fast and
dynamic lifecycle.

When I need A and B but not C I have many options which vary depending
on esactly what A, B, and C are and where they are found. Rolling back
the entire tree and rewriting code verbatim isn't a great option.
Branching pieces to a new line has the same result as piecemeal
labeling, just with a lot of overhead and complexity added for everyone
making the process infact *less* reliable.
Clearcase is complex largely because it is flexible. What you ask for sounds great if you're managing the /etc of a number of servers, but for a software development group would severely limit workflow options while doing absolutely nothing to improve build reliability. When i spoke about snapshot of a filesystem, it's not necessary for the whole filesystem, but only a part of it: the directory which contain our CSCI. IMHO, robustess is more important than flexibility in SCM. Using a "configuration oriented" SCM tool limit possibilies but that's really not a misfeature.

Again I'd argue it's a balance and what is actually more important and
to what degree varies greatly between organizations and projects.

And I'd still argue that used sensibly, piecemeal configurations are no
less robust then "snap shot" configs. Indeed, in many situations they
will be the *most* reliable option available (all things considered,
much including time).
But to turn this around, from what I'm reading it looks to me like you're putting the tool before and above the process. SCM is 90% process, 10% tools. Treat it otherwise and you'll be working too hard. We can't ignore tools. It is seen by people which use it and we also rely on it to identify our software. Clearcase experience has shown that a "flexible" and "sophisticated" need a hard work and a full time SCM administrator. This kind of tool choice error is in fact very expensive. The process is always up to me, so there is no problem or limitation. The point for me is to avoid tool choice error made by management.

This I'll agree with. Large tools brought in without full consideration
are almost always a mistake, they become huge boat anchors. Clearcase
is very much in this category. They are great when they fit, but when
they don't fit they *really* don't fit.
IMHO, "sophisticated", "feature rich" or "complexity" often sounds evil is software.

Much agreed. For the record I deal mostly with one of the most
primitive SCM tools around, CVS. :)

-Zenin

[William]

"jerome . devivie at free . fr" <"jerome . devivie at free . fr"> wrote in
message news:41052f7f$0$30098$626a14ce@news.free.fr... By curiosity, which tool do you use ?

Perforce. -Wm

[FredBear]

Zenin wrote: The *ability* to do something different, demonstrable or not, is *not* a bug. Simply because you do not find it useful doesn't mean that a great many others haven't either. I'd go further to say that if it wasn't such a wildly popular feature, demonstrated to be very reliable when used correctly, that it wouldn't be included in nearly every SCM repo tool ever made.

"popular feature" or "reliable when used correctly" is not enough for me.

In a group with long development lifecycles such as developing a shrink-wrap product you can probably manage to do this. In a typical IT setting however, with very fast lifecycles and very dynamic requirements practicality trumps all. There is no one size fits all and the ability to quickly (often within minutes) rearrange a configuration drastically and reliably is of critical value to me and many others.

Agree. Very fast lifecycles in SCM generaly mean very often
synchronisation. That's good to reduce the size of "change sets" you
have to merge. Rearranging drastical configuration in only a few minutes
can raise errors. It was so critical for me that i have asked each
developpers to merge my "baseline branch" into their "developpement
branch" before i merge back their work on my "baseline branch". With
this process, i was able to avoid a big part of the integration work and
thus errors.

I should note here that shrinkwrap software groups are but a fraction of the software engineering landscape; the typical case has a very fast and dynamic lifecycle. When I need A and B but not C I have many options which vary depending on esactly what A, B, and C are and where they are found. Rolling back the entire tree and rewriting code verbatim isn't a great option. Branching pieces to a new line has the same result as piecemeal labeling, just with a lot of overhead and complexity added for everyone making the process infact *less* reliable.


Ok. I see your point: being able to select A & B but not C. In this
case, i admit that direct selection is easier. When you need to do this,
this probably mean that C has been merged to early and is not stable
enough, so it should be removed (temporary ?) from baseline branch. It's
often a needed regression and, if the software is modular enough, you
could probaly removed it with a one line patch in the Makefile.

Again I'd argue it's a balance and what is actually more important and to what degree varies greatly between organizations and projects. And I'd still argue that used sensibly, piecemeal configurations are no less robust then "snap shot" configs. Indeed, in many situations they will be the *most* reliable option available (all things considered, much including time).


With balance and good SCM level, you are able to deal reliabily with
piecemeal configurations and i saw your point. I dislike to limit SCM
possibilities but when i saw complexity, i just try to made people aware
of possible risks. It's a bit like using C++ langage: we can do
everything but it's also easy to do bad things with this langage. If
everybody in the team have stong knowledge of software engineering,
there is no problems !


This I'll agree with. Large tools brought in without full consideration are almost always a mistake, they become huge boat anchors. Clearcase is very much in this category. They are great when they fit, but when they don't fit they *really* don't fit.


Fully agree.

IMHO, "sophisticated", "feature rich" or "complexity" often sounds evil issoftware. Much agreed. For the record I deal mostly with one of the most primitive SCM tools around, CVS. :)

;-)

j.