SCM tool for multi-site development

[Sianturi Saleh]

Hi,

imagine this scenario: 2 developer sites. One of them hosts a SCM Server
and is connected to the internet via DSL. The other site has ISDN
connection. Development is done on both sites. Network connection is not
really reliable.

What SCM tool would you recommend? I'm particularly thinking about CVS
vs. Clearcase.

Thanks,
Sianturi

[Stephen Baynes.]

"Sianturi Saleh" <sianturi_saleh@web.de> wrote in message
news:be37e3$ncfi$1@ID-165034.news.dfncis.de... Hi, imagine this scenario: 2 developer sites. One of them hosts a SCM Server and is connected to the internet via DSL. The other site has ISDN connection. Development is done on both sites. Network connection is not really reliable. What SCM tool would you recommend? I'm particularly thinking about CVS vs. Clearcase.


Telelogic Synergy supports data replication over different sites in a
similar way to Clearcase.
Perforce offers various features for accessing remote repositories with some
local caching which can probably cope with network not being available at
times.

--
Stephen Baynes CEng MBCS
DTG-S&S, Philips Semiconductors Southampton
Tel +44 23 80316431

[Stefan Aufischer]

Robert Cowham wrote: Sianturi Saleh <sianturi_saleh@web.de> wrote:Hi,imagine this scenario: 2 developer sites. One of them hosts a SCMServer and is connected to the internet via DSL. The other site hasISDN connection. Development is done on both sites. Network connectionis not really reliable.What SCM tool would you recommend? I'm particularly thinking about CVSvs. Clearcase.

I'd recommend CVS, because working offline (in a local sandbox) or with
a slow connection is not a problem.
ClearCase MultiSite needs a very reliable and fast network connection,
when using its network file system (don't remember how it is called).
If money is not an issue, you could go with ClearCase and snapshot views
(same concept as CVS sandboxes).
ClearCase is the only tool as far as I am aware that has true multi-site capabilities, and even then one site is the Master for a branch (though I believe mastership can move to other sites). That said, mlutisite comes at quite a price.

I believe Sianturi meant multi-site development, not mutli-site
repositories!

-Stefan

[Chuck Karish]

Stephen Baynes. wrote: "Sianturi Saleh" <sianturi_saleh@web.de> wrote in message news:be37e3$ncfi$1@ID-165034.news.dfncis.de...imagine this scenario: 2 developer sites. One of them hosts a SCM Serverand is connected to the internet via DSL. The other site has ISDNconnection. Development is done on both sites. Network connection is notreally reliable.What SCM tool would you recommend? I'm particularly thinking about CVSvs. Clearcase. Telelogic Synergy supports data replication over different sites in a similar way to Clearcase.

PVCS Dimensions and Bitkeeper claim to manage replicated remote
repositories, too.
Perforce offers various features for accessing remote repositories with some local caching which can probably cope with network not being available at times.

Perforce doesn't do replication that preserves a consistent view
of the metadata. It's painful to use through a small pipe if your
users need up-to-date access to a large file tree.

[Shayne Wissler]

Kaz Kylheku wrote:
Commits can be inconsistent for reasons other than lack of atomicity. Even with an atomic commit feature in place, CVS could still be told to selectively commit specific files. That could result in an inconsistency. Developers can create inconsistencies in other ways, like forgetting to add newly created local files under version control.

It's one thing to purposely commit a subset of files. It's an entirely
different thing to have the system inadvertently do that.
Atomicity is quite important in other domains: such as updating multiple bank accounts as one transaction, etc. It's not that important in version control, where we never lose anything: commits don't erase any history so there is no hazard condition. There may be transient periods of inconsistency, but consistency is achieved eventually.

Perhaps you have a high tolerance for easily avoided nuisances. I don't. CVS
is a relic that deserves to be applauded for how well it served in the
past, but buried nonetheless. Lack of atomicity is just one out of a list
of problems that should not be put up with year after year.


Shayne Wissler

[Mike Castle]

In article <cnKOa.12333$N7.2057@sccrnsc03>,
Shayne Wissler <thales000@yahoo.com> wrote:Perhaps you have a high tolerance for easily avoided nuisances. I don't. CVSis a relic that deserves to be applauded for how well it served in thepast, but buried nonetheless. Lack of atomicity is just one out of a listof problems that should not be put up with year after year.


In all of my time working with Perforce (with atomic transactions) and CVS
(without atomic transactions), I've had about the same number of transient
inconsistencies. And those were for the reasons Kaz pointed out:
developers submitted an improper set of files, either not all of the ones
appropriate for a change or new files, occasionally even a deleted file.

Even if your SCM software does provided a transactional feature, it doesn't
mean it gets used properly.

Only one time in years of using CVS have I ever actually had a situation
where an update while commit was in progress resulted in an unusable
combination. And a quick 'cvs up' took care of that.

In my opinion, it's simply not a significant enough of a problem to bother
with. (I mean, something like W2K crashing is more problematic and should
require more attention than CVS and atomic updates.)

mrc

--
Mike Castle dalgoda@ix.netcom.com www.netcom.com/~dalgoda/
We are all of us living in the shadow of Manhattan. -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc

[=?ISO-8859-15?Q?Jes=FAs_M=2E_NAVARRO?=]

Hi, Shayne:

Shayne Wissler escribió en comp.software.config-mgmt:
Mike Castle wrote:
[...]
In my opinion, [talking about atomicity] it's simply not a significant
enough of a problem to bother with. (I mean, something like W2K crashing is more problematic and should require more attention than CVS and atomic updates.) Well, W2K crashing is yet another *reason* to have atomicity. It's an easy problem to solve, and is in fact solved with modern SCM systems. It's hard for me to see why anyone would try to argue against having this feature.

Well, talking about it is cheaper than taking care of it. Since it is a
problem most of us haven't seen *ever* or, as most, just once, as Mike
recognices, it's reasonable nobody has taken the work of resolve it (which
is more work than just talking about how horrible is CVS not having atomic
multifile checkins). Still, if you feel it is so horrible, you are free to
publish your patch to solve it.
--
SALUD,
Jesús
***
jesus_navarro@undominio.net
***

[Chuck Karish]

Mike Castle wrote:
Only one time in years of using CVS have I ever actually had a situation where an update while commit was in progress resulted in an unusable combination. And a quick 'cvs up' took care of that.

'cvs up' doesn't take care of the case where two commits move
overlapping sets of files in different directions. Somebody has to
re-merge to fix that.

[Kaz Kylheku]

Shayne Wissler <thales000@yahoo.com> wrote in message news:<4%oPa.27146$wk6.6116@rwcrnsc52.ops.asp.att.net>... Mike Castle wrote: In article <cnKOa.12333$N7.2057@sccrnsc03>, Shayne Wissler <thales000@yahoo.com> wrote:Perhaps you have a high tolerance for easily avoided nuisances. I don't.CVS is a relic that deserves to be applauded for how well it served in thepast, but buried nonetheless. Lack of atomicity is just one out of a listof problems that should not be put up with year after year. In all of my time working with Perforce (with atomic transactions) and CVS (without atomic transactions), I've had about the same number of transient inconsistencies. And those were for the reasons Kaz pointed out: developers submitted an improper set of files, either not all of the ones appropriate for a change or new files, occasionally even a deleted file. Atomicity-related failures are rare for small projects with reliable hardware. That doesn't mean that atomicity isn't a value.

It also doesn't mean that there is a need for a mass exodus of users
from the system that doesn't have coarse-grained atomicity.

CVS has atomicity. A file is either committed or not committed. The
crash recovery procedure is simple: recurse through the repository,
clean away stale locks and temporary files.

With CVS you can repeat the commit as many times as you need to to get
the files in; each individual commit may fail partway through, but
nevertheless make some permanent progress toward consistency.

A design that incorporates atomic transactions must be able to support
this use case in order to be useful. It's not good enough to abort the
whole transaction when the connection fails; there must be a way to
get the materials to the server in as many connection attempts as it
takes, and then trigger the actual commit.

You can hack atomicity with CVS like this: have two identical
sandboxes; one locally and one on the server. Before you commit, use
rsync or a similar tool to synchronize your changes from the remote
sandbox to the local sandbox. Do it as many times as needed to
overcome the unreliable network. Then issue a remote commit command on
the remote sandbox.
Even if your SCM software does provided a transactional feature, it doesn't mean it gets used properly. So? Only one time in years of using CVS have I ever actually had a situation where an update while commit was in progress resulted in an unusable combination. And a quick 'cvs up' took care of that. So?

So it means that it's not a real issue that actually affects a
significant fraction of CVS users a significant fraction of their
time, so people who bring it up are typically just blowing smoke.
``Buy my proprietary system because it solves a case that happens one
in a million times under CVS, and is taken care of by repeating a
command''.

[Mike Castle]

In article <4%oPa.27146$wk6.6116@rwcrnsc52.ops.asp.att.net>,
Shayne Wissler <thales000@yahoo.com> wrote:It's an easy problem to solve, and is in fact solved with modern SCMsystems. It's hard for me to see why anyone would try to argue againsthaving this feature.

I'm not arguing against having the feature. I'm arguing that the feature,
in real life, really doesn't offer you a whole lot.

There are just more important things to worry about.

mrc
--
Mike Castle dalgoda@ix.netcom.com www.netcom.com/~dalgoda/
We are all of us living in the shadow of Manhattan. -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc

[Shayne Wissler]

Kaz Kylheku wrote:
Atomicity-related failures are rare for small projects with reliable hardware. That doesn't mean that atomicity isn't a value. It also doesn't mean that there is a need for a mass exodus of users from the system that doesn't have coarse-grained atomicity.

Ah, well that explains it...

But, if my purpose was to list all the very good reasons for leaving CVS
and/or VSS behind, I would have.


Shayne Wissler

[Shayne Wissler]

Mike Castle wrote:
In article <4%oPa.27146$wk6.6116@rwcrnsc52.ops.asp.att.net>, Shayne Wissler <thales000@yahoo.com> wrote:It's an easy problem to solve, and is in fact solved with modern SCMsystems. It's hard for me to see why anyone would try to argue againsthaving this feature. I'm not arguing against having the feature. I'm arguing that the feature, in real life, really doesn't offer you a whole lot. There are just more important things to worry about.

Fortunately, we don't have to face this alternative in the SCM world. We can
deal with the "more important things", and with the atomicity.


Shayne Wissler

[Chuck Karish]

Kaz Kylheku wrote:
Shayne Wissler <thales000@yahoo.com> wrote in message news:<4%oPa.27146$wk6.6116@rwcrnsc52.ops.asp.att.net>...Atomicity-related failures are rare for small projects with reliablehardware. That doesn't mean that atomicity isn't a value. It also doesn't mean that there is a need for a mass exodus of users from the system that doesn't have coarse-grained atomicity.

If it works for you, use it.
With CVS you can repeat the commit as many times as you need to to get the files in; each individual commit may fail partway through, but nevertheless make some permanent progress toward consistency. A design that incorporates atomic transactions must be able to support this use case in order to be useful. It's not good enough to abort the whole transaction when the connection fails; there must be a way to get the materials to the server in as many connection attempts as it takes, and then trigger the actual commit.

I don't see why this is needed when it's possible to make the system
reliable enough that the case doesn't arise.
You can hack atomicity with CVS like this: have two identical sandboxes; one locally and one on the server. Before you commit, use rsync or a similar tool to synchronize your changes from the remote sandbox to the local sandbox. Do it as many times as needed to overcome the unreliable network. Then issue a remote commit command on the remote sandbox.

That's not an atomic transaction. That's doing what you can to win in a
race condition.
So it means that it's not a real issue that actually affects a significant fraction of CVS users a significant fraction of their time, so people who bring it up are typically just blowing smoke.

Or they're working in projects that are big enough that they need the
feature.
``Buy my proprietary system because it solves a case that happens one in a million times under CVS, and is taken care of by repeating a command''.

If you want CVS, you know where to find it. Have a nice day.

The pathological case that's solved by the transactional checkin model
is a race condition between two checkins, not the one you keep
describing that's a race between reading and writing.

Anyway, the ability to do transactional checkins is just one consequence
of a design choice that offers significant improvements in speed, in
usability, and in reporting capabilities.

[Robert Cowham]

"Mark Bools" <mb.usenet@REMOVETHISinspect-uk.com> wrote in
news:beptpt$cgd$1$8302bc10@news.demon.co.uk:
Atomic replacement of changes (sets of files) reduces cognitive load. This is _good_ because it means I, as a developer, have less to remember. On a busy work schedule were my time is divided between developments and fixing problems I really like tools that help me. I have more important things to think about, that's precisely the reason why change sets are a good idea. As a configuration manager I am more comfortable when a developer tells me "this set of file changes will fix that fault". It's neat and tidy, it reduces my cognitive work load.

Absolutely.

When looking at the history of a file, it is often really useful when
looking at a particular version to be able to easily answer the question
"what else was changed together with this version" (i.e. as part of the
same changeset)? In CVS and some other tools you have to search for items
changed around the same time and probably with a similar checking comment
which is rather less useful. Subversion has this capability.

It is also particularly useful to be able to propagate changesets between
branches rather than individual changes to individual files.

Now developers can make life difficult by submitting massive changesets
with lots of unrelated changes together, but a little eduction usually
sorts this out...

--
Robert Cowham

[Chuck Karish]

NotMy RealName wrote:
For example, backing out a change that that altered content of 10 files, added one file, and removed another. This should be easy, but most solutions fall down here, regardless of whether they use atomic database commits or not.

It works fine in Perforce, as long as there have not been subsequent
modifications to the files that depend on what's being backed out.

[NahtMy ReelName]

Chuck Karish wrote: NotMy RealName wrote: For example, backing out a change that that altered content of 10 files, added one file, and removed another. This should be easy, but most solutions fall down here, regardless of whether they use atomic database commits or not. It works fine in Perforce, as long as there have not been subsequent modifications to the files that depend on what's being backed out.

Which basically says you can't back out the change weeks or months
later. This unfortunate restriction tends to be an artifact of
delta-based file version management.

Changeset offerings that are built on composite versioning
don't have this limitation, it's a major feature of their
implementation. You can enable/disable the cset at any time.

That isn't to say that composite versioning is a magic bullet, there
/can/ be conflicts and such that require reconciliation in the form of
additional csets, but by and large it works great.

Technically the limitation can be removed for delta-based
(non-composite) implementations too, but it's a lot more work and
generally isn't done. I suspect this is how klearkase attempts to
implement its cset behavior but don't have experience with klearkase
csets. (Any experienced users care to comment?)

[Jorgen Grahn]

On 14 Jul 2003 11:15:05 GMT, Robert Cowham <rc@vaccaperna.co.uk> wrote:
.... When looking at the history of a file, it is often really useful when looking at a particular version to be able to easily answer the question "what else was changed together with this version" (i.e. as part of the same changeset)? In CVS and some other tools you have to search for items changed around the same time and probably with a similar checking comment which is rather less useful. [...]

The 'cvs2cl' script can help somewhat, at least if you don't branch much.
Similar tools might exist for other version control systems.

/Jorgen

--
// Jorgen Grahn <jgrahn@ ''If All Men Were Brothers,
\X/ algonet.se> Would You Let One Marry Your Sister?''

[Kaz Kylheku]

Chuck Karish <karish@well.com> wrote in message news:<tgUPa.5656$_K5.1548@fe01.atl2.webusenet.com>... Kaz Kylheku wrote: A design that incorporates atomic transactions must be able to support this use case in order to be useful. It's not good enough to abort the whole transaction when the connection fails; there must be a way to get the materials to the server in as many connection attempts as it takes, and then trigger the actual commit. I don't see why this is needed when it's possible to make the system reliable enough that the case doesn't arise.

The reason is that you don't need atomic transactions at all in that
case. Atomic transactions are there to provide fault tolerance when
there are interruptions.

When everything is working reliably, you just need mutual exclusion
that is not necessarily fault tolerant. CVS already locks down
directory trees; there can't be concurrent commits, or concurrent
commits and updates, in the same tree.
You can hack atomicity with CVS like this: have two identical sandboxes; one locally and one on the server. Before you commit, use rsync or a similar tool to synchronize your changes from the remote sandbox to the local sandbox. Do it as many times as needed to overcome the unreliable network. Then issue a remote commit command on the remote sandbox. That's not an atomic transaction. That's doing what you can to win in a race condition.

There is no race condition; I didn't make it sufficiently clear that
there is a 1:1 correspondence between these local and remote
sandboxes. There is no race condition during the transfer of material
from one to the other, because there is no concurrency! They are
effectively functioning as one sandbox that has been distributed
across two machines. It's like synchronization between a PDA and a
desktop computer, or whatever.
So it means that it's not a real issue that actually affects a significant fraction of CVS users a significant fraction of their time, so people who bring it up are typically just blowing smoke. Or they're working in projects that are big enough that they need the feature.

Hardly. The probability that a commit will be interrupted by a network
outage or server failure increases with the size of the commit, not to
the size of the project. Larger projects don't necessarily have larger
commits.

I'm no more likely to be affected by a network problem when committing
half a day's work to a 2 gigabyte CVS repository than to a 500
kilobyte repository. (All else being equal, which it rarely is. For
all we know, the larger project may be better funded and have better
hardware all round).
``Buy my proprietary system because it solves a case that happens one in a million times under CVS, and is taken care of by repeating a command''. If you want CVS, you know where to find it. Have a nice day.

Whether or not I want CVS or know where to find it is irrelevant. I
oppose the spreading of clueless misinformation and distortions about
CVS.
The pathological case that's solved by the transactional checkin model is a race condition between two checkins, not the one you keep

Usually, concurrency solutions that address a read-write hazard also
implicitly address a write-write hazard. Write-write hazards aren't
really interesting as a separate category of mutual exclusion
problems.

If something that is being written is locked out against reading, why
wouldn't
it also be locked out against concurrent writing?

This is why I tend not to touch upon the write-write hazard in
discussion; that doesn't mean I'm overlooking it.
describing that's a race between reading and writing.

Your understanding of CVS is imperfect. CVS has no race condition
between two checkins. It has locks which effectively give rise to
transactions.

The problem being discussed is that if the connection breaks, the
locks go away. You get a half-done commit; sandboxes can be now
updated to that half-done commit, and new work can be based on it,
etc.

There are (at least) two kinds of atomicity: mutual exclusion, and
fault-tolerance.

CVS transactions assure mutual exclusion, but only limited fault
tolerance, to the extent that in a network interruption or crash you
don't lose individual files, but you lose the integrity of the commit
as a whole. By the time you re-try it, another commit may have taken
place; you have to integrate with that and keep going.

Additionally, the mutual exclusion is relaxed when commits or updates
span multiple directories, directories, as documented. If one sandbox
does a ``cvs commit dirA dirB'' and some other does a ``cvs update
dirA dirB'', that sandbox may pick up the new version of dirA and its
children, but the old version of dirB. If dirA and dirB have a common
ancestor directory, you can do commits and updates at that level
instead. CVS recursively locks a directory and all its children.

[Chuck Karish]

NahtMy ReelName wrote: Chuck Karish wrote: NotMy RealName wrote: For example, backing out a change that that altered content of 10 files, added one file, and removed another. This should be easy, but most solutions fall down here, regardless of whether they use atomic database commits or not. It works fine in Perforce, as long as there have not been subsequent modifications to the files that depend on what's being backed out. Which basically says you can't back out the change weeks or months later. This unfortunate restriction tends to be an artifact of delta-based file version management.

No, it doesn't say that. The deltas can be backed out at any
time. The problem is in dealing with the side effects of the
removals, since code written later may depend on them.

The SCM system can't fix this because the content of the changes
isn't under the SCM system's control.

[NahtMy ReelName]

David Masterson wrote:>>>NahtMy ReelName writes:A compositely versioned scm system knows everything about thephysical nature of changes. But that doesn't mean it can producecode that will compile in response to arbitrary cset add/removeoperations. Often a cset add/remove will require some kind of'resolution cset' that fixes logical/semantic shortcomings in theresulting view of content after adding/removing csets. How is this different then than a delta based system? For instance, using appropriate branch management, it should be possible to subtractively merge out a change. The result, though, may require a "resolution fix" to correct the problems.

It's an implementation detail that isn't of particular interest to most
people.

But a compositely versioned system typically doesn't store strings
of deltas with any particular branch affinity, and deltas are NEVER
copied. A special and simple index is created for each file that
provides assignment of record IDs to each record ever inserted, and
knows for any insertion what other record ID it's relative to, and for
any deletion which record id was deleted. This is independent of any
future deactivations, reactivations, or migrations of the changes to
other version contexts.

VERSIONS then are simply sets of change IDs. Changes are either on or
off in some version, and migrating the change to another branch is
simply promoting the change iD to be active in that version.

I won't go into more detail, but that starts to touch on it.

You can emulate this behavior to some degree with delta relative
implementations. But it isn't quite as natural an act, and generally
there deltas must be copied for every branch to which they apply.
Then it's a slippery implementation slope to provide other cset-oriented
behaviors. For example, it becomes harder to say if a whole logical
change was promoted into some other branch, or whether just a piece of
the change.

Most of this is of no interest to end-users. What's important (and
started this sub-topic) was the general point that cset-oriented change
flow models are tremendously useful, and a good one observes not only
checkin-atomic transaction behavior, but also atomic behavior in every
aspect of change flow. And you should be able to ask, for instance,
whether a given branch at a given time does or does not contain a given
change(set). If it's a delta based implementation, that's ok. But they
definitely had to work harder to implement it, and historically they
tend not to have completely implemented all the features associated with
compositely-based cset implementations because of the extra
implementation work.

[Rahul Bhargava]

This is a dated thread but I just wanted to update it with new
information. If you are looking at multisite options with CVS, you
should consider WANdisco CVS Replicator -
http://www.wandisco.com/cvs

I have posted more information on another thread -

http://groups-beta.google.com/group/comp.software.config-mgmt/browse_frm/thread/e4b8113d1be7ca1e/f7f8c6077225b336?_done=%2Fgroup%2Fcomp.software.config-mgmt%3F&_doneTitle=Back+to+topics&_doneTitle=Back&&d#f7f8c6077225b336