How best handle large files

[Tom]

I recently converted from Quicken 2000 to 2004 and have more than 10
years of data on my QDATA file. Although this worked fine with the 2000
version, the "upgrade" is very sluggish and prone to crash if I have more
than one or two other programs running.

Intuit suggests that the problem is the large size of my data file
(15,058 KB) and that I need to "COPY" the last 3 years of data to create a
new data file.

Is COPY the best way to create a new, smaller file? How about "Archive"?
(I want to keep the past 5 years of data.)

Your suggestions will be appreciated.

Thanks.

[Rick Hess]

"Tom" <tom@excite.com> wrote in message
news:Jnhfb.12009$Rd4.7380@fed1read07... I recently converted from Quicken 2000 to 2004 and have more than 10 years of data on my QDATA file. Although this worked fine with the 2000 version, the "upgrade" is very sluggish and prone to crash if I have more than one or two other programs running.

What OS? How much RAM? What RAM-resident programs run at startup? What's
swapfile size?

Intuit suggests that the problem is the large size of my data file (15,058 KB) and that I need to "COPY" the last 3 years of data to create a new data file.

Although I don't disagree that smaller is better for Q files, mine is over
20MB and stable. I can't help but wonder if you have other factors that are
the real problem -- and not your Q filesize.

Is COPY the best way to create a new, smaller file? How about
"Archive"? (I want to keep the past 5 years of data.)

FILE>COPY will defragment your file, but the decrease (if any) might not be
very significant. It's not a bad idea to do this occasionally to keep your
file consolidated. You can keep only the last __years of data if you're
absolutely sure you don't need all of it in one file. Many here do just
that. Personally, I have all data since "day one" in one file with no
noticeable problems. However, I may be a good candidate for future file
corruption.

--


Rick Hess
New Orleans

[Dick Ballard]

The COPY function within Quicken does remove garbage from the data
files and reduces the file size. However, Intuit tech support people
have a habit of blaming everything they can't explain otherwise on a
large file size. I'd look elsewhere for the cause of your problems.

Dick Ballard
ballardr@att.net


On Fri, 3 Oct 2003 09:23:26 -0700, "Tom" <tom@excite.com> wrote:
I recently converted from Quicken 2000 to 2004 and have more than 10years of data on my QDATA file. Although this worked fine with the 2000version, the "upgrade" is very sluggish and prone to crash if I have morethan one or two other programs running. Intuit suggests that the problem is the large size of my data file(15,058 KB) and that I need to "COPY" the last 3 years of data to create anew data file. Is COPY the best way to create a new, smaller file? How about "Archive"?(I want to keep the past 5 years of data.)

[Dick Ballard]

A small point I neglected in my last post is that the COPY function
within Quicken creates smaller data files even if you copy the whole
file. You don't need to limit it to a shorter period unless you want
to dramatically reduce the file size.

Dick Ballard
ballardr@att.net


On Fri, 3 Oct 2003 09:23:26 -0700, "Tom" <tom@excite.com> wrote:
I recently converted from Quicken 2000 to 2004 and have more than 10years of data on my QDATA file. Although this worked fine with the 2000version, the "upgrade" is very sluggish and prone to crash if I have morethan one or two other programs running. Intuit suggests that the problem is the large size of my data file(15,058 KB) and that I need to "COPY" the last 3 years of data to create anew data file. Is COPY the best way to create a new, smaller file? How about "Archive"?(I want to keep the past 5 years of data.)

[Rick Hess]

"Dick Ballard" <ballardr@att.net> wrote in message
news:knnunvg4qmhis248j9ojnid3o13rlv67r1@4ax.com...
A small point I neglected in my last post is that the COPY function within Quicken creates smaller data files even if you copy the whole file.

An interesting phenomenon I have noticed with COPY is that it doesn't always
decrease the filesize. In fact, in my experience, the routine can actually
increase the filesize. That's why I chose my words the way I did to the OP.

I don't know why this happens. I've posted about this here and, IIRC, a
couple of other posters said they occasionally observed the same thing. My
best guess is that the COPY routine leaves some "slack" space in the data
file so that when data is added to an individual record within the file, the
record tends to remain contiguous.

All that aside, I think we're in agreement that the end result of COPY is a
good thing, and the OP's problems may not be due to his filesize.
--


Rick Hess
New Orleans

[Hank Arnold]

That's my experience, also... Q's file size increases if I COPY the file. I
also agree with the others on this whole issue. File size is not likely to
be the answer. It could be some file corruption. Copying the file and then
validate/super validate may clean up corruption (please be sure to
validate/super validate only on a copy...). It's more likely, though, that
it's the environment that's doing it......

--
Regards,
Hank Arnold

"Rick Hess" <RHess51295@aol.com> wrote in message
news:blnre3$dehus$1@ID-199374.news.uni-berlin.de... "Dick Ballard" <ballardr@att.net> wrote in message news:knnunvg4qmhis248j9ojnid3o13rlv67r1@4ax.com... A small point I neglected in my last post is that the COPY function within Quicken creates smaller data files even if you copy the whole file. An interesting phenomenon I have noticed with COPY is that it doesn't
always decrease the filesize. In fact, in my experience, the routine can
actually increase the filesize. That's why I chose my words the way I did to the
OP. I don't know why this happens. I've posted about this here and, IIRC, a couple of other posters said they occasionally observed the same thing.
My best guess is that the COPY routine leaves some "slack" space in the data file so that when data is added to an individual record within the file,
the record tends to remain contiguous. All that aside, I think we're in agreement that the end result of COPY is
a good thing, and the OP's problems may not be due to his filesize. -- Rick Hess New Orleans

[Steve]

"Hank Arnold" <rasilon@aol.com> wrote:That's my experience, also... Q's file size increases if I COPY the file. Ialso agree with the others on this whole issue. File size is not likely tobe the answer. It could be some file corruption. Copying the file and thenvalidate/super validate may clean up corruption (please be sure tovalidate/super validate only on a copy...).

I know Intuit recommends this, but I've never been quite sure why. If
there's no problem, seems like it wouldn't matter. And if there is a
problem, and you fix the copy, you'd have to replace the original with
the fixed copy anyway. ???

[John Pollard]

Steve wrote: I know Intuit recommends this, but I've never been quite sure why. If there's no problem, seems like it wouldn't matter. And if there is a problem, and you fix the copy, you'd have to replace the original with the fixed copy anyway. ???

If doing the copy creates a problem in the new file, you can easily revert
back. You could phrase the advice differently and just say make a backup
before you perform any "file operations" on your Q data.

[Hank Arnold]

The problem comes in if the validation process finds an error. Any time you
try to "fix" a database (and Q *is* a database) you run the risk of losing
data. It's the nature of the beast. Even the process of doing a COPY could
conceivably cause a problem. All you are doing is preserving your dataset in
the event of a problem. This is simply good sensible computing. If there are
no errors found and the COPY does not shrink the file size, then just go
back to the original copy. If errors are found and corrected, then, yes,
you'll have to rename the database files. Q has a menu function (File/File
Operations/Rename) where you can do the renaming.

If that is a problem, then backup the database to a safe location and do the
operations on the original file. If there is a problem, restore the backup.

--
Regards,
Hank Arnold


"Steve" <abc@def.inv> wrote in message
news:5va0ovks1s959ni98kpc168qd5jfbrs5mj@4ax.com... I know Intuit recommends this, but I've never been quite sure why. If there's no problem, seems like it wouldn't matter. And if there is a problem, and you fix the copy, you'd have to replace the original with the fixed copy anyway. ???

[Rick Hess]

"Hank Arnold" <rasilon@aol.com> wrote in message
news:1P9gb.46945$E95.23847492@news4.srv.hcvlny.cv.net... The problem comes in if the validation process finds an error. Any time
you try to "fix" a database (and Q *is* a database) you run the risk of losing data. It's the nature of the beast. Even the process of doing a COPY could conceivably cause a problem. All you are doing is preserving your dataset
in the event of a problem. This is simply good sensible computing. If there
are no errors found and the COPY does not shrink the file size, then just go back to the original copy. If errors are found and corrected, then, yes, you'll have to rename the database files. Q has a menu function (File/File Operations/Rename) where you can do the renaming. If that is a problem, then backup the database to a safe location and do
the operations on the original file. If there is a problem, restore the
backup.

Hank, I may misunderstand your post, but here's my 2 cents: At first when I
read your post I thought you were talking about an OS copy of the file. But
when you say "and the COPY does not shrink the file size, then just go back
to the original copy" it sounds like you're referring to Q's COPY routine.
I'm under the impression that Q's FILE>COPY is never a bad thing and the
(supposedly) resulting cleaner file is a good thing. Therefore, even if
COPY does not decrease the file size, there's still more potential good in
keeping the new file.

Of course, this is not the same thing as an OS copy of the file to be used
as a backup in preparation for a validate.
--


Rick Hess
New Orleans

[Tom]

Rick,

Thanks for your help - and that of everyone who posted comments. In
response to your questions:

I'm using Win98 SE with 384 RAM. Resident programs include Norton
Anti-Virus, Zone Alarm, GO Back, and a backup program - the same as I used
with Quicken 2000. My swap file is set for a minimum of 383 and Maximum of
768.

Any additional comments you have will be appreciated.

Thanks.

"Rick Hess" <RHess51295@aol.com> wrote in message
news:blkn9j$cvhdi$1@ID-199374.news.uni-berlin.de... "Tom" <tom@excite.com> wrote in message news:Jnhfb.12009$Rd4.7380@fed1read07... I recently converted from Quicken 2000 to 2004 and have more than 10 years of data on my QDATA file. Although this worked fine with the 2000 version, the "upgrade" is very sluggish and prone to crash if I have
more than one or two other programs running. What OS? How much RAM? What RAM-resident programs run at startup?
What's swapfile size? Intuit suggests that the problem is the large size of my data file (15,058 KB) and that I need to "COPY" the last 3 years of data to create
a new data file. Although I don't disagree that smaller is better for Q files, mine is over 20MB and stable. I can't help but wonder if you have other factors that
are the real problem -- and not your Q filesize. Is COPY the best way to create a new, smaller file? How about "Archive"? (I want to keep the past 5 years of data.) FILE>COPY will defragment your file, but the decrease (if any) might not
be very significant. It's not a bad idea to do this occasionally to keep
your file consolidated. You can keep only the last __years of data if you're absolutely sure you don't need all of it in one file. Many here do just that. Personally, I have all data since "day one" in one file with no noticeable problems. However, I may be a good candidate for future file corruption. -- Rick Hess New Orleans

[John Pollard]

Rick Hess wrote:
Hank, I may misunderstand your post, but here's my 2 cents: At first when I read your post I thought you were talking about an OS copy of the file. But when you say "and the COPY does not shrink the file size, then just go back to the original copy" it sounds like you're referring to Q's COPY routine. I'm under the impression that Q's FILE>COPY is never a bad thing and the (supposedly) resulting cleaner file is a good thing. Therefore, even if COPY does not decrease the file size, there's still more potential good in keeping the new file. Of course, this is not the same thing as an OS copy of the file to be used as a backup in preparation for a validate.

Another take.

My thinking is that an os copy would be the "safest" type of copy; an os
copy has no knowledge of the underlying format, it is designed to copy
bit-for-bit - as programs go, this is a trivial exercise.

The copy that Quicken does is a "logical" copy, that is, it knows the format
of the incoming data and it uses that knowledge in the copy process. both
reading the records from the old file and writing the records to the new
file. And if, as we believe, the Quicken copy is designed to actually
physically remove some records from the input file under certain conditions
(basically by not writing them to the output file), that would be yet
another point at which the copy could introduce a problem. I believe any
"logical" copy would be more likely to create problems than an os copy,
because it uses more code and requires more human knowledge to write the
code and get it correct.

Having said that, I do not recall ever hearing of a Quicken copy adding any
corruption to a file; then again, it is not clear to me that we would likely
recognize that that had happened even if it did. But I do feel reasonably
comfortable doing a Quicken copy of my file.

[Rick Hess]

"John Pollard" <willnotworkatall@hotmail.com> wrote in message
news:Oozgb.25591$9a7.4290@bignews6.bellsouth.net... Another take. My thinking is that an os copy would be the "safest" type of copy; an os copy has no knowledge of the underlying format, it is designed to copy bit-for-bit - as programs go, this is a trivial exercise. The copy that Quicken does is a "logical" copy, that is, it knows the
format of the incoming data and it uses that knowledge in the copy process. both reading the records from the old file and writing the records to the new file. And if, as we believe, the Quicken copy is designed to actually physically remove some records from the input file under certain
conditions (basically by not writing them to the output file), that would be yet another point at which the copy could introduce a problem.

I agree.

I believe any "logical" copy would be more likely to create problems than an os copy, because it uses more code and requires more human knowledge to write the code and get it correct.

I agree. But how "likely" is "likely"? I think there's SOME risk in almost
all file operations. Even with an OS copy: power surges or ESD could
create a defective copy that might not be readily apparent. Perhaps Q's
FILE>COPY is even as risky as a disk defragment, which certainly is risky.

Having said that, I do not recall ever hearing of a Quicken copy adding
any corruption to a file; then again, it is not clear to me that we would
likely recognize that that had happened even if it did. But I do feel reasonably comfortable doing a Quicken copy of my file.

Although you say you feel "reasonably comfortable" with the routine, I'm not
sure what your conclusion/recommendation is.

Some in this NG have recommended a regular FILE>COPY as part of file
maintenance -- even if there are no apparent problems. If FILE>COPY does
what I think it does, then I think this sounds like a good practice -- one
that far outweighs the risk. But I say that without really knowing what the
risk probability is. I would love to hear from a knowledgeable Intuit rep
on all of this.

And, of course, I think we've both seen FILE>COPY solve some problems for
users in this NG (one last week, I believe).

--


Rick Hess
New Orleans

[John Pollard]

Rick Hess wrote: Although you say you feel "reasonably comfortable" with the routine, I'm not sure what your conclusion/recommendation is. Some in this NG have recommended a regular FILE>COPY as part of file maintenance -- even if there are no apparent problems. If FILE>COPY does what I think it does, then I think this sounds like a good practice -- one that far outweighs the risk. But I say that without really knowing what the risk probability is. I would love to hear from a knowledgeable Intuit rep on all of this. And, of course, I think we've both seen FILE>COPY solve some problems for users in this NG (one last week, I believe).

I was really just trying to put some additional perspective on the
discussion; not inveighing against Quicken's file copy. I only wanted to
point out that I thought that one should not blindly believe that a Quicken
file copy will always leave your Q file better - or even no worse - than it
found it. I think whenever you do one of Quicken's file operations, you
should give the resulting file some extra scrutiny, for a while anyway; I
don't feel that way about a Windows file copy.

And you are certainly right, there is are a lot of subjective and relative
terms in this discussion - no way to avoid them without inside information.
But I will retract one of my weasel words and say, I feel comfortable doing
a Quicken file copy.