Skybuck's Universal Code 5 (The Native Version)

[Skybuck Flying]

That is not how Windows really does it. For FAT devices, it has a
needlessly complex way of selecting which sector to use. The first
fit method tends not to lead to fragmentation. The Windows method is
carefully tuned to lead to a great deal of fragmentation. This is why
you have to defrag so often on a windows box.

Since windows is closed source how are you gonna to proof your claims ?

One could analyze the binary data but this still requires knowledge of the
file system layout and such.

Also I wouldn't be too surprised if windows xp x64 does a little back of
defragging in the background ?

Finally I have a very large harddisk and try to delete as few things as
possible to prevent any fragmentation from occuring.

So far it seems to be working quite nice... though I must say the system
bootup time has increased a little bit... whatever it causes is hard to
tell... many applications installed by now... and tortuisesvn installed
too... that could be doing things as well <- pure speculation.

Bye,
Skybuck.

 

[Chris Thomasson]

Since windows is closed source how are you gonna to proof your claims ?

[...]

http://www.microsoft.com/presspass/press/2006/jan06/01-25EUSourceCodePR.mspx

 

[Chris Thomasson]

Since windows is closed source how are you gonna to proof your claims ?

[...]

http://www.microsoft.com/presspass/press/2006/jan06/01-25EUSourceCodePR.mspx

Here is the license:

http://www.microsoft.com/resources/sharedsource/licensingbasics/referencelicense.mspx

 

[StickThatInYourPipeAndSmokeIt]

Simple, it writes to the first clustor that is NOT marked as used.

It is "cluster" and no, generally not. It writes to the next
"available" write space, and newly "released", just previously written to
areas get put at the end of the list of available space as they get
marked "free", so the first cluster to get written to is the first one
that was originally available in the "newly formatted" original "list".
This insures that all areas on a drive's platter gets "used", instead of
keeping all your drive activity to a confined area on the drive, inviting
an early failure mode.

 

[StickThatInYourPipeAndSmokeIt]

That is not how Windows really does it. For FAT devices, it has a
needlessly complex way of selecting which sector to use. The first
fit method tends not to lead to fragmentation. The Windows method is
carefully tuned to lead to a great deal of fragmentation. This is why
you have to defrag so often on a windows box.

Horseshit. The only way a file gets fragmented is if it gets edited
since the original write and after other files have been written after
the original write. NO OTHER WAY.

That said, only dopes that want to endlessly thrash their drive
constantly defrag.

I defrag less than once a year. I have several partitions.

System screams right along.

Over-maintenance is idiocy.

FAT and FAT32 FS are accessed the same way they were in the DOS days.
NTFS is a bit more OS managed, but not much.

FAT can frag up. NTFS doesn't frag much at all.

Your knowledge of the subject is fragmented.

 

[MooseFET]

MooseFET wrote: [....]

I have 4GB and it isn't even close to enough. Just the raw data file
for one of the things I am doing is over 2GB. Forunately, I can
operate on it in sections. By carefully arranging how it is broken
up, I have got the processing time to be less than 8 hours.

What is it, if this is not a secret? RF design?

The highest frequency is about 500Hz but yes I guess you could call it
RF if you wanted to. I was reprocessing some old data that showed the
natural variation in the earths magnetic field. It was a little over
a week's worth of data.

 

[MooseFET]

Simply read file data sequentially from 0 to N-1.

This is garantueed to provide the fastest performance.
False


If it doesn't then it's a file system problem.
False


I would recommend to defrag the file system.

Add that to the processing time if you must do it.

It's up to the file system to make sure the fragments of the files are as
good as possibly de-fragmented.

This isn't what happens on Windows.

Then it's up to me to make sure I read the files fast with a good algorithm.

Which you aren't

For little files this is easy for big files over 4 GB special handling will
be necessary.

No, I just simply don't use anything like what you suggest and I get
much better speed and no limit on size. No special handling is
needed.

So I am not really seeing the problem.

Ays the blind man.

If I store many, many, many, many integer lists as described, then what is
the problem ?

If you need to get to the values in a specific order and they won't
all fit into memory, you can't just read it all in nor can you go
sequentially.

For a very simple example. Imagine you are doing a simple LOG2:

function LOG2(X : tNumber) : tNumber;
......
while X <> 1 do begin
if X>=2 then begin
X := X/2;
Y := Y + Delta;
end;
X := X * X;
Delta := Delta /2;
end;

If X, Y and Disp won't fit in memory, getting to the MSB of X first is
very handy since the conditional all can be answered quickly from that
end.

 

[MooseFET]

On Wed, 27 Feb 2008 18:17:26 -0800 (PST), MooseFET




I do believe that that 'magic number' is a 0 (Zero).

It doesn't matter so long as no real sector can have that number.

FAT32 is supposed to keep a pointer to the lowest available cluster to
avoid searching the whole FAT, but who knows if it works?

Many other systems make the free sectors into a big linked list.

As well as some use a bit map for both free and allocated sectors.

<snip>
--
ArarghMail802 at [drop the 'http://www.'from ->]http://www.arargh.com
BCET Basic Compiler Page:http://www.arargh.com/basic/index.html

To reply by email, remove the extra stuff from the reply address.

 

[MooseFET]

Since windows is closed source how are you gonna to proof your claims ?

It is simple with some tricky software that tracks the disk
operations. It is a little harder but still quite easy to do if you
just have software such as some defraging programs that show you where
the sectors that are used are. You run a program under a good
debugger that writes a file and the defraging program at the same
time. You add something to the file and then look to see where the
sector goes.

You can also disassemble the part of windows that deals with the hard
disk operation and look to see what it does. Because it is needed to
do the swapping the disk driver is always in RAM.

One could analyze the binary data but this still requires knowledge of the
file system layout and such.

The layout is documented for FAT so that is no problem.

Also I wouldn't be too surprised if windows xp x64 does a little back of
defragging in the background ?

A well written disk system doesn't need degragging.

So far it seems to be working quite nice... though I must say the system
bootup time has increased a little bit.

If boot time is increasing, something is wrong. The part of the disk
that is needed for booting should be unchanging/

 

[MooseFET]

Horseshit. The only way a file gets fragmented is if it gets edited
since the original write and after other files have been written after
the original write. NO OTHER WAY.

Wrong.

They can be written fragmented it happens all the time.

 

[Skybuck Flying]

How you suggest we read files then ?

Do I hear your mind thinking: Random(N) ?

Bye,
Skybuck

 

[[email protected]]

It doesn't matter so long as no real sector can have that number.

FATs don't store sector numbers, they store Allocation Unit Numbers,
which start at 2.

<snip>

 

[MooseFET]

How you suggest we read files then ?

Since you again snipped the context, I will answer woth
"efficiently". Much earlier I gave a very quick explanation of the
problem with your suggest method. Go back and look for it. I am way
too lazy to do it for you.

Do I hear your mind thinking: Random(N) ?

No, the code I posted was LOG2(). It is a rather poor way to do the
function but it is much simpler than the realk code.

 

[MooseFET]

It doesn't matter so long as no real sector can have that number.

FATs don't store sector numbers, they store Allocation Unit Numbers,
which start at 2.

<snip>
--
ArarghMail802 at [drop the 'http://www.'from ->]http://www.arargh.com
BCET Basic Compiler Page:http://www.arargh.com/basic/index.html

To reply by email, remove the extra stuff from the reply address.

Why are you introducing the extra complexity when the OP haven't yet
grasped the over simplified version I posted? You will only further
confuse him.

BTW: very early versions had the sector number equal to the
allocation (cluster) number.

 

[StickThatInYourPipeAndSmokeIt]

Wrong.

They can be written fragmented it happens all the time.

On a nearly full, already heavily fragmented drive, yes.

On any drive with huge amounts of space left on them, particularly if
it has never yet been written to, the file write will ALWAYS be
contiguous and monolithic in nature. Also, constantly "defragged" drives
will nearly always have their free space "defragged" as well, and that
will result in several GB of new file writes, of which all will be
contiguous.

Very large files will not obey the rule as much, but a few pieces of a
for the most part contiguous huge file will not take a hit for it.

A database file is the most likely candidate for fragmentation as any
changes made to the database results in a file fragment which will not be
contiguous with the rest of the file.

Another is if one is DLing two files at once to the same partition.

So, it doesn't happen "all the time". It happens when you volume is
nearly full already, and already has a highly fragmented fill on it.

If one's drive has never been filled since its original format, a new
file write will always fill a new, contiguous space on the volume.

 

[Skybuck Flying]

For a harddrive manufacturer it would make sense to optimize for 0 to N-1,
to keep things simple for programmers.

You agree with that ?

I think they do that.

So file system should do that too.

<contexted snipped>

If you need it read previous ones.

Because I don't need it, because what you wrote is BS.

Bye,
Skybuck.

 

[Skybuck Flying]

The download example you gave is an interesting example.

I might be prevented by allocating the complete file size for each download
up front, if ofcourse the file size is known up front.

This would prevent the download from being fragmented.

Which could help prevent fragmentation in the future after one of the
download is deleted and replaced by other files.

Though this would require one of the downloads to remain and the other to be
deleted for fragmentation to occur... but still a pretty realistic
possibility.

Doing so is possible by seeking to the end of the file-1 and performing a
one byte zero write there or so.

This could immediatly allocate the file, unless maybe the file system is
working in sparse mode... but that is probably not the case (?)

Finally it has adventages and disadventages:

Adventage:
1. The user will immediatly know if it has enough space for the downloads.
This could prevent wasting bandwidth on download failures.

Disadventage:
2. The user has no longer time to free up space in case he suspects the
download might not fit.

Adventage:
3. The user is garantueed the download will succeed if he leaves, because
space has been pre-allocated. This could again prevent bandwidth being
wasted on unnecessary try-again-packets.

Downloads could fail if space is not pre-allocated, other processess might
be filling the harddisks.

In your scenerio it's totally possible that both downloads would fail.

4. Adventage:

File system remains unfragmented.

5. Disadventage:

If download fails, and partial file remains... then user could simply
continue from the last byte, if all previous bytes are garantueed to have
arrived, partial files makes evaluating if everything arrived more easy too,
by simply comparing file size.

For example support os crashed during download... pre-allocated files could
remain creating the illusion that they were downloaded fully.

It's a difficult decision to make.

Robustness vs Assurance vs Bandwidth Wasted vs Speed.

Bye,
Skybuck.

 

[MooseFET]

On a nearly full, already heavily fragmented drive, yes.

So you are retracting your "NO OTHER WAY', I assume. Thankyou for
correcting the record for others who may be reading along. You also
have it wrong on the "nearly full heavily fragmented". If you write
a large file to a very slightly fragmented drive, the new file can
also be fragmented.

On any drive with huge amounts of space left on them, particularly if
it has never yet been written to, the file write will ALWAYS be
contiguous and monolithic in nature.

This is not always true for a disk that has been written to in the
past. If the OS always allocates the new file after the last used
sector in an effort to try to prevent fragmented files, eventually it
will be allocating near the end of the disk. At that point it ends up
fragmenting.

 

[RoyLFuchs]

Because I don't need it, because what you wrote is BS.

You're an idiot. Stop gumming up our newsgroups with you retarded baby
bullshit. That really is all that you post.

 

[RoyLFuchs]

It's a difficult decision to make.

Robustness vs Assurance vs Bandwidth Wasted vs Speed.

Bye,
Skybuck.

You're an idiot.