DIR and COPY misbehaving with lots of files

  • Thread starter Thread starter PhilHibbs
  • Start date Start date
P

PhilHibbs

I have an XP server with a directory with over half a million files in
it, with long numerical names such as 123456_234567.TIF. I wanted to
get a sample of them, so I ran a command prompt and executed "dir
12345*". It returned a list of files that mostly - but not entirely -
began with "12345". One might have been, for example,
"124679_124680.PCX". I tried "dir 12345*.*" but that did the same.

I then ran a "copy" command to copy the files to another empty
directory on a different drive. The command ran for a while, then
stopped with an "overwirite Yes, No, or All" prompt. The destination
directory was empty at the start, yet it was clearly trying to copy
the file a second time. I replied 'N', and it carried on for a while
and stopped again with the same prompt. After a couple of these I
pressed 'A' because I couldn't wait around and nurse it all night.

Is there a bug with doing directory listings or copying from
directories with a large number of files?

The files are in a TrueCrypt container file on an external USB drive,
might that be a factor?

Phil Hibbs.
 
"PhilHibbs" <snarks@gmail.com> wrote in message
news:d2bf50db-81a7-4fd4-941d-47013aa9dd24@o42g2000hsc.googlegroups.com...
>I have an XP server with a directory with over half a million files in
> it, with long numerical names such as 123456_234567.TIF. I wanted to
> get a sample of them, so I ran a command prompt and executed "dir
> 12345*". It returned a list of files that mostly - but not entirely -
> began with "12345". One might have been, for example,
> "124679_124680.PCX". I tried "dir 12345*.*" but that did the same.
>
> I then ran a "copy" command to copy the files to another empty
> directory on a different drive. The command ran for a while, then
> stopped with an "overwirite Yes, No, or All" prompt. The destination
> directory was empty at the start, yet it was clearly trying to copy
> the file a second time. I replied 'N', and it carried on for a while
> and stopped again with the same prompt. After a couple of these I
> pressed 'A' because I couldn't wait around and nurse it all night.
>
> Is there a bug with doing directory listings or copying from
> directories with a large number of files?
>
> The files are in a TrueCrypt container file on an external USB drive,
> might that be a factor?
>
> Phil Hibbs.

This is not a bug but a limiation in the LFN/SFN naming system
that manifests itself when you have lots of similar file names
whose SFNs are generated by the operating system. Try this
little experiment:
- Click Start / Run / cmd {OK}
- Type these commands:
md \SFNTest {Enter}
cd \SFNTest {Enter}
for /L %a in (1,1,1000) do @echo. > "ABCDEFGH %a.txt" {Enter}
dir abb*.* {Enter}

You will see a number of files, none of which start with the
letters "ABB". Why? Try this command:

dir /x /p {Enter}

Having half a million files in the one folder is IMHO not a good
idea. Not only are you likely to run into LFN/SFN problems but
the performance of your machine will suffer badly. I think that
5,000 files is a reasonable limit.
 
Pegasus wrote:
> This is not a bug but a limiation in the LFN/SFN naming system...

You say potato, I say potato.

> Having half a million files in the one folder is IMHO not a good
> idea. Not only are you likely to run into LFN/SFN problems but
> the performance of your machine will suffer badly. I think that
> 5,000 files is a reasonable limit.

We've been thinking of partitioning the files based on the prefix of
the file name already. If I create subdirectories based on the first
two digits, that should give us 99 directories with just under 6,000
files each. Thanks for the info, nice to know why these things happen.

Phil Hibbs.
 
Back
Top