CopyFile slowness

[ElectroLund]

A long time ago, I posted this question about INI files over a network.  A lot has changed with my application since then.  I've moved away from INI to a basic ASCII columnar format which is easier to maintain and parse.

What hasn't changed for is my topology.  I still have multiple client computers running the same software, all accessing the same file server for their logging.  My scheme looks like this:

1) load up a UUT on a station

2) find the log file on the server

3) CopyFile it to the local machine for updating

4) when done, CopyFile back to the server for archival

I've left out various error checking specifics in the above outline.

The Problem

What I've observed is that this process can take 3-6 seconds for each UUT!  I did some more analysis and found the following interesting results:

• if the file server destination folder has very few files, the above takes only 0.2-0.8 seconds
• as soon as the folder grows in size to over 100 files, that time grows to the aforementioned 6 seconds

So it appears that CopyFile must be indexing the entire directory before it then works its magic. 

Possible Workarounds

1) lowlvlio.h

I've looked at using the open() and read(), write() low level I/O ANSI C library functions.  These work nicely, but you must have an allocated buffer to hold the data first.  My log files can sometimes be as large as 8MB.  And though the CVI help states that numberOfBytes is an unsigned int, I found that I can only use a value of about 100,000 before the function fails.

So with this driver, I'd need to break up the files into chunks first.  Yuck!

2) Utility Library 

Using the same mechanism as the lowlvio, I tried using ReadLine and WriteLine in a loop, but this took an enormous amount of time (69 seconds!) on an 8MB file.  The obvious advantage of the Utility library would have been the ability to declare numberOfBytes as -1.

3) CopyFile from the Win32 API

The help file talks about including windows.h and setting SDK_CONFLICT_PRIORITY macro.  Trouble is, this has ramifications elsewhere in my code where I use other CVI Utility Library functions.

Any other thoughts?  This can't be this hard!

[ElectroLund]

Well, shoot.  Of course I would find a better solution shortly after I post that.  A modification of #2 but with gets() and puts() works very fast. 

I found with a 650KB file, I can copy the file in 0.26 sec

with an 8MB file, 2.79 sec

[gdargaud]

gets() and puts() ?!? Why would you copy a file character by character ? Define a buffer, preferably a multiple of the filesystem blocksize and fread/fwrite your blocks until the last one which will have a smaller size, as returned by fread.

And BTW, you can include windows.h in one of your .c files without messing anything about the rest of your project.