Welcome to Emulationworld

Forum Index | FAQ | New User | Login | Search

*View All ThreadsShow in Flat Mode*


SubjectZipMax - New setting to assist in distributed processing Reply to this message
Posted byPacFan
Posted on01/29/04 11:56 AM



Roman --

I have an idea that would be small to implement and greatly increase the functionality of ZipMax.

Ideally, I'd like some Client/Server way to manage distributing the Rezipping over numerou machines connected to a network.. Sending the smallest zip's to remote (and/or slower) machines and woking on the larger zips on the mail (and/or faster) machines.

Obiously that would take a lot of rework.

However, I have an idea that would accomplish this in a different way with probably a couple lines of code and 1 new ini setting.

Here's the plan:
- Add an ini file setting called "SkipReadOnlyFiles=0"
- The default would be 0.
- Add a line of code in the routine parsing the files to check if the readonly bit is set, and IF it is, then skip the file and continue to the next one (Something it might want to do regardless of this feature?)
- If it is not readonly, mark it as read only immediately and start processing
- Mark it as writeable after processing is complete (just before the source zip is updated) or on any failure cleaup routines. Continue to set the Archive bit of course upon succeeding.
- Since the default value would be 0, it is the same functionality as you currently have and wouldn't break anyone who doesn't know how to set/clear the readonly flag (though I'd doubt anyone would have the stuff marked readonly to begin with)

Benefits:
This would then allow you to set up a single file share on the main machine, then from remote machines, highlight the files and drag and drop into zip maxes (you could sort by file size first to get the smaller ones to the remote machines to save network bandwidth or control sizes to smaller processors/etc..)

Drawbacks:
- It does mark the files as read only and could leave something in a bad state if something totally falls apart. But it doesn't delete the file or anything "bad"
- It is possible, though not very likely, for 2 machines to accidentially grab the same file at the same time and mark it readonly and both work on it. (or catch it not marked read only as the original zip is being replaced) This is lessened by network latency and processor speed getting to the same file at the same time.


I'd be glad to make the changes myself as I am a programmer, and in fact did look at the ZipMax code 2 years ago in an early version (and added support on my own machine for 99 zippers before BJW got his update in) However I currently dont have the correct compiler and library versions and figured you could make this update rather quickly if you thought it would work for most people.

You might also chose to use the Hidden flag but I think ReadOnly makes the most sense as those probably should be skipped in any case (even without a flag or this feature)

It would significantly help the speed of recompression... ESPECIALLY when using BJWFlate which is awesome for the extra compression it can do over 7za and numerous other compressors. I have 5 computers at home and 3 at work each networked and this would make the chore of a large rezip easy with virtually no manual segmentation/work to keep things running.

LMK your thoughts and thanks for the work on a great program!

-
Entire Thread
Subject  Posted byPosted On
.ZipMax - New setting to assist in distributed processing  PacFan01/29/04 11:56 AM
.*Re: ZipMax - New setting to assist in distributed processing  PacFan01/29/04 12:11 PM
..*Re: ZipMax - New setting to assist in distributed processing  Roman01/29/04 12:40 PM
...*Re: ZipMax - New setting to assist in distributed processing  PacFan01/29/04 07:14 PM
....*Done -- Multi-Machine processing for ZipMax  PacFan01/30/04 11:50 AM
.....*Re: Done -- Multi-Machine processing for ZipMax  Roman02/02/04 04:13 AM
....*Re: ZipMax - New setting to assist in distributed processing  f205v01/30/04 04:31 AM