Here's the situation. I have about 30,000 zip files in a folder. Many are duplicates. I want to be able to eliminate the duplicates, and have as few zips as possible without losing any roms.

Example:
a.zip has 0.bin, 1.bin, 2.bin, 3.bin
b.zip has exactly the same, same crc's
c.zip has 0.bin only (same crc as above)
d.zip has 5.bin (different crc from anything above)

I need c.zip to get deleted. I need one of a.zip and b.zip to be deleted, the other kept. d.zip must not be touched as it has roms for a different game.

There is no dat file, and no such thing as split-merged or whatever.

Can cmp do this? If not, what can?


-----------------------------------------
Robert

well...the used datfile (and without any datfile, you can't do anything) defines what's needed and how a rom is spread over sets. With a dat, the scanner and rebuilder does the rest for you.


Roman Scherzer

> Here's the situation. I have about 30,000 zip files in a folder. Many are
> duplicates. I want to be able to eliminate the duplicates, and have as few zips
> as possible without losing any roms.
>
> Example:
> a.zip has 0.bin, 1.bin, 2.bin, 3.bin
> b.zip has exactly the same, same crc's
> c.zip has 0.bin only (same crc as above)
> d.zip has 5.bin (different crc from anything above)
>
> I need c.zip to get deleted. I need one of a.zip and b.zip to be deleted, the
> other kept. d.zip must not be touched as it has roms for a different game.
>
> There is no dat file, and no such thing as split-merged or whatever.
>
> Can cmp do this? If not, what can?
>
>
> -----------------------------------------
> Robert
>
there is a long way to get there.
1) use cmpro to create a datfile from you unsorted folder
2) use listtool or datutil to convert it from cmpro to romcenter format
3) use a text editor to replace "," with "-"
4) use a text editor to replace "¬" with ","
5) use a spreadsheet to import the comma separated file
6) use a spreadsheet to eliminate lines with duplicate crc
7) save it back to comma separated file
8) use text editor to replace "," with "¬"
9) use clrmame to import this file and check your collection.

Maybe there are better solution, this is what I'm doing anyway.

ciao
f205v

hmm...I should add an option "remove dupes" in dir2dat ;)
Something like 'keep only unique sha1'....but then dir2dat decides which file (in case you got e.g. 2 files with the same sha1) to keep...


Roman Scherzer

> hmm...I should add an option "remove dupes" in dir2dat ;)
> Something like 'keep only unique sha1'....but then dir2dat decides which file
> (in case you got e.g. 2 files with the same sha1) to keep...

OK, when you have a collection of 30.000+ unsorted roms, usually you do not care much about the file name...
If I may suggest, "eliminate dupes" should check both SHA1 and size, just in case there are 2 files with same SHA1 and different size.

ciao
f205v

> 1) use cmpro to create a datfile from you unsorted folder
>

How do I do that? I'm a noob at cmp.

-----------------------------------------
Robert

> > 1) use cmpro to create a datfile from you unsorted folder
> >
>
> How do I do that? I'm a noob at cmp.
>
> -----------------------------------------
> Robert
>
With the Dir2dat feature (in profiler).
You can look at the clrmamepro docs in your install folder for more infos.

profiler->dir2dat


Roman Scherzer