Hello,
Now that I've gotten the header filter created, I notice that I've got some dupes in the DATs I've generated. Does anyone know of a tool, or method in clrMAME to remove duplicate 'rom' tags from a single 'game' tag?

Also, Roman, thanks for implementing this feature. It will save me a lot of time renaming files with slightly different headers. I found that to load the filters, I had to open a DAT and set up the 'Headers' for that profile with the header xml file I want to use. Then when running Dir2DAT, it would properly generate the DAT File. Clearing the cache will keep it loaded, but loading another DAT (that doesn't have the 'Headers' set in it's profile) will cause Dir2DAT to generate CRCs without using the header filter.

Here are the steps to take to replicate what I am describing:

- Open a new instance of clrMAME Pro
- Generate a DAT for some files that should return a true response for the header detector's rule(s)
- Examine the generated DAT in a text editor, making a note of a couple of the CRC values
- Load any DAT file
- In the settings, choose the Header you intended to use above
- Repeat the Dir2DAT procedure above.
- You should see that the CRC values have changed, and are the correct, headerless values.
- Return to the Profiler and clear the cache
- Repeat the Dir2DAT procedure. The CRCs are still the correct ones.
- Return to the Profiler, and load a DAT that does not have the Headers set
- Return to the Profiler from the 5 button 'menu'
- Repeat the Dir2DAT, and notice that the initial (header included) CRCs return.

That should show you what I am describing. In any case, if you get some time, I would love to see a section in Dir2DAT to choose a header detector to apply during DAT creation. I cannot find a way to load a detector without modifying a (potentially) unrelated DAT profile.

Thanks again.

The header settings are per-profile, not general ones. This is intended because you don't want to scan a MAME collection for example with active NES headers. So with every new profile load, you automatically load the possible other set header settings of that other profile.

For the double-rom remover in the dat...well....a good texteditor with sort options should do 8) Don't know an easy way without manual work.


Roman Scherzer

> The header settings are per-profile, not general ones. This is intended because
> you don't want to scan a MAME collection for example with active NES headers. So
> with every new profile load, you automatically load the possible other set
> header settings of that other profile.
>
> For the double-rom remover in the dat...well....a good texteditor with sort
> options should do 8) Don't know an easy way without manual work.
>
>
> Roman Scherzer
>

Thanks for the prompt reply. Now that I know the Dir2DAT is tied to a profile, I can just create a dummy dat/profile for my header detector and load it as needed.

As for the dupes, I've decided to just throw together some code to do the duplicate checking. Hopefully I've got enough RAM to handle some giant hash tables ;)

Thanks again for all your efforts.

Not dir2dat is tied, the currently loaded profile is. So if you loaded a profile which got NES enabled and you enter Dir2Dat, NES is active. When nothing is loaded, no header should be active.

By the way...if you created new header xmls yourself (or did I misinterpret this?), let me know. I'd like to add them officially.


*Edit* Hmmm....I guess an option to select the currently used header xml would be fine within dir2dat...I will think about it.



Roman Scherzer

> Not dir2dat is tied, the currently loaded profile is. So if you loaded a profile
> which got NES enabled and you enter Dir2Dat, NES is active. When nothing is
> loaded, no header should be active.

This seems to hold true, except for when the "clear cache" is clicked. As I described in the steps above, the header detector(s) will remain loaded in Dir2DAT.

>
> By the way...if you created new header xmls yourself (or did I misinterpret
> this?), let me know. I'd like to add them officially.

I've currently created header detectors for NSF, PSID, and PSID v2 NG. I want to perform more testing though, since I just created the PSID ones at work today ;) I'm also looking into some more, but we'll see where that goes...

>
>
> *Edit* Hmmm....I guess an option to select the currently used header xml would
> be fine within dir2dat...I will think about it.
>

What I was hoping for would be a Dir2DAT independent of any profile: adding a small subsection to select the detector(s) desired to be applied during DAT creation. Is there a particular reason for tying the Dir2DAT to a loaded profile? Perhaps some hidden functionality I could be harnessing? Or is it to prevent having to reload a profile upon closing Dir2DAT?

In any case, I am very grateful for this tool no matter what you decide to do, and the header detectors already work when the appropriate steps are taken. I recognize that this work is done in your own free time, and appreciate it.

I'll be sure to contact you when I am comfortable with the correctness of my header detectors.

> This seems to hold true, except for when the "clear cache" is clicked. As I
> described in the steps above, the header detector(s) will remain loaded in
> Dir2DAT.

Hmm...I will check this. *update* fixed in the next version


> I've currently created header detectors for NSF, PSID, and PSID v2 NG. I want
> to perform more testing though, since I just created the PSID ones at work today
> ;) I'm also looking into some more, but we'll see where that goes...

Yay...congrats! Finally someone finds the time to check the xml mania I've added 8) Me myself unfortunalety didn't have much time lately due to real life job. It would be great if I can add your definitions officially later (of course your author tag will hold your name).


> What I was hoping for would be a Dir2DAT independent of any profile:

Well..there are some options (match tagdata for example) which work with the currently loaded profile. You can for example automatically fill in manufacturer, year, description tags automatically this way. I guess for headers , a header selector box like in settings can be added to dir2dat and that should be enough for your purposes.


Roman Scherzer

> Yay...congrats! Finally someone finds the time to check the xml mania I've added
> 8) Me myself unfortunalety didn't have much time lately due to real life job. It
> would be great if I can add your definitions officially later (of course your
> author tag will hold your name).

Of course, no problem. I'll keep you posted if any issues arise as well. For anyone building these detectors, heed the advice in the spec about ignoring size when creating the DATs, the rebuilder will not work if a size field is present, since it compares the actual (header-included) file size to the (headerless) file size in the DAT. I'm not sure if checking the 'ignore size' in the rebuilder options fixes this but it may be worth investigating.

> Well..there are some options (match tagdata for example) which work with the
> currently loaded profile. You can for example automatically fill in
> manufacturer, year, description tags automatically this way.

This sounds very interesting. Does this mean that if there is a 'game' entry with the same name the Dir2DAT will grab the description, etc... from the loaded profile and populate that in the new DAT, allowing all of the manual work of DAT creation to be reused as much as possible? If so, can you tell me the matching criteria? This sounds extremely useful, I can understand why a profile and Dir2DAT would be linked if this is the case.

>I guess for headers
> , a header selector box like in settings can be added to dir2dat and that should
> be enough for your purposes.

Thanks much, that would be great.

> anyone building these detectors, heed the advice in the spec about ignoring size
> when creating the DATs, the rebuilder will not work if a size field is present

Hmm...the size in the dat should represent the size without the header and the rebuilder should be aware of that. I will recheck that, too.




> This sounds very interesting. Does this mean that if there is a 'game' entry
> with the same name the Dir2DAT will grab the description, etc... from the
> loaded profile and populate that in the new DAT, allowing all of the manual work
> of DAT creation to be reused as much as possible? If so, can you tell me the
> matching criteria?

Well..the matching criteria is the chosen setname. If dir2dat created a set called pacman for example and a current MAME profile is loaded, it'll try to match 'pacman' with the sets in MAME and (in case of a match) it takes manufacturer, year, description etc. from that. Guess how I create the sample dat ;)



Roman Scherzer

Thanks for all the help and information. In experimenting with a simple NSF header detector, I discovered a few cases where duplicates were generated due differences in the actual header.

For example, the NSF header (http://tinyurl.com/yznzln) contains certain load/play/init adresses (0x0008) and a region setting (0x007a) which controls playback frequency. Sandwiched between those two important pieces of data, are the label fields (0x000e), which often vary considerably.

Because certain .nsf files can have their load/play/init addresses or region setting corrected, while the data segment(0x0080) remains the same, I am left with duplicates. Someone rebuilding to this DAT could think they have the correct/fixed version, when in fact it contains the wrong load/play/init addresses or region frequency.

So...do you (or anyone) know of a way to skip a portion of the header, specifically the labels, while retaining the load/play/init addresses and the region setting? It seems that the current implementation of the header detector can trim segments from either end of the file, but not (to my understanding) remove a chunk from the middle of a file. Being a "header" detector, this makes sense, I'm just hoping someone has a trick that could help me solve this particular problem.

I have included my NSF detector here as a reference (http://tinyurl.com/v6pbv) for anyone that has some ideas.

hmmm...interesting....so you need something like:

- take some bytes from offset x
- skip y bytes
- take some bytes from offset x + y
- skip
- etc...

I will think about that one.


Roman Scherzer

> hmmm...interesting....so you need something like:
>
> - take some bytes from offset x
> - skip y bytes
> - take some bytes from offset x + y
> - skip
> - etc...
>
> I will think about that one.
>
>
> Roman Scherzer
>

Yeah, that sounds great. Good to see the programmer mind at work, already putting a solution in algorithmic form ;)

> Yeah, that sounds great. Good to see the programmer mind at work, already
> putting a solution in algorithmic form ;)


well...it was more a "so you want an operation to skip blocks"....currently I'm sitting at work and wonder how this could get implemented and defined easily...maybe just as an skip attribute:

rule start_offset="0" end_offset="EOF" operation="wordswap" skip_start="1B" skip_size="5B"

meaning: if rule is true, set start = 0, end = EOF, do a wordswap on the block and skip offsets 0+1B to 0+1B+5B. Hmm...only wonder how or better when the operation should be applied and if that overlaps the memblock....hmmmm

...and that would allow only one skipping area by rule....don't know if that's enough


Roman Scherzer

After reading this again I wonder...if the datasegment is identical while the load/play/init address can vary (e.g. because it was fixed), why isn't a skip 0x80 byte simple header check enough? Sample plain data, same checksum, ignore stuff from the header completely....or do I miss something here?


Roman Scherzer

> After reading this again I wonder...if the datasegment is identical while the
> load/play/init address can vary (e.g. because it was fixed), why isn't a skip
> 0x80 byte simple header check enough? Sample plain data, same checksum, ignore
> stuff from the header completely....or do I miss something here?

I've been pondering this as well. Where does the data actually begin if the header can make a noticable difference in some of these files? When I originally started working on this detector, my goal was to filter out all the duplicates that have different labels. I didn't expect these kinds of results, so I think I'll need to take a closer look at some of these dupes.

In any case thanks for the help. I'm going to rexamine some of these duplicates side-by-side and see what I can make out. I'll keep you posted.

Well, after doing some side-by-side hex comparisons in the wonderful Total Commander, I noticed that a lot of things varied in these dupes. Some things such as number of songs and region will not affect the actual data, but will dramtically affect playback of the NSF file. So for now, I will have to keep the header. If you find the time, adding functionality to perform multiple skips (so I could skip over the tags) would be extremely helpful. In any case, thanks for all the effort and a great utility.

Currently I have nearly zero free time. Real life job's taking all of my time and brings me to 3 different countries within a week...So...I doubt I can add it in the near future. It's noted though.


Roman Scherzer