Offline Editor For Project Files (*.NRD)

[XORadmin]

By the way. The Pattern files are very easy and straight forward. The .NRD files are very complex and grow with every new version and that is why I prefer to use an export/import tool which generates XML or so.

But they are more important things to do. (Got to keep the company running)
What I actually easily could do is that you could bulk import patterns which would include all 175 patterns in one file. That would probably be something I guess?

@Thomas
Yes, a pattern file seems straight forward to import/edit and export back to a .PTN file. So that can open up already a lot of fun possibilities to edit offline and to prepare live sets etc...

Can you be more specific on your idea of having a import/export tool as the way you see it fit? 
What do you mean by "bulk import patterns which would include all 175 patterns in one file"? (If I understand correctly, we then still need to understand the .NRD structure to be able to extract the patterns from it?)

Import export would more be like that a tool could read and generate XML from and to .NRD files. A XML is much easier to maintenance and 'readable' and can be used for a fully external editor. Maybe I will write one in BASIC...mmhuuahahahaha

The Bulk one would be only a 'in between step' and you would only need to know the pattern files as I described here but then 175 times.
Could be as easy as the layout from up here but instead of:
[16] Name 'NerdSEQ Pattern' 
it would be [16] Name 'NerdSEQ Bulk xxx'     (where xxx is the pattern number or xx in hex whatever)
and that 175 times.
Both sides could detect the 'bulk' and keep on reading the patterns until EOF. On import it would detect and 'overwrite' the current patterns (not creating new ones). Possibly the sequencer data would need to be cleaned up in case the used patterns don't match the tracktypes.

Just a fast idea. Shoot on it.  (and yes..the XML one is low prio, so don't expect that anytime soon.)

[eliclement]

By the way. The Pattern files are very easy and straight forward. The .NRD files are very complex and grow with every new version and that is why I prefer to use an export/import tool which generates XML or so.

But they are more important things to do. (Got to keep the company running)
What I actually easily could do is that you could bulk import patterns which would include all 175 patterns in one file. That would probably be something I guess?

@Thomas
Yes, a pattern file seems straight forward to import/edit and export back to a .PTN file. So that can open up already a lot of fun possibilities to edit offline and to prepare live sets etc...

Can you be more specific on your idea of having a import/export tool as the way you see it fit? 
What do you mean by "bulk import patterns which would include all 175 patterns in one file"? (If I understand correctly, we then still need to understand the .NRD structure to be able to extract the patterns from it?)

Import export would more be like that a tool could read and generate XML from and to .NRD files. A XML is much easier to maintenance and 'readable' and can be used for a fully external editor. Maybe I will write one in BASIC...mmhuuahahahaha

The Bulk one would be only a 'in between step' and you would only need to know the pattern files as I described here but then 175 times.
Could be as easy as the layout from up here but instead of:
[16] Name 'NerdSEQ Pattern' 
it would be [16] Name 'NerdSEQ Bulk xxx'     (where xxx is the pattern number or xx in hex whatever)
and that 175 times.
Both sides could detect the 'bulk' and keep on reading the patterns until EOF. On import it would detect and 'overwrite' the current patterns (not creating new ones). Possibly the sequencer data would need to be cleaned up in case the used patterns don't match the tracktypes.

Just a fast idea. Shoot on it.  (and yes..the XML one is low prio, so don't expect that anytime soon.)

Ok I'm getting somewhere for the first steps of the offline editor. I can import the .PTN file successfully. I am able to unpack the raw .PTN files as you described (for the "modular" track type ("13") only for now) and parse it as a JSON file (can be XML as well) and roughly looks like:

Code:

{
    'header': {
        'file_type': 'NerdSEQ Pattern',
        'version_number': (1, 38),
        'track_type': 10,
        'length': 64,
        'clock': 9,
        'name': '\x00\x00\x00\x00\x00\x00\x00\x00',
        'reserved': '((((((((((((((((((((((((((((((((((((((((',
        'track_type_name': 'Modular'
    },
    'pattern': {
        0: {
            'note': 254,
            'placeholder': b'\xff\xff',
            'patch': 0,
            'table': 0,
            'trigger': 0,
            'mod': 4095,
            'fx1': {'command': 0, 'value': 127},
            'fx2': {'command': 255, 'value': 65535},
            'fx3': {'command': 255, 'value': 65535},
            'fx4': {'command': 255, 'value': 65535},
            'groove': 1
        },
        1: {
            'note': 0,
            'placeholder': b'\xff\xff',
            'patch': 1,
            'table': 1,
            'trigger': 1,
            'mod': 4096,
            'fx1': {'command': 1, 'value': 4095},
            'fx2': {'command': 255, 'value': 65535},
            'fx3': {'command': 255, 'value': 65535},
            'fx4': {'command': 255, 'value': 65535},
            'groove': 2
        },
        2: {
            'note': 1,
            'placeholder': b'\xff\xff',
            'patch': 2,
            'table': 2,
            'trigger': 254,
            'mod': 4217,
        .
        .
        .

Converting back to the .PTN file is works also.

Some questions:

1. How do I interpret the 2 bytes of the version_number of the pattern file? As 2 separate chars, as a 2 byte short int or as 2 unsigned chars (as it is in the example above)?
   
2. Can you provide the structure for the other pattern track types  ?
   
3. Bulk import/export is certainly possible, but then I need to be able to extract those 174 patterns out of the .NRD file... If you can give me more info on where those patterns are located, we can start from there by extracting only the patterns (hopefully the wil remain in the same position in the .NRD file. As you mentioned the .NRD files may change over different versions)
   

Cheers!

[XORadmin]

By the way. The Pattern files are very easy and straight forward. The .NRD files are very complex and grow with every new version and that is why I prefer to use an export/import tool which generates XML or so.

But they are more important things to do. (Got to keep the company running)
What I actually easily could do is that you could bulk import patterns which would include all 175 patterns in one file. That would probably be something I guess?

@Thomas
Yes, a pattern file seems straight forward to import/edit and export back to a .PTN file. So that can open up already a lot of fun possibilities to edit offline and to prepare live sets etc...

Can you be more specific on your idea of having a import/export tool as the way you see it fit? 
What do you mean by "bulk import patterns which would include all 175 patterns in one file"? (If I understand correctly, we then still need to understand the .NRD structure to be able to extract the patterns from it?)

Import export would more be like that a tool could read and generate XML from and to .NRD files. A XML is much easier to maintenance and 'readable' and can be used for a fully external editor. Maybe I will write one in BASIC...mmhuuahahahaha

The Bulk one would be only a 'in between step' and you would only need to know the pattern files as I described here but then 175 times.
Could be as easy as the layout from up here but instead of:
[16] Name 'NerdSEQ Pattern' 
it would be [16] Name 'NerdSEQ Bulk xxx'     (where xxx is the pattern number or xx in hex whatever)
and that 175 times.
Both sides could detect the 'bulk' and keep on reading the patterns until EOF. On import it would detect and 'overwrite' the current patterns (not creating new ones). Possibly the sequencer data would need to be cleaned up in case the used patterns don't match the tracktypes.

Just a fast idea. Shoot on it.  (and yes..the XML one is low prio, so don't expect that anytime soon.)

Ok I'm getting somewhere for the first steps of the offline editor. I can import the .PTN file successfully. I am able to unpack the raw .PTN files as you described (for the "modular" track type ("13") only for now) and parse it as a JSON file (can be XML as well) and roughly looks like:

Code:

{
    'header': {
        'file_type': 'NerdSEQ Pattern',
        'version_number': (1, 38),
        'track_type': 10,
        'length': 64,
        'clock': 9,
        'name': '\x00\x00\x00\x00\x00\x00\x00\x00',
        'reserved': '((((((((((((((((((((((((((((((((((((((((',
        'track_type_name': 'Modular'
    },
    'pattern': {
        0: {
            'note': 254,
            'placeholder': b'\xff\xff',
            'patch': 0,
            'table': 0,
            'trigger': 0,
            'mod': 4095,
            'fx1': {'command': 0, 'value': 127},
            'fx2': {'command': 255, 'value': 65535},
            'fx3': {'command': 255, 'value': 65535},
            'fx4': {'command': 255, 'value': 65535},
            'groove': 1
        },
        1: {
            'note': 0,
            'placeholder': b'\xff\xff',
            'patch': 1,
            'table': 1,
            'trigger': 1,
            'mod': 4096,
            'fx1': {'command': 1, 'value': 4095},
            'fx2': {'command': 255, 'value': 65535},
            'fx3': {'command': 255, 'value': 65535},
            'fx4': {'command': 255, 'value': 65535},
            'groove': 2
        },
        2: {
            'note': 1,
            'placeholder': b'\xff\xff',
            'patch': 2,
            'table': 2,
            'trigger': 254,
            'mod': 4217,
        .
        .
        .

Converting back to the .PTN file is works also.

Some questions:

1. How do I interpret the 2 bytes of the version_number of the pattern file? As 2 separate chars, as a 2 byte short int or as 2 unsigned chars (as it is in the example above)?
   
2. Can you provide the structure for the other pattern track types  ?
   
3. Bulk import/export is certainly possible, but then I need to be able to extract those 174 patterns out of the .NRD file... If you can give me more info on where those patterns are located, we can start from there by extracting only the patterns (hopefully the wil remain in the same position in the .NRD file. As you mentioned the .NRD files may change over different versions)
   

Cheers!

Nice!!

Version number Major, Minor each one byte the 38 is decimal => 26 hex where the 26 is for 26. This has historical reasons and it will go up to 29 and then 30 (not 2A). = Binary coded decimal. 

Next pattern types will follow another time. I am busy with the release now.

3: The bulk would be in the pattern files and not the NRD file. The NRD file is way to complex to explain here and I got no time for it.
The Pattern File is then multifunctional and got with an export all patterns included back to back as described before.

I attached an example which you can check out. On top of that I added also the sequencer screen data, so you could work on both, patterns and arrangement. That should cover already 90% of what you want to do.
I got to admit that this would be a shortcut and a bit of a hack and not the way it should be. But you could then already do what you want to do without waiting for me for years until I made the JSON or XML importer/exporter of the NRD files.
So it is the complete pattern data where instead of:
[16] Name 'NerdSEQ Pattern' 
It is this:
[16] Name 'NerdSEQ Bulk 001'          (where 001 is the pattern number. Decimal and leading zeros
This then 175 times.
After this the sequencer data like this:
[16] Name 'NerdSEQ Sequence'
[8] Track type 8 x one byte with the track type numbers (9 Audio, 10 Modular, 12 Video, 13 Trigger16, 14 CV16, 15 Midi)
[8] Sequencer rows 254 times 8 byte. FF is empty, FE is **
[8] Sequencer extra rows 254 times 8 byte. Which includes extra data like the colour. Will describe later.
[8] Sequencer extra extra rows 254 times 8 byte. Which is unused yet but reserved for some fancy future features I got in mind

So it is up to you to detect through the name field if it is only 1 pattern or a bulk or sequencer data.
The importing will be able to just import what is in the file. So either import some patterns (and overwrite them) or only sequencer data or everything.
Bulk import always overwrites patterns. Single pattern import creates a new pattern.
It is your responsibility to take care that if there is also sequencer data or bulk imports that the track types match both the patterns and the sequencer assignments. Else it will be a mess, trust me :-)

[eliclement]

By the way. The Pattern files are very easy and straight forward. The .NRD files are very complex and grow with every new version and that is why I prefer to use an export/import tool which generates XML or so.

But they are more important things to do. (Got to keep the company running)
What I actually easily could do is that you could bulk import patterns which would include all 175 patterns in one file. That would probably be something I guess?

@Thomas
Yes, a pattern file seems straight forward to import/edit and export back to a .PTN file. So that can open up already a lot of fun possibilities to edit offline and to prepare live sets etc...

Can you be more specific on your idea of having a import/export tool as the way you see it fit? 
What do you mean by "bulk import patterns which would include all 175 patterns in one file"? (If I understand correctly, we then still need to understand the .NRD structure to be able to extract the patterns from it?)

Import export would more be like that a tool could read and generate XML from and to .NRD files. A XML is much easier to maintenance and 'readable' and can be used for a fully external editor. Maybe I will write one in BASIC...mmhuuahahahaha

The Bulk one would be only a 'in between step' and you would only need to know the pattern files as I described here but then 175 times.
Could be as easy as the layout from up here but instead of:
[16] Name 'NerdSEQ Pattern' 
it would be [16] Name 'NerdSEQ Bulk xxx'     (where xxx is the pattern number or xx in hex whatever)
and that 175 times.
Both sides could detect the 'bulk' and keep on reading the patterns until EOF. On import it would detect and 'overwrite' the current patterns (not creating new ones). Possibly the sequencer data would need to be cleaned up in case the used patterns don't match the tracktypes.

Just a fast idea. Shoot on it.  (and yes..the XML one is low prio, so don't expect that anytime soon.)

Ok I'm getting somewhere for the first steps of the offline editor. I can import the .PTN file successfully. I am able to unpack the raw .PTN files as you described (for the "modular" track type ("13") only for now) and parse it as a JSON file (can be XML as well) and roughly looks like:

Code:

{
    'header': {
        'file_type': 'NerdSEQ Pattern',
        'version_number': (1, 38),
        'track_type': 10,
        'length': 64,
        'clock': 9,
        'name': '\x00\x00\x00\x00\x00\x00\x00\x00',
        'reserved': '((((((((((((((((((((((((((((((((((((((((',
        'track_type_name': 'Modular'
    },
    'pattern': {
        0: {
            'note': 254,
            'placeholder': b'\xff\xff',
            'patch': 0,
            'table': 0,
            'trigger': 0,
            'mod': 4095,
            'fx1': {'command': 0, 'value': 127},
            'fx2': {'command': 255, 'value': 65535},
            'fx3': {'command': 255, 'value': 65535},
            'fx4': {'command': 255, 'value': 65535},
            'groove': 1
        },
        1: {
            'note': 0,
            'placeholder': b'\xff\xff',
            'patch': 1,
            'table': 1,
            'trigger': 1,
            'mod': 4096,
            'fx1': {'command': 1, 'value': 4095},
            'fx2': {'command': 255, 'value': 65535},
            'fx3': {'command': 255, 'value': 65535},
            'fx4': {'command': 255, 'value': 65535},
            'groove': 2
        },
        2: {
            'note': 1,
            'placeholder': b'\xff\xff',
            'patch': 2,
            'table': 2,
            'trigger': 254,
            'mod': 4217,
        .
        .
        .

Converting back to the .PTN file is works also.

Some questions:

1. How do I interpret the 2 bytes of the version_number of the pattern file? As 2 separate chars, as a 2 byte short int or as 2 unsigned chars (as it is in the example above)?
   
2. Can you provide the structure for the other pattern track types  ?
   
3. Bulk import/export is certainly possible, but then I need to be able to extract those 174 patterns out of the .NRD file... If you can give me more info on where those patterns are located, we can start from there by extracting only the patterns (hopefully the wil remain in the same position in the .NRD file. As you mentioned the .NRD files may change over different versions)
   

Cheers!

Nice!!

Version number Major, Minor each one byte the 38 is decimal => 26 hex where the 26 is for 26. This has historical reasons and it will go up to 29 and then 30 (not 2A). = Binary coded decimal. 

Next pattern types will follow another time. I am busy with the release now.

3: The bulk would be in the pattern files and not the NRD file. The NRD file is way to complex to explain here and I got no time for it.
The Pattern File is then multifunctional and got with an export all patterns included back to back as described before.

I attached an example which you can check out. On top of that I added also the sequencer screen data, so you could work on both, patterns and arrangement. That should cover already 90% of what you want to do.
I got to admit that this would be a shortcut and a bit of a hack and not the way it should be. But you could then already do what you want to do without waiting for me for years until I made the JSON or XML importer/exporter of the NRD files.
So it is the complete pattern data where instead of:
[16] Name 'NerdSEQ Pattern' 
It is this:
[16] Name 'NerdSEQ Bulk 001'          (where 001 is the pattern number. Decimal and leading zeros
This then 175 times.
After this the sequencer data like this:
[16] Name 'NerdSEQ Sequence'
[8] Track type 8 x one byte with the track type numbers (9 Audio, 10 Modular, 12 Video, 13 Trigger16, 14 CV16, 15 Midi)
[8] Sequencer rows 254 times 8 byte. FF is empty, FE is **
[8] Sequencer extra rows 254 times 8 byte. Which includes extra data like the colour. Will describe later.
[8] Sequencer extra extra rows 254 times 8 byte. Which is unused yet but reserved for some fancy future features I got in mind

So it is up to you to detect through the name field if it is only 1 pattern or a bulk or sequencer data.
The importing will be able to just import what is in the file. So either import some patterns (and overwrite them) or only sequencer data or everything.
Bulk import always overwrites patterns. Single pattern import creates a new pattern.
It is your responsibility to take care that if there is also sequencer data or bulk imports that the track types match both the patterns and the sequencer assignments. Else it will be a mess, trust me :-)

Super! Thanks for the quick response, I can continue from here :-) There will be always new questions on the horizon...
Is the bulk pattern export/import already planned in your roadmap?

[XORadmin]

Is the bulk pattern export/import already planned in your roadmap?

In the next release (todayish) and already working (at least with the file provided).

[eliclement]

@Thomas

When programming my value validator function I was going through all the commands and have the following questions:

1. The manual on p.28 says "...the note scaling begins from C-0 to A#9." should it be: "...the note scaling begins from C-0 to C-A." (I checked it and indeed the NerdSeq highest note input value the different columns is always C-A.
   
2. Is it correct that the FX command "CBPB" has values for the Swing from -64 to 62 (FX value range (FX column value range 0x600 up to 0x67E)?
   
3. Is it correct that the FX command "CBPB" has values for the "Trk jump to row" from 0x00 to 0xFC (0-252) (FX column value range for 0x700 up to 0x7FC)? I should expect it to go up to 255
   
4. Is it correct that the FX command "CBPB" has values for the "All jump to row" from 0x00 to 0xFC (0-252) (FX column value range for 0x800 up to 0x8FC)? I should expect it to go up to 255
   
5. What does the following FX command "TABL" with value: "STP" mean? Does this mean the current table stop command?
   

Thanks for the great work on the new release 1.27!
Cheers!

[XORadmin]

@Thomas

When programming my value validator function I was going through all the commands and have the following questions:

1. The manual on p.28 says "...the note scaling begins from C-0 to A#9." should it be: "...the note scaling begins from C-0 to C-A." (I checked it and indeed the NerdSeq highest note input value the different columns is always C-A.
   
2. Is it correct that the FX command "CBPB" has values for the Swing from -64 to 62 (FX value range (FX column value range 0x600 up to 0x67E)?
   
3. Is it correct that the FX command "CBPB" has values for the "Trk jump to row" from 0x00 to 0xFC (0-252) (FX column value range for 0x700 up to 0x7FC)? I should expect it to go up to 255
   
4. Is it correct that the FX command "CBPB" has values for the "All jump to row" from 0x00 to 0xFC (0-252) (FX column value range for 0x800 up to 0x8FC)? I should expect it to go up to 255
   
5. What does the following FX command "TABL" with value: "STP" mean? Does this mean the current table stop command?
   

Thanks for the great work on the new release 1.27!
Cheers!

1: C-A it is.
5: Table 'Stop'  (same as ** in the table column)
Following is updated in V1.27A:
2: -63 to +63 from now
3,4: Set to FE now (they are 254 rows)

[eliclement]

@Thomas

When programming my value validator function I was going through all the commands and have the following questions:

1. The manual on p.28 says "...the note scaling begins from C-0 to A#9." should it be: "...the note scaling begins from C-0 to C-A." (I checked it and indeed the NerdSeq highest note input value the different columns is always C-A.
   
2. Is it correct that the FX command "CBPB" has values for the Swing from -64 to 62 (FX value range (FX column value range 0x600 up to 0x67E)?
   
3. Is it correct that the FX command "CBPB" has values for the "Trk jump to row" from 0x00 to 0xFC (0-252) (FX column value range for 0x700 up to 0x7FC)? I should expect it to go up to 255
   
4. Is it correct that the FX command "CBPB" has values for the "All jump to row" from 0x00 to 0xFC (0-252) (FX column value range for 0x800 up to 0x8FC)? I should expect it to go up to 255
   
5. What does the following FX command "TABL" with value: "STP" mean? Does this mean the current table stop command?
   

Thanks for the great work on the new release 1.27!
Cheers!

1: C-A it is.
5: Table 'Stop'  (same as ** in the table column)
Following is updated in V1.27A:
2: -63 to +63 from now
3,4: Set to FE now (they are 254 rows)

@Thomas,
Ok thanks!
Do you check the version of the pattern files upon importing? Do you allow only pattern files to be imported with the same NerdSeq firmware version? 
I tried to import a single pattern file (not the bulk one) and got "Pattern import failed". It can be my bad constructed pattern file during testing here, but I just want to make sure if you do allow or restrict only certain versions for imports...

[XORadmin]

Older pattern files can always be imported. = Backward compatible.
I don't check the version number yet, since nothing changed so far. (I do in the NRD files)

I see I didn't compile with the finished version of the importer/exporter which causes the error you see.... will be updated in a few minutes.

[XORadmin]

Updated. Sorry for the trouble.