Sony Network Walkman NW-S23 MP3 File Storage Introduction The NW-S23 apparently supports real MP3 playback; it just obfuscates the files before writing them to the device. I've reverse-engineered the obfuscation sufficiently that I can read and write files from the device, and the few bits I haven't managed to explain away don't seem to matter - they're probably "magic bytes" (version, etc.) or other non-critical data. File Mechanism and Layout Please note that this describes v1.2 and earlier of the MP3FileManager application. V2.0 and later uses a different method of obfuscating files which is covered in FILE_FORMAT_v2.txt When you load an MP3 file to the device via the MP3FileManager application, the first thing it does is to create a folder for the file. If you've just dropped a single MP3 file onto the application, your folder will be called "New Folder" or similar; if you drag an entire folder to the application, the folder name will be copied. The folder data is stored in a file called PBLIST1.DAT; whenever this file is modified, it is backed up to PBLIST0.DAT and the new data written to a fresh PBLIST1.DAT. Once the folder has been created, the MP3 file's ID3 information is stripped, some of which ends up in the PBLIST1.DAT file, the MP3 file is obfuscated and written to the device, and the track number and folder/playlist position of the obfuscated file are copied to the PBLIST1.DAT file. I'm not clear on the exact order of how this happens, but it seems logical that the application would attempt to first write the obfuscated file, and only if that succeeds update the PBLIST1.DAT file. The obfuscated file is named MPXXXX.DAT, where XXXX is the track number in zero-padded hex format. Note that this has no relation to the ID3 track number; it's simply an internal index used by the NW-S23 to identify the track. It's also used in the obfuscation algorithm, as you'll see below. To finish with the grosser details of the file handling, the files are located on the device as follows: [device root] (e.g. /media/NW-S23 on Linux, E: on Windows) | +-control (various files I don't know/care about live here) +-esys | | | +-nw-mp3 ** MPXXXX.DAT files go here | | | +-PBLIST0.DAT (backup playlist) | +-PBLIST1.DAT (live playlist) | +-hifi (more files I don't know/care about, probably ATRAC area) Specifics of File Formats General All multi-byte integers are stored in big-endian format, which means if you're writing code for glibc on Intel chips to interface with this you'll need to do an amount of byteswapping. The library code I've written does this for you where appropriate, e.g. in extracting track numbers, but foldernames and suchlike are left in their on-disk format. All text strings appear to be big-endian UTF-16 with null termination. Folder names run to a maximum of 126 characters + NULL, and other metadata runs to 127 characters + NULL. PBLIST format * The file starts with an 8-byte signature consisting of the characters "WMPLESYS". * The next 4-byte longword is a timestamp matching the creation time of the ESYS folder. It's stored in FAT timestamp format, which is awkward, but well-documented elsewhere. It also doesn't appear to actually matter if this is set correctly or not. * The next 4-byte longword is the media serial number of the device. This is a 4-byte sequence at offset 0x27 in the FAT bootsector. The NW-S23 doesn't appear to care if this is correct or not, but it's possible that other devices check it. * The last longword appears to vary only when the device is reformatted, and neither the NW-S23 nor the NW-E55 seem to care what it's set to. * Next, there's a pair of 4-byte longwords containing the number of folders and the number of tracks on the device. * The last four bytes of header data are a longword XOR checksum of the header bytes. If you take the entire header including this field as longwords, and XOR them, you should end up with 0. * Next we get to actual data. First comes the folder list: for each folder on the device, there's a 256-byte block. The first 252 bytes are are 126 words containing the folder name in big-endian UTF-16 format. The last four bytes make up a longword pointing to the start of the tracklist for this folder as an absolute file offset - you should be able to fseek() to this offset and start reading the tracklist for the folder. * After all the folders comes the tracklist which the folders' longword pointer points into. There is only a single tracklist on the device, consisting of a list of words representing each track. Thus for multiple folders the list is something like "Folder 1 Track 1", "Folder 1 Track 2" ... "Folder 2 Track 1". Obviously this allows you to trivially move tracks between folders or change the order of tracks without having to reencode the files. The block is rounded up to the nearest multiple of eight bytes by zero-padding, and the only way you can find out how big the block is is by using the number of tracks field in the header plus a bit of math. When writing files to the device, the Sony application appears to try to fill holes in the existing tracklist before allocating new numbers, so for example if you've got tracks 1, 2 and 4 and you add a new track, it will be numbered 3 rather than 5. Your tracklist will still be written out in the correct order, i.e. 1, 2, 4, 3. * After the folder list comes the track metadata. For each track the device stores the original filename, the track title, and the artist, in that order, in fixed-sized blocks of 128 words. Unused space (i.e. for short strings) is zero-padded. * The file ends at the last block of metadata - there's no trailer. MPDAT format This is the fun one. * The file starts with a 4-byte signature, "WMMP" * Next is a 4-byte longword giving the total file-size in bytes. This includes the file header, i.e. it's exactly what you'd see displayed in a directory listing of the file. * Next is the duration of the track in milliseconds, again in a 4-byte longword. * The third 4-byte longword gives the number of frames in the file. If you're trying to write a file to the device using your own code, I recommend ripping bits out of XMMS or mp3info to get this number, as I had difficulty locating a library that would calculate it without actually decoding the entire file. * There are 16 bytes of magic: there's the media serial number sequence that occurs in the PBLIST file, followed by 0x01, and padded out to 16 bytes with 0x00. Beyond the MSN, the data doesn't appear to be significant. * The rest of the file is the obfuscated MP3 data, with no ID3 frames - strip those out before you encode or your file will not play in the device. The Obfuscation Mechanism The obfuscation mechanism is a trivial "substitution cypher" based on the track number. Start off with a 256-byte array (one for each possible byte value) and fill it with array[index] = 256 - index. Then, start working your way through powers of 2 from 1 up to the biggest power of 2 less than or equal to the track number. For each power N, if the track number has bit N set, go through your array in blocks of 2N, and swap the first N bytes of the block with the second N bytes. Here's the C code I've written to do this: void mple_build_conv_array( guint16 trackno, guint8 *conv ) { guint16 bit; guint16 i; for ( i = 0; i < 256; i++ ) { conv[i] = 255 - i; } bit = 1; while( bit <= trackno ) { if ( trackno & bit ) { guint16 j; guint16 k; for ( j = 0; j < 256; j+= bit * 2 ) { for ( k = 0; k < bit; k++ ) { guint8 temp; temp = conv[j + k]; conv[j + k] = conv[j + k + bit]; conv[j + k + bit] = temp; } } } bit <<= 1; } } Note that this array works for conversion in either direction. However, before it can be used, a further XOR must be applied to the conversion array; the entire array is xor'd with a bitflipped version of the last octet of the media serial number. Device Formatting A standard Win2k format of the device rendered it unusable. Having examined the differences between that and the result of using the Initialize option in SonicStage, it appears that the differences are in the drive geometry. A suitable mformat command line to completely erase the NW-S23 is 'mformat -t 975 -h 16 -s 32 -H 37 :'. I would recommend not reformatting the device, however, as it shouldn't be necessary to make use of the rest of this information. History v1.2 August 7, 2005: Include device formatting information. v1.1 August 6, 2005: Include media serial number and E55 XOR comment. v1.0 April 10, 2005: Initial version =============================================================================== v1.1 / Ronan Waide / August 6th, 2005 / Distribute as you see fit