did.dat

A new thread for discussion of did.dat extraction techniques.

To start is a post I made on another thread describing the did.dat layout for TAW:

The TAW (and most other) did.dat files are of this form:

The first 4 bytes are pointer to to the index section at the end of the file.
In the case of TAW’s this is 0x03cc22ae which gets us to this point:

At this point are 4 16-bit numbers:
0x312c=12588 The number of files in the archive
0x0069=105 Number of path specifiers
0x0047=71 Number of file extensions
0x06e1=1761 Number of bytes until the start of the index (ie to the point after the paths and extensions) are listed:

Now we have the list proper and there are 12 bytes for each of the 12588 files:
The first 4 bytes are a mysterious index, then 4 bytes for the offset in did.dat, finally 4 bytes for the file size.
In this case we have:
Index=0x8006c719, Offset=0x02bd558a, Size=0x00000068

The hope would be that this index is a compressed version of the filename, but hope is fading …

These index numbers increment in a strange way, the differences don’t correspond to the compressed or uncompressed sizes of the files they relate to.

This index is of TFX’s 3460 files, but you get the idea. I’ve left the index as hex, but converted the offsets and sizes to decimal:

0 80117f36 1572576 415
1 8020f12b 1714256 421
2 802fc869 3142886 40
3 80310755 1706269 432
4 80416965 1068735 206
5 8045e661 3154385 240
6 80487169 1397285 587
7 8052fd69 1248725 526
8 805fc74a 2916069 940
9 80624825 3132188 40
10 806d494a 1881578 1702
11 8075a665 3155401 267
12 8078b479 3117002 40
13 807b5549 1708292 536
14 8080b759 1371479 859
15 80861761 1068095 437
16 80897c35 3169674 152
17 808b6635 3171708 152
18 808d6835 3173492 152
19 808dcb35 1571740 836
20 808fdb50 3274013 4610
21 80903553 2814718 607
22 80910579 1227034 346
23 80913f53 2819051 156
24 80923953 2819948 156
25 809c1532 1674678 122
26 80a31f35 1583159 196
27 80a38c23 3162430 183
28 80a51935 1585754 308
29 80ab154d 1712500 655
30 80b93c35 3171252 152
31 80bb2635 3173120 124
32 80bd2835 3176176 152
…
2857 afcfa293 10413207 169
2858 afd60c8a 2541536 5756
2859 afd7a493 10376338 214
2860 afd8a291 10378145 246
2861 afda332b 1715744 1156
2862 afdaa5d4 1939414 141
2863 afdfaaa8 3213531 82
2864 afe9ffd9 1594625 302
2865 aff993a9 1885281 130
2866 affbb7a0 1961241 552
2867 30209303 1511452 565
2868 30db9c02 1981640 15589
2869 3210a202 3159339 30
2870 3966dc01 333934 5678
2871 3c168102 3496343 133
2872 3e3cb702 1840737 204
2873 3f3eb502 1895112 1830
2874 40060605 1043223 4895
2875 40ba5904 1640423 5796
2876 41080405 1048118 3733
…
3445 7f095d29 2809104 603
3446 7f31fc35 3156283 40
3447 7f3f522e 2265448 140
3448 7f498535 1502889 431
3449 7f4d9d35 1503320 412
3450 7f632629 7075180 6017
3451 7f845429 3194679 40
3452 7f87912d 1656202 756
3453 7fb41429 3194919 40
3454 7fccf939 1222871 565
3455 7fd5d638 3390152 8639
3456 7fd9282a 2092604 198
3457 7fe9a529 1918678 113
3458 7feadc2c 3196081 50
3459 7ff3e635 4247747 120866

The unknown indices would seem to be a hash of the filename, and I’ve managed to recreate the process for a least one file, coltab.dat which is the first file extracted from did.dat.

This stuff is a real pain to work with, so in case I can’t be bothered looking at this again here’s some crude Python code which emulates the TAW subroutine. The subroutine is fed with the string coltabA where the last A may be a key for the .dat.
EDIT: Confirmed. The number 0x41 gives the code for .dat. It’s a coincidence that this is also the ASCII code for A.

This would appear to be a one way process, meaning we can’t get the filename from the hash. If we know the name of a missing file, it should now be possible to locate.

instring='coltabA'     #The first file TAW reads is coltab.dat, 'A' may be a key for '.dat'
length=len(instring)
if length>15:
    length=15

var=83          #53 hex.....a variable from elsewhere, seems to be constant
mod=1048571     #ffffb hex
temp=0          #Accumulator
i=0             #loop variable
h=0             #a bitmap, where 1 is added from the right if char code > the last one, 0 otherwise
while i<length:
    t=instring[i]  #read in char
    f=ord(t)    #Treat as decimal
    if f==95:    #Test for the underscore '_'
        temp-=i
    else:
        if i>0:
            if f>oldf:
                h=h*2+1
            else:
                h=h*2
            temp=((var*temp)+f) % mod
        else:
            temp=f
    oldf=f
    i+=1
a=length & 15
a=a*268435456   #shift 28 bits to the left
b=temp*256      #shift 8 bits to the left
hash=a^b^h      #xor them
print instring, hex(hash)

Krishty

We need to know where the EXE gets the filename from before it is fed into the hash function! It would be alright if it’s just disassembly; I could probably read it.

Also, we need to find out if all names are capital letters, or non-capital letters or both: If they are mixed, then there are (26+26+10+1)^8 possible names for each file (letters, capital letters, digits, underscore) – that’s 57,778 possible names per available hash (2^32). If we could reduce it down to capital letters, then it would be only (26+10+1)^8 ÷ 2^32 = ~818 possible names per hash. If we could definitely say that all file names are at least three characters long, this would go down further. And so forth. In the end, we may reduce the possibility space (don’t know the English word here) far enough to build a rainbow table.

'A' could be a reference to the compression. Compressed files are marked 'RA', I guess uncompressed files just had the attribute 'A'.

The problem is the the names come from all over the place. There’s only a few places where it all comes together.
It’s not all bad though, some code upstream of this hash function removes and encodes the file path and extensions and convert any letters to lower case. So, the hash function gets fed a one byte code for the file path, then the name consisting of lower case letters, numbers and underscore and finally a one byte code for the extension.

The leading and trailing codes are as follows:

Path Identifier, first byte into hash function

0x30  pcx\scenario\scenario\campaign\
0x31  pcx\scenario\scenario\arcade\
0x32  pcx\scenario\scenario\sim\
0x33  samples\sfx\engines.old\
0x34  pcx\scenario\campaign\
0x35  pcx\mainmenu\spanish\
0x36  pcx\mainmenu\english\
0x37  pcx\mainmenu\italian\
0x38  pcx\scenario\arcade\
0x39  pcx\mainmenu\french\
0x3a  pcx\mainmenu\german\
0x3b  fonts\hudfont.old\
0x3c  pcx\scenario\tod\
0x3d  briefing\spanish\
0x3e  briefing\italian\
0x3f  pcx\scenario\sim\
0x40  briefing\german\
0x41  f22data\italian\
0x42  f22data\spanish\
0x43  briefing\french\
0x44  fonts\topazpro\
0x45  huddle\spanish\
0x46  huddle\italian\
0x47  pcx\background\
0x48  f22data\german\
0x49  fonts\fifth_av\
0x4a  fonts\helvetic\
0x4b  f22data\french\
0x4c  fonts\sansserf\
0x4d  pcx\quitscreen\
0x4e  fonts\topazmon\
0x4f  samples\speech\
0x50  fonts\guifont\
0x51  fonts\didfont\
0x52  fonts\hudfont\
0x53  fonts\univers\
0x54  huddle\german\
0x55  samples\beeps\
0x56  huddle\french\
0x57  samples\music\
0x58  samples\betty\
0x59  fonts\granite\
0x5a  samples\noise\
0x5b  fonts\newhud\
0x5c  fonts\triumv\
0x5d  fonts\tuxedo\
0x5e  pcx\campaign\
0x5f  pcx\insignia\
0x60  pcx\missions\
0x61  pcx\multipla\
0x62  pcx\quickcom\
0x63  pcx\squadron\
0x64  pcx\titlebar\
0x65  pcx\options\
0x66  pcx\debrief\
0x67  pcx\gamemap\
0x68  fonts\times\
0x69  kdl\spanish\
0x6a  kdl\italian\
0x6b  iff\256\gui\
0x6c  samples\gui\
0x6d  fonts\micro\
0x6e  samples\sfx\
0x6f  pcx\credits\
0x70  pcx\warning\
0x71  pcx\winlose\
0x72  hardware.tm\
0x73  cfg\weapons\
0x74  kdl\german\
0x75  pcx\pilots\
0x76  fonts\kara\
0x77  pcx\stamps\
0x78  fonts\blob\
0x79  kdl\french\
0x7a  pcx\skies\
0x7b  pcx\misc\
0x7c  winfonts\
0x7d  waypoint\
0x7e  briefing\
0x7f  red0200\
0x80  red0600\
0x81  red1000\
0x82  red1400\
0x83  red1800\
0x84  red2200\
0x85  f22data\
0x86  iff\256\
0x87  colours\
0x88  samples\
0x89  huddle\
0x8a  mp_col\
0x8b  fonts\
0x8c  cgdat\
0x8d  hwacl\
0x8e  music\
0x8f  mdl\
0x90  kdl\
0x91  cdl\
0x92  lev\
0x93  dat\
0x94  agp\
0x95  pdl\
0x96  udl\
0x97  ss\
0x98  3\

Extension identifier, last byte into hash function

0x41  .dat
0x42  .txt
0x43  .ini
0x44  .gdd
0x45  .win
0x46  .cfg
0x47  .saf
0x48  .lad
0x49  .and
0x4a  .hud
0x4b  .ian
0x4c  .ins
0x4d  .rog
0x4e  .dmo
0x4f  .taw
0x50  .sjp
0x51  .fig
0x52  .icd
0x53  .sav
0x54  .gd2
0x55  .brf
0x56  .lst
0x57  .smk
0x58  .col
0x59  .fad
0x5a  .fnt
0x5b  .ext
0x5c  .lbm
0x5d  .iff
0x5e  .pcx
0x5f  .4e2
0x60  .4ev
0x61  .asc
0x62  .env
0x63  .map
0x64  .trg
0x65  .wld
0x66  .lab
0x67  .opt
0x68  .ssd
0x69  .pak
0x6a  .pdl
0x6b  .udl
0x6c  .mdl
0x6d  .kdl
0x6e  .cdl
0x6f  .tcs
0x70  .hlp
0x71  .hpj
0x72  .sss
0x73  .act
0x74  .raw
0x75  .wav
0x76  .bnk
0x77  .sam
0x78  .bin
0x79  .idx
0x7a  .mds
0x7b  .rst
0x7c  .sf2
0x7d  .ok
0x7e  .13
0x7f  .tm
0x80  .in
0x81  .fn
0x82  .sh
0x83  .pk
0x84  .4
0x85  .5
0x86  .3
0x87  .6

I manually tried out my hash routine on 10 or so filenames and got the same answer as TAW, even with numbers and underscore in the name.

Rainbow tables? Looks complicated. 🙂
http://en.wikipedia.org/wiki/Rainbow_table

Looks interesting … I have a similar list of paths and extensions around here somewhere, from quite some time ago. I’ll have to see what happened with that process. I do remember trying to get the .4, .5 and .6 files to function similarly to the .3 models without much reward 🙁

The undocumented fonts (Triumv, Granite, Fifth_av, etc.) are in the extracted group, however, I got sidetracked before exploring them further. I seem to remember that the specific font file-names are contained within the individual files at a specific address 0x810(?) This one you’re on is a good path to follow 🙂

Krishty’s rainbow table idea is the frontrunner right now since it appears we have all we need:

1. The hash algorithm
2. The hash list to test against
3. The character set

The character set still might be too big, since:
The first byte can be in the range 0x30–0x98
The last byte has a range 0x41–0x87, assuming all files have extensions
The middle bytes can have values 0x30–0x39, 0x5f, 0x61–0x7a

We could speed things up at the start by just looking at one file type, e.g. .3, which would fix the first byte to 0x98 and the last byte to 0x86.

bored

The problem is the the names come from all over the place. There’s only a few places where it all comes together.
It’s not all bad though, some code upstream of this hash function removes and encodes the file path and extensions and convert any letters to lower case.

That’s good news already!

bored

So, the hash function gets fed a one byte code for the file path, then the name consisting of lower case letters, numbers and underscore and finally a one byte code for the extension.

Then again, this is bad news, because now we have to test all folders and file extensions, too.

bored

The leading and trailing codes are as follows:

Awesome 🙇

bored

The character set still might be too big, since:
The first byte can be in the range 0x30–0x98
The last byte has a range 0x41–0x87, assuming all files have extensions
The middle bytes can have values 0x30–0x39, 0x5f, 0x61–0x7a

That’s 25,570,850,424,544,880 possibilities per hash, or 1,775 hours on a Core i7. Impossible to crack brute force if I don’t use GPU & cloud computing 😕 Sticking to a specific file extension is a good idea, I guess. But still I think that analyzing the exe will be the quicker solution in the long run.

Krishty

… But still I think that analyzing the exe will be the quicker solution in the long run.

Maybe so … and fine if you like that sort of thing. 🙂

I’ve spent the last 3 evenings with IDA and just about had enough.

By the way, I’m using the Glide version of f22.dat, so any addresses I quote from the dissassembler will relate to that.

Yeah, great.

Also: As I see, the four most significant bytes of the hash give the length. This already helps a lot: By reading the first hex number of the hash, you know the length of the string. (That’s why there are no hashes beginning with 0, 1 or 2: there is no empty file name, no filenames with one character and no path or extension etc).

You can replace the last two XOR’s in the algorithm with OR’s, because the values won’t overlap (I guess). It’s just combining the three kinds of bits: Four bits for the length, 20 bits for ??? and eight bits for ???.

I’m sure my code can be made much more efficient … I was just amazed it worked. I should try some experiments with known files to make sure it’s universal, but confidence is high.

Here is the dissassembled hash function from IDA:

AUTO:0057B2B8 sub_57B2B8      proc near               ; CODE XREF: sub_4D0A7C+D␘p
AUTO:0057B2B8                                         ; sub_4D0B88+8B␘p ...
AUTO:0057B2B8
AUTO:0057B2B8 var_20          = byte ptr -20h
AUTO:0057B2B8 var_1C          = byte ptr -1Ch
AUTO:0057B2B8
AUTO:0057B2B8                 push    ebx
AUTO:0057B2B9                 push    ecx
AUTO:0057B2BA                 push    edx
AUTO:0057B2BB                 push    esi
AUTO:0057B2BC                 push    edi
AUTO:0057B2BD                 push    ebp
AUTO:0057B2BE                 sub     esp, 8
AUTO:0057B2C1                 mov     edi, ds:dword_6ED7FC
AUTO:0057B2C7                 mov     ebx, eax
AUTO:0057B2C9                 call    strlen_
AUTO:0057B2CE                 mov     esi, eax
AUTO:0057B2D0                 cmp     eax, 0Fh
AUTO:0057B2D3                 jle     short loc_57B2DA
AUTO:0057B2D5                 mov     esi, 0Fh
AUTO:0057B2DA
AUTO:0057B2DA loc_57B2DA:                             ; CODE XREF: sub_57B2B8+1B␘j
AUTO:0057B2DA                 mov     edi, ds:dword_6ED7FC
AUTO:0057B2E0                 xor     ah, ah
AUTO:0057B2E2                 xor     edx, edx
AUTO:0057B2E4                 xor     ecx, ecx
AUTO:0057B2E6                 mov     [esp+20h+var_20], ah
AUTO:0057B2E9                 test    esi, esi
AUTO:0057B2EB                 jle     short loc_57B326
AUTO:0057B2ED                 mov     ebp, 0FFFFBh
AUTO:0057B2F2
AUTO:0057B2F2 loc_57B2F2:                             ; CODE XREF: sub_57B2B8+6C␙j
AUTO:0057B2F2                 mov     al, [ebx]
AUTO:0057B2F4                 cmp     al, 5Fh
AUTO:0057B2F6                 jz      short loc_57B350
AUTO:0057B2F8                 test    ecx, ecx
AUTO:0057B2FA                 jz      short loc_57B312
AUTO:0057B2FC                 cmp     al, [ebx-1]
AUTO:0057B2FF                 jbe     short loc_57B348
AUTO:0057B301                 mov     [esp+20h+var_1C], 1
AUTO:0057B306
AUTO:0057B306 loc_57B306:                             ; CODE XREF: sub_57B2B8+96␙j
AUTO:0057B306                 mov     ah, [esp+20h+var_20]
AUTO:0057B309                 add     ah, ah
AUTO:0057B30B                 or      ah, [esp+20h+var_1C]
AUTO:0057B30F                 mov     [esp+20h+var_20], ah
AUTO:0057B312
AUTO:0057B312 loc_57B312:                             ; CODE XREF: sub_57B2B8+42␘j
AUTO:0057B312                 imul    edx, edi
AUTO:0057B315                 and     eax, 0FFh
AUTO:0057B31A                 add     eax, edx
AUTO:0057B31C                 xor     edx, edx
AUTO:0057B31E                 div     ebp
AUTO:0057B320
AUTO:0057B320 loc_57B320:                             ; CODE XREF: sub_57B2B8+9A␙j
AUTO:0057B320                 inc     ecx
AUTO:0057B321                 inc     ebx
AUTO:0057B322                 cmp     ecx, esi
AUTO:0057B324                 jl      short loc_57B2F2
AUTO:0057B326
AUTO:0057B326 loc_57B326:                             ; CODE XREF: sub_57B2B8+33␘j
AUTO:0057B326                 and     esi, 0Fh
AUTO:0057B329                 shl     edx, 8
AUTO:0057B32C                 xor     eax, eax
AUTO:0057B32E                 shl     esi, 1Ch
AUTO:0057B331                 mov     al, [esp+20h+var_20]
AUTO:0057B334                 xor     edx, esi
AUTO:0057B336                 xor     eax, edx
AUTO:0057B338                 mov     ds:dword_6ED7FC, edi
AUTO:0057B33E                 add     esp, 8
AUTO:0057B341                 pop     ebp
AUTO:0057B342                 pop     edi
AUTO:0057B343                 pop     esi
AUTO:0057B344                 pop     edx
AUTO:0057B345                 pop     ecx
AUTO:0057B346                 pop     ebx
AUTO:0057B347                 retn

EDIT:
Oh, I think the last two parameters in my code could overlap if there are more than 9 characters in the input string.

The bitmap at the end of the hash is interesting. Consider:

If the file name is n characters long, then there are n-1 bit positions.

Most file names are eight characters long, with one additional character for the directory and for the extension.

So there are nine bits occupied. I am optimistic that these are only eight bits in most files.

They give the direction, how the hash must be inverted when the inversion is ambiguous. Very intersting

I’ve had a quick look through the TAW files and can’t find any greater than 8 characters long, not including the extension. Well, there are a few but I’m fairly sure they have been added in the modding process.

We could really do with a fresh set of filenames from did.dat. If only I could remember how to script the IDA debugger, which is how I obtained those EF2000 names back in 2007. It would be useful to be able to do stuff like that again. 😠

One thing to notice:

Suppose you have eight numbers, each in the range of [0; 15], and want to pack them into one 32-bit number. Then you’d go like this:

for(each number)
	result = result * 16 + number;

You could reverse this by doing:

for(each number)
	number = result mod 16;
	result = result / 16;

From what I see, the inner loop of the hash algorithm does:

for(each character)
	result = result * 83 + character;

It’s the same thing, just not with individual bits but on sub-bits. This should somehow be reversible, too. But I haven’t found out yet, how. Math class is waaaay to long back 🙁

Hmmn, I never did pay much attention in school, but then again we didn’t have computers in those days. 🙂

I can’t see how anything can be reversible in our case since we lose information when we do the mod 0xffffb operation.

Krishty

Also: As I see, the four most significant bytes of the hash give the length. This already helps a lot: By reading the first hex number of the hash, you know the length of the string. (That’s why there are no hashes beginning with 0, 1 or 2: there is no empty file name, no filenames with one character and no path or extension etc).

I just realised the significance of that statement. 👍

Here’s the central section of TAW’s hashes showing a few with length 15:

10740 aff09445 1427013 901
10741 aff5922b 47086863 1104
10742 aff5e52b 47041103 1108
10743 aff6382b 1074367 1122
10744 aff68b2b 47128197 1114
10745 affbf341 46104167 5464
10746 b4f9572d 49653285 3206
10747 bda8a45b 16658446 420651
10748 bda8f75b 17079097 198103
10749 bda94a5b 17277200 220064
10750 c3e4664c 30852667 1071
10751 c557214c 31046589 1722
10752 c6ffb24c 30690342 1032
10753 ccf2a098 15995681 15738
10754 ce107c4c 30949127 1796
10755 e4e5c138 32688490 56370
10756 e4e5c338 32744860 44303
10757 e711ba3a 9516747 52437
10758 e93eca18 32937430 10144
10759 ea24263c 9460220 56527
10760 f3d05d24 1992645 1125
10761 f7c81708 1996147 1029
10762 fc8682e7 1994893 1254
10763 fe1c8946 1993770 1123
10764 30057800 18956062 343
10765 36458f01 3457019 3408
10766 3645e201 3460427 3854
10767 36463501 3464281 3408
10768 36468801 3467689 4082
10769 38b07e01 46071788 1095
10770 38b0d101 46072883 1549
10771 38b12401 46075436 1094
10772 38b17701 46080413 1549
10773 39a30201 3498076 1543
10774 39a35501 3499619 945

I’m trying to remember why I’m getting the 100 byte offset, starting at 0x03cc23ae instead of 0x03cc22ae, any ideas?

seg000:0100  AE 22 CC 03 01 00 05 00  03 00 00 00 52 41 00 02  «"¦␃␁.␅.␃...RA.␂
seg000:0110  B7 46 01 00 00 7D 3B 7A  EE 9D 0A 41 0C 03 04 02  +F␁..};ze¥␤A␌␃␄␂
seg000:0120  08 03 09 04 13 05 1C 81  41 41 E0 61 50 60 D8 E4  ␈␃␉␄␓␅␜üAAaaP`+S
seg000:3CC22A0  A6 B4 15 14 31 38 D6 82  FB 31 82 DD D6 A3 C4 FE  ª¦§¶18+év1é¦+ú-¦
seg000:3CC22B0  3B E6 9E 63 B9 E0 78 A3  3B 5E 84 1E 29 D7 3B A1  ;µPc¦axú;^ä␞)+;í
seg000:3CC22C0  60 32 33 5B F5 77 C5 6A  36 5F 72 61 16 6B 3B D2  `23[)w+j6_ra␖k;-
seg000:3CC22D0  BB 32 D4 51 C6 AE 0B 5F  2B 6D 38 A8 FE 2D E4 76  +2+Q¦«␋_+m8¿¦-Sv
seg000:3CC22E0  61 72 69 6F 75 73 C9 B7  6C 6C 69 38 3E D0 20 65  arious++lli8>- e
seg000:3CC22F0  66 66 65 63 8E E1 00 D3  D7 6E 1B 82 A3 AC 87 C4  ffecÄß.++n␛éú¼ç-
seg000:3CC2300  40 0A 8C B6 ED 84 E0 2A  C5 36 BC 62 6F 6D 62 76  @␤î¦fäa*+6+bombv
seg000:3CC2310  36 E2 99 A4 67 72 D9 81  6E 64 8B 59 BF 36 BC 9B  6GÖñgr+ündïY+6+¢
seg000:3CC2320  FE 5F 79 B2 2C 83 B9 D8  A2 1C 80 20 F9 F8 EC 33  ¦_y¦,â¦+ó␜Ç ·°83
seg000:3CC2330  EC 8C 77 E3 4A 28 81 48  19 0B 11 FB 35 63 3C 08  8îwpJ(üH␙␋␑v5c<␈
seg000:3CC2340  FB 13 63 9B 76 0C 63 72  65 74 65 1E 01 5B 31 A6  v␓c¢v␌crete␞␁[1ª
seg000:3CC2350  51 00 83 79 66 AF 61 69  72 BC 46 32 66 74 56 71  Q.âyf»air+F2ftVq
seg000:3CC2360  4E 53 A5 69 DD 38 07 3D  5A 37 39 41 AA 62 75 69  NSÑi¦8␇=Z79A¬bui
seg000:3CC2370  6C 77 9E 41 44 EA 47 77  BE 67 42 4E B9 75 DC 73  lwPADOGw+gBN¦u_s
seg000:3CC2380  B2 31 8B 43 20 86 A1 62  9B C2 14 09 20 1B 69 B5  ¦1ïC åíb¢-¶␉ ␛i¦
seg000:3CC2390  1F 33 35 BA 5B 1F 4F F2  3B 2F 83 EA 1C C4 56 F6  ␟35¦[␟O=;/âO␜-V÷
seg000:3CC23A0  1D 37 5F 76 E5 67 62 7A  1F FB 38 70 2F 00 2C 31  ␝7_vsgbz␟v8p/.,1
seg000:3CC23B0  69 00 47 00 E1 06 1F 70  63 78 5C 73 63 65 6E 61  i.G.ß␆␟pcx\scena
seg000:3CC23C0  72 69 6F 5C 73 63 65 6E  61 72 69 6F 5C 63 61 6D  rio\scenario\cam

No. What hex editor are you using?

bored

No. What hex editor are you using?

IDA Pro Free

I use a separate hex editor (AXE3) when working with IDA, since with my versions 5.2/5.3 you can’t edit the hex code.

I took a closer look at these files which have length >=15 filenames and there is something weird with them. When extracted, they come out as RA compressed files, but it takes 4 or 5 passes through the decompressor before they get to their plain text form.

This indicates that they are malformed in some way.

10760 f3d05d24 1992645 1125
10761 f7c81708 1996147 1029
10762 fc8682e7 1994893 1254
10763 fe1c8946 1993770 1123

bored

I use a separate hex editor (AXE3) when working with IDA, since with my versions 5.2/5.3 you can’t edit the hex code.

Okay, I’ll go with the AXE3.

bored

I took a closer look at these files which have length >=15 filenames and there is something weird with them. When extracted, they come out as 'RA' compressed files, but it takes 4 or 5 passes through the decompressor before they get to their plain text form.

This indicates that they are malformed in some way.

10760 f3d05d24 1992645 1125
10761 f7c81708 1996147 1029
10762 fc8682e7 1994893 1254
10763 fe1c8946 1993770 1123

Okay, well that’s new. Those are f22data text files aren’t they? In the previously extracted group, I remember having to decompress some files more than once but also discovered that the TAW game process worked normally, recognizing the content of those files in their compressed form.

I dug out the file list for Krusade’s original extractor and cross referenced it to the index of all the 12588 files in TAW’s did.dat to put all known info into one file.
This was then updated for the ~200 .3 files we got the names for yesterday.

Since there is a focus on .3 files right now, I then extracted a fresh set of .3 files with known names from did.dat. Then I extracted all files without filenames (about ~2600) and scanned through to sift out those that look like .3 files, which resulted in about 600 files.

I’ve uploaded these two sets of files with the index to box.net:
http://www.box.net/files#/files/0/f/105927630/Extracted

Right now we have 1842 known .3 files, and 595 unknown.

Note that some of the 595 unknown are probably already identified. That’ll require some further effort to find out which …

bored

I dug out the file list for Krusade’s original extractor and cross referenced it to the index of all the 12588 files in TAW’s did.dat to put all known info into one file.
This was then updated for the ~200 .3 files we got the names for yesterday.

Since there is a focus on .3 files right now, I then extracted a fresh set of .3 files with known names from did.dat. Then I extracted all files without filenames (about ~2600) and scanned through to sift out those that look like .3 files, which resulted in about 600 files.

I’ve uploaded these two sets of files with the index to box.net:
http://www.box.net/files#/files/0/f/105927630/Extracted

Right now we have 1842 known .3 files, and 595 unknown.

Note that some of the 595 unknown are probably already identified. That’ll require some further effort to find out which …

👍 That will be quite helpful. When I trace your offsets back (AXE3: access denied … therefore using WinHex 😩) for the file doesn’t exist group, I have been getting a not found response just as you did but I know I have many (all?) of those files (eg: su331.3 = noname11673(4)). I was up late last night going thru Krus’ 4-extract and 4-extract_en_d tawfiles.txt to cross reference … and don’t get me wrong, I do appreciate what you are doing here, but I must admit … this work is painful 🙄

Hmmn, I see what you mean about su331.3. I have a similar plane at noname11672 (seems like we started counting from 1 before).

Also uk747.3 is not in the Krus list I’ve been working with, but appears in my 3 folder, so I may not have the latest list. Thus uk747.3 is one of the 595 unknown files for now.

Do you have proof that su331.3 is noname11673, or is it an educated guess?

There is a disturbing possibility that one of the terms in my hash algorithm (what I’ve called var) is not constant. This could explain why I calcuate an incorrect hash for su331.3 and the others … but I really hope it’s something else.

I’m trying to automate as much of the donkey work as possible, but will probably still end up asking for help in trying to find out all the various places where the filenames come from …

bored

Hmmn, I see what you mean about su331.3. I have a similar plane at noname11672 (seems like we started counting from 1 before).

Also uk747.3 is not in the Krus list I’ve been working with, but appears in my 3 folder, so I may not have the latest list. Thus uk747.3 is one of the 595 unknown files for now.

Do you have proof that su331.3 is noname11673, or is it an educated guess?

There is a disturbing possibility that one of the terms in my hash algorithm (what I’ve called var) is not constant. This could explain why I calcuate an incorrect hash for su331.3 and the others … but I really hope it’s something else.

I’m trying to automate as much of the donkey work as possible, but will probably still end up asking for help in trying to find out all the various places where the filenames come from …

The proof remains indirect, however, I believe it to be fairly substantial. noname11673 is a Sukhoi model, the header and code and model are unique but very similar to that of the Su-27 (Su-33 is a naval version of the Su-27). Viewing it in AC3d does not display a tail-hook (but nothing’s perfect 🙄). Hopefully continued progress will allow you to tweak the hash algorithm and we can confirm it 🙂 I’ll continue working on some of the other files. You have really opened an exciting new can of worms with your TAW hash algorithm 👍

OK thanks. I need to do a few more sanity checks on the TAW hash algorithm before we can rely on it 100%.

Once that’s done, we can look at the files again.

Wow. AWESOME! Many of them are glitchy, some have unknown opcodes in them. Some are beautiful. There’s so many unused icons … looks like there were entire game modes planned but never implemented!

This Su-27 has afterburner textures (the su271.3 used in the game hasn’t):

noname7248 is AGP Moon

noname7329 is beautiful dust trails

noname10879 is a blue explosion

noname10987 is the F-22 cockpit

noname12508 is an F-16XL with alternative painting scheme

Many models look familar, but their damage models are different to the final game.

But – it’s a pity – there’s no complete EF2000 hidden in there 😉 I spent a whole hour just looking at the new files; can’t wait for the weekend to begin!

Krishty

there’s no complete EF2000 hidden in there 😉 I spent a whole hour just looking at the new files; can’t wait for the weekend to begin!

… Yeah, I’m also having a hard time getting anything useful done because of Boreds recent posts, it really allows for a little better perspective on what these additional game modes might be. Simply going back and reviewing many of the previous model sets to revise my theories 😉 This is a virtual AGP re-texturing paradise … and there’s more 😁

PS: 11672 (su331.3?) also has the afterburner textures, some time ago I modified the F22 afterburners with some new and current textures and it’s a very nice effect 🙂
What’s causing the duplication of the HOTAS textures on the cockpit sideboards? Why does the Viewer not display the transparent textures (rain drops?) that are a part of 10987? They appear to be erroneous calls to refect.tm

DrKevDog

What’s causing the duplication of the HOTAS textures on the cockpit sideboards? Why does the Viewer not display the transparent textures (rain drops?) that are a part of 10987? They appear to be erroneous calls to refect.tm

Excellent questions 🙂 Answering them will take a lot of time to analyze the specific opcodes, but I don’t have it now. I remember, though, that this is not the first transparent texture to appear wrong. I guess there’s some rarely-used texture mode hidden in one of the rare opcodes …

Krishty

Excellent questions 🙂 Answering them will take a lot of time to analyze the specific opcodes, but I don’t have it now. I remember, though, that this is not the first transparent texture to appear wrong. I guess there’s some rarely-used texture mode hidden in one of the rare opcodes …

Okay, I’ll begin working on the analysis, when time permits. Currently I’m engaged in dialogue with AMD regarding the marginally supported AGP Hotfix drivers. I am hoping to get a slightly more stable testing platform for the TAW AGP textures. It’s not yet perfect, however, at least we’re making good progress now:

noname7248 is AGP Moon

DrKevDog

What’s causing the duplication of the HOTAS textures on the cockpit sideboards? Why does the Viewer not display the transparent textures (rain drops?) that are a part of 10987? They appear to be erroneous calls to refect.tm

Okay, the answer is quite easy here:

The file is just broken.

The cockpit used in the final game is vrcpt.3 and there’s not the slightest glitch in it (except it’s not at the coordinate origin, so it’s hard to look at); I guess noname10987 is from an early development stage and that’s why it looks buggy.

The Moon is just as deformed here. I really don’t get why it was so hard for DiD to make a decent-looking Moon 🙁

Anyone has an idea what’s wrong with noname11695.3? Looks like different scale factors on the coordinate axes …

Edit: There’s definitely something wrong with the viewer; texture tiling won’t work properly any more. I need to look into it tomorrow before I can analyze unknown files.

Krishty

Anyone has an idea what’s wrong with noname11695.3? Looks like different scale factors on the coordinate axes …

Interesting the way the depth coordinate seems to change in such a fluid manner in the Viewer.

Can’t recall seeing the following redundant pattern in any other file:

006200770000000000000066000000660000006600000066000000660000006600000066000000660000006600000066000000660000006600000066000000660000006600000066000000660000006600000066000000660000006600000066000000660000006600000066000000660000006600000066000000660000006600000066000000660000006600000066000000660000006600000066000000660000006100270077

An update on the the hash effort:

For TAW, I’ve calculated the hash values for all 9972 known filenames and compared them with the hash values stored in did.dat. This revealed 98 (or about 1%) that did not match.
For these 98, the hashes are identical except for the 6th character. I’m not sure what causes this, but could be explained if the extension key can vary.

For EF2000, I’ve located the hash routine in the exe for SuperEF2000 and can confirm that the pathname and extension symbols are as follows:

0x30  german\supergui\
0x31  french\supergui\
0x32  german\iff\256\
0x33  french\iff\256\
0x34  french\cfg\
0x35  french\gui\
0x36  german\gui\
0x37  german\cfg\
0x38  missions\
0x39  supergui\
0x3a  iff\256\
0x3b  colours\
0x3c  samples\
0x3d  waybin\
0x3e  german\
0x3f  views\
0x40  music\
0x41  anim\
0x42  iff\
0x43  spr\
0x44  dat\
0x45  lev\
0x46  cfg\
0x47  gui\
0x48  tm\
0x49  ss\
0x4a  3\

0x41  .hdd
0x42  .fad
0x43  .gui
0x44  .cfg
0x45  .dat
0x46  .txt
0x47  .exe
0x48  .tfx
0x49  .256
0x4a  .brf
0x4b  .sav
0x4c  .ne1
0x4d  .doc
0x4e  .dbn
0x4f  .mds
0x50  .and
0x51  .col
0x52  .pal
0x53  .clt
0x54  .clu
0x55  .par
0x56  .plt
0x57  .mis
0x58  .hud
0x59  .flt
0x5a  .lbm
0x5b  .ste
0x5c  .ef2
0x5d  .bin
0x5e  .abm
0x5f  .bbm
0x60  .pcx
0x61  .pmi
0x62  .pmm
0x63  .opt
0x64  .trg
0x65  .xmi
0x66  .le2
0x67  .lev
0x68  .sta
0x69  .4e2
0x6a  .4ev
0x6b  .stp
0x6c  .env
0x6d  .map
0x6e  .wld
0x6f  .ppp
0x70  .stm
0x71  .zip
0x72  .mid
0x73  .bmp
0x74  .asc
0x75  .sam
0x76  .wnd
0x77  .saf
0x78  .spr
0x79  .pos
0x7a  .ssd
0x7b  .lab
0x7c  .way
0x7d  .tm
0x7e  .fn
0x7f  .ad
0x80  .ra
0x81  .cd
0x82  .3

The hash routine probably does something really simple, but I haven’t as yet unpicked the assembly code to create a Python version. I’ll look at it later after drinking some beer which is required for this sort of thing. 🙂

:004FC200 sub_4FC200      proc near               ; CODE XREF: sub_4AC110+D␘p
BEGTEXT:004FC200                                         ; sub_4AC210+A1␘p ...
BEGTEXT:004FC200
BEGTEXT:004FC200 var_18          = dword ptr -18h
BEGTEXT:004FC200
BEGTEXT:004FC200                 push    ebx
BEGTEXT:004FC201                 push    ecx
BEGTEXT:004FC202                 push    edx
BEGTEXT:004FC203                 push    esi
BEGTEXT:004FC204                 push    edi
BEGTEXT:004FC205                 sub     esp, 4
BEGTEXT:004FC208                 mov     esi, eax
BEGTEXT:004FC20A                 call    strlen_
BEGTEXT:004FC20F                 mov     [esp+18h+var_18], eax
BEGTEXT:004FC212                 cmp     ax, 0Fh
BEGTEXT:004FC216                 jle     short loc_4FC21F
BEGTEXT:004FC218                 mov     [esp+18h+var_18], 0Fh
BEGTEXT:004FC21F
BEGTEXT:004FC21F loc_4FC21F:                             ; CODE XREF: sub_4FC200+16␘j
BEGTEXT:004FC21F                 mov     edi, [esp+18h+var_18]
BEGTEXT:004FC222                 xor     bl, bl
BEGTEXT:004FC224                 xor     dl, dl
BEGTEXT:004FC226                 xor     ch, ch
BEGTEXT:004FC228                 xor     eax, eax
BEGTEXT:004FC22A                 mov     dh, [esi]
BEGTEXT:004FC22C                 xor     cl, cl
BEGTEXT:004FC22E                 test    di, di
BEGTEXT:004FC231                 jle     short loc_4FC273
BEGTEXT:004FC233
BEGTEXT:004FC233 loc_4FC233:                             ; CODE XREF: sub_4FC200+71␙j
BEGTEXT:004FC233                 test    ax, ax
BEGTEXT:004FC236                 jz      short loc_4FC253
BEGTEXT:004FC238                 cmp     dh, 5Fh
BEGTEXT:004FC23B                 jz      short loc_4FC253
BEGTEXT:004FC23D                 movsx   edi, ax
BEGTEXT:004FC240                 add     bl, bl
BEGTEXT:004FC242                 cmp     dh, [edi+esi]
BEGTEXT:004FC245                 setb    dh
BEGTEXT:004FC248                 movzx   edi, dh
BEGTEXT:004FC24B                 and     ebx, 0FFh
BEGTEXT:004FC251                 or      ebx, edi
BEGTEXT:004FC253
BEGTEXT:004FC253 loc_4FC253:                             ; CODE XREF: sub_4FC200+36␘j
BEGTEXT:004FC253                                         ; sub_4FC200+3B␘j
BEGTEXT:004FC253                 movsx   edi, ax
BEGTEXT:004FC256                 mov     dh, [edi+esi]
BEGTEXT:004FC259                 cmp     dh, 5Fh
BEGTEXT:004FC25C                 jz      short loc_4FC268
BEGTEXT:004FC25E                 add     cl, dh
BEGTEXT:004FC260                 add     dl, dl
BEGTEXT:004FC262                 add     ch, cl
BEGTEXT:004FC264                 xor     dl, dh
BEGTEXT:004FC266                 jmp     short loc_4FC26A
BEGTEXT:004FC268 ; ---------------------------------------------------------------------------
BEGTEXT:004FC268
BEGTEXT:004FC268 loc_4FC268:                             ; CODE XREF: sub_4FC200+5C␘j
BEGTEXT:004FC268                 sub     dl, al
BEGTEXT:004FC26A
BEGTEXT:004FC26A loc_4FC26A:                             ; CODE XREF: sub_4FC200+66␘j
BEGTEXT:004FC26A                 mov     edi, [esp+18h+var_18]
BEGTEXT:004FC26D                 inc     eax
BEGTEXT:004FC26E                 cmp     ax, di
BEGTEXT:004FC271                 jl      short loc_4FC233
BEGTEXT:004FC273
BEGTEXT:004FC273 loc_4FC273:                             ; CODE XREF: sub_4FC200+31␘j
BEGTEXT:004FC273                 mov     eax, [esp+18h+var_18]
BEGTEXT:004FC276                 xor     ah, ah
BEGTEXT:004FC278                 and     al, 0Fh
BEGTEXT:004FC27A                 cwde
BEGTEXT:004FC27B                 mov     esi, eax
BEGTEXT:004FC27D                 mov     al, cl
BEGTEXT:004FC27F                 shl     esi, 1Ch
BEGTEXT:004FC282                 and     al, 0Fh
BEGTEXT:004FC284                 and     eax, 0FFh
BEGTEXT:004FC289                 shl     eax, 18h
BEGTEXT:004FC28C                 or      esi, eax
BEGTEXT:004FC28E                 xor     eax, eax
BEGTEXT:004FC290                 mov     al, ch
BEGTEXT:004FC292                 shl     eax, 10h
BEGTEXT:004FC295                 or      esi, eax
BEGTEXT:004FC297                 xor     eax, eax
BEGTEXT:004FC299                 mov     al, dl
BEGTEXT:004FC29B                 shl     eax, 8
BEGTEXT:004FC29E                 or      esi, eax
BEGTEXT:004FC2A0                 xor     eax, eax
BEGTEXT:004FC2A2                 mov     al, bl
BEGTEXT:004FC2A4                 or      eax, esi
BEGTEXT:004FC2A6                 add     esp, 4
BEGTEXT:004FC2A9                 pop     edi
BEGTEXT:004FC2AA                 pop     esi
BEGTEXT:004FC2AB                 pop     edx
BEGTEXT:004FC2AC                 pop     ecx
BEGTEXT:004FC2AD                 pop     ebx
BEGTEXT:004FC2AE                 retn
BEGTEXT:004FC2AE sub_4FC200      endp

bored

The hash routine probably does something really simple, but I haven’t as yet unpicked the assembly code to create a Python version. I’ll look at it later after drinking some beer which is required for this sort of thing. 🙂

You’re a better man than I. 🙇

Hmmn, maybe too much beer …

I’m struggling with this. The mix of 16-bit and 32-bit commands is annoying.

If anybody wants to help: if the input string pointed to at line 4fc208 is coltab+0x45, then the correct result appearing in register EAX after line 4fc2a4 is 7a67452a.

Here’s what I’ve got so far:

// … (prologue)

Int16b length = strlen(filename);
if(15 < length)
	length = 15

BEGTEXT:004FC208                 mov     esi, eax				; copy string pointer to ESI
BEGTEXT:004FC20A                 call    strlen_
BEGTEXT:004FC20F                 mov     [esp+18h+var_18], eax
BEGTEXT:004FC212                 cmp     ax, 0Fh
BEGTEXT:004FC216                 jle     short loc_4FC21F
BEGTEXT:004FC218                 mov     [esp+18h+var_18], 0Fh

Int32b bx = 0;
Int8b totalHash = 0;
Int16b hash = 0;
Int16b i = 0;

char currentCharacter = filename[0];
do {

BEGTEXT:004FC21F loc_4FC21F:
BEGTEXT:004FC21F                 mov     edi, [esp+18h+var_18]	; copy length to EDI
BEGTEXT:004FC222                 xor     bl, bl					; zero bits 0-8 of EBX
BEGTEXT:004FC224                 xor     dl, dl					; zero bits 0-8 of totalHash (EDX 0-8)
BEGTEXT:004FC226                 xor     ch, ch					; zero bits 8-16 of hash (ECX)
BEGTEXT:004FC228                 xor     eax, eax				; zero i (EAX)
BEGTEXT:004FC22A                 mov     dh, [esi]				; copy first character to bits 8-16 of EDX
BEGTEXT:004FC22C                 xor     cl, cl					; zero bits 0-8 of hash (ECX)
BEGTEXT:004FC22E                 test    di, di					; if bits 0-16 of length (EDI) are below or equal to zero, …
BEGTEXT:004FC231                 jle     short loc_4FC273		; … then skip the loop and ???

	if((0 != i) && ('_' != currentCharacter)) {

		BEGTEXT:004FC233
		BEGTEXT:004FC233 loc_4FC233:
		BEGTEXT:004FC233                 test    ax, ax				; if i (EAX) …
		BEGTEXT:004FC236                 jz      short loc_4FC253	; … equals zero, then jump
		BEGTEXT:004FC238                 cmp     dh, 5Fh			; if currentCharacter (EDX 8-16) compares to 0x5F (ASCII code for underscore) …
		BEGTEXT:004FC23B                 jz      short loc_4FC253	; … equally, then jump

		bx = (bx << 1) & 0xFF;
		if(currentCharacter < filename[i]) {
			bx = bx | 1;
		}

		BEGTEXT:004FC23D                 movsx   edi, ax			; copy i (EAX) to EDI
		BEGTEXT:004FC240                 add     bl, bl				; multiply bits 0-8 of EBX by two
		BEGTEXT:004FC242                 cmp     dh, [edi+esi]		; currentCharacter (EDX 8-16), ESI points to the string and EDI holds i. If bits 8-16 of EDX compare to the current character …
		BEGTEXT:004FC245                 setb    dh					; … lower, then set bits 8-16 of EDX to 1.
		BEGTEXT:004FC248                 movzx   edi, dh			; copy EDX bits 8-16 to EDI
		BEGTEXT:004FC24B                 and     ebx, 0FFh			; bitwise AND of EBX and 255
		BEGTEXT:004FC251                 or      ebx, edi			; bitwise OR of EBX and EDX, result in EBX

	} // if

	BEGTEXT:004FC253 loc_4FC253:

	currentCharacter = filename[i];

	BEGTEXT:004FC253                 movsx   edi, ax			; copy i (EAX) to EDI
	BEGTEXT:004FC256                 mov     dh, [edi+esi]		; copy filename[i] to currentCharacter (EDX 8-16)

	if('_' != currentCharacter) {

		BEGTEXT:004FC259                 cmp     dh, 5Fh			; if currentCharacter (EDX 8-16) compares to 0x5F (ASCII code for underscore) …
		BEGTEXT:004FC25C                 jz      short loc_4FC268	; … equally, then jump

		BEGTEXT:004FC25E                 add     cl, dh				; add currentCharacter (EDX 8-16) to bits 0-8 of hash (ECX)
		BEGTEXT:004FC260                 add     dl, dl				; multiply bits 0-8 of EDX by two
		BEGTEXT:004FC262                 add     ch, cl				; add bits 0-8 of hash (ECX) to bits 8-16 of hash (ECX)
		BEGTEXT:004FC264                 xor     dl, dh				; bitwise XOR of EDX bits 0-8 with EDX bits 8-16
		BEGTEXT:004FC266                 jmp     short loc_4FC26A	; jump

	}

	BEGTEXT:004FC268 loc_4FC268:
	BEGTEXT:004FC268                 sub     dl, al				;

} while(++i < length);

BEGTEXT:004FC26A loc_4FC26A:
BEGTEXT:004FC26A                 mov     edi, [esp+18h+var_18]; copy length to EDI
BEGTEXT:004FC26D                 inc     eax				; increment i (EAX)
BEGTEXT:004FC26E                 cmp     ax, di				; if i (EAX) compares to length (EDI) …
BEGTEXT:004FC271                 jl      short loc_4FC233	; … lower, then proceed with the loop

BEGTEXT:004FC273 loc_4FC273:                             ; CODE XREF: sub_4FC200+31␘j
BEGTEXT:004FC273                 mov     eax, [esp+18h+var_18]
BEGTEXT:004FC276                 xor     ah, ah
BEGTEXT:004FC278                 and     al, 0Fh
BEGTEXT:004FC27A                 cwde
BEGTEXT:004FC27B                 mov     esi, eax
BEGTEXT:004FC27D                 mov     al, cl
BEGTEXT:004FC27F                 shl     esi, 1Ch
BEGTEXT:004FC282                 and     al, 0Fh
BEGTEXT:004FC284                 and     eax, 0FFh
BEGTEXT:004FC289                 shl     eax, 18h
BEGTEXT:004FC28C                 or      esi, eax
BEGTEXT:004FC28E                 xor     eax, eax
BEGTEXT:004FC290                 mov     al, ch
BEGTEXT:004FC292                 shl     eax, 10h
BEGTEXT:004FC295                 or      esi, eax
BEGTEXT:004FC297                 xor     eax, eax
BEGTEXT:004FC299                 mov     al, dl
BEGTEXT:004FC29B                 shl     eax, 8
BEGTEXT:004FC29E                 or      esi, eax
BEGTEXT:004FC2A0                 xor     eax, eax
BEGTEXT:004FC2A2                 mov     al, bl
BEGTEXT:004FC2A4                 or      eax, esi

// … (epilogue)

Obviously, the loop has been unrolled for its first iteration … a reverse compiler should handle that. Symbolic variable names are still *much* easier to analyze than this hell of 8-, 16- and 32-bit values.

Wow, I didn’t expect an answer …

Here’s what my decompiler says about it … Personally, I usually have better luck with the assembly.

int __usercall sub_4FC200<eax>(unsigned __int8 *a1<eax>)
{
  unsigned __int8 *v1; // esi@1
  int v2; // eax@3
  unsigned __int8 v3; // dl@3
  unsigned __int8 v4; // dh@3
  __int16 v5; // cx@3
  unsigned __int8 v6; // bl@3
  int v8; // eax@1
  int v9; // [sp+0h] [bp-18h]@1

  v1 = a1;
  v8 = strlen_();
  v9 = v8;
  if ( (_WORD)v8 > 15 )
    v9 = 15;
  v6 = 0;
  v3 = 0;
  v2 = 0;
  v4 = *v1;
  v5 = 0;
  if ( (_WORD)v9 > 0 )
  {
    do
    {
      if ( (_WORD)v2 )
      {
        if ( v4 != 95 )
          v6 = v4 < v1[(signed __int16)v2] | 2 * v6;
      }
      v4 = v1[(signed __int16)v2];
      if ( v4 == 95 )
      {
        v3 -= v2;
      }
      else
      {
        LOBYTE(v5) = v4 + (_BYTE)v5;
        HIBYTE(v5) += v5;
        v3 = v4 ^ (unsigned __int8)(2 * v3);
      }
      ++v2;
    }
    while ( (_WORD)v2 < (_WORD)v9 );
  }
  return (v3 << 8) | (HIBYTE(v5) << 16) | ((v5 & 0xF) << 24) | ((v9 & 0xF) << 28) | v6;
}

Thanks. Here’s what I’ve got so far:

// … (prologue)
Int16b length = strlen(filename);
if(15 < length)
	length = 15

BEGTEXT:004FC208                 mov     esi, eax				; copy string pointer to ESI
BEGTEXT:004FC20A                 call    strlen_
BEGTEXT:004FC20F                 mov     [esp+18h+var_18], eax
BEGTEXT:004FC212                 cmp     ax, 0Fh
BEGTEXT:004FC216                 jle     short loc_4FC21F
BEGTEXT:004FC218                 mov     [esp+18h+var_18], 0Fh

Int32b bitmap = 0;		// EBX
Int8b totalHash = 0;	// EDX bits 0-8
Int16b hash = 0;		// ECX bits 0-16
Int16b i = 0;			// EAX

char currentCharacter = filename[0];
do {

BEGTEXT:004FC21F loc_4FC21F:
BEGTEXT:004FC21F                 mov     edi, [esp+18h+var_18]	; copy length to EDI
BEGTEXT:004FC222                 xor     bl, bl					; zero bits 0-8 of bitmap (EBX)
BEGTEXT:004FC224                 xor     dl, dl					; zero bits 0-8 of totalHash (EDX 0-8)
BEGTEXT:004FC226                 xor     ch, ch					; zero bits 8-16 of hash (ECX)
BEGTEXT:004FC228                 xor     eax, eax				; zero i (EAX)
BEGTEXT:004FC22A                 mov     dh, [esi]				; copy first character to bits 8-16 of EDX
BEGTEXT:004FC22C                 xor     cl, cl					; zero bits 0-8 of hash (ECX)
BEGTEXT:004FC22E                 test    di, di					; if bits 0-16 of length (EDI) are below or equal to zero, …
BEGTEXT:004FC231                 jle     short loc_4FC273		;  …then skip the loop and ???

	if((0 != i) && ('_' != currentCharacter)) {

		BEGTEXT:004FC233
		BEGTEXT:004FC233 loc_4FC233:
		BEGTEXT:004FC233                 test    ax, ax				; if i (EAX) …
		BEGTEXT:004FC236                 jz      short loc_4FC253	; … equals zero, then jump
		BEGTEXT:004FC238                 cmp     dh, 5Fh			; if currentCharacter (EDX 8-16) compares to 0x5F (ASCII code for underscore) …
		BEGTEXT:004FC23B                 jz      short loc_4FC253	; … equally, then jump

		bitmap = (bitmap << 1) & 0xFF;
		if(currentCharacter < filename[i]) {
			bitmap = bitmap | 1;
		}

		BEGTEXT:004FC23D                 movsx   edi, ax			; copy i (EAX) to EDI
		BEGTEXT:004FC240                 add     bl, bl				; multiply bits 0-8 of bitmap (EBX) by two
		BEGTEXT:004FC242                 cmp     dh, [edi+esi]		; currentCharacter (EDX 8-16), ESI points to the string and EDI holds i. If bits 8-16 of EDX compare to the current character …
		BEGTEXT:004FC245                 setb    dh					; … lower, then set bits 8-16 of EDX to 1.
		BEGTEXT:004FC248                 movzx   edi, dh			; copy EDX bits 8-16 to EDI
		BEGTEXT:004FC24B                 and     ebx, 0FFh			; bitwise AND of bitmap (EBX) and 255
		BEGTEXT:004FC251                 or      ebx, edi			; bitwise OR of bitmap (EBX) and EDX

	} // if

	BEGTEXT:004FC253 loc_4FC253:

	currentCharacter = filename[i];

	BEGTEXT:004FC253                 movsx   edi, ax			; copy i (EAX) to EDI
	BEGTEXT:004FC256                 mov     dh, [edi+esi]		; copy filename[i] to currentCharacter (EDX 8-16)

	if('_' != currentCharacter) {

	BEGTEXT:004FC259                 cmp     dh, 5Fh			; if currentCharacter (EDX 8-16) compares to 0x5F (ASCII code for underscore) …
	BEGTEXT:004FC25C                 jz      short loc_4FC268	; … equally, then jump

		hash = hash + currentCharacter;
		totalHash = totalHash << 1;
		hash = hash + (hash << 8);
		totalHash = totalHash ^ currentCharacter;

		BEGTEXT:004FC25E                 add     cl, dh				; add currentCharacter (EDX 8-16) to bits 0-8 of hash (ECX)
		BEGTEXT:004FC260                 add     dl, dl				; multiply bits 0-8 of totalHash (EDX 0-8) by two
		BEGTEXT:004FC262                 add     ch, cl				; add bits 0-8 of hash (ECX) to bits 8-16 of hash (ECX)
		BEGTEXT:004FC264                 xor     dl, dh				; bitwise XOR of totalHash (EDX 0-8) with EDX bits 8-16
		BEGTEXT:004FC266                 jmp     short loc_4FC26A	; jump

	}

	totalHash = totalHash - i;

	BEGTEXT:004FC268 loc_4FC268:
	BEGTEXT:004FC268                 sub     dl, al				; subtract i (EAX) from totalHash (EDX 0-8)

} while(++i < length);

BEGTEXT:004FC26A loc_4FC26A:
BEGTEXT:004FC26A                 mov     edi, [esp+18h+var_18]; copy length to EDI
BEGTEXT:004FC26D                 inc     eax				; increment i (EAX)
BEGTEXT:004FC26E                 cmp     ax, di				; if i (EAX) compares to length (EDI) …
BEGTEXT:004FC271                 jl      short loc_4FC233	; … lower, then proceed with the loop

BEGTEXT:004FC273 loc_4FC273:                             ; CODE XREF: sub_4FC200+31␘j
BEGTEXT:004FC273                 mov     eax, [esp+18h+var_18]
BEGTEXT:004FC276                 xor     ah, ah
BEGTEXT:004FC278                 and     al, 0Fh
BEGTEXT:004FC27A                 cwde
BEGTEXT:004FC27B                 mov     esi, eax
BEGTEXT:004FC27D                 mov     al, cl
BEGTEXT:004FC27F                 shl     esi, 1Ch
BEGTEXT:004FC282                 and     al, 0Fh
BEGTEXT:004FC284                 and     eax, 0FFh
BEGTEXT:004FC289                 shl     eax, 18h
BEGTEXT:004FC28C                 or      esi, eax
BEGTEXT:004FC28E                 xor     eax, eax
BEGTEXT:004FC290                 mov     al, ch
BEGTEXT:004FC292                 shl     eax, 10h
BEGTEXT:004FC295                 or      esi, eax
BEGTEXT:004FC297                 xor     eax, eax
BEGTEXT:004FC299                 mov     al, dl
BEGTEXT:004FC29B                 shl     eax, 8
BEGTEXT:004FC29E                 or      esi, eax
BEGTEXT:004FC2A0                 xor     eax, eax
BEGTEXT:004FC2A2                 mov     al, bl
BEGTEXT:004FC2A4                 or      eax, esi

// … (epilogue)

I’ll compare it to the decompiler output tomorrow. Do need sleep badly 🙁

That last block from 4fc273 is fairly strightforward but I’m just not in the mood to go through the loops today. Maybe tomorrow …

I was quite close to the decompiler’s output … here’s what I got after

1. replacing abstract names with metaphoricals
2. removing copies (e.g. v5, v8 and v9 are the same variables)
3. removing redundant type casts (e.g. by knowing length will never exceed 15, so casting it to __int8 is not necessary
4. resolving HIBYTE and LOWBYTE access:

int hashFromLowerCaseFilename(char const * filename) {

	int		length		= strlen(filename);
	unsigned char	bitmap		= 0;
	int		i		= 0;

	unsigned __int8 v3 = 0; // dl@3
	__int16 v5 = 0; // cx@3

	if(length > 15) {
		length = 15;
	}
	if(0 < length) {
		unsigned char character = filename[0];
		do {
			if(0 < i && '_' != character) {
				bitmap = bitmap << 1;
				if(character < filename[i]) {
					bitmap = bitmap | 1;
				}
			}
			character = filename[i];
			if('_' == character) {
				v3 = v3 - i;
			} else {
				unsigned char v5low = v5 & 0xFF;
				signed char v5high = (v5 >> 8) & 0xFF;
				v5low = character + v5low;
				v5high = v5high + v5;
				v5 = (__int16)v5high | v5low;
				v3 = character ^ (unsigned __int8)(2 * v3);
			}
		} while(++i < length);
	}
	return (length << 28) | ((v5 & 0xF) << 24) | ((v5 & 0xFF00) << 8) | (v3 << 8) | bitmap;
}

I can’t test it, though, because I don’t have any actual combinations of EF2000 filenames and hashes here. I don’t know if Python supports bit shifting, so you might have to replace

x << 8
y >> 16

by

x * 256
y / 65536

and so on. & is bitwise AND, | is bitwise OR and ^ is bitwise XOR; char is an 8-bit integer or an ASCII character (C doesn’t differentiate here).

Thanks, although I’m still having problems. Might only be one character out though …

Here’s a list of a couple of input strings and the correct output:

coltab(0x45)       → 7a67452a (coltab.dat)
check(0x81)        → 6f6e1313 (check.cd)
(0x3f)views2(0x45) → 847f2549 (views\views2.dat)
(0x4a)(0x32)(0x82) → 3ec4ce01 (3\2.3)
(0x4a)2m1_tc(0x82) → 83bbec15 (3\2m1_tc.3)

Yes, it doesn’t work here either. There’s definitely problems with the underscore (it’s counted, but it shouldn’t) and with the bitmap (some hashes come out one too low) – I’ve marked my wrong results:

I’ll keep you informed if I get it fixed.

Edit: Made my characters unsigned; bitmap is working now. Underscore remaining.

Note that last one is 2m1_tc not 2m1_tc2 …

You are correct; this is so emberassing 😳

It’s all working now. Here’s my code:

int sub_4FC200(
	UInt1B const * filename
) {
	UInt1B v3(0);
	SInt2B v5(0);
	UInt1B bitmap(0);

	auto length(::strlen((char const *)filename));
	if(15 < length) {
		length = 15;
	}

	auto character = filename[0];
	if(0 < length) {
		unsigned int i(0);
		do {
			if((0 < i) && (character != '_')) {
				bitmap <<= 1;
				if(character < filename[i]) {
					bitmap |= 1;
				}
			}
			character = filename[i];
			if('_' == character) {
				v3 = UInt1B(v3 - i);
			} else {
				v5 = (v5 & 0xFF00) | (character + v5) & 0xFF;
				v5 = (v5 & 0xFF) | ((((v5 >> 8) & 0xFF) + v5) << 8);
				v3 = character ^ UInt1B(2 * v3);
			}
		}
		while(++i < length);
	}
	return (v3 << 8) | ((v5 & 0xFF00) << 8) | ((v5 & 0xF) << 24) | ((length & 0xF) << 28) | bitmap;
}

And here are the hashes of the texture files I asked you for:

831d7d7b	citycr
8d11417b	citycl
824f5d75	block2
7f0be329	tmsea
97e8ebd1	groundc
a3a40da5	groundm2
8adfed45	trees2
5ae8650d	run
801fe55b	martin
9ba3f1eb	mount3p
8181ff75	mounti
9aa0f5eb	mount2p
7834d53b	mount
89114f75	mount1
9708cbd3	grounds
91fcf7d3	groundm
6ef5dd1d	hill
7ae27923	wheel
aca35985	cliffcn2
af22cbb7	glaciers
a916f7b5	glacierm
a50effb5	glacieri
af02ebb5	glacierc
a4a60fa5	groundm3
a81191ab	grasstop
ae654911	xrndsmid
ad816113	xrndslit
a56a7315	xrndsdrk
8b95cb75	mounts
888fcd75	mountp
9df4ffd3	groundi
ad31610b	xrndclit
a51a730d	xrndcdrk
8b75eb75	mountc
8c97c361	cliffc
809e0151	gcliff
7401af29	bang7
7ff7a529	bang2
81481b45	tmmark
7ae7113d	cityl
77e11b3d	cityi
70f32d3d	cityr
71d50f3d	cityc
73de9333	burbs
8c925f65	stripe
a856a9ab	coclconr
7fb7c137	coast
97c97ba5	valleye
af92e5b5	coastcon
9a22ddc3	cliffcn
78108723	trees
6532f311	tree
83fff96b	cldtst
7cc6b93d	cloud
71f52f3d	citys
8f22d555	ffyord
637d0319	bud3
814d5b75	block1
af55894b	smoketst
afc19d95	hulltex4

@Bored,

I know this is late and probably OBE, but here’s a 7zip of my unmodded SEF2000 folder:
http://www.mediafire.com/?nrziaqztolrp3lt

EDIT (OT): If you want s savegame for a simple campaign, then check out Beta 3 and activate the _Revert Pilot Level campaign series from the Launcher. Starting a Revert campaign with the cheat pilot should provide you with a simple enough campaign to do some analysis. Also, if you go through with losing the campaign with the cheat pilot, there is little enough in the file that the differences (score and level) between the baseline cheat pilot and the newly reverted pilot might help decode the pilot file as well.

HF, thanks but I’ve got SEF2000 working on a Vista32 machine. All I wanted to do was find the hash subroutine.

Krishty,
Here are the extracted textures:
http://www.box.net/shared/evyn33ed7korou3qch1i

There are three that I couldn’t find: hill, cityl and cityr.

Thank you very much 👍

Looks like we’ve found a new texture mode … or this texture was meant for TFX’s color palette. But similar artifacts appear on single polygons in explosion files, too …

Neat! They probably weren’t expecting you to be so close to that can of beer. Probably looks OK at 20000 feet …

Oh sh*t. I just realized why supershapes won’t load with EF2000: There’s four names in EF2000’s ssinfo.fn which are too long to process (nine characters):

• NVPAST0_C (1132)
• NVPAST0_E (1133)
• CLIF1C_NE (1158)
• KUZNETSOV (1292)

What should I do? The identifier system cannot handle more than eight characters (something like that never occurred in TAW); should I just ignore them? (Seems like you did the same with some of the extracted .3 files.)

Also: How about going through ssinfo.fn and hashing all names? Just like in TAW, this could bring up even more .3 files …

Hmmn, don’t know. I haven’t ignored anything. The list of known EF2000 filenames (until today) was produced by logging which files were requested by the game while playing it.
I notice that EF2000’s ssinfo.fn contains both kuznetsov.3 (not called) and kuzntsov.3 which appears in the 3 folder so must have been called.

It is my intention to go through through all sources of filenames to try and indentify the noname files. We’re gradually (or quickly in your case 🙂) putting the tools together to be able to do this.

One thing I noticed is that lambo and lambo2 don’t appear in ssinfo.fn … so where do they come from?
ssinfo.fn contains 1370 names, but we have over 2000 files in the 3 folder. As noted earlier, we have files named in ssinfo.fn that haven’t appeared in the 3 folder (like kuznetsov). Strange …

bored

One thing I noticed is that lambo and lambo2 don’t appear in ssinfo.fn … so where do they come from?

lambo.3 is the full model and lambo2.3 is only the body component of the model. diablo.3 (Lamborghini Diablo) is the most detailed model and, if you look on the rear bumper it carries the signature of Andy Bates. This suggests a possible R&D process. This question of where do they come from? is a bit elusive at the current time. The only answer I have to offer at this time would have to be … Italy.

But I’m still workin on it 😉

Umm, thanks DKD… 😕

I’ve had to to take a break for a few days, but hopefully will be back in action by next weekend … and still never got the EF2000 hash routine working exactly right in Python. 🙁 A situation. I think, where the strong types of C++ might be an advantage.

EDIT: TAW and SEF2000 hash routines working fine now. In TAW, the bitmap is limited to one byte which explains the anomoly I noted earlier.

Some notes on the TAW extraction package that I sent to the dropbox:

I’ve given up making a grand unified unpacker/packer for all types of did.dat for the time being, and have broken down the job into separate scripts in order that some progress can be made. Eventually, it could all go together into one tool.
Note this only works for TAW’s did.dat for now.

It is advised that the package is extracted to a clean directory. You then need to put your TAW did.dat file in the same directory.

First run get_index_list.py which reads the list of hashes, offsets and sizes of each file in did.dat and whether each file is compressed. The result is stored a text file taw_index_list.txt. This program only needs to be run once and if it fails it means that your TAW did.dat is different to mine …

Next comes match_files.py which takes a list of candidate file names from raw_names_list.txt and calculates their hashes and compares them with those in taw_index_list.txt. A new list is then produced, taw_extract_list.txt which is used to provide input for the actual extraction.

The raw_names_list.txt file that I compiled is based on Krus’s original list plus some recent suggestions. I’m certain that there are many files that we already know about that could go into this list, but I haven’t had time to sort through the current TAW2.0 directories.
The list as uploaded identifies 10007 of the 12588 files in did.dat.

taw_extract.py can then be run to extract the files into the TAW directory structure. First, the files are written on one pass, then decompressed as necessary in a second pass.
Since the decompressor is a separate program that I launch and have no control over, it can hang if there are multiple instances running. I’ve added some wait loops in my script to help with this.

For TAW it should be possible to repackage did.dat for TAW2.0 to quicken load times. It may not be necessary to recompress the files. We'll get a larger did.dat file, but we don’t have to include the noname files from the original did.dat.
It would be nice to have our own compressor though, and I’m sure one could be developed by studying the code from the current decompressor.

bored

and if it fails it means that your TAW did.dat is different to mine …

Okay so mine is probably bigger than yours, I think you should put it in the pubic box 😉

A few more to add to the list:

lev\arcade.lst = noname132
lev\arcade.lbm = noname133
lev\arcade.asc = noname135
lev\arcade.env = noname136
lev\arcade.trg = noname137
lev\arcade.wld = noname138

These are to accompany the seaworld map/level. I have only made minimal changes to the level since I extracted the complete number of additional files for implementation.

This is the D.I.D. planned implementation configuration:

WINDOW_TYPE PANE
WINDOW "WND_ARCADE"
USE_DID_PATH
PALETTE 	  "\iff\256\gui\brushmtl.iff"
DESKTOP 1024x768
3D_WIN 320x200
AT 0 0
DIM 800 562
BMPDIM 800 600
BACKGROUND "\iff\256\gui\brushmtl.iff"
GADGET "BMPREG_ARCADE_HISCORES"				 AT 389 27	 DIM 371 373 BMPFILE "\iff\256\gui\brushmtl.iff" END
GADGET "BTN_ARCADE_STRAFE_ATTACK"				 AT 194 290	 DIM 100 40 END
GADGET "BTN_ARCADE_TARGET_RACE"				 AT 64 290	 DIM 100 40 END
GADGET "BTN_ARCADE_SWEEP_CARPET"				 AT 194 240	 DIM 100 40 END
GADGET "BTN_ARCADE_SEEK_CARRIER"				 AT 64 240	 DIM 100 40 END
GADGET "BTN_ARCADE_TURKEY_SHOOT"				 AT 194 190	 DIM 100 40 END
GADGET "BTN_ARCADE_PARA_DROP"				 AT 64 190	 DIM 100 40 END
GADGET "GRP_ARCADE_ARCADE_OPTIONS"				 AT 19 10	 DIM 320 350 END
GADGET "BTN_ARCADE_COUNTER_TERRORIST"				 AT 64 40	 DIM 100 40 END
GADGET "BTN_ARCADE_DEFECTORS"				 AT 194 40	 DIM 100 40 END
GADGET "BTN_ARCADE_BILLS_LIMO"				 AT 64 90	 DIM 100 40 END
GADGET "BTN_ARCADE_DECAP"				 AT 194 90	 DIM 100 40 END
GADGET "BTN_ARCADE_SCUD_HUNT"				 AT 64 140	 DIM 100 40 END
GADGET "BTN_ARCADE_DOODLE_BUG"				 AT 194 140	 DIM 100 40 END
GADGET "GRP_ARCADE_GAME_SETTINGS"				 AT 18 368	 DIM 320 203 END
GADGET "GRP_ARCADE_HISCORE_TABLE"				 AT 384 9	 DIM 400 400 END
GADGET "VS_ARCADE_SCROLL"				 AT 763 27	 DIM 16 373 END
END_WINDOW
END_DESKTOP
END_3D

If I get some time I’ll work on reconstructing the second world environment again to see what we could do with this.
Do you have any of the arcade mission briefing files?

PS: You probably have this one already. I keep referring to noname8319 which is 3/hardware.ini

Well done. I was wondering what the seaworld island would be called. I tried lev\seaworld.env then gave up. 🙂

I have no idea what the .wld file is for, but there is also a lev\redsea.wld at noname1299.

bored

Well done. I was wondering what the seaworld island would be called. I tried lev\seaworld.env then gave up. 🙂

I have no idea what the .wld file is for, but there is also a lev\redsea.wld at noname1299.

I believe this is the full monty, .wld’s, 4ev’s, 4e2’s and all 😉

lev\redsea.dat
lev\redsea.lbm
lev\redsea.4ev
lev\redsea.4e2
lev\redsea.trg
lev\redsea.wld
lev\arcade.dat
lev\arcade.txt
lev\arcade.lst
lev\arcade.lbm
lev\arcade.4ev
lev\arcade.4e2
lev\arcade.asc
lev\arcade.env
lev\arcade.trg
lev\arcade.wld
lev\clds0600.dat
lev\clds0600.lst
lev\clds0600.lbm
lev\clds0600.4ev
lev\clds0600.4e2
lev\clds0600.asc
lev\clds0600.env
lev\clds0600.trg
lev\clds0600.wld
lev\clds0800.dat
lev\clds0800.lst
lev\clds0800.lbm
lev\clds0800.4ev
lev\clds0800.4e2
lev\clds0800.asc
lev\clds0800.env
lev\clds0800.trg
lev\clds0800.wld
lev\clds1000.dat
lev\clds1000.lbm
lev\clds1000.4ev
lev\clds1000.4e2
lev\clds1000.trg
lev\clds1000.wld
lev\clds1200.dat
lev\clds1200.lst
lev\clds1200.lbm
lev\clds1200.4ev
lev\clds1200.4e2
lev\clds1200.asc
lev\clds1200.env
lev\clds1200.trg
lev\clds1200.wld
lev\clds1400.dat
lev\clds1400.lst
lev\clds1400.lbm
lev\clds1400.4ev
lev\clds1400.4e2
lev\clds1400.asc
lev\clds1400.env
lev\clds1400.trg
lev\clds1400.wld
lev\clds1600.dat
lev\clds1600.lst
lev\clds1600.lbm
lev\clds1600.4ev
lev\clds1600.4e2
lev\clds1600.asc
lev\clds1600.env
lev\clds1600.trg
lev\clds1600.wld
lev\clds1800.dat
lev\clds1800.lst
lev\clds1800.lbm
lev\clds1800.4ev
lev\clds1800.4e2
lev\clds1800.asc
lev\clds1800.env
lev\clds1800.trg
lev\clds1800.wld
lev\clds2000.dat
lev\clds2000.lst
lev\clds2000.lbm
lev\clds2000.4ev
lev\clds2000.4e2
lev\clds2000.asc
lev\clds2000.env
lev\clds2000.trg
lev\clds2000.wld

👍

10756 found out of 12588

bored

👍

10756 found out of 12588

It’s embarrassing to say but, I have not been keeping a master list, I hope you have 😕

Question: I am wondering if there could be a method to extract the sub-files. For example, the 12kb AGP .raw files are not in the did.dat extracted group in the 12kb format.

Below is a sample from hardware.ini:

;201=exp1a
;202=exp1b
;203=exp1c

;341=alexp1a
;342=alexp1b
;343=alexp1c

341-343 are 16kb raws, 201-203 are the same files only without the (al prefix) alpha channel.

Considering the hash/extraction process, is there a way to extract the file, extract the alpha channel from it and then assign the name, directory and extension?

The list I have is basically Krusade’s original list together with what you, me and Krishty have added over the last few months. There may be some obvious stuff missing though.

Regarding file manipulation, anything is possible but those particular files don’t seem to exist, at least in the hardware.tm directory.

bored

The list I have is basically Krusade’s original list together with what you, me and Krishty have added over the last few months. There may be some obvious stuff missing though.

Regarding file manipulation, anything is possible but those particular files don’t seem to exist, at least in the hardware.tm directory.

Good, as long as you are adding to the one list, I’ll feed you what I have as I get new hits.
Those particular files are the AGP files and would be in the agp directory. I have not yet attempted to run them through the program because I’m fairly sure I have all that exist and none of the ones I have are 12kb, I had to create mine from the 16kb files. I’ll confirm that tonight when I get home.

The same thing is true for the AGP sky textures. The ones in the extracted set are 384kb, however, according to hardware.ini, the sky0000 and skynvg are loaded in slots not supporting 384kb textures (768 and 12kb-clamped, respectively). I can scale the 384’s to get the others, but I am curious as to how this transformation is supposed to happen natively, assuming the CPU or GPU is involved.

Just wanted to let you know: I’m still alive, and I’m still working on reverse engineering TFX 2 & 3.

The latest thing I’m on is implementing the virtual file system, i.e. did.dat.
I’ve got the hashing algorithms; I’ve got the RA decompression algorithm and I’m putting it all together for TAW (loading its did.dat takes about three seconds on a Core i7 and at first glance, all extraction seems to have succeeded).

However, EF2000’s did.dat behaves differently. Any tip on its structure and where the list of directories and file extensions comes from?

Good to hear from you!

The only did.dat structure that is understood 100% is that from TAW.
For TFX, the structure is known but we don’t have the hash algorithm.

For EF2000, the structure and hash algorithm differs between the european and US versions, at least for the european SEF2000, US SEF2000 & US 3dfx versions that I’ve looked at.
I can extract from all these, but it’s done by manually finding the start of each section. I’d need to dig out the numbers …

Coincidentally, I was looking at the RA decompressor source code and was contemplating making a Python version, but on second thoughts it’s probably best leaving it as C++ with a SWIG wrapper if I go down that road … but now I know you’re looking at this, I may not have to do anything. 🙂

By the way, have you looked at making a RA compressor? This is maybe not needed as I don’t think the files have to be compressed, but it would make a reconstituted did.dat much quicker to load.
If we want to mod EF2000, it would seem to be essential to repackage the dat file since it doesn’t run very well with the files extracted.

EDIT: What EF2000 did.dat file(s) do you have? If you give me the exact size, I can compare it with mine.

bored

For EF2000, the structure and hash algorithm differs between the european and US versions, at least for the european SEF2000, US SEF2000 & US 3dfx versions that I’ve looked at.

Good to know; I’m already trying to consider this in my design since I saw the hash algorithms differ between the US and UK versions, too.

bored

I can extract from all these, but it’s done by manually finding the start of each section. I’d need to dig out the numbers …

Oh, so I guess they’re hard-coded. I wouldn’t be surprised, since I could not find the file extension list either – so I guess it’s hard-coded, too.

bored

This is maybe not needed as I don’t think the files have to be compressed, but it would make a reconstituted did.dat much quicker to load.

Are you sure it would be quicker? I don’t see where a modern hard drive with a prefetching operating system would be significantly slower than a (in comparison) weak compression algorithm. In fact, I could load TAW’s did.dat an order of magnitude faster if not so many files were compressed.

bored

If we want to mod EF2000, it would seem to be essential to repackage the dat file since it doesn’t run very well with the files extracted.

Can’t we repackage them without compression?

I just don’t see the big advantage of a compressor which would justify the effort (which probably wouldn’t be too much, since RA seems to be some kind of LZ77 or Deflate, and existing encoders could be ported), considering that I’ve promised a new version of the viewer months ago.

bored

EDIT: What EF2000 did.dat file(s) do you have? If you give me the exact size, I can compare it with mine.

I’ve got HomeFries’ vanilla Super EF2000, with a did.dat size of 40,484,862 B.

Krishty

Can’t we repackage them without compression?

I guess we need to try this to see what happens.

The US SEF2000 and 3dfx did.dats are exactly the same size, although I haven’t tested them to see if the files are identical.
Anyway, the 12 byte index(hash+address+size) list starts at address 40385730 decimal.
Just before this in the file comes the path and extension information, which apart from the first byte in each entry is encrypted by flipping bits 7,2 & 0 in each byte. I only have the encrypted list to hand:

09e8ecf6f6eceaebf6d9
09f6f0f5e0f7e2f0ecd9
08e6eae9eaf0f7f6d9
08f6e4e8f5e9e0f6d9
08ece3e3d9b7b0b3d9
07f2e4fce7ecebd9
06f3ece0f2f6d9
06e8f0f6ece6d9
05e4ebece8d9
04e2f0ecd9
04ece3e3d9
04f6f5f7d9
04e6e3e2d9
04e9e0f3d9
04e1e4f1d9
03f6f6d9
03f1e8d9
02b6d9
04abf5e4f7
04abe3e4e1
04abe2f0ec
04abe6e3e2
04abe1e4f1
04abf1fdf1
04abf1e3fd
04abb7b0b3
04abe7f7e3
04abf6e4f3
04abebe0b4
04abe1eae6
04abe1e7eb
04abe8e1f6
04abe4ebe1
04abe6eae9
04abf5e4e9
04abe6e9f1
04abe6e9f0
04abe7eceb
04abf5e9f1
04abede1e1
04abedf0e1
04abe3e9f1
04abe9e7e8
04abe0e3b7
04abe7e8f5
04abe4e7e8
04abe7e7e8
04abf5e8ec
04abf5e8e8
04abeaf5f1
04abf1f7e2
04abe4f6e6
04abe9e0b7
04abe9e0f3
04abf6f1e4
04abb1e0b7
04abb1e0f3
04abf6f1f5
04abe0ebf3
04abe8e4f5
04abf2e9e1
04abf5f5f5
04abf6f1e8
04abe8ecf6
04abe8ece1
04abfde8ec
04abffecf5
04abf6e4e8
04abf2ebe1
04abf6e4e3
04abf6f5f7
04abf5eaf6
04abf6f6e1
04abe9e4e7
04abf2e4fc
03abf1e8
03abe3eb
03abe6e1
03abe4e1
02abb6

… but the last entry will be .3

There are 12 bytes at the start of the file that probably point to these sections, but are also encryped in some different way. I still need to work this out.

You have my datlib.py script. This contains the hash algorithm, which is the same as TAW’s apart from one byte. This file also has the unencrypted paths and extensions in a function called EFHash_US … or similar.

Thank you very much! It would have taken me a long time to find out about that bit flipping …

Here’s the Red Sea rendered with textures, shapes and supershapes from did.dat:

I’ve got the impression that rendering is actually about one third slower than before. So, either I did a mistake somewhere, or my extracted files differ slightly from did.dat’s (that already happened when my planes didn’t have LoD), or 3View can now load more files than before. Interesting anyway.

Great!

… and if I get some time I’ll produce a set of scripts for filename hunting in EF2000’s hash list, similar to what I did for TAW last week.

EDIT: I’ve uploaded a set of crap scripts which should help with extraction, filename hunting etc for both EF2000 and TAW. The TAW scripts are a slightly rationalised version of those uploaded previously.

Krishty

Just wanted to let you know: I’m still alive, and I’m still working on reverse engineering TFX 2 & 3.

Good. I know you started with an interest in the TAW radio messages so I thought this might be of some interest to you.

In the TAW Samples directory, the speech files include twenty-two, each, of .idx and .bin files.
There is a third counterpart set which is not extracted in the regular game, the .in files. Using bored’s hash extractor I found the .in files below. They seem to be responsible for the text, which accompanies the speech, in the AWACS Message Window in AWACS mode:

samples\speech\digits.in
samples\speech\planes.in
samples\speech\angels.in
samples\speech\oclock.in
samples\speech\flevel.in
samples\speech\direct.in
samples\speech\format.in
samples\speech\wingpos.in
samples\speech\letters.in
samples\speech\airtower.in
samples\speech\compass.in
samples\speech\plusmin.in
samples\speech\airbase.in
samples\speech\callsign.in
samples\speech\missions.in
samples\speech\nmeweaps.in
samples\speech\climdesc.in
samples\speech\messages.in
samples\speech\hml.in
samples\speech\speed.in
samples\speech\alleg.in
samples\speech\teams.in

Haven’t had the time to really analyze them in-game, but perhaps they will be of some benefit in the future.

Krishty

Are you sure it would be quicker? I don’t see where a modern hard drive with a prefetching operating system would be significantly slower than a (in comparison) weak compression algorithm. In fact, I could load TAW’s did.dat an order of magnitude faster if not so many files were compressed.

I’ve hit a bit of a problem investigating this, since the following sequence does not work:

1. Extract and decompress 10752 files from TAW’s original did.dat which we have names for.
2. Repack these files uncompressed into a new did.dat. This results in a file about 107MB in size.
3. Try to run the original game with the new file.

This results in the process f22.dat just hanging without any error message. 🙁

bored

The only did.dat structure that is understood 100% is that from TAW.

I withdraw this rather rash statement.

The program was hanging since it does some sort of sanity check on did.dat using bytes 4 to 7. If this failed, an error saying something like Incorrect dat file for this version of the game should be raised according to the debugger but the program just hangs.

I can get past this point, but now I get a CTD:

Program fault at 0xF8830845, cleanup up and exiting!
000CF324 000CF308 F8830845 00210202
FFFFF92C 1484C0DA 00000000 00539D11 0053A514 0053A4C8
0000002B 0000002B 00000053 0000002B 00000023 0000002B

This is not good, particularly since 0xF8830845 is not a valid address.

bored,

I remember you saying, you wanted to go through all resources and collect their filenames in order to identify the unknown files in did.dat.

Since I have a neat virtual file system now, I can gather all failed first-chance requests (files that are not found in the extracted directory):

3\ball_3.3
3\ball_2.3
3\ball_4.3
3\ball_1.3
3\ball_5.3
3\f22.3
3\wakem_1.3
3\rlrldt_1.3
3\nlarbs1c.3
3\lerx.3
3\video_10.3
3\tbigfact.3
3\tbgfa180.3
3\tbigf_90.3
3\tbigfa90.3
3\su331.3
3\desal_3.3
3\desvil_1.3
3\iantest.3
3\fxagm65h.3
3\fxrocket.3
3\yaim_9s.3
3\ymk82.3
3\xmav_ir.3
3\zmav_ir.3
3\eri_cont.3
3\eth_cont.3
3\brdm2.3
3\misfix.3
ss\frtw_1ra.ssd
ss\frtw_2ra.ssd
ss\multil_4.ssd
ss\multir_4.ssd
ss\rvrvab_1.ssd
ss\ctrest_4.ssd
ss\rdrddn_1.ssd
ss\rddune_3.ssd
ss\rdrdab_1.ssd
ss\rdfort_3.ssd
ss\rdfort_4.ssd
ss\rdrdfo_1.ssd
ss\rdfore_2.ssd
ss\rlrldt_1.ssd
ss\rldste_2.ssd
ss\rldunt_1.ssd
ss\rldunt_2.ssd
ss\rldunt_3.ssd
ss\rldunt_4.ssd
ss\rlrldn_1.ssd
ss\rldune_1.ssd
ss\rldune_3.ssd
ss\rldune_4.ssd
ss\rlarbt_1.ssd
ss\rlarbt_3.ssd
ss\rlarbt_4.ssd
ss\rlrlab_1.ssd
ss\rlarbe_1.ssd
ss\rlarbe_4.ssd
ss\rlfors_1.ssd
ss\rlfors_2.ssd
ss\rlforc_1.ssd
ss\rlforc_2.ssd
ss\rlforc_3.ssd
ss\rlforc_4.ssd
ss\rlfort_1.ssd
ss\rlfort_2.ssd
ss\rlfort_3.ssd
ss\rlfort_4.ssd
ss\rlrlfo_1.ssd
ss\rlfore_1.ssd
ss\rlfore_2.ssd
ss\rlfore_3.ssd
ss\rlfore_4.ssd
ss\rlforn_1.ssd
ss\rlforn_2.ssd
ss\rlforn_3.ssd
ss\rlforn_4.ssd
ss\pldstt_1.ssd
ss\pldstt_2.ssd
ss\pldstt_4.ssd
ss\plpldt_1.ssd
ss\pldste_1.ssd
ss\pldunc_2.ssd
ss\pldunc_4.ssd
ss\pldunt_1.ssd
ss\pldunt_2.ssd
ss\pldunt_3.ssd
ss\pldunt_4.ssd
ss\plpldn_1.ssd
ss\pldune_1.ssd
ss\pldune_2.ssd
ss\pldune_3.ssd
ss\rdrldt_1.ssd
ss\rlrddt_1.ssd
ss\rdrlab_1.ssd
ss\rlrdab_1.ssd
ss\rlrvfo_1.ssd
ss\rvrlfo_1.ssd
ss\rvplfo_1.ssd
ss\rdrlfo_1.ssd
ss\rlrdfo_1.ssd
ss\rdplfo_1.ssd
ss\rlplfo_1.ssd
ss\rdrldn_1.ssd
ss\rlrddn_1.ssd
ss\plrddn_1.ssd
ss\rlpldn_1.ssd
ss\plrldn_1.ssd
ss\rlfors_1.ssd
ss\rlfors_1.ssd
ss\rlfors_2.ssd
ss\rlfors_2.ssd
ss\rldunj_1.ssd
ss\rldunj_3.ssd
ss\rlforj_1.ssd
ss\rlforj_2.ssd
ss\rlforj_3.ssd
ss\rlforj_4.ssd
ss\rldstn_1.ssd
ss\rldstn_3.ssd
ss\rldstn_4.ssd
ss\desvil_1.ssd
ss\twn_cst1.ssd
ss\desal_3.ssd
ss\duninds3.ssd
ss\duninds4.ssd
ss\dunsups1.ssd
ss\dunsups2.ssd
ss\dunsups3.ssd
ss\dunsups4.ssd
ss\desewrs3.ssd
ss\dunewrs2.ssd
ss\dunewrs3.ssd
ss\dunewrs4.ssd
ss\dunarms2.ssd
ss\dunarms3.ssd
ss\dunarms4.ssd
ss\descoms3.ssd
ss\duncoms3.ssd
ss\duncoms4.ssd
ss\dunrefs1.ssd
ss\dunrefs3.ssd
ss\dunrefs4.ssd
ss\pliest_3.ssd
ss\pldstn_3.ssd
ss\dcrlct_1.ssd
ss\dcrlct_2.ssd
ss\cit1arl1.ssd
ss\arbsups3.ssd
ss\arbarms4.ssd

Many of these should be present in the did.dat. There may be false positives, though – the RA decompressor stopped working late in the extraction process.

I hope when I update my supershape implementation soon, I will find every single .ssd and .3 file ever referenced in TAW’s terrain. I’ll keep you up to date.

Thanks!

That list resulted in 123 newly identified files.
The score is now 10908 0f 12588.

Well, I don’t know. This reconstructed did.dat does work with the original game:
https://rapidshare.com/files/1006987826/did.zip

The only difference I can see between this and earlier attempts is that there is one file (m401.3) that is left compressed by mistake.

… and Krishty is right, it loads very very fast. 🙂

Now, I need to look at the TAW2.20 files. For this I need to change the path and extension lists since the folder red2000 for example, does not appear in the original game.

Thank you for the file!

It is not only very fast – somehow, the HUD fonts have changed and now dgVoodoo rendering performance is drastically improved (must have almost doubled, I guess) …

Also, my 3View debug build now loads in less than 0.5 seconds – it was more than four before. This makes debugging significantly more comfortable!

That’s good to hear. At least this is not a completely wasted effort then.

I still don’t know why I get a crash under certain circumstances, but my theory is that TAW’s exception handling is rather basic …

I’ve now developed the packing script to take a nominal set of files and produce a folder and extension list on the fly. I’m not convinced the game runs any faster though … but at least it runs. 🙂

Here’s a did.dat produced from the TAW2.20 set which unpacks to about 123MB.

https://rapidshare.com/files/3981852657/did.zip

bored

I’ve now developed the packing script to take a nominal set of files and produce a folder and extension list on the fly. I’m not convinced the game runs any faster though … but at least it runs. 🙂

Here’s a did.dat produced from the TAW2.20 set which unpacks to about 123MB.

https://rapidshare.com/files/3981852657/did.zip

without trying to sound like a groupie … OMG OMG OMG 😁 🙇

seriously, this is really awesome stuff 🙂

haven’t tried this yet but i will pretty soon i think! having the loading speeds of the original game would be a very nice bonus!

i'm not sure if implementing mods would be as easy though? does the game require all files to be in the did.dat? any possibility of having some files in a .dat and some in folders?

i'm guessing if a file is in the did.dat, it will not even look for it in the main folder?

now, if it does find the did.dat - will it still try to load files it doesn’t find in the did.dat from the main game folder?

I believe it checks the dat file first, then if it can’t find a file, it will then try the folders.

Only just found this out though. I tried packing all the directories (that were packed in the original game) of the latest TAW2.20 beta8.
The packing went OK, but the game claims it can’t find fonts/TOPAZPRO/10.fnt. I have packed the file, but as fonts/topazpro/10.fnt. So, the DiD virtual file system must be case sensitive since I don’t believe Windows cares.

Anyway, I put the fonts directory back and the game then worked fine.

There must be another problem: TOPAZPRO is not part of the file name, it’s part of the directory name. These are hard-coded in the .dat file, and they are always lowercase in the original did.dat. So, if the search for the directory name was case-sensitive, the game couldn’t the file with the original .dat either, because it looks for the uppercase version there, too. Are you sure you have re-encoded this directory name correctly, i.e. all lowercase, with backslashes, and without trailing whitespace?

Also, did you check if the file has gotten the hash 0x46802801? That’s where I find it in the original did.dat.

I do hope it’s something like this, because if the virtual file system really was case-sensitive, I’d have to rewrite lots of code 🙁

Yes, the directory name is as follows in did.dat, together with a leading 0x0f giving its length:
fonts\topazpro\

There’s over 11000 files in the archive which are found without any problems, and the few that are complained about have some upper case letters in the directory or file name.

Since I now have directory names that didn’t exist in the orininal game, there is now a new list, which in turn means that my hardcoded lookup table for the leading (and trailing for the file extension) characters is no longer valid for the hash function.
Thus, the hash function now needs not only the path, name and extension but the path and extension list as well, which is read from the dat file. While, this seems to be the way it is meant to be used, obtaining a hash takes somewhat longer now.

@Eagle,

Unfortunately, the did.dat overrides any existing files/folders, as opposed to files/folders overwriting what is present in did.dat. Kind of a back-asswards way of doing it, esp. from a modding/patching standpoint. However, since this was before the time that people wanted flight sims to be moddable (Quake II was just starting the modding frenzy), this may have been there to protect the content.

What this means is that we can recompile a did.dat, but lose all moddability. This includes custom tails, skins, and UI as well as the obvious things like sounds, tiles, and objects. Additionally, TAWBC integration, though doable, will require a rewrite of the launcher integration routine.

If we could figure a way to make mods work on top of the existing did.dat, we would really be on to something.

bored

Since I now have directory names that didn’t exist in the orininal game, there is now a new list, which in turn means that my hardcoded lookup table for the leading (and trailing for the file extension) characters is no longer valid for the hash function.
Thus, the hash function now needs not only the path, name and extension but the path and extension list as well, which is read from the dat file. While, this seems to be the way it is meant to be used, obtaining a hash takes somewhat longer now.

I’ve already implemented it that way; so, does the TAW 2.20 did.dat you’ve uploaded show these problems? If you gave me a list of those few unrecognized files, I could try and see if my implementation loads correctly …

Update: Reproducable with 3View and the did.dat you’ve just uploaded. My implementation is reporting an unrecognizable directory; I’m investigating.

Update II: Your did.dat is faulty – your entries are not all lower-case. Here’s the invalid directories:

fonts\TOPAZPRO\
pcx\OPTIONS\
fonts\MICRO\
MUSIC\
UDL\
PDL\
MDL\

The file extension list, on the other hand, looks O.K.

I’ll implement rejection of DAT files with uppercase letters in their directories since TAW obviously cannot handle them either.

Also, I’ve got a ton of other problems with the .dat invalid indices in .lst files, bogus .ssd names, invalid terrain tile indices and so on … but I don’t know if it’s a did.dat problem. If I just use the shapes and texture maps with the original terrain data, it works fine:

My bad – I have not yet implemented .asc file interpretation, that’s why the .lst’s wouldn’t be accepted.

Krishty

Your did.dat is faulty – your entries are not all lower-case. Here’s the invalid directories:

fonts\TOPAZPRO\
pcx\OPTIONS\
fonts\MICRO\
MUSIC\
UDL\
PDL\
MDL\

The file extension list, on the other hand, looks O.K.

I’ll implement rejection of DAT files with uppercase letters in their directories since TAW obviously cannot handle them either.

Ah yes, sorry. The did.dat file that I checked yesterday wasn’t the right one.
I’ll make sure that the lists in the dat file are lowercase … but I’m struggling to see exactly what is going wrong. I convert everything to lowercase before doing the hash operation, which is what I believe that TAW does as well.
Also, doesn’t the original did.dat have uppercase entries? I’ll check later when I get back to my hex editor …

Krishty

My bad – I have not yet implemented .ASC file interpretation, that’s why the .lst’s wouldn’t be accepted.

I think it’s only the number in the first line of the .asc file that must match one of the parameters in the .lst file. The rest of the file appears not to be used at all at runtime and can be deleted.
This file is interesting though, as it contains information about the tile edges that could be used when checking the integrity of a map.
Like this entry:

q:\projects\tfx3\sh\TARGETS\STATIC\TOWN\DESERT\ROAD\1\
0 3 21 3 3 40 3 3 21 3

The numbers can be arranged so the tile is split into 9 parts with each number giving some attribute for each part, something like this:

3 21 3
3 40 3
3 21 3

Where 3 = desert, 21 = desert road, 40 = town

HomeFries

If we could figure a way to make mods work on top of the existing did.dat, we would really be on to something.

You could have a set of base files in the did.dat, then leave the relatively few files that are selected depending on Launcher settings outside in their directories.

The system appears to be flexible regarding what can be put inside the .dat file.

bored

I’ll make sure that the lists in the dat file are lowercase … but I’m struggling to see exactly what is going wrong. I convert everything to lowercase before doing the hash operation, which is what I believe that TAW does as well.
Also, doesn’t the original did.dat have uppercase entries? I’ll check later when I get back to my hex editor …

The original entries are all lowercase. When looking up a file path, it is entirely converted to lowercase characters by f22.dat. After having identified the directory and the extension parts (i.e. finding the last backslash and the last dot), these are looked up in the .dat file. Since the entries are sorted by length, all entries with another length are skipped. Then, all entries with the same length are compared byte-wise (i.e. case-sensitive, which is effectively case-insensitive because the .dat entries as well as the filename are both lowercase) until the correct entry is found (and an error is raised on no matches). Then, TAW uses the indices of the lookup (plus 0x30/0x41) to build the shortened path (up to 15 characters) and, eventually, compute the hash which is looked up in the .dat list via binary searching.

When you are writing uppercase characters into the .dat, the byte-wise comparison will fail and the according directory is never found.

bored

I think it’s only the number in the first line of the .asc file that must match one of the parameters in the .lst file. The rest of the file appears not to be used at all at runtime and can be deleted.

Yes; I’ve quickly implemented it yesterday in order to get TAW 2.0 running, as seen in the snapshot thread 🙂

bored

This file is interesting though, as it contains information about the tile edges that could be used when checking the integrity of a map.
Like this entry:

q:\projects\tfx3\sh\TARGETS\STATIC\TOWN\DESERT\ROAD\1\
0 3 21 3 3 40 3 3 21 3

The numbers can be arranged so the tile is split into 9 parts with each number giving some attribute for each part, something like this:

3 21 3
3 40 3
3 21 3

Where 3 = desert, 21 = desert road, 40 = town

This is interesting indeed. I always felt like this file belonged to source codes, e.g. as input to DID’s shape compiler. It seemed very strange to me to keep this file in a released version just for the sake of one number at the start …

Krishty

Yes; I’ve quickly implemented it yesterday in order to get TAW 2.0 running, as seen in the snapshot thread 🙂
This is interesting indeed. I always felt like this file belonged to source codes, e.g. as input to DID’s shape compiler. It seemed very strange to me to keep this file in a released version just for the sake of one number at the start …

It might be important to consider that there are multiple .asc files with EF2000 and they were necessary and used extensively there to produce the maps (which I believe included nodal attrition). With TAW, only the first line is used with the current format, however, the files we have include all the files for the map legacy format and the entire file would be necessary for that implementation.

DrKevDog

It might be important to consider that there are multiple .asc files with EF2000 and they were necessary and used extensively there to produce the maps (which I believe included nodal attrition). With TAW, only the first line is used with the current format, however, the files we have include all the files for the map legacy format and the entire file would be necessary for that implementation.

Interesting theory DKD. 😉

As far as I can see, EF2000’s .asc is similar to TAW’s … but as we can have rotated tiles, there can be up to 4 variants of the same tile:

n:\3dshapes\tfx2\sh\enviro\mount3\s\
0 5 4 4 5 4 4 5 4 4
n:\3dshapes\tfx2\sh\enviro\mount3\s\
16384 4 4 4 4 4 4 5 5 5
n:\3dshapes\tfx2\sh\enviro\mount3\s\
-32768 4 4 5 4 4 5 4 4 5
n:\3dshapes\tfx2\sh\enviro\mount3\s\
-16384 5 5 5 4 4 4 4 4 4

Here is a new and improved TAW did.dat … now with extra lowercase letters:
https://rapidshare.com/files/1006987826/did.zip

I have some idea what bytes 8 to 11 of TAW’s did.dat might do now. In the code of f22.dat there is this sub which appears to decide which hash routine to use based on result:

int __usercall sub_57B354<eax>(int result<eax>)
{
  int v1; // edx@4
  int (__usercall *v2)<eax>(unsigned __int8 *<eax>); // ecx@4

  if ( result == 1 )
  {
    v2 = sub_57B2B8;
    v1 = result;
    dword_6ED7FC = 36;
    goto LABEL_5;
  }
  if ( result == 2 )
  {
    v2 = sub_57B2B8;
    v1 = result;
    dword_6ED7FC = 52;
    goto LABEL_5;
  }
  if ( result != 3 )
  {
    v2 = sub_57B200;
    v1 = 0;
LABEL_5:
    off_6ED7F8 = (int (__fastcall *)(_DWORD, _DWORD))v2;
    dword_6ED7F4 = v1;
    return result;
  }
  dword_6ED7FC = 83;
  off_6ED7F8 = (int (__fastcall *)(_DWORD, _DWORD))sub_57B2B8;
  dword_6ED7F4 = result;
  return result;
}

In TAW’s did.dat bytes 8 to 11, considered as a 32 bit integer have a value of 3. So, if this corresponds to result, we set dword_6ED7FC = 83 and perform sub_57B2B8 which calculates the hash.

For the US version of EF2000’s did.dat, it was noticed that the hash function is the same but a value of 36 is used instead of 83 which corresponds to result=1 in the code. Thus we can expect bytes 8 through 11 to have a value 00000001.

Further analysis of the EF2000’s did.dat shows it to have exactly the same structure as TAW’s but with the pointers and path/extension list scrambled using bit flipping. However, since we know what these values should be, the flipped bits reveal themselves:

|-----------------32 bit--------------|
                    |------16 bit-----|
                              |--char-|
00XX 0X0X X000 000X 00X0 X0XX X000 0X0X

where X represents the flipped bits.

That is impressive!

If I am interpreting this correctly, it looks like the ADF did.dat has a value of 00000002 and therefore a value of 52 is used instead of 83 or 36.

That is my theory as well, but I haven’t looked at ADF’s did.dat yet, and there may be no reason to do so now.

I’m not sure I understand that code sequence above, but I interpret it as sub_57b200 will be run if 'result' is not 1,2 or 3 which takes us to the hash routine encountered earlier in the euro version of EF2000. Interesting that it is still present in the TAW code.

Anyway, in the short term this could be a way to differentiate did.dat files from the different games.

OK, The US EF2000 did.dat works as expected. The game runs with a rebuilt dat file, but I’m still missing a lot of filenames so can’t do so much. The rebuilt uncompressed file is only about 30MB, while the original dat file is about 40MB …

I still need a bunch of the TAW filenames, so if you have any please share. Here are the new ones I have, although the list is random and disorganized 😁 :

lev\redsea.dat = noname1292
lev\redsea.lbm = noname1294
lev\redsea.4ev = noname1295
lev\redsea.4e2 = noname1306
lev\redsea.trg = noname1298
lev\redsea.wld = noname1299
lev\arcade.dat = noname130
lev\arcade.txt = noname131
lev\arcade.lst = noname132
lev\arcade.lbm = noname133
lev\arcade.4ev = noname134
lev\arcade.4e2 = noname1104
lev\arcade.asc = noname135
lev\arcade.env = noname136
lev\arcade.trg = noname137
lev\arcade.wld = noname138
lev\clds0600.dat = noname8741
lev\clds0600.lst = noname8742
lev\clds0600.lbm = noname8743
lev\clds0600.4ev = noname8744
lev\clds0600.4e2 = noname10620
lev\clds0600.asc = noname8745
lev\clds0600.env = noname8746
lev\clds0600.trg = noname8747
lev\clds0600.wld = noname8748
lev\clds0800.dat = noname9024
lev\clds0800.lst = noname9025
lev\clds0800.lbm = noname9026
lev\clds0800.4ev = noname9027
lev\clds0800.4e2 = noname10662
lev\clds0800.asc = noname9028
lev\clds0800.env = noname9029
lev\clds0800.trg = noname9030
lev\clds0800.wld = noname9031
lev\clds1000.dat = noname8677
lev\clds1000.lbm = noname8679
lev\clds1000.4ev = noname8680
lev\clds1000.4e2 = noname8952
lev\clds1000.trg = noname8683
lev\clds1000.wld = noname8684
lev\clds1200.dat = noname8981
lev\clds1200.lst = noname8982
lev\clds1200.lbm = noname8983
lev\clds1200.4ev = noname8984
lev\clds1200.4e2 = noname8972
lev\clds1200.asc = noname8985
lev\clds1200.env = noname8986
lev\clds1200.trg = noname8987
lev\clds1200.wld = noname8988
lev\clds1400.dat = noname9287
lev\clds1400.lst = noname9288
lev\clds1400.lbm = noname9289
lev\clds1400.4ev = noname9290
lev\clds1400.4e2 = noname9013
lev\clds1400.asc = noname9291
lev\clds1400.env = noname9292
lev\clds1400.trg = noname9293
lev\clds1400.wld = noname9294
lev\clds1600.dat = noname9587
lev\clds1600.lst = noname9588
lev\clds1600.lbm = noname9589
lev\clds1600.4ev = noname9590
lev\clds1600.4e2 = noname9059
lev\clds1600.asc = noname9591
lev\clds1600.env = noname9592
lev\clds1600.trg = noname9593
lev\clds1600.wld = noname9594
lev\clds1800.dat = noname9925
lev\clds1800.lst = noname9926
lev\clds1800.lbm = noname9927
lev\clds1800.4ev = noname9928
lev\clds1800.4e2 = noname9107
lev\clds1800.asc = noname9929
lev\clds1800.env = noname9930
lev\clds1800.trg = noname9931
lev\clds1800.wld = noname9932
lev\clds2000.dat = noname9540
lev\clds2000.lst = noname9541
lev\clds2000.lbm = noname9542
lev\clds2000.4ev = noname9543
lev\clds2000.4e2 = noname10739
lev\clds2000.asc = noname9544
lev\clds2000.env = noname9545
lev\clds2000.trg = noname9546
lev\clds2000.wld = noname9547
pcx\scenario\arcade\arc01.pcx = noname11673
pcx\scenario\arcade\arc02.pcx = noname11674
pcx\scenario\arcade\arc03.pcx = noname11675
pcx\scenario\arcade\arc04.pcx = noname11676
pcx\scenario\arcade\arc05.pcx = noname11677
pcx\scenario\arcade\arc06.pcx = noname11678
pcx\scenario\arcade\arc07.pcx = noname11679
pcx\scenario\arcade\arc08.pcx = noname11680
pcx\scenario\arcade\arc10.pcx = noname11688
pcx\scenario\arcade\arc01.pcx = noname11673
pcx\scenario\arcade\arc02.pcx = noname11674
pcx\scenario\arcade\arc03.pcx = noname11675
pcx\scenario\arcade\arc04.pcx = noname11676
pcx\scenario\arcade\arc05.pcx = noname11677
pcx\scenario\arcade\arc06.pcx = noname11678
pcx\scenario\arcade\arc07.pcx = noname11679
pcx\scenario\arcade\arc08.pcx = noname11680
pcx\scenario\arcade\arc10.pcx = noname11688
pcx\scenario\scenario\arcade\arc01.pcx = noname11787
pcx\scenario\scenario\arcade\arc02.pcx = noname11788
pcx\scenario\scenario\arcade\arc03.pcx = noname11789
pcx\scenario\scenario\arcade\arc04.pcx = noname11790
pcx\scenario\scenario\arcade\arc09.pcx = noname11795
pcx\scenario\scenario\arcade\arc10.pcx = noname11797

hardware.tm\tex_45.tm = noname463
hardware.tm\tex_47.tm = noname464
hardware.tm\tex_49.tm = noname466
hardware.tm\tex_52.tm = noname468

colours\atc.col = noname10934
colours\tree.col = noname11167
mp_col\acmi.col = noname11524
colours\acmi.col = noname11109
colours\acmi2.col = noname12445
colours\att_pat.col = noname5651
colours\hei_pal.col = noname1885
colours\drone.col = noname12258
colours\fuzzpal.col = noname3559
colours\att_pat2.col = noname8614
colours\drone2.col = noname1112
mp_col\att_pat2.col = noname9848
mp_col\drone2.col = noname850
lev\rednvg.txt = noname230
lev\rednvg2.txt = noname4141

3\misfix.3 = noname1623

3\PLARBS_1.3 = noname10519
3\PLARBS_2.3 = noname10521
3\PLARBC_1.3 = noname10186
3\PLARBC_2.3 = noname10187
3\PLARBC_3.3 = noname10188
3\PLARBC_4.3 = noname10189
3\PLARBT_1.3 = noname10546
3\PLARBT_2.3 = noname10547
3\PLARBT_3.3 = noname10548
3\PLARBT_4.3 = noname10549
3\PLPLAB_1.3 = noname8749
3\PLARBE_1.3 = noname10234
3\PLARBE_2.3 = noname10235
3\PLARBE_3.3 = noname10236
3\PLARBE_4.3 = noname10237
3\PLFORS_1.3 = noname9982
3\PLFORS_2.3 = noname9983
3\PLFORC_1.3 = noname9580
3\PLFORC_2.3 = noname9582
3\PLFORC_3.3 = noname9583
3\PLFORC_4.3 = noname9584
3\PLFORT_1.3 = noname9997
3\PLFORT_2.3 = noname9999
3\PLFORT_3.3 = noname10002
3\PLFORT_4.3 = noname10004
3\PLPLFO_1.3 = noname8035
3\PLFORE_1.3 = noname9633
3\PLFORE_2.3 = noname9634
3\PLFORE_3.3 = noname9637
ss\frtw_1ra.ssd = noname7665
ss\frtw_2ra.ssd = noname9517
ss\multil_4.ssd = noname9911
ss\multir_4.ssd = noname10064
ss\rvrvab_1.ssd = noname10252
ss\ctrest_4.ssd = noname10530
ss\rdrddn_1.ssd = noname9506
ss\rddune_3.ssd = noname8513
ss\rdrdab_1.ssd = noname10540
ss\rdfort_3.ssd = noname8052
ss\rdfort_4.ssd = noname8053
ss\rdrdfo_1.ssd = noname9873
ss\rdfore_2.ssd = noname7743
ss\rlrldt_1.ssd = noname9933
ss\rldste_2.ssd = noname7599
ss\rldunt_1.ssd = noname8832
ss\rldunt_2.ssd = noname8833
ss\rldunt_3.ssd = noname8834
ss\rldunt_4.ssd = noname8835
ss\rlrldn_1.ssd = noname9785
ss\rldune_1.ssd = noname8460
ss\rldune_3.ssd = noname8463
ss\rldune_4.ssd = noname8464
ss\rlarbt_1.ssd = noname8591
ss\rlarbt_3.ssd = noname8596
ss\rlarbt_4.ssd = noname8597
ss\rlrlab_1.ssd = noname10704
ss\rlarbe_1.ssd = noname8255
ss\rlarbe_4.ssd = noname8258
ss\rlfors_1.ssd = noname7981
ss\rlfors_2.ssd = noname7982
ss\rlforc_1.ssd = noname7661
ss\rlforc_2.ssd = noname7662
ss\rlforc_3.ssd = noname7663
ss\rlforc_4.ssd = noname7664
ss\rlfort_1.ssd = noname7996
ss\rlfort_2.ssd = noname7997
ss\rlfort_3.ssd = noname7998
ss\rlfort_4.ssd = noname7999
ss\rlrlfo_1.ssd = noname10166
ss\rlfore_1.ssd = noname7702
ss\rlfore_2.ssd = noname7703
ss\rlfore_3.ssd = noname7704
ss\rlfore_4.ssd = noname7705
ss\rlforn_1.ssd = noname7881
ss\rlforn_2.ssd = noname7882
ss\rlforn_3.ssd = noname7883
ss\rlforn_4.ssd = noname7884
ss\pldstt_1.ssd = noname9434
ss\pldstt_2.ssd = noname9435
ss\pldstt_4.ssd = noname9437
ss\plpldt_1.ssd = noname7455
ss\pldste_1.ssd = noname9134
ss\pldunc_2.ssd = noname9952
ss\pldunc_4.ssd = noname9954
ss\pldunt_1.ssd = noname10360
ss\pldunt_2.ssd = noname10361
ss\pldunt_3.ssd = noname10362
ss\pldunt_4.ssd = noname10363
ss\plpldn_1.ssd = noname10699
ss\pldune_1.ssd = noname10003
ss\pldune_2.ssd = noname10005
ss\pldune_3.ssd = noname10006
ss\rdrldt_1.ssd = noname9956
ss\rlrddt_1.ssd = noname9627
ss\rdrlab_1.ssd = noname10738
ss\rlrdab_1.ssd = noname10502
ss\rlrvfo_1.ssd = noname8768
ss\rvrlfo_1.ssd = noname7496
ss\rvplfo_1.ssd = noname10400
ss\rdrlfo_1.ssd = noname10195
ss\rlrdfo_1.ssd = noname9839
ss\rdplfo_1.ssd = noname9575
ss\rlplfo_1.ssd = noname9549
ss\rdrldn_1.ssd = noname9810
ss\rlrddn_1.ssd = noname9496
ss\plrddn_1.ssd = noname7577
ss\rlpldn_1.ssd = noname9219
ss\plrldn_1.ssd = noname7838
ss\rlfors_1.ssd = noname7981
ss\rlfors_1.ssd = noname7981
ss\rlfors_2.ssd = noname7982
ss\rlfors_2.ssd = noname7982
ss\rldunj_1.ssd = noname8580
ss\rldunj_3.ssd = noname8582
ss\rlforj_1.ssd = noname7799
ss\rlforj_2.ssd = noname7802
ss\rlforj_3.ssd = noname7804
ss\rlforj_4.ssd = noname7805
ss\rldstn_1.ssd = noname7787
ss\rldstn_3.ssd = noname7789
ss\rldstn_4.ssd = noname7790
ss\desvil_1.ssd = noname10257
ss\twn_cst1.ssd = noname10503
ss\desal_3.ssd = noname4322
ss\duninds3.ssd = noname7608
ss\duninds4.ssd = noname7610
ss\dunsups1.ssd = noname8167
ss\dunsups2.ssd = noname8168
ss\dunsups3.ssd = noname8169
ss\dunsups4.ssd = noname8170
ss\desewrs3.ssd = noname10437
ss\dunewrs2.ssd = noname9865
ss\dunewrs3.ssd = noname9866
ss\dunewrs4.ssd = noname9867
ss\dunarms2.ssd = noname8689
ss\dunarms3.ssd = noname8690
ss\dunarms4.ssd = noname8692
ss\descoms3.ssd = noname10560
ss\duncoms3.ssd = noname9990
ss\duncoms4.ssd = noname9991
ss\dunrefs1.ssd = noname7503
ss\dunrefs3.ssd = noname7506
ss\dunrefs4.ssd = noname7508
ss\pliest_3.ssd = noname8503
ss\pldstn_3.ssd = noname9317
ss\dcrlct_1.ssd = noname7951
ss\dcrlct_2.ssd = noname7953
ss\cit1arl1.ssd = noname7676
ss\arbsups3.ssd = noname9252
ss\arbarms4.ssd = noname9763

3\descrpc1.3 = noname8131
3\descrpc2.3 = noname8132
3\descrpc3.3 = noname8133
3\descrpc4.3 = noname8134
3\descrps1.3 = noname8521
3\descrps2.3 = noname8522
3\descropx.3 = noname10049
3\descrpt1.3 = noname8534
3\descrpt2.3 = noname8535
3\descrpt3.3 = noname8536
3\descrpt4.3 = noname8537
3\descrpe1.3 = noname8200
3\descrpe2.3 = noname8201
3\descrpe3.3 = noname8202
3\descrpe4.3 = noname8204
ss\descrpc1.ssd = noname8171
ss\descrpc2.ssd = noname8172
ss\descrpc3.ssd = noname8173
ss\descrpc4.ssd = noname8174
ss\descrps1.ssd = noname8528
ss\descrps2.ssd = noname8530
ss\descropx.ssd = noname10066
ss\descrpt1.ssd = noname8553
ss\descrpt2.ssd = noname8554
ss\descrpt3.ssd = noname8555
ss\descrpt4.ssd = noname8556
ss\descrpe1.ssd = noname8220
ss\descrpe2.ssd = noname8222
ss\descrpe3.ssd = noname8223
ss\descrpe4.ssd = noname8224

3\rlrldt_1.3 = noname10365
3\nlarbs1c.3 = noname10743
3\desal_3.3 = noname5898
3\desvil_1.3 = noname10625

samples\speech\digits.in = noname127
samples\speech\planes.in = noname735
samples\speech\angels.in = noname743
samples\speech\oclock.in = noname817
samples\speech\flevel.in = noname1030
samples\speech\direct.in = noname1178
samples\speech\format.in = noname1284
samples\speech\wingpos.in = noname1938
samples\speech\letters.in = noname3703
samples\speech\airtower.in = noname8149
samples\speech\compass.in = noname6289
samples\speech\plusmin.in = noname7103
samples\speech\airbase.in = noname6010
samples\speech\callsign.in = noname8330
samples\speech\missions.in = noname8600
samples\speech\nmeweaps.in = noname9141
samples\speech\climdesc.in = noname9244
samples\speech\messages.in = noname9921
samples\speech\hml.in = noname10969
samples\speech\speed.in = noname11851
samples\speech\alleg.in = noname12308
samples\speech\teams.in = noname12507

ss\zaim_9x.ssd = noname6257
ss\zaim120c.ssd = noname8370
ss\zalarm.ssd = noname263
ss\zfab500r.ssd = noname8275
ss\zmk82r.ssd = noname97
ss\zmk82f.ssd = noname96
ss\zfuelo.ssd = noname1017
ss\zgbu24.ssd = noname988

3\zagm65g.3 = noname2641
3\zrbk500.3 = noname3195

ss\fzjsow.ssd = noname855
ss\fzjsowas.ssd = noname9145
ss\fzjsowat.ssd = noname9146
ss\zaim_9x.ssd = noname6257
ss\zagm88.ssd = noname1
ss\zaim120c.ssd = noname8370
ss\zalarm.ssd = noname263
ss\zfab500r.ssd = noname8275
ss\zmk82r.ssd = noname97
ss\zmk82f.ssd = noname96
ss\zfuelo.ssd = noname1017
ss\zgbu24.ssd = noname988
ss\zjsow.ssd = noname11976
ss\zagm65g.ssd = noname1972
ss\zpod.ssd = noname11018
ss\zdecoy.ssd = noname256
ss\zs5_rock.ssd = noname9562
ss\zb8w.ssd = noname11019
ss\zub32_57.ssd = noname7653
ss\zagm84a.ssd = noname2431
ss\zlau68.ssd = noname296
ss\zs_eagle.ssd = noname9778
ss\zmk83f.ssd = noname106
ss\zjdam.ssd = noname11706
ss\gen.ssd = noname10837
ss\t.ssd = noname10764
ss\test.ssd = noname11272

ss\fra_oils.ssd = noname8991
ss\zagm86n.ssd = noname2498
ss\fzbomb.ssd = noname285
ss\fzjsowap.ssd = noname9143
ss\fzjsowar.ssd = noname9144
ss\fzlau68.ssd = noname5573
ss\gen2.ssd = noname11051
ss\grdmiss.ssd = noname3556
ss\missis.ssd = noname1330
ss\rolmiss.ssd = noname5338
ss\harpoon.ssd = noname5278
ss\patriot.ssd = noname3323
ss\rapmiss.ssd = noname3104
ss\gndtoair.ssd = noname7959
ss\sabot.ssd = noname12387
ss\yaim_9s.ssd = noname5452
ss\ymk82.ssd = noname12494
ss\dt8.ssd = noname10902
ss\dt2.ssd = noname10896
ss\smb2.ssd = noname11263
ss\dt6.ssd = noname10900
ss\smb3.ssd = noname11264
ss\smb4.ssd = noname11265
ss\smb5.ssd = noname11266
ss\smb6.ssd = noname11267

ss\smb7.ssd = noname11268
ss\smb8.ssd = noname11269
ss\dt4.ssd = noname10898
ss\trnsmk2.ssd = noname4077
ss\trnsmk10.ssd = noname7652
ss\trnsmk4.ssd = noname4079
ss\trnsmk6.ssd = noname4081
ss\trnsmk8.ssd = noname4084
ss\neil.ssd = noname11509
ss\andtest.ssd = noname6241
ss\starter.ssd = noname3875
ss\decoal.ssd = noname1605
ss\wpnlift.ssd = noname7356
ss\osarop.ssd = noname1129
ss\temp.ssd = noname11257
ss\setup.ssd = noname12310
ss\andyg.ssd = noname12472
ss\arbinds1.ssd = noname8726
ss\duninds3.ssd = noname7608
ss\misfix1.ssd = noname6170

agp\sky0800.raw = noname5025
agp\sky0600.raw = noname4529

hardware.tm\cam4_21.tm = noname4809
cgdat\djbtimap.sh = noname8967
cgdat\egyptmap.sh = noname7809
cgdat\ertremap.sh = noname10305
cgdat\ethpamap.sh = noname9812
cgdat\saudimap.sh = noname10295
cgdat\smliamap.sh = noname10421
cgdat\sudanmap.sh = noname10371
cgdat\yemenmap.sh = noname9297
3\djbtimap.3 = noname8720
3\egyptmap.3 = noname7562
3\ertremap.3 = noname10019
3\ethpamap.3 = noname9537
3\saudimap.3 = noname10007
3\smliamap.3 = noname10198
3\sudanmap.3 = noname10116
3\yemenmap.3 = noname9050

colours\nvgpal.col = noname264

ss\iantest.ssd = noname2962
ss\test.ssd = noname11272
3\mig21.3 = noname12441

ss\smb2.ssd = noname11263
ss\smb3.ssd = noname11264
ss\smb4.ssd = noname11265
ss\smb5.ssd = noname11266
ss\smb6.ssd = noname11267
ss\smb7.ssd = noname11268
ss\smb8.ssd = noname11269

ss\ssinfo.opt = noname1626
ss\andyg.ssd = noname12472

🍻

Wow!

I haven’t looked at any of this stuff since my last post on 29th Dec, so have forgotten what to do …

Will try and get back to it in the next few days.

I notice my match_files.py script gives an incorrect score since it doesn’t handle where a filename is repeated in raw_names_list.txt

The real score is:
Known: 11042
Unknown: 1546

Probably, the best strategy to categorise what’s left, starting with the .3 files of which there are 272.

270 now

3\ab.3 = noname10809
3\da.3 = noname10811

Now 256 unknown .3 files. I can try every combination by brute force for the shorter filenames. This tactic soon gets out of hand though.

3\bab.3 = noname10911
3\bar.3 = noname10912
3\bri.3 = noname10928
3\bun.3 = noname10930
3\dav.3 = noname10927
3\dot.3 = noname10937
3\fb2.3 = noname10936
3\fe3.3 = noname10943
3\fu2.3 = noname10960
3\pow.3 = noname10858
3\taz.3 = noname10868
3\tem.3 = noname10873
3\tkc.3 = noname10878
3\tmp.3 = noname10879

… and sometimes different combinations of characters will lead to the same hash, so we have to guess which one is right. In the following I guess noname11071 is really bab2.3??

3\a102.3 = noname11209
3\bab2.3 = noname11071
3\lp9_.3 = noname11071
3\ls_5.3 = noname11071
3\n_p5.3 = noname11071
3\nder.3 = noname11518
3\nkha.3 = noname11389

3\neilc.3 = noname12449
3\tvil2.3 = noname12359
3\tvil4.3 = noname12360
3\il76x.3 = noname11695
3\b_tmp.3 = noname11637
3\zcrv7.3 = noname11781
3\tsu27.3 = noname11672
3\st_mf.3 = noname11897
3\st_rf.3 = noname11921
3\st_mt.3 = noname11898
3\c7471.3 = noname11783
3\st_ff.3 = noname11848
3\tmpbr.3 = noname11697
3\nk290.3 = noname11635
3\icarr.3 = noname11660
3\lig01.3 = noname11769
3\lig02.3 = noname11770
3\lig03.3 = noname11771
3\lig04.3 = noname11772
3\st_lt.3 = noname11894
3\st_rt.3 = noname11925
3\nk180.3 = noname12140
3\st_lw.3 = noname11895
3\st_rw.3 = noname11926

Amazing! Your statement, sometimes different combinations of characters will lead to the same hash, so we have to guess which one is right. In the following I guess noname11071 is really bab2.3?? Is true, however, I have not had much of a problem with sorting out the correct names.

The problem gets worse with the longer file names with maybe 10 hits per hash for the 5 character names above, although the real name becomes easier to spot.

I’ve started a run for 6 character names, but each time I add a character the program takes 37 times longer to run, so this is going to take a couple of days.

I know the character script takes longer but it is the most probable for the longer filenames. Some previously unrecognized naming convention patterns have already started to surface 🙂

two more:

3\st_mf1.3 = noname1704
3\st_mf2.3 = noname1705

You might as well leave the 6 character filenames to my script, it should finish tomorrow.
I think that’s about as far as we can go since 7 characters will take over a month, 8 characters 3 years etc.

There are much smarter ways to do this, but I don’t have the development time. 🙁

Okay, that’s great 👍. I’ll start work on the 7 and 8 characters, the old fashioned way:

It may be better to look at the longest filenames first. I’ll make a list of the unknown .3 files against name length to help.

Something else I can do is rewrite the program in C++, which is what it was designed for anyway. This may give enough of a speed boost to bring the 7 character filenames into range.

If I get time, I’ll see if there is any easy way to do this using CUDA on the graphics card. That’ll probably be way over my head though.

Perhaps that could accelerate the process, sounds like your C++ is coming along, wish I could say the same for my Python. I suspect you could use the CUDA python wrapper if that C++ seems too burdensome. The only thing over your head is a bunch of hair:

I’ve been meaning to ask you … have you seen a doctor lately 😁

DrKevDog

I’ve been meaning to ask you … have you seen a doctor lately 😁

Yeah, cost a fortune in plastic surgery getting that look …

The 6 character script has finished. There are 45 hashes in this category, but the script found 4246 matches. I need to sort them into noname order before trying to find which ones to discard, so there’ll be a slight delay.

There are 59 7-character and 124 8-character filenames left to find. Thankfully, there are no .3 files with longer names.

… and there only 10 files of any sort with filenames >8 chars, and there is something weird with those since at least some of them appear to have been compressed multiple times.

Very helpful information, now it’s getting interesting (as if the current crop of new filenames, in and of themselves, aren’t already interesting to explore and analyze 😁).

I could use your help sorting through the results of the 6 character run.
There are 45 files in this category and for 40 of them, it’s been quite obvious what the real filename is, although sometimes I need to use 3View to check my guess.
For the other 5, no filename is jumping out at me from the list of possibles.

I’ve uploaded sorted8_redux.txt to boxnet so that you can have a go …

A C++ version of the tool is running on the 7-character names and it’s about 20 times faster. It’ll still take about 3 days to plough through them though.

Be glad to help if I can, I’ll get right on it 🙂

noname1143 = flbg90: Fuel Bag
noname1221 = stotmp: Storage Bldg with 2 Temple mounts
noname1003 = fbhawk: Blackhawk
noname1630 = lndp90: Pipeline
noname1198 = jedt90: Pavillion Bldg.

Thanks!

That was quick! 👍

I should have spotted 'fbhawk' …

Sometimes, too much plastic surgery can effect not only the way you look but also the way you look … at the things you see 😁

Well, you can laugh now … but when the 7-character script finishes (currently on lap 9 of 37), I’ll just associate the hits with each of the 59 nonameXX and leave you to it.
It looks like there will be hundreds of permutations which lead to the same hash, even though we are only checking the 59 hashes that are associated with .3 files.

I will gladly add these to my list. Hundreds of permutations … could be a challenge though 😉

Thanks!

Great work with the remaining file names! Sorry I couldn’t help much in the last months.

I’m still implementing did.dat identification & loading for the different versions of TFX games. Maybe now that I’m home ill, I’ll finally have time to complete this …

Good to hear from you!

I’m currently running a brute force approach to the 7-character .3 filenames, this will take 3 days in a single thread on my PC.

Any ideas on how to handle the final run of 8-character filenames?
I’ve been reading up on CUDA/OpenCL but don’t understand much so far …

Like Bored said, it’s good to hear from you. Glad you’re home. The 8-character filenames are going to be a real challenge but we’re getting there 😉

I guess the real problem with the eight-character run won’t be computing power but all the false positives: each time you add a character, it takes 37 times longer because there are 37 times as many possibilities, but the hash function remains the same size – i.e. there will also be 37 times as many false positives …

Anyway – I guess CUDA is a good way to go. I’m afraid to say I cannot help you with the language since I’m using an AMD GPU (which does not support CUDA) … also, it’s been at least two years since I’ve done GPU computing.

What I would do is:

1. Write all hashes we have yet to decode into an array (or whatever CUDA calls it, maybe a one-dimensional texture)
2. For each thread, use the thread ID and a counter you pass from the CPU to assemble a file name (e.g. thread 1 handles all filenames starting with an a; in the first pass it will be aaaaaaaa, aaaaaaab in the second, and so on)
3. Compute the hash of the assembled file name
4. Compare it to all hashes in the array of step 1.
5. This is the hardest part: I have no idea on how to get the results back to the CPU. Maybe CUDA offers you something like synchronized queues (HLSL does). Or you write a thread-safe queue yourself (I think it’s unlikely that many threads find a match in the same pass, so this won’t be a bottleneck anyway).
6. Increase the counter to compute the next warp of file names. After each million or so steps, safe the results so they are not lost if the program crashes. If it does, you can use the counter to resume where you’ve left.

I see what you mean about the false positives. There were ~90 hits per hash for the 6-char names, so we can expect over 3000 per hash for 7-chars.
While I’m sure our pattern matching computer (DrKevDog) is up to the task, 8-chars might be a bit much.

… in which case, I’ll leave the CUDA stuff for the time being … as interesting as it is. Don’t have the time for it right now anyway.
Regarding step 5), it looks like you can use printf from each thread.

Well, the 7 character script has finished …

I’ve uploaded dot_3_7.zip which contains the sorted data split into 58 separate files, one for each noname that is yet to be identified.

There were 59 files, but in order to check my scripts didn’t produce complete garbage, I could ascertain that:

noname2243 = 3\tvild90.3

… after looking at the .3 file in the viewer, then searching for tvil which similar objects have in their names.

Glad it’s finished. I’ll work on these later when I get free. The viewer does come in mighty handy for this work 👍

Great!
When you get bored, I can take over.

In the meantime, I’ll start the same process with the .ssd files. I expect most of these will match the new .3 that we’ve found.

Here’s the .ssd files with 3, 4 & 5 characters:

ss\abu.ssd = noname10941
ss\aby.ssd = noname10942
ss\car.ssd = noname10955
ss\dar.ssd = noname10872
ss\man.ssd = noname10861
ss\map.ssd = noname10862
ss\amad.ssd = noname11438
ss\dt10.ssd = noname11049
ss\flak.ssd = noname11316
ss\fsam.ssd = noname11227
ss\horn.ssd = noname11461
ss\miss.ssd = noname11492
ss\para.ssd = noname11197
ss\su33.ssd = noname11294
ss\xr73.ssd = noname11172
ss\xr77.ssd = noname11173
ss\xsam.ssd = noname11033
ss\elsib.ssd = noname12373
ss\flare.ssd = noname12230
ss\crate.ssd = noname11920
ss\xjsow.ssd = noname11816
ss\ftank.ssd = noname12286
ss\echo1.ssd = noname11977
ss\echo2.ssd = noname11979
ss\echo3.ssd = noname11981
ss\echo4.ssd = noname11983
ss\echo5.ssd = noname11985
ss\echo6.ssd = noname11987
ss\echo7.ssd = noname11988
ss\xcrv7.ssd = noname12561
ss\fcarr.ssd = noname11774
ss\scale.ssd = noname12357
ss\xkh58.ssd = noname11702
ss\xjdam.ssd = noname11573
ss\su27a.ssd = noname11962
ss\xr77r.ssd = noname11800

… and for the 6-chars, these are the fairly obvious ones:

ss\xfueli.ssd = noname497
ss\xfuelo.ssd = noname500
ss\f0mk20.ssd = noname1045
ss\xalarm.ssd = noname1565
ss\f3mk20.ssd = noname507
ss\delete.ssd = noname1615
ss\fxfuel.ssd = noname484
ss\xagm86.ssd = noname1343
ss\xagm88.ssd = noname1345
ss\f1mk20.ssd = noname244
ss\fshipl.ssd = noname798
ss\fpjsow.ssd = noname1174
ss\fxjsow.ssd = noname502
ss\group2.ssd = noname565
ss\fpfuel.ssd = noname1151
ss\fships.ssd = noname800
ss\bigfac.ssd = noname245
ss\fpjdam.ssd = noname730
ss\fxmk20.ssd = noname516
ss\fymk20.ssd = noname1517
ss\fpmk20.ssd = noname1186
ss\f0jdam.ssd = noname480
ss\f1jdam.ssd = noname1477
ss\ball_1.ssd = noname557
ss\ball_2.ssd = noname558
ss\ball_3.ssd = noname559
ss\ball_4.ssd = noname560
ss\ball_5.ssd = noname562
ss\f2mk20.ssd = noname1240
ss\fxjdam.ssd = noname5
ss\fyjdam.ssd = noname1098
ss\whouse.ssd = noname1015
ss\xdecoy.ssd = noname1556
ss\xgbu24.ssd = noname473
ss\xkh59m.ssd = noname512
ss\f2jdam.ssd = noname814
ss\f3jdam.ssd = noname1709
ss\xkh29t.ssd = noname1185
ss\xlau68.ssd = noname1606
ss\xtiald.ssd = noname176
ss\xmk82r.ssd = noname1398
ss\xkh31a.ssd = noname260
ss\xmk82f.ssd = noname1397
ss\xmk83f.ssd = noname1401

but there’s also these which I’m not sure about. These are the most likely possibilities:

noname881
amxf5r
bilfer
biutil
fmzget

noname1501
aphen2
arriw3
mybase
opfai6
orper7

noname720
eeaird
olilpg
sq_ft2

noname723
eeaire
hfaiz4
sq_ft3

noname716
eeairb
hfaiz1
sq_ft0

noname718
eeairc
hfaiz2
sq_ft1

Here are the 7-character, I’ll clean up the loose ends tomorrow:

1713 temple2
1729 tempusa
1814 tnav_hq
2069 tempamx
2262 t_rdr90
2349 tcyl_90
2425 lightly
2575 tdome90
2587 store90
2588 f22shad
2619 0600sun
2648 redsea1
2649 redsea2
2657 t_crate
2841 ?
2842 ?
2860 frafale
2931 ex_phit
3036 tamy_hq
3042 gen2rwy
3283 tvil180
3297 eject_1
3298 eject_3
3451 fmirage
3475 ianmap1
3670 sunseth
3816 tair_hq
3985 mikehor
4104 agaim9x
4153 abtest1
4154 abtest2
4382 fgalaxy
4425 faurora
4582 flats01
4583 flats02
4749 xcannon
4825 fapache
4899 flatmap
4911 ex_bhit
5103 Odryhot (?)
5420 horizon
5470 1800sun
5914 ?
6062 ?
6346 sud7471
6349 hide_90
6441 oilcont
6502 fef2000
6574 fx_flak
6629 smokews
6705 bedford
6825 sud7671
6848 tprfb90
7051 t_radar
7116 riy_msq (?)
7192 tvil_90
7248 ?
7329 ib_dust

🙇
You've got more patience than me!

The 7-char .ssds should be available later today, hopefully they’ll provide more of a challenge. 😉

I created 8 separate programs, each handling a part of the problem and have two i7 PCs running 4 each. This brings the computation time down from over 3 days to about 10 hours.

bored

I created 8 separate programs, each handling a part of the problem and have two i7 PCs running 4 each. This brings the computation time down from over 3 days to about 10 hours.

Congratulations on figuring out how to harness that i7 computing power. You most definitely are our resident computer genius! 🍺

Thanks, but starting from a Python script gives considerable room for improvement in speed. 🙂

I’ve uploaded the 7-char .ssds. I thought that I’d messed something up since I couldn’t find anything recognisable, but I think they are all hard to spot like fpmk82f.

Here are the 7-char .ssds:

noname1892 f1aim9x
noname2424 awc_sa6
noname2558 f2aim9x
noname3334 fxagm88
noname3383 fpagm88
noname3655 fxmk82f
noname3658 fxmk82r
noname3697 fxmk83f
noname3711 fpmk82f
noname3719 fpmk82r
noname3743 fpmk83f
noname3778 f0agm88
noname3959 fyagm88
noname4043 fpdecoy
noname4056 f0mk82f
noname4058 f0mk82r
noname4093 f0mk83f
noname4172 fxlau68
noname4227 fplau68
noname4273 ex_hit2
noname4277 ex_hit4
noname4287 fymk82f
noname4296 fymk82r
noname4337 fymk83f
noname4349 xasraam
noname4459 f1agm88
noname4511 f0decoy
noname4667 f0lau68
noname4772 f1mk82f
noname4776 f1mk82r
noname4818 xaim_9x
noname4822 f1mk83f
noname4851 xagm84a
noname4921 xfab250
noname4936 xagm86n
noname4969 xrbk500
noname5127 f2agm88
noname5172 f1decoy
noname5329 xmav_ir
noname5330 f1lau68
noname5412 f2mk82f
noname5416 f2mk82r
noname5483 f2mk83f
noname5810 f3agm88
noname5874 f2decoy
noname6110 f2lau68
noname6135 fxgbu24
noname6189 fpgbu24
noname6224 f3mk82f
noname6233 f3mk82r
noname6266 f3mk83f
noname6309 fxaim9x
noname6352 fpaim9x
noname6444 xrocket
noname6606 f3decoy
noname6749 f0aim9x
noname6788 f3lau68
noname6797 fygbu24
noname6970 fyaim9x
noname7097 xr27_er
noname7100 xr27_et
noname2237 xdurand
noname5117 wakem_1

Outstanding!

It would now seem that it’s worth taking on the 8-character names even though we may end up with ~100000 possibilities per hash.

I’ve downloaded the CUDA SDK and can compile and run Nvidia’s sample code, so I’ll try and implement what Krishty suggests.

That will take some time and effort, so what should we look at in the meantime?

I started with .3 and .ssd, since:

1. We can scan the unknown files and identify which are .3 or .ssd thus we only check the hashes for those files.
2. .3 and .ssd files are only found in the 3 and ss folders respectively
3. .3 and .ssd files always have extension .3 and .ssd (… obviously)

Because of 2. and 3., the first and last bytes of the candidate string that is to the hash function is fixed, and 1. significantly reduces the false positives.

bored

It would now seem that it’s worth taking on the 8-character names even though we may end up with ~100000 possibilities per hash.

My opinion is that it would be better to have it than not have it, all the while hoping for a more user friendly method.

bored

That will take some time and effort, so what should we look at in the meantime?

I can think of a number of things, such as, concurrent work on analyzing the newly named .3 and/or .ssd files, naming the unused palette and fadetable files (I need to find the files for the different modes, if they exist), work on finding a method to distinguish / identify files specific to individual modes (e.g. software vs. hardware, etc …), defining the prefix and general naming conventions for better clarity and understanding. It seems to me that most of the common names used for the .3 and .ssd files are those of individuals listed in the credits as being on the DID graphics team, it also seems as if Mike (mike.3, mikehor.3, etc …) was working on the implementation of the hardware mode and truecolor texturing.

I’m still working on the hardware mode theory and have found some files which I believe are native to that mode.

Damn, I need to travel around for a few days, so probably won’t have much time for for TAW. 🙁

We’ve had a good week though. 🙂

I agree, it was a productive week. It’s unfortunate you now have to travel around but I hope to still be here when you return and I’ll keep working on stuff while your gone 😉

While trying to look at CUDA (or OpenCL) in more detail, I am reminded of what a piece of crap Microsoft Visual Studio is if you just want to do something basic. 😠

I have only have source file, but VS deems it necessary to produce whole directories full of crap when all I need is exactly one .exe file. 🙁 🙁

A pity there is no equivalent of Dev-C++ for CUDA …

A real pity that is 😠 I hope you can find alternate route so we can continue …

Here are a couple more which are of interest:

lev\smoke.dat = noname11859
ss\ex_vols5.ssd = noname9894

noname11859 is interesting because although it is identical to the default smoke.dat, in TAW the smoke.dat is in the f22data directory. This is a continuing pattern I have found with a number of files and may lend credence to my theory regarding the support for an alternate EF2K-like system of execution.

Noname12355 is the TFX3 extracted smoke.dat, and noname12356 is in the smoke.dat format but has unusual content.

Still working on hashing the noname12356, perhaps it contains a clue to its function.

Noname9894 is of interest because, IIRC, it is not normally extracted. It is listed as the last (41st) entry in the redxxxx.txt files explosions index. This has been a persistent source of TAW explosion-related game crashes (bug). It’s curious to me why it is not extracted?

Good work!

DrKevDog

A real pity that is 😠 I hope you can find alternate route so we can continue …

Oh, I can continue but fighting environment issues just puts me in a bad mood. 🙁
I’ve got CUDA sort of working in that I can get the same answer as the CPU code in a much shorter time, at least for 4-character .3 filenames. It takes less than a second as opposed to about 5 seconds.
I’ve hit a bit of a snag in that if I try and calculate longer hashes in that the graphics driver resets the GPU after a few seconds, I guess because it thinks it’s died.

Maybe you’re not supposed to do CUDA stuff on the primary display GPU …

That is good news, I’ll keep working on file code issues. Currently I am curious about how the particle system works with the smoke, dust, wake, etc. files. I’ll post something over in one of the other threads while you figure out why the GPU resets, good luck 🙂

The GPU reset is a Windows safety feature which can be turned off (at my own risk) by adding a Registry key.

Since I have 2 GPUs on my card, I tried just using GPU0 for the screen and GPU1 for CUDA, but I can’t seem to allocate memory on GPU1 for some reason, although I haven’t exactly read the manual yet. The display seems to work OK, even with the CUDA stuff running on the same GPU though.

Anyway, for the 5-char .3 filenames, of which we have a list of 24 hashes, it takes 2mins 10 seconds to produce the names. This is faster than the i7, but not by enough. When I reduced the hash list to 12 hashes, it took 1min 15 seconds and then only 23 seconds for one hash. So, it would seem that we are wasting a lot of time checking through the list of hashes which are stored as a 1D texture common to all threads.

Using both GPUs and spreading the workload over more threads will probably help, but that will have to wait until next weekend.

Still don’t really know what I’m doing, but got the 5-char time down from 130 to 34 seconds by copying the 1D texture to a local array as each thread starts.
We’re still only about 5 times faster than the single i7 though. 🙁

Need to create more threads …

EDIT: Well, that seemed to work. 🙂 Got the 5-char down to about 3 seconds by giving each possible character in the second position of the test string its own thread.

So moving on to the 6-chars and we meet the next problem. The calculation takes about 75 seconds to find 45 hashes, but when the results are spit out at the end my PC freezes … although it does seem to complete the task.
I need to find a better way to get the results from the GPU, since this will only get worse with longer hashes …

On the bright side, we’re about 100 times faster than the i7 now. 🙂

I have no personal experience with CUDA so I’m not sure what is causing the crashes, nevertheless, it sounds like the efficiency and speed are developing, and that is promising 🙂

Probably just about fast enough now, but need to sort out the freezing problem which I guess is the CUDA stuff interfering with the board’s display driver activities.

I thought I might be able to set the video driver into some basic VGA mode, like that used at bootup … but can’t see an option in the Boot menu.

I have an old GTX580 that I could use as the primary display driver, but I’m not sure the PSU can handle a 580 and 590 even though the 580 would be effectively idling.

Who’s the hardware expert around here?

No time for any progress recently, although I have two options to move forward. Either have a dedicated CUDA GPU board as described above, or leave the PC hardware as it is and restrict individual CUDA kernel run times to less than 5 seconds.

hey, don’t know if this is any help, but in my newly bought rig, i have an i5 2500k and a radeon hd 6970 … i think the programming for it might be different … and i know it doesnt support cuda.

however i also know that for certain hash calculations (like for bitcoin mining) ATIs have superior speed compared to nvidia cards.

/// EDIT:
for example, in GUI Miner, mine (which is also overclocked) gets on average 412 Mega Hashes / sec

in comparison:

gtx 580 - 112.3 MHashes / sec
gtx 590 - 174.4 MHashes / sec
hd 6970 - 395.9 MHashes / sec

source: http://www.hardocp.com/article/2011/07/13/bitcoin_mining_gpu_performance_comparison/2

EDIT ///

i gladly offer to run scripts to help find those file names … if it can be used, please dont hesitate to ask 🙂

cheers!

Great! Some help would certainly be useful. 🙂

I was looking for my CUDA code to post, but I don’t have it with me this weekend. 🙁

If we want to run OpenCL, it would need some rewrite anyway …

bored

No time for any progress recently …

I have only a small contribution thus far. In the way of some clean-up of the loose ends on the 7-character .3’s:

2841 jetty_1
2842 jetty_2
5103 neilder
5914 chan_9
7248 agpwisp

Every little bit helps 🙂

Eagle_Flight

however i also know that for certain hash calculations (like for bitcoin mining) ATIs have superior speed compared to nvidia cards.

…

i gladly offer to run scripts to help find those file names … if it can be used, please dont hesitate to ask 🙂

cheers!

Yes, ATI appear to be better at raw computing than NV’s graphics cards. The conspiracy theory is that NV are doing this on purpose to push their expensive Tesla cards.
Apparently, the GTX680 is slower than the GTX580 running CUDA.

Here’s what my code has devolved into, which in this case is configured for 7-char .3 files. I have a problem running this right now since the CUDA locks up the PC after a few seconds.

#include <stdio.h>
#include "cuPrintf.cu"
#include "shrUtils.h"
#include "cutil_inline.h"
#include <cstdlib>
#include <iostream>
#include <fstream>
#include <string>
using namespace std;
texture<int, cudaTextureType1D, cudaReadModeElementType> tex;
__global__ void testKernel(int val,int gpu) {
char c,x,j,d,y;
int hashes[150];
x=threadIdx.x;
if (gpu==0) {
if (x<10) c=x+48;
if (x==10) c=95;
if (x>10) c=x+86;
} else {
c=x+19+86;
}
y=threadIdx.y;
if (y<10) d=y+48;
if (y==10) d=95;
if (y>10) d=y+86;
unsigned char test[9]={152,0,0,0,0,0,0,0,134}; //&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;
// unsigned char out[4]={'X','X','X',0};
//	  cuPrintf("%s %s\n",out,88);
test[1]=c;
test[2]=d;
// out[0]=c;
unsigned char a1,a2,a3,a4,a5;
  unsigned char v1;
  int v2;
  unsigned int v3=0;
  int result,rescheck;
  unsigned char v9=0;
  int var=83;
  unsigned char i;
  unsigned char oldv1=0;

	 for (j=0;j<val;j++) {
  hashes[j]=tex1D(tex,j);
  }

for (a5=48;a5<123;a5++) {
	if (a5==58) {a5=95;}
	if (a5==96) {a5=97;}
for (a4=48;a4<123;a4++) {
	if (a4==58) {a4=95;}
	if (a4==96) {a4=97;}
for (a3=48;a3<123;a3++) {
	if (a3==58) {a3=95;}
	if (a3==96) {a3=97;}
  for (a2=48;a2<123;a2++) {
  if (a2==58) {a2=95;}
  if (a2==96) {a2=97;}
//	  cuPrintf("%u",a2);
for (a1=48;a1<123;a1++) {
	if (a1==58) {a1=95;}
	if (a1==96) {a1=97;}

test[3]=a5;
test[4]=a4;
test[5]=a3;
test[6]=a2;
test[7]=a1;
// out[1]=a2;
// out[2]=a1;
result=0;
v3=0;
v9=0;
oldv1=0;
  v2=9; //&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;
   for ( i = 0; i < v2; i++ )
  {
   v1=test[i];
   if ( v1 == 95 )
   {
	  v3 -= i;
	  }
	  else
	  {
	  if ( i )
	  {
		   if (v1>oldv1) {
			  v9 = v9*2+1;
		   } else {
			 v9=v9*2; }
	  v3 = (var * v3 + v1) % 0xFFFFB;
	  } else {
			 v3=v1;
	  }

	}
	oldv1=v1;
  }
  result = ((v2 & 0xF) << 28) ^ (v3 << 8) ^ v9;

   for (j=0;j<val;j++) {
  rescheck=hashes[j];
	  if (result==rescheck) {
	  cuPrintf("%u %s%s%s%s%s%s%s\n",j,test[1],test[2],test[3],test[4],test[5],test[6],test[7]); //&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;
	  }
	  }
}//a1
}//a2
}//a3
}//a4
}//a5
//   cuPrintf("\tValue is:%s %u\n", c,href);
}
int main(int argc, char *argv[]) {
	string line;
  int count=0;
  int gpun;
  int linsiz;
  int sp0;
  string shash,snoname;
  string ahash[126];
  string anoname[126];
  string sp=" ";
   fstream myfile;
   fstream outfile;
  string hash;
  int j,k;
  string tnam="";
  string test;
  int hint;
  int cint;
  int harray[125];
  dim3 tPB1(19, 37);
  dim3 tPB2(18, 37);

	myfile.open(".\\unknown9_list.txt"); //&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;&sect;
  if (myfile.is_open())
  {
  while ( myfile.good() )
  {
	  getline (myfile,line);
   linsiz=line.length();
   sp0=line.find(sp);
   shash=line.substr(0,sp0);
   snoname=line.substr(sp0+1,linsiz-sp0-1);
	  ahash[count]=shash;
	  anoname[count]=snoname;
//   cout << count << " " << ahash[count] << " " << anoname[count] << endl;
   count++;
   }
	myfile.close();
  }
  count --;

   for (j=0;j<count;j++) {
   line=ahash[j];
   hint=0;
  for (k=0;k<8;k++) {
  shash=line[k];
  if (shash=="0") cint=0;
  if (shash=="1") cint=1;
  if (shash=="2") cint=2;
  if (shash=="3") cint=3;
  if (shash=="4") cint=4;
  if (shash=="5") cint=5;
  if (shash=="6") cint=6;
  if (shash=="7") cint=7;
  if (shash=="8") cint=8;
  if (shash=="9") cint=9;
  if (shash=="a") cint=10;
  if (shash=="b") cint=11;
  if (shash=="c") cint=12;
  if (shash=="d") cint=13;
  if (shash=="e") cint=14;
  if (shash=="f") cint=15;
  if (k==7) hint=hint + (cint);
  if (k==6) hint=hint + (cint << 4 );
  if (k==5) hint=hint + (cint << 8 );
  if (k==4) hint=hint + (cint << 12 );
  if (k==3) hint=hint + (cint << 16 );
  if (k==2) hint=hint + (cint << 20 );
  if (k==1) hint=hint + (cint << 24 );
  if (k==0) hint=hint + (cint << 28 );
  }//for k
   harray[j]=hint;
//   cout << line << " " << harray[j] << endl;
   }//for j

   cudaSetDevice(0);
   cudaArray* cu_array;
  cudaChannelFormatDesc description = cudaCreateChannelDesc<int>();
   cutilSafeCall(cudaMallocArray(&cu_array, &description, count));
	 cout << count << endl;
  // Copy image data to array
  cudaMemcpyToArray(cu_array, 0,0,harray, count*4, cudaMemcpyHostToDevice);

  // Bind the array to the texture
  cudaBindTextureToArray(tex, cu_array);
	 cudaSetDevice(1);
  cudaChannelFormatDesc description2 = cudaCreateChannelDesc<int>();
   cutilSafeCall(cudaMallocArray(&cu_array, &description2, count));
	 cout << count << endl;
  // Copy image data to array
  cudaMemcpyToArray(cu_array, 0,0,harray, count*4, cudaMemcpyHostToDevice);

  // Bind the array to the texture
  cudaBindTextureToArray(tex, cu_array);

	cudaSetDevice(0);
   cudaPrintfInit();
   gpun=0;
   testKernel<<< 1,tPB1 >>>(count,gpun);
  cudaPrintfDisplay(stdout, false);
   cudaPrintfEnd();

cudaSetDevice(1);
	cudaPrintfInit();
  gpun=1;
testKernel<<< 1,tPB2 >>>(count,gpun);
	cudaPrintfDisplay(stdout, false);
   cudaPrintfEnd();
   return 0;
}

I think I have a solution to the runime problem in that I can use a USB-VGA adapter to drive the screen while the GTX590 does the CUDA stuff. While not affecting the PC hardware, this may take some configuring, which I don’t want to risk now …

bored

I have some idea what bytes 8 to 11 of TAW’s did.dat might do now. In the code of f22.dat there is this sub which appears to decide which hash routine to use based on result

I’ve implemented the .dat version recognition, so the current version of Red Sea Explorer works with ADF, too. I’ve already spotted a glitch that has been fixed for TAW:

Splendid!

I know those pictures aren’t taken from exactly the same point, but according to your figures there are many more triangles rendered in the TAW version. I don’t know why though.

That’s just by accident – I had trouble finding Djibouti with the TAW version, so I increased the viewing range. The program adapts LOD choices to viewing range, therefore there’s more triangles being drawn. (You can tell because the TAW screenshot has less fog at the same distance.)

Any chance to enumerate EF2000’s did.dat files?

I.e.: Are there noticable differences between the initial version, v2.02, v2.04, Super EF2000, TACTCOM / Evolution, Graphics+, v2? I have only access to the U.S. EF2000 did.dat …

I guess U.S. version 2 for DOS equals TACTCOM / Evolution with Graphics+, right? And U.S. version 2 for Windows 95 equals Super EF2000? At least that’s how I interpreted this and this…

It might be nice to differentiate them for the sake of completeness, but up to now I’ve only looked at the dat files for the euro SEF and US V2.0 3dfx versions.

Krishty

I guess U.S. version 2 for DOS equals TACTCOM / Evolution with Graphics+, right?

Yes, although the V2.0 disk contains four versions, DOS/TACTCOM, DOS/TACTCOM/3dfx,DOS/TACTCOM/Rendition, SEF(US).
I’ve just run a diff test on the dat files for the V2.0 3dfx and SEF(US), and they are identical.

Krishty

And U.S. version 2 for Windows 95 equals Super EF2000?

The US version of SEF is slightly different to the euro version in menus and multiplay options. I assume the basic game is the same though.
The did.dat files are quite a bit different, with the euro version about 10MB larger and a slightly different format.
It’s been a while since I looked at this, but I seem to remember the euro version having some embedded exe files as well as language directories for french, german and spanish.

I’ve reread this thread and had another look at the did.dat files that I have immediate access to, and note the following:

The euro SEF2000 did.dat has a size of 49938786 bytes and is slightly different in format in that the data file content starts from the 5th byte.
Presumably, the first 4 bytes give a pointer to the start of the index section, but must be encrypted/obfuscated in a different way to that for the V2.0 EF2000.
The index section itself is not encrypted and is the same format as TAW.
The first 4 bytes converted from intel format create 0x812b8435, but the start of the index section is 0x02f86ced.

For the two games immediately preceding EF2000, which are TFX and Inferno, the first 4 bytes give a direct pointer to the index section where the same rules as for TAW apply.
We don’t know the hash functions for these games though.

Yesterday, I had another go at finding the remaining names from TAW’s did.dat, but ended up totalling my PC. This is a pity, since even if I have some data backed up, the development environment that I’m used to is lost and would need to be recreated from scratch.

I had refined the CUDA code somewhat so that it was now acceptably fast for 8 char names and easier to use, and also created scripts to convert the CUDA output into separate files for manual analysis. Tests on 4 and 6 char filenames went OK, but with the 6 char filenames I started encountering the problem that graphics freezes when the CUDA runs for more than a few seconds.

So, to try and get around this, I used a USB-DVI converter to run a second monitor as the primary display driver, assuming that the main graphics card can now be used exclusively for CUDA. From a display point of view, this worked fine, but got a no CUDA device detected when I tried to run the CUDA code. crap!

When I installed the driver for the USB-DVI converter, I seemed to lose network access. As the CUDA wasn’t working, I gave up on it and uninstalled the driver using it’s own uninstall program and was prompted to restart, which turned out to be a bad move.

Now, on startup I get an almost immediate BSOD, and pressing F8 gets me to the Safe Mode menu, but will BSOD as soon as I select anything. The only useful option is the one which stops the cyclic reset by stopping at the BSOD, where it seems I have error code 0x000000B7.

If I try the repair option, it always fails.

Not sure what to do next …

I’d try removing the network adapter and one the GPUs, so whatever driver causes the fault, it’s not loaded any more. I hope it’s no hardware failure.

It was the USB GPU’s drivers that likely caused the problem, and I used a USB device to avoid this sort of thing. 🙁
Removing the device from the USB port doesn’t help.

I think I’ll just try reading the drives on another PC to try and salvage some data, then take the PC back to the shop and get my money back as it’s only 8 months old.

Taking the PC back to the shop sounds like the best next step. I had a recent problem with screen freezes and BSOD, I was able to trace it to memory modules. I have been able to correct it by removing the problem memory and … so far, so good.

Well, that was weird. The hardware is fine, but every attempt to repair the boot/mbr area with the RecoveryEnvironment tools failed.
I’ve never had such a serious problem with any version of Windows, never mind the supposedly awesome Win7.

Windows is reinstalled now, and the only data I lost was the CUDA code. Luckily I had the foresight to paste a copy of it in this thread. 🙂

It shouldn’t take too long to get back to where I was, and this time I’ll try and put the project data on the D: drive, which hasn’t been affected by all this.

EDIT:
Turns out I haven’t lost anything. The files I thought were missing were actually hidden by default.

bored

EDIT:
Turns out I haven’t lost anything. The files I thought were missing were actually hidden by default.

I’m glad for you and that’s a relief. I am looking forward to a new set of names to decode 🙂

DrKevDog

… I am looking forward to a new set of names to decode 🙂

Really?

I might as well upload the results of a CUDA run on the 8-char nonameXX.3 hashes despite there being a problem.
https://www.box.com/s/6066ece18193fef9b4d8

The problem is that I got less matches with a CUDA run on the 7-char names when checked against the CPU only run. What was produced seems correct, but there are gaps in the output.

This means that despite the output of the 8-char run being huge, it should be even bigger.

I obviously need to troubleshoot the CUDA script when I get time, but it may be worth seeing if some files can be found in order to cut down on the unknowns.
The run time is proportional to the number of unknown filenames, and as there are 124 hashes to match, this run took nearly 3 days.

Bear in mind that:

1. If you find an obvious name in a file … then great! that’s one less for the next run.
2. If there is no obvious match, then the name is probably missing and we’ll have to run it again.

I’ve started things off by finding that:

noname7495=riy_base.3

(This file is still in the zip, since I forgot to remove it)

EDIT:

noname7792=azi_base.3

EDIT2:
… beginning to see a pattern here.

noname8314=lux_base.3

… oh, and for the shorter .3 filenames, I have suggestions for everything except the 7-char noname6062.
After a quick scroll through the candidates I’m picking hfusbig.3. Looking at it in 3View, I don’t think it matters if it’s wrong. 🙂

Oh boy, you really don’t know when people are really just joking, do you? 😁

No, actually I think this is very good. I’m amazed at how quickly you’ve bounced back with your data.
And I’m also already amazed at what you’ve found. I thought that att_base.3 was the only one of its kind, very interesting 🙂

I’ll get on it ASAP!

Just in case it helps: I have generated two lists with the paths of all files accessed in did.dat through my viewer; one for Red Sea and one for arcade. For each list, I flew over the whole terrain and switched damage levels and time of day. Sadly, I could not fly over the terrain for each time of day and each damage level individually, so other texture lists than red1400 are probably not complete. However, since it includes file names read from .ssd/.3/.INI files, it might include one or two files you’ve not discovered yet.

It’s 3676 filenames for arcade; 3791 for Red Sea. They're sorted alphabetically (in order to filter duplicate entries).

I don’t know how many of these requests were actually satisfied by did.dat; there may be some non-existent files.

Thanks!

We now have the technology to scan those lists and compare them with what we already have, but unfortunately they add nothing new.
So, we’re still at 11286 of 12588 (not including 40 or so .3 files that I forgot to add to the known list).

As a matter of interest, why is your viewer looking for 707.3, for example? I assume it’s as a consequence of reading the ssinfo.fn file, but there may be other sources …

Yes, it is very likely read from ssinfo.fn.

I’ll keep the technology in mind – hopefully, when I can load more game files, we’ll explore some of the remaining files 🙂

I’ve updated the known list to include the missing .3 files including the three 8-char names from a couple of post ago. The score is now 11336 of 12588.

Note that I haven’t looked for any of the 121 remaining 8-char .3 filenames since I uploaded that batch of files.

If I get a bit of peace and quiet over the weekend, I’ll try to categorize the rest of the unknown files and fix the CUDA problem.
Why I’m doing this, I just don’t know. 🙂

Well, not much peace and quiet but I can work around the CUDA problem by having two separate programs each targeting its own GPU with half the task. These can run concurrently, so shouldn’t affect total execution time.

I’ll probably wait until sunday night before starting another 8-char run, as the PC becomes very sluggish.

DrKevDog, have you had any luck with the results of the incomplete first run? I might have a look, but I don’t want to duplicate anything you’ve already done.

Well, I had a quick look anyway …
… and here is where I’m at with the 8-char .3 files

7495=riy_base
7792=azi_base
8314=lux_base
9208=asy_base
7536=kha_base
10482=khr_base
8841=genarfld
7877=dome_180
10001=kha1_180
8369=kha2_han
9501=kha1ha90

We can leave any further analysis until I do the next run … but let me know if you have any more.

Not much time lately, although I did take a swipe at a few of the last 8-char .3 files:

7440 = awc_j707
7806 = awc_r767
9903 = awc_f15e
7947 = awc_su30
8309 = flearjet
7914 = desdrk_1
8348 = plrdfo_1
8990 = plrdab_1

The last two are rather interesting in that neither has the designated road. Regular files, in use in-game, like plrddt_1 have the road in their code. We have extracted the non-krus road.3 (noname11528), which is probably the road we need to take. Perhaps it is the road less traveled, however, when I get the time I think I will follow this road to see where it leads and if it offers any clues to either how to implement this more flexible road in TAW or to the true meaning of life 😁

Great, thanks!
Only 105 .3 filenames left to get. There’ll be a bit of a delay before I can get around to doing that last .3 CUDA run. This is due to operational reasons …

Speaking of roads, there are some terrain tiles which have both road and rail river crossings (eg noname10727.3). The only bridges I can recall in TAW are single road bridges.

Very nice! Currently, there are no bridges of that type in TAW, however, I believe Tem.3 (10874) is the bridge you are describing.

3View is not showing tem.3, but I don’t mean a combined road/rail bridge if that’s what tem.3 is. The tiles I’m pointing out would have two bridges some distance apart, but on the same tile.

bored

3View is not showing tem.3, but I don’t mean a combined road/rail bridge if that’s what tem.3 is. The tiles I’m pointing out would have two bridges some distance apart, but on the same tile.

Tem.3 is a combined road / rail bridge. Hmm, now I see what you are referring to and 10726 is related:

noname10726:

There are at least 2 bridges, in the unused files, which will facilitate the river crossing of both wheeled vehicles and trains. I have not yet found a rails only bridge, however, I must say I was surprised to find that noname10727 was not in my comprehensive list 😲

I believe the question would be answered by finding the correct corresponding supershape file(s).

Note, that in my list tem.3 is noname10873 … but that would make my noname10727 be noname10728 for you, but your picture is definitely my noname10727. My head hurts. 🤷

noname873.3 is drawing; you just need to move close enough (140 is doing fine here) and look directly upwards. Then, you’ll see this:

No parameters, no matching textures. I’m clueless.

Ah yes, I see it now.

The name tem.3 doesn’t give much away, and there’s quite a few objects that fall into this category … EF2000 is even worse.

Is this what you see, suspended?

Is that tem.3 in game? If so, that’s definitely not what we see in the viewer which is the picture above. With some imagination, there are some similarities though.

yes, tem.3. Viewing it in-game in Custom Combat, it is suspended @ 4000ft. I believe you are viewing the inferior surface and my screenshot is of the superior surface. How is it suspended 🤷

Could it be 3840 feet? All the Y coordinates have a value of between 480–483 for some reason. Looking up at it would explain the disparity in what we see.

EDIT:
Just realised I can edit .3 files, so changed the Y coordinate in the first 0062 line to zero, et voila:

The textures of the bridge supports come from cam3_9 which are F22 parts. So that’s a rather expensive bridge. 🙂

bored

Could it be 3840 feet?

Yep.

bored

The textures of the bridge supports come from cam3_9 which are F22 parts. So that’s a rather expensive bridge. 🙂

The combined weight of the rail and road vehicles crossing the dual-lane bridge would require a special metal bridge construct such as the above Titanium bridge. It has me curious as to the purpose of these dual-lane bridges in TAW

Here is the 8-char .3 file aliquot for the day:

7782 = lux_term
9401 = tprfb_18 (now they’re breaking the rules and making things even more difficult 😁)
9586 = tprfb_90
9717 = strg_90a
10306 = tcmhq_90

DrKevDog

Here is the 8-char .3 file aliquot for the day:

Splendid! Only 100 to get.

Should be back to the CUDA PC in a couple of days. Next time I should try and set up Remote Desktop so that I can control it remotely although I’m not sure Win7 Home edition allows this.

I’ve also been looking at OpenCL (specifically pyOpenCL), which would mean that anybody could run the scripts on a reasonably modern machin … but it’s not trivial.

Bored, the OpenCL idea sounds splendid.

Okay, lets move away from the titanium bridge constructed from salvaged F22 parts, and get back to the road (noname11528) less traveled 😁

The positive thing about this road is 1.) it has three functioning LOD’s, which are selected via the 0027 jumps, and 2.) it has more utility.
Although it is less traveled, it is more prone to crashing TAW. I believe the crash is related to the second LOD:

Distance transition:

LOD #2:

The crash may be related to either the 001B or 004A opcodes:

004A0000
002703E802EC
000100340000001B
001B0000003400000004001B001E
0007FB94
0000
004A001E
002703E802DE
00010034001B0005
001B00000034001E001B00060005
0007FBA8
0000
004A0005
002703E802D0
0001003400050009
001B000000340005000600090008
0007FBBC
0000
004A0008
002703E802C2
0001003400090011
001B000000340008000900100011
0007FBD0
0000…

Is this an opacity issue?

Not sure, but I’ve just noticed that some roads have different lods and some don’t.
Take noname10725 and 10727 (you may have them as 10726&10728), the first one has 2 lod levels of the tracks, while the second only one … but looking at the parsed files, I don’t see anything unexpected.

Which file are you using in the above picture?

Both are 11528. IIRC, we left off, with the discussion on 004b, that it was for disabling sprites. I am wondering if this road is supposed to involve some other files / sprites.

Scratch that, I’m imposing a 004b onto a 004a problem. Hmmm.

Hard to say. That’s road.3, and would seem to be some sort of experiment. It doesn’t crash 3View, but I don’t know offhand if those 004a/001b opcodes are implemented.

Any idea on how I can tell apart an extracted ADF directory from an extracted TAW directory? Are there, for example, elementary mission files which exist in one or the other only?

Going from memory here, but check the value of bytes 8-11 of did.dat. TAW should have a value 3, ADF=2 and the EF2000V2 file should = 1, but I think that is coded using the bit flipping scheme I mentioned earlier in this thread.
With this info, you can select the correct magic number for the hash routine.

This is true for did.dat, but what if it had been extracted and deleted? I’d like the viewer to work with extracted folders, too …

Oops, sorry, was watching the football …

I don’t know of a fundamental difference, but I’ve never seen an extracted version of ADF with known filenames as I haven’t got around to doing it.

My suggestion would be to use some image files, like the menu backgrounds as a marker … or the size of the f22.dat binary if we assume that it will be present.

I’d need to see an ADF extracted folder tree before giving a definitive answer …

Would using the TAW Campaign specific files work?

ADF also has a second .dat file called did1st.dat which is loaded … first.

I’m not sure what’s in this, but this file is looked for first, even in TAW’s code.

Indeed, there’s lots of options and GUI files in it, and even some textures. So I guess, the basic principle of the file system is: did1st.dat; if not found, did.dat; if not found, actual file in the folder. I’m going to implement it like that. I could, however, not determine if there is ADF-only files in did1st.dat.

Edit: I just checked, and none of the files I’m currently using in the Red Sea Explorer (shapes, terrain supershapes and textures) are overriden in did1st.dat for ADF.

hey Bored,

The 8-character file sets are proving problematic. One issue is, obviously, the size (some are 750,000 characters long), and another is the uncertainty of the name actually being contained in the list, i.e. gaps in the output? did you have a chance to troubleshoot the CUDA script?

I thought I had a workaround to the CUDA problem which involved running two separate processes, one for each GPU. This worked perfectly for the 7 char filenames for which I have reference data. The run took about 35 minutes for 5 hashes.
While trying the same trick fo the 8 chars, the first process started OK and is still running now. Starting the second process resulted in a not enough CUDA resources error, which I don’t understand and haven’t had time to look into further.
For now, I’ll run the second process when the first one has finished, but this will take a while as we still have 100 hashes to match.

Unfortunately, I haven’t had much time for TAW of EF2000 during the last few months. 🙁

An update:
The first run finished and produced a text file of about 24MB, and as the total produced last time was about 34MB it seems likely that I’m not losing data this time. The second run is now underway and will finish sometime on tuesday.

I don’t really have the time to learn how to do this properly, so for now it’s going to take 5 days or so for a 8-char run and 100 hashes. On the bright side, this would have taken over 7 years with the original python script. 🙂

The annoying thing is that it is virtually impossible to use the PC for anything else while it is doing the CUDA run, but apparently there’s not that much that can be done to improve the situation on a Win7 machine.
It would be much better to use Linux, but I’m sure this will have its own issues …

My x64 machine is still chugging through the 8-char .3 files … which is why i haven’t downloaded the latest Viewer yet. I suppose a Linux version is out of the question. 😉

In the meantime, I’ve gone through the remaining noname files and categorised them with some simple rules, to try and determine at least the file extension.
The result can be found here:
https://www.box.com/s/efe41c8214bbff537914

If I get time tomorrow, I’ll try and sub-categorise the .txt files.

bored

In the meantime, I’ve gone through the remaining noname files and categorised them with some simple rules, to try and determine at least the file extension.

👍

Thanks!
I suspect most of the 16kb .raw files in the Unknown/agp directory are actually .dat and .fad files. Also, 11650, 11640, 11639: look more like they belong to the hwacl directory. I’ll do some further work on them as I can.

Yes, probably a bit early to be naming them /agp and .raw as, the only criteria I used was that they are exactly 16384 bytes in size.

I haven’t really looked at the actual file contents, although while coming up for a .fnt detection method I noticed the font name is usually at location 0x812 of the binary file.

EDIT: All that be ascertained from loc 0x812 is that it is a font, but it doesn’t really help with the name.
Anyway, here are some of the shorter filenames. The remaining 5 fonts have at least 5 chars …

fonts\nh13.fnt = noname11127
fonts\didfont\160.fnt = noname10840
fonts\didfont\161.fnt = noname10841
fonts\hf6.fnt = noname10884
fonts\hf7.fnt = noname10885
fonts\hf9.fnt = noname10886
fonts\nh9.fnt = noname10944
fonts\guifont\11.fnt = noname10829
fonts\helvetic\26.fnt = noname10819
fonts\hudfont\10.fnt = noname10785
fonts\hudfont\15.fnt = noname10786
fonts\topazmon\10.fnt = noname10822
fonts\topazpro\11.fnt = noname10802
fonts\triumv\47.fnt = noname10807
fonts\hudfont.old\10.fnt = noname10804
fonts\hudfont.old\15.fnt = noname10805
fonts\sansserf\10.fnt = noname10821
fonts\tuxedo\26.fnt = noname10788
fonts\10.fnt = noname10794
fonts\15.fnt = noname10795
fonts\hudfont.old\6.fnt = noname10765
fonts\hudfont.old\7.fnt = noname10766
fonts\hudfont.old\8.fnt = noname10767
fonts\hudfont.old\9.fnt = noname10768
fonts\blob\6.fnt = noname10777
fonts\7.fnt = noname10778
fonts\8.fnt = noname10779
fonts\newhud\15.fnt = noname10827
fonts\newhud\10.fnt = noname10825
fonts\times\52.fnt = noname10792

Great work! My math may be off because I count 6 remaining, 2 of which don’t belong in the fonts directory and can be removed:

winfonts/ f22_1.ext = noname12579
winfonts/ f22_2.ext = noname12580

Great! I was wondering what to do with those. On inspection, they are truetype fonts, but there is no .ttf extension in the list.

So, there’s just the remaining … er … 4 to get. I should have 12358 in a few hours, but the last 3 will have to wait their turn with CUDA unless we can work them out with traditional sleuthing. 🙂

Thanks to missing a %s in a printf statement, the 5 day 8-char run was a complete washout and I only have the first 7 chars of each filename. 🙁

I’ve started it again, but it’s going to be next tuesday, at the earliest, before I can collate the results.

Next Tuesday is okay, it'll give us time to see if the Greek vote kicks them out of the euro, at which time the world, reportedly, will implode and we will have to suspend the TAW project in order to scavenge for food 🙄

Just wanted to say I’m moving. Eventually, I’ll have four more hours a day for programming, but currently, I don’t have much time at all for TFX.

I did a little tuning this weekend though – you’ll find the current release in our box, as usual.

Glad to hear it, and I’m having the same problem getting any quality TFX time as well.

As I was away in southern europe over the weekend, I left my PC doing the 8-char CUDA run. Anyway, I got back to find it switched off … but the other half claims she hasn’t touched it.
Bottom line is that it’s now going to be at least thursday before I can sort the results.

DrKevDog, a present for you:
https://www.box.com/s/e1e7fbd8c3a100ae27c7

Enjoy!

Oh Great! Now I can quit my job, kick the wife and kids out, grow a beard and just stare at the monitor for the next 2-3 weeks 🙃

bored

DrKevDog, a present for you:

That is a much better present 😉

I took a quick look at the first several files, and immediately these fell out:

7467 = tempoliv   (ship: Oliver)
9213 = nkha1_90   (Building: Khartoum)
9049 = tatna_90   (Antenna)
8737 = temfb_90   (FuelBag)
10405 = tdome_90   (Building: radome)
9167 = fchinook   (Chinook-flat)
9178 = t_cntl90   (Building: Control)
9876 = tempchim   (Building: Chimney)
8432 = nkha1han   (Building: Khartoum Hanger)
8557 = scrufc_1*   (Terrain Tile: Scruf)
8558 = scrufc_2*
8559 = scrufc_3*
8560 = scrufc_4*

*to be comfirmed

Great!

I think the scruf is correct as the following also exist, although I haven’t checked to see what they look like.

scrufm_1
scrufi_1,2,3,4
scrufs_1,2,3,4

That’s 22 down. 🙂

Thanks, that is confirmation.

A few more:

10261 = tdome_18
8363 = tdomebui
10401 = plarbn_1
10402 = plarbn_2
10403 = plarbn_3
10404 = plarbn_4
10442 = riy_prk1
10443 = riy_prk3
9638 = plfore_4
9847 = plforn_1
9849 = plforn_2
9850 = plforn_3
9851 = plforn_4

… and taking your temp theme:

7791 = tempcrat
9102 = tempanlo
7924 = tempudal
9703 = temparli
10700 = tempporm
7471 = tempneut

The temp theme may be complete.

Others:

8751 = chan_180
8892 = tatna_18
9799 = fcomanch
10410 = tantenna

I’ve had a look at the .txt files that I categorised earlier, and could find enough clues to get the following:

f22data\spanish\serial.gdd = noname1300
f22data\italian\serial.gdd = noname874
f22data\german\serial.gdd = noname793
f22data\french\serial.gdd = noname465
f22data\serial.gdd = noname774
f22data\muldemo.txt = noname3329
f22data\arcade.txt = noname895
f22data\cg_amt.txt = noname978
samples\sfx\attack.wav = noname1150
samples\sfx\distort1.wav = noname10427
samples\sfx\distort2.wav = noname10428
samples\sfx\distort3.wav = noname10429
samples\sfx\crackbn.wav = noname6849
samples\sfx\engine4.wav = noname5378
samples\sfx\fx061lp.wav = noname6353
samples\sfx\fx055a.wav = noname1099
samples\sfx\fx053b.wav = noname1075
samples\sfx\afterin2.wav = noname10622
samples\sfx\fx033a.wav = noname972
samples\sfx\zap1lp.wav = noname1048
samples\sfx\zap2lp.wav = noname200
samples\sfx\zap3lp.wav = noname1173
samples\sfx\zap4lp.wav = noname323
samples\sfx\zap5lp.wav = noname1301
samples\sfx\zap6lp.wav = noname496
samples\sfx\fx031alp.wav = noname10419
samples\sfx\fx031blp.wav = noname8866
samples\sfx\fx004alp.wav = noname10562
samples\sfx\fx029lp.wav = noname2567
pcx\scenario\sim\SIM28a.pcx = noname1183
pcx\scenario\sim\SIM28b.pcx = noname1184
pcx\scenario\sim\SIM32.pcx = noname12207
pcx\scenario\sim\SIM14cj.pcx = noname4910
pcx\scenario\sim\SIM13b.pcx = noname111
pcx\scenario\tod\image1.pcx = noname1670
pcx\scenario\tod\image2.pcx = noname1671
pcx\scenario\tod\image3.pcx = noname1672
pcx\scenario\tod\saudi.pcx = noname12409
pcx\scenario\tod\redsea.pcx = noname1356
cgdat\ff16.sh = noname11330
cgdat\fmirage.sh = noname6567
cgdat\fr767.sh = noname12034
pcx\squadron\badges02.pcx = noname9368
pcx\squadron\badges01.pcx = noname9367
huddle\f22hi.hud = noname11634
huddle\f22lo.hud = noname11659
f22data\spanish\mp_lobby.gdd = noname8701
f22data\spanish\trg_area.gdd = noname10287
f22data\spanish\bcom.gdd = noname11242
f22data\spanish\scenario.gdd = noname8676
f22data\spanish\squadron.gdd = noname9211
f22data\spanish\editpilt.gdd = noname10656
f22data\spanish\tour.gdd = noname11383
f22data\spanish\login.gdd = noname12142
f22data\spanish\quick0.gdd = noname395
f22data\spanish\simultor.gdd = noname7672
f22data\italian\mp_lobby.gdd = noname8273
f22data\italian\trg_area.gdd = noname9827
f22data\italian\bcom.gdd = noname11467
f22data\italian\scenario.gdd = noname8709
f22data\italian\squadron.gdd = noname9241
f22data\italian\editpilt.gdd = noname10669
f22data\italian\tour.gdd = noname11072
f22data\italian\login.gdd = noname11910
f22data\italian\quick0.gdd = noname1610
f22data\italian\simultor.gdd = noname7684
f22data\german\mp_lobby.gdd = noname7727
f22data\german\trg_area.gdd = noname9282
f22data\german\bcom.gdd = noname11537
f22data\german\scenario.gdd = noname8542
f22data\german\squadron.gdd = noname9105
f22data\german\editpilt.gdd = noname10555
f22data\german\tour.gdd = noname11122
f22data\german\login.gdd = noname11798
f22data\german\quick0.gdd = noname1540
f22data\german\simultor.gdd = noname7545
f22data\french\mp_lobby.gdd = noname8971
f22data\french\trg_area.gdd = noname10553
f22data\french\bcom.gdd = noname11350
f22data\french\scenario.gdd = noname8475
f22data\french\squadron.gdd = noname9046
f22data\french\editpilt.gdd = noname10468
f22data\french\tour.gdd = noname11538
f22data\french\login.gdd = noname12289
f22data\french\quick0.gdd = noname1260
f22data\french\simultor.gdd = noname7470
f22data\mp_lobby.gdd = noname8943
f22data\trg_area.gdd = noname10471
f22data\bcom.gdd = noname11191
f22data\scenario.gdd = noname10648
f22data\squadron.gdd = noname7769
f22data\campaign.win = noname9773
f22data\editpilt.gdd = noname9186
f22data\tour.gdd = noname11287
f22data\login.gdd = noname12438
f22data\quick0.gdd = noname1520
f22data\simultor.gdd = noname9653
f22data\missplan.win = noname7758
fonts\top10.fnt = noname12358

Outstanding!

The huddle\f22hi.hud line caught my attention. It may give some direction as to how to categorize the HUD components.

I believe we see an evolution:

From this:

LOADHUD "f22lo"
LOADHUD "f22hi"
LOADHUD "f22800"
LOADHUD "f221024"

To This:

LOADHUD "f22" VP 640 400 320 200 FONT "hudfont" 8
LOADHUD "f22" VP 640 400 640 400 FONT "newhud" 9
LOADHUD "f22" VP 640 400 800 600 FONT "newhud" 13
LOADHUD "f22" VP 640 400 1024 768 FONT "hudfont" 9

Could be used to better tweak the 1024×768 Rez3 modification.

Maybe so. There’s quite a few HUD files left to name.

Of the 565 files that I’ve categorised as text, about half are non-english. There are about 60 cfg\weapons\*.txt files and about 20 .sh files, and 33 .gdd.

That leaves a couple of hundred to go through manually.

Today’s outfiles8 bundle:

10246 = t_cnt_90 *
10495 = tdfhq_90
10677 = fharrier
8759 = store180
10420 = store_90
8386 = t_contro
8571 = t_cnt180
9363 = fdarksta
9533 = fstallio
9845 = tamyhq90

👍

Only 45 to go, by my reckoning. 🙂

I’m basically leaving you to it, but have we got this one:

9548=ftornado

44.

More scraping the bottom of the barrel …

There are no obvious solutions for these 4 three letter names, so I’ve picked a value from the hash list so that they get extracted.

samples\sfx\err.act = noname10871
red0600\nei.rog = noname10962
iff\256\sah.cdl = noname10847 (actually .lbm file)
iff\256\sam.cdl = noname10848 (actually .lbm file)

These are OK, although poo.txt seems rather apt.

cgdat\fu2.sh = noname10939
3\poo.txt = noname10856
samples\gui\rps.bnk = noname10973
mp_col\map.col = noname10965
colours\map.col = noname10846
f22data\french\out.txt = noname10940
f22data\out.txt = noname10953
f22data\gbv.txt = noname10867
f22data\f22.rog = noname10892
f22data\f22.ian = noname10890
f22data\f22.and = noname10888
f22data\tfx.fig = noname10882
f22data\ian.ins = noname10877
huddle\ian.ins = noname10883
f22data\f22.ins = noname10891
huddle\efa.lad = noname10865
f22data\efa.lad = noname10859
f22data\f22.lad = noname10887
f22data\f22.hud = noname10889
agp\sun.raw = noname10966
samples\speech\foo.in = noname10957
samples\sfx\foo.wav = noname10964
f22data\intelsim.txt = noname10353
f22data\icondesc.txt = noname9938
f22data\windesc.win = noname6100
cdl\demo.cdl = noname11459
cdl\demo.tcs = noname11460
f22data\win.win = noname10845
cgdat\f22.hlp = noname10863

Note to self. The following 2 files are erroneously flagged as unknown, probably because they live in the root directory.

airbase.dat = noname477
coltab.dat = noname12259

Now this is getting VERY interesting. Great work!
I believe you are correct, poo it is. I went through that file some time ago and concluded the alpha-prefixed models were discards. It seems the opposite pattern is present, with respect to volume, for the temp and t models compared to the current game models?

😲

Some more:

f22data\acmi.win = noname11003
briefing\demo.txt = noname11439
cfg\weapons\f117.pak = noname11132
kdl\kots.kdl = noname11319
briefing\french\kots.txt = noname11547
briefing\german\kots.txt = noname11124
briefing\italian\kots.txt = noname11015
briefing\spanish\kots.txt = noname11249
briefing\kots.txt = noname11089
f22data\coll.dat = noname11023
f22data\french\options.cfg = noname5222
f22data\german\options.cfg = noname3257
f22data\italian\options.cfg = noname4070
f22data\spanish\options.cfg = noname4770
f22data\options.cfg = noname5144

… and any idea what files looking like this do?

CENTRE 0 0
SHAPE 17 0
PT -74 -82
PT -59 -103
PT -44 -119
PT -29 -131
PT -14 -138
PT 0 -140
PT 15 -138
PT 29 -131
PT 44 -119
PT 58 -103
PT 73 -82
SHAPE 17 0
PT 1 -63
PT -12 -63

That looks like: \HUDDLE\XXX.fig

Thanks!

huddle\tfx3.fig = noname10981
f22data\tfx3.fig = noname11349

Heh, doesn’t look like they bothered giving any win/lose messages for the french. Must have assumed they’d give up before finishing a campaign. 🙂

briefing\spanish\lose10.txt = noname20
briefing\italian\lose10.txt = noname543
briefing\german\lose10.txt = noname1509
briefing\spanish\lose9.txt = noname11943
briefing\italian\lose9.txt = noname12169
briefing\german\lose9.txt = noname12397
briefing\spanish\lose8.txt = noname11942
briefing\italian\lose8.txt = noname12168
briefing\german\lose8.txt = noname12396
briefing\spanish\lose7.txt = noname11941
briefing\italian\lose7.txt = noname12167
briefing\german\lose7.txt = noname12395
briefing\spanish\lose6.txt = noname11939
briefing\italian\lose6.txt = noname12166
briefing\german\lose6.txt = noname12394
briefing\spanish\lose5.txt = noname11937
briefing\italian\lose5.txt = noname12165
briefing\german\lose5.txt = noname12393
briefing\spanish\lose4.txt = noname11935
briefing\italian\lose4.txt = noname12164
briefing\german\lose4.txt = noname12392
briefing\spanish\lose3.txt = noname11933
briefing\italian\lose3.txt = noname12163
briefing\german\lose3.txt = noname12391
briefing\spanish\lose2.txt = noname11931
briefing\italian\lose2.txt = noname12162
briefing\german\lose2.txt = noname12390
briefing\spanish\lose1.txt = noname11929
briefing\italian\lose1.txt = noname12161
briefing\german\lose1.txt = noname12389
briefing\spanish\win10.txt = noname11701
briefing\italian\win10.txt = noname11814
briefing\german\win10.txt = noname12158
briefing\spanish\win9.txt = noname11183
briefing\italian\win9.txt = noname11458
briefing\german\win9.txt = noname10996
briefing\spanish\win8.txt = noname11182
briefing\italian\win8.txt = noname11457
briefing\german\win8.txt = noname10995
briefing\spanish\win7.txt = noname11181
briefing\italian\win7.txt = noname11456
briefing\german\win7.txt = noname10994
briefing\spanish\win6.txt = noname11180
briefing\italian\win6.txt = noname11455
briefing\german\win6.txt = noname10993
briefing\spanish\win5.txt = noname11179
briefing\italian\win5.txt = noname11454
briefing\german\win5.txt = noname10992
briefing\spanish\win4.txt = noname11178
briefing\italian\win4.txt = noname11453
briefing\german\win4.txt = noname10991
briefing\spanish\win3.txt = noname11177
briefing\italian\win3.txt = noname11452
briefing\german\win3.txt = noname10990
briefing\spanish\win2.txt = noname11176
briefing\italian\win2.txt = noname11451
briefing\german\win2.txt = noname10989
briefing\spanish\win1.txt = noname11175
briefing\italian\win1.txt = noname11450
briefing\german\win1.txt = noname10988

I was hoping to add more, however, with the focus of today’s earlier Terrain discussions, I may not get back to this immediately.

9548=ftornado
8651 = 2200moon
8496 = 0200star
9428 = t_dishes
10251 = tvild_90
8843 = awc_mig3
10496 = lux_park
10253 = lux_hard
10191 = ib_trail
7963 = rvplab_1

A few more:

cgdat\f707.sh
cgdat\map1.sh
cgdat\map2.sh
cgdat\map3.sh
cgdat\map4.sh
cgdat\map5.sh
cgdat\map6.sh
cgdat\map7.sh
cgdat\map8.sh
cgdat\map9.sh
cgdat\test.sh
colours\grey.col
colours\map2.col
colours\matt.col
colours\norm.col
mp_col\map2.col
mp_col\matt.col
mp_col\norm.col
dat\grey.fad
dat\grey.dat
dat\acmi.fad
dat\3dfx.dat
pcx\misc\bcom.pcx
pcx\misc\star.pcx
pcx\scenario\scenario\arcade\arc05.pcx
pcx\scenario\scenario\arcade\arc06.pcx
pcx\scenario\scenario\arcade\arc07.pcx
pcx\scenario\scenario\arcade\arc08.pcx
pcx\scenario\tod\tod1.pcx
pcx\scenario\tod\tod2.pcx
pcx\scenario\tod\tod3.pcx
pcx\scenario\tod\tod4.pcx
pcx\scenario\tod\tod5.pcx
pcx\scenario\scenario\sim\sim1.pcx
pcx\scenario\scenario\sim\sim2.pcx
pcx\scenario\scenario\sim\sim3.pcx
pcx\scenario\scenario\sim\sim4.pcx
pcx\scenario\scenario\sim\sim5.pcx
f22data\tm.hlp
pcx\quickcom\qc.pcx
briefing\spanish\qc.txt
briefing\italian\qc.txt
briefing\german\qc.txt
briefing\french\qc.txt
samples\sfx\ew.dat
hwacl\x.txt

Some of these are very good. Although I wasn’t able to test it extensively, x.txt seems to add a dimension to the flares. Some, like tm.hlp, are just baffling 🤷

Yes, the majority of remaining files are unlikely to lead to great revelations but as this activity doesn’t require much overhead, I might as well continue as time permits.

Today’s batch:

samples\sfx\_sfx.cfg
samples\sfx\_sfx.bnk
samples\sfx\taxi.wav
samples\sfx\null.wav
cfg\weapons\list.txt
cfg\weapons\king.txt
briefing\french\sim9.txt
briefing\german\sim9.txt
briefing\italian\sim9.txt
briefing\spanish\sim9.txt
ss\xecm.ssd
ss\xb8w.ssd
f22data\tfxa.fig
huddle\tfxa.fig
mp_col\tree.lbm
mp_col\acmi.lbm

agp\wisp2.raw
agp\wisp3.raw
agp\wispy.raw
f22data\tcp_ip.gdd
f22data\french\tcp_ip.gdd
f22data\german\tcp_ip.gdd
f22data\italian\tcp_ip.gdd
f22data\spanish\tcp_ip.gdd
f22data\french\modem.gdd
f22data\german\modem.gdd
f22data\italian\modem.gdd
f22data\spanish\modem.gdd
f22data\modem.gdd
samples\sfx\flyby.wav
samples\sfx\noise.wav
samples\sfx\windy.wav
samples\gui\click.wav
cgdat\ff15e.sh
cgdat\ff16a.sh
cgdat\flare.sh
cgdat\fsu30.sh
cgdat\ftri1.sh
cgdat\ftri2.sh
cgdat\ftri3.sh
cgdat\ff16u.sh
briefing\spanish\sim18.txt
briefing\spanish\sim20.txt
briefing\spanish\sim21.txt
briefing\spanish\sim23.txt
briefing\spanish\sim31.txt
briefing\spanish\sim32.txt
briefing\spanish\tour1.txt
briefing\spanish\tour2.txt
briefing\spanish\tour3.txt
briefing\italian\sim18.txt
briefing\italian\sim20.txt
briefing\italian\sim21.txt
briefing\italian\sim23.txt
briefing\italian\sim31.txt
briefing\italian\sim32.txt
briefing\german\sim18.txt
briefing\german\sim20.txt
briefing\german\sim21.txt
briefing\german\sim23.txt
briefing\german\sim31.txt
briefing\german\sim32.txt
briefing\german\tour1.txt
briefing\german\tour2.txt
briefing\german\tour3.txt
briefing\french\sim18.txt
briefing\french\sim20.txt
briefing\french\sim21.txt
briefing\french\sim23.txt
briefing\french\sim31.txt
briefing\french\sim32.txt
briefing\french\tour1.txt
briefing\french\tour2.txt
briefing\french\tour3.txt
pcx\scenario\tod\sim1.pcx
cgdat\grey.col
briefing\german\more.txt
samples\speech\kots.bin
samples\speech\kots.idx
samples\speech\kots.in

Okay, so now I can modify my file placements and move the wisp files: 11639, 11640 and 11650 from hwacl to the agp directory. It seems reasonable that these are related to noname7248 (agpwisp.3).

noname7248:

00000000:ff ec 0f a0 00 00 00 00 7f ff 00 00 00 00 00 00
00000010:00 00 00 00 00 02 00 03 00 03 00 00 00 00 00 00
00000020:00 83 00 07 00 04 00 05 00 06 00 00 00 00 00 00
00000030:00 20 00 00 00 20 00 00 00 20 00 00 00 04 00 4b
00000040:00 03 00 04 00 00 00 62 00 00 ff ec ff ec 00 00
00000050:00 64 00 28 00 65 00 28 00 64 ff d8 00 61 00 04
00000060:00 00 00 47 01 94 00 04 00 00 00 00 00 3f 00 00
00000070:00 3f 00 3f 00 00 00 3f 00 88 00 04 00 00 00 01
00000080:00 02 00 03 00 00 ff ff

This shape calls the AGP texture moon.raw. Wispy, wisp2 and wisp3 are most probably an interesting set of moon overlay treatments. Being 16kb, 32bit raws, brings up the question of where they are to correctly reside within the scheme of the texture index. They meet the obligatory material properties of index slots 341–450. 341–404 are occupied with the explosion files. Index slot 405 is the last known occupied slot with moon, it therefore seems reasonable that the wisp files would occupy those immediately behind moon.

Seems reasonable …

File identification is starting to bog down now since I have to guess the file path and extension, then customize a script, run it and hope for the best.
Luckily, as it’s fairly quiet in the summer, I can do this at work. 🙂
A pity the computers at work are not CUDA/OpenCL capable though. 🙁

We always strive to be reasonable 🙂 I believe it reasonable, considering our project priorities, to walk right up to the proprietor and demand he immediately replace the computers with CUDA/OpenCL capable ones … and give you a big pay increase while he’s at it 😁

They would probably go for it. The problem isn’t really access to the hardware, but the time required to learn and implement CUDA (or similar) properly.
If I really wanted to, and had the time, I could either remote desktop to my home PC or maybe use Google Compute which is free for now.

Here’s an uninspiring batch:

pcx\scenario\scenario\campaign\camp01.pcx
pcx\scenario\scenario\campaign\camp02.pcx
pcx\scenario\scenario\campaign\camp03.pcx
pcx\scenario\scenario\campaign\camp04.pcx
pcx\scenario\scenario\campaign\camp05.pcx
pcx\scenario\scenario\campaign\camp06.pcx
pcx\scenario\scenario\campaign\camp07.pcx
pcx\scenario\scenario\campaign\camp08.pcx
pcx\scenario\scenario\campaign\camp09.pcx
pcx\scenario\scenario\campaign\camp10.pcx
pcx\scenario\scenario\campaign\camp11.pcx
pcx\scenario\scenario\campaign\camp12.pcx
pcx\scenario\scenario\campaign\camp13.pcx
pcx\scenario\scenario\campaign\camp14.pcx
pcx\scenario\scenario\campaign\camp15.pcx
pcx\scenario\campaign\camp01.pcx
pcx\scenario\campaign\camp02.pcx
pcx\scenario\campaign\camp03.pcx
pcx\scenario\campaign\camp04.pcx
pcx\scenario\campaign\camp05.pcx
pcx\scenario\campaign\camp06.pcx
pcx\scenario\campaign\camp07.pcx
pcx\scenario\campaign\camp08.pcx
pcx\scenario\campaign\camp09.pcx
pcx\scenario\campaign\camp10.pcx
pcx\scenario\campaign\camp11.pcx
pcx\scenario\campaign\camp12.pcx
pcx\scenario\campaign\camp13.pcx
pcx\scenario\campaign\camp14.pcx
pcx\scenario\campaign\camp15.pcx
pcx\campaign\camp11.pcx
pcx\campaign\camp12.pcx
pcx\campaign\camp13.pcx
pcx\campaign\camp14.pcx
pcx\campaign\camp15.pcx
pcx\scenario\scenario\sim\sim6engf.pcx
pcx\scenario\scenario\sim\sim7elcf.pcx
pcx\scenario\scenario\sim\sim7hydf.pcx
pcx\scenario\tod\tod16a.pcx
pcx\scenario\tod\tod16b.pcx
pcx\scenario\tod\tod27a.pcx
pcx\scenario\tod\tod27b.pcx
pcx\scenario\scenario\sim\sim10a.pcx
pcx\scenario\scenario\sim\sim10b.pcx
pcx\scenario\scenario\sim\sim11a.pcx
pcx\scenario\scenario\sim\sim11b.pcx
pcx\scenario\scenario\sim\sim12a.pcx
pcx\scenario\scenario\sim\sim12b.pcx
pcx\scenario\scenario\sim\sim13a.pcx
pcx\scenario\scenario\sim\sim13c.pcx
pcx\scenario\scenario\sim\sim14a.pcx
pcx\scenario\scenario\sim\sim14b.pcx
pcx\scenario\scenario\sim\sim14c.pcx
pcx\scenario\scenario\sim\sim24a.pcx
pcx\scenario\scenario\sim\sim24b.pcx
pcx\scenario\scenario\sim\sim13b.pcx
pcx\scenario\scenario\sim\sim15a.pcx
pcx\scenario\scenario\sim\sim15b.pcx
pcx\scenario\scenario\sim\sim16a.pcx
pcx\scenario\scenario\sim\sim16b.pcx
pcx\scenario\scenario\sim\sim16c.pcx
pcx\scenario\scenario\sim\sim17a.pcx
pcx\scenario\scenario\sim\sim17b.pcx
pcx\scenario\scenario\sim\sim28a.pcx
pcx\scenario\scenario\sim\sim28b.pcx
briefing\spanish\sim13b.txt
briefing\spanish\sim15a.txt
briefing\spanish\sim15b.txt
briefing\spanish\sim16a.txt
briefing\spanish\sim16b.txt
briefing\spanish\sim16c.txt
briefing\spanish\sim16d.txt
briefing\spanish\sim17a.txt
briefing\spanish\sim17b.txt
briefing\spanish\sim19a.txt
briefing\spanish\sim19b.txt
briefing\spanish\sim19c.txt
briefing\spanish\sim19d.txt
briefing\spanish\sim19e.txt
briefing\spanish\sim19g.txt
briefing\spanish\kots10.txt
briefing\spanish\sim28a.txt
briefing\spanish\sim28b.txt
briefing\italian\sim13b.txt
briefing\italian\sim15a.txt
briefing\italian\sim15b.txt
briefing\italian\sim16a.txt
briefing\italian\sim16b.txt
briefing\italian\sim16c.txt
briefing\italian\sim16d.txt
briefing\italian\sim17a.txt
briefing\italian\sim17b.txt
briefing\italian\sim19a.txt
briefing\italian\sim19b.txt
briefing\italian\sim19c.txt
briefing\italian\sim19d.txt
briefing\italian\sim19e.txt
briefing\italian\sim19g.txt
briefing\italian\kots10.txt
briefing\italian\sim28a.txt
briefing\italian\sim28b.txt
briefing\german\sim13b.txt
briefing\german\sim15a.txt
briefing\german\sim15b.txt
briefing\german\sim16a.txt
briefing\german\sim16b.txt
briefing\german\sim16c.txt
briefing\german\sim16d.txt
briefing\german\sim17a.txt
briefing\german\sim17b.txt
briefing\german\sim19a.txt
briefing\german\sim19b.txt
briefing\german\sim19c.txt
briefing\german\sim19d.txt
briefing\german\sim19e.txt
briefing\german\sim19g.txt
briefing\german\kots10.txt
briefing\german\sim28a.txt
briefing\german\sim28b.txt
briefing\french\sim13b.txt
briefing\french\sim15a.txt
briefing\french\sim15b.txt
briefing\french\sim16a.txt
briefing\french\sim16b.txt
briefing\french\sim16c.txt
briefing\french\sim16d.txt
briefing\french\sim17a.txt
briefing\french\sim17b.txt
briefing\french\sim19a.txt
briefing\french\sim19b.txt
briefing\french\sim19c.txt
briefing\french\sim19d.txt
briefing\french\sim19e.txt
briefing\french\sim19g.txt
briefing\french\kots10.txt
briefing\french\sim28a.txt
briefing\french\sim28b.txt
pcx\scenario\scenario\sim\sim01.pcx
pcx\scenario\scenario\sim\sim02.pcx
pcx\scenario\scenario\sim\sim03.pcx
pcx\scenario\scenario\sim\sim04.pcx
pcx\scenario\scenario\sim\sim05.pcx
pcx\scenario\scenario\sim\sim06.pcx
pcx\scenario\scenario\sim\sim07.pcx
pcx\scenario\scenario\sim\sim08.pcx
pcx\scenario\scenario\sim\sim09.pcx
pcx\scenario\scenario\sim\sim10.pcx
pcx\scenario\scenario\sim\sim11.pcx
pcx\scenario\scenario\sim\sim12.pcx
pcx\scenario\scenario\sim\sim22.pcx
pcx\scenario\scenario\sim\sim25.pcx
pcx\scenario\scenario\sim\sim26.pcx
pcx\scenario\scenario\sim\sim27.pcx
pcx\scenario\scenario\sim\sim30.pcx
pcx\scenario\scenario\sim\sim31.pcx
pcx\scenario\scenario\sim\sim32.pcx
pcx\scenario\scenario\sim\sim34.pcx
pcx\scenario\scenario\sim\sim35.pcx
pcx\scenario\scenario\sim\sim6a.pcx
pcx\scenario\scenario\sim\sim7a.pcx
pcx\scenario\scenario\sim\sim8a.pcx
pcx\scenario\tod\tod11.pcx
pcx\scenario\tod\tod12.pcx
pcx\scenario\tod\tod13.pcx
pcx\scenario\tod\tod14.pcx
pcx\scenario\tod\tod15.pcx
pcx\scenario\tod\tod18.pcx
pcx\scenario\tod\tod19.pcx
pcx\scenario\tod\tod20.pcx
pcx\scenario\tod\tod21.pcx
pcx\scenario\tod\tod22.pcx
pcx\scenario\tod\tod23.pcx
pcx\scenario\tod\tod24.pcx
pcx\scenario\tod\tod25.pcx
pcx\scenario\tod\tod26.pcx
pcx\scenario\tod\tod29.pcx
pcx\scenario\tod\tod30.pcx
pcx\scenario\tod\tod31.pcx
pcx\scenario\tod\tod6a.pcx
pcx\scenario\tod\tod8a.pcx
pcx\scenario\tod\tod8b.pcx
pcx\scenario\tod\tod8c.pcx
pcx\scenario\sim\sim01.pcx
pcx\scenario\sim\sim02.pcx
pcx\scenario\sim\sim03.pcx
pcx\scenario\sim\sim04.pcx
pcx\scenario\sim\sim05.pcx
pcx\scenario\sim\sim06.pcx
pcx\scenario\sim\sim07.pcx
pcx\scenario\sim\sim08.pcx
pcx\scenario\sim\sim09.pcx
pcx\scenario\sim\sim10.pcx
pcx\scenario\sim\sim11.pcx
pcx\scenario\sim\sim12.pcx
pcx\scenario\sim\sim28.pcx
pcx\debrief\war_a.pcx
pcx\gamemap\plate.pcx
pcx\misc\blank.pcx
pcx\misc\leave.pcx

Getting this:

ERROR: Path not recognised pcx\scenario\scenario\campaign\
Traceback (most recent call last):
  File "G:\guess.py", line 68, in <module>
    h=datlib.TawHash(path,name,ext)
  File "G:\datlib.py", line 519, in __init__
    teststring=pre+name+post
UnboundLocalError: local variable 'pre' referenced before assignment

Will have to troubleshoot it later,

Thanks

That happened to me as well …
Line 161 of datlib.py should read:

  elif path.lower()=='pcx\\scenario\\scenario\\campaign\\':

and not:

  elif path.lower()=='pcxsescenario\\scenario\\campaign\\':

… which I guess you have in your version.

Thanks, I’ll make that correction.

You’re scaring the children with your talk of cloud computing, google compute, EC2 and powerful platforms like CUDA. There will no longer be freedom, only control! 🙇

But that’s all it is … just talk. 🙁

briefing\spanish\sim14cj.txt
briefing\italian\sim14cj.txt
briefing\german\sim14cj.txt
briefing\french\sim14cj.txt
briefing\spanish\endtour1.txt
briefing\german\endtour1.txt
briefing\french\endtour1.txt
briefing\spanish\endtour2.txt
briefing\german\endtour2.txt
briefing\french\endtour2.txt
briefing\spanish\endtour3.txt
briefing\german\endtour3.txt
briefing\french\endtour3.txt
f22data\hiscores.txt
cgdat\borders.txt
f22data\borders.saf
f22data\awachelp.txt
f22data\briefing.win
3\fogdefs.txt
f22data\4_cow.txt
f22data\tempo.txt
samples\betty\betty.txt
f22data\sample.brf
f22data\options.win
samples\music\music.cfg
samples\music\music.bnk
ss\paddyair.ssd
pcx\quickcom\quickcom.pcx
briefing\spanish\credits1.txt
briefing\french\credits1.txt
briefing\german\credits1.txt
briefing\italian\credits1.txt
cgdat\icondesc.txt
f22data\italian\td_tours.txt
f22data\german\td_tours.txt
f22data\french\td_tours.txt
briefing\german\multipla.txt
f22data\menuintl.txt
f22data\mainmenu.sav
f22data\mainmenu.txt
f22data\squadron.txt
cgdat\airbases.dat
f22data\scenario.lst
cfg\weapons\attax.txt
cfg\weapons\stuff.txt

Apologies for 4_cow.txt but couldn’t see anything better. At least it’s marked as a text file.

MOON ASIDE: I decided to take a look @ the AGP moon thing. The file names struck me as peculiar, and the only other place wisp is used in TAW is in reference to the clouds. I implemented the three different agp wisp raws to replace the three cloud TM textures (I can do that easily using the 27th header word) and the floating moon overlay effect at night is rewarding 😉

8586 = nlarbs_2
10262 = t_rdr_90
8179 = tcommsbu
8313 = tcommshq
9272 = tcmbl_90
8456 = rwyendn7
10725 = nlrlab_1
10727 = nlrlab_2
10624 = rlplab_1
9256 = plrlab_1
8624 = plrlfo_1
10668 = plrvfo_1

bored

Apologies for 4_cow.txt but couldn’t see anything better. At least it’s marked as a text file.

Confidence in 4_cow.txt is low. Do you still have the lists?

Ok, instead of f22data\4_cow.txt for noname12451 you can have 3\neily.txt … which probably makes more sense.

To get all possibities, I’d have to run all possible paths with all possible 5-char names, with all possible extensions.
That’s 106×37×37×37×37×37×71 permutations. While this can be done, it takes time, and if I open noname12451 I don’t see anything earth shattering.

I know anything involving transtextures is your bag though …

Forgive me for hijacking the thread:

No new version this week, but a sneak peak at the current progress:
https://www.box.com/s/d925b92eba02525bc37c
(It’s HD in lossless mode, so you might need a fast CPU to watch it flawlessly.)

I’m currently implementing joystick controls. DirectInput is deprecated and Microsoft recommends using the Raw Input API, but this is quite a hassle because it means communicating directly with the USB driver, which is done using their Driver Development Kit. I couldn’t download it until just now; and without the driver kit, I could only get my own joystick working, which is why I don’t release the current build for you.

Anyway: There is no physics; what you see is just the joystick’s throttle interpreted as movement speed. As you see, the sky is not correct because I’m not done with decoupling geometry transformations from controls yet. I need to complete the fixed-point number implementation or otherwise, steering becomes very unaccurate at some points in the world.

In parallel, the re-implementation of the SSD loader does a good job. I hope I can suck out collision information and drawing order soon.

Finally: My moving went well. I’ll hopefully have more time soom when my girlfriend’s holidays are over 😉

I’m online at work, but I cannot log in; i.e., I will not be able to read any posts in the Air Dominance Forum before weekend and I cannot post anything anywhere before weekend. I won’t be able to access our box either. If there is feedback, instructions, or new discoveries you’d like to share with me, just hijack the public 3View thread, I’ll surely read it then.

See you next weekend!

P.S.: I don’t have the time to upload the video to YouTube or Vimeo; so, if anyone wants to do it to push public interest in TFX reverse engineering, I’d be fine with it.

Looks great!

Have you thought about using SDL (libsdl.org) for the joystick handling?

I think it looks good. What I need to know is, what is the issue with the skybox?

bored

Looks great!

Have you thought about using SDL (libsdl.org) for the joystick handling?

My experiences with 3rd-party libraries (no matter whether they’re open source like libpng or closed-source like PhysX) have not been quite positive, and I’m trying to keep the program as streamline as possible, so I avoid foreign code wherever I can.

DrKevDog

I think it looks good. What I need to know is, what is the issue with the skybox?

I have not yet written the code that rotates the skybox according to the viewer’s orientation. Currently, it is just rendered in cockpit coordinate space. The transformation is simple and it will probably be implemented in less time than I need to explain here, but there were just other priorities 🙂

The RawInput API seems to work as expected now. There is one open issue: Although the API is guaranteed to work with Windows XP, I could not find the XP libraries in my Driver Development Kit. Therefore, the current build (I’ll upload it in a few minutes) might work on XP and Vista, but I cannot guarantee it. If one of you uses a Vista or XP system, please try it and tell me if it works fine or if it quits with an application couldn’t start error. (I’ll try and test it myself on a virtual machine on Monday, but if we find out earlier, maybe I can find help online.)

Finally: Due to some stupidity at synchronizing, my netbook misses some header files and I cannot compile 3View until I’m back at my offline home. That’s why you’ll get a slightly older build (last Wednesday) now instead of an up-to-date build on Sunday. Sorry for that.

Edit: The build is in the box.

Full stop: The API is not referenced by the EXE I uploaded, so it’s almost certainly a wrong build. It might just become uncontrollable with any other joystick than mine, so I deleted it. Sorry again. I’ll see if I can restore some of those headers, maybe then I can compile a current build here. Else, we need to wait until next weekend …

bored, I didn’t know if you had gotten the list I posted earlier, so here it is again:

8586 = nlarbs_2
10262 = t_rdr_90
8179 = tcommsbu
8313 = tcommshq
9272 = tcmbl_90
8456 = rwyendn7
10725 = nlrlab_1
10727 = nlrlab_2
10624 = rlplab_1
9256 = plrlab_1
8624 = plrlfo_1
10668 = plrvfo_1

I have been sending the filenames to you to keep, but now I could use the comprehensive list of filenames, how can I get that list?

Yes, thanks. Here is a list which may be the latest. I need to check a PC at work tomorrow to be sure.
https://www.box.com/s/413885e5a72ecd82fafa

Okay. That’s a useful list, thanks! That list contains f22data\4_cow.txt, which should be, IIRC, neily.txt.
Let me know if your work PC has a later list.

Thanks

Here is the latest list of known files, plus a categorised dump of the 500ish remaining unknowns.
https://www.box.com/s/70105b4e6032b805921d

I’ve added these today, and they are taken into account in the zip file:

f22data\mp_new.gd2
dat\0200pal.fad
iff\256\gui\multi.iff
hardware.tm\trg_19.tm
3\tasker.ini
hardware.tm\tasker.tm
hardware.tm\tasker4.tm
hardware.tm\tasker3.tm
hardware.tm\tasker1.tm
hardware.tm\refect.tm
hardware.tm\deadcam1.tm
red0600\txe_4.tm
red0200\txe_4.tm
hardware.tm\trg_5.tm
hardware.tm\trg_6.tm
hardware.tm\mob_2.tm
agp\marco.raw
hardware.tm\exp_5.tm
red2200\exp_5.tm
red1800\exp_5.tm
red1000\mob_6.tm

So, there are still 23 8-char .3 files still to identify, but here’s 6 of them:

7697 storage1
7814 tcylindr
7961 taxi_tmp
8105 taihq_18
9710 f22marco
10117 landpipe

Very nice, my hat’s off to you sir. 👍 I will work on the remaining 17. This is a timely installment, as I was working on the hardware mode files. Your tracker.ini was one of the files I was analyzing earlier yesterday. There are only 3 .ini files with the expanded [transtxtr] block and tracker.ini is one of the three. It is especially interesting in that it does not have a [dir] header and is unique in that respect. That is a point (clue?) not worth missing. Being that we now have enough files to make inferences, I would say the nomenclature is going to be of particular importance.

Eventually:
https://www.box.com/s/60f36818870c2720b517

Don’t know if it works with XP. Tested it on Windows 7 with my Joystick and with my PlayStation controller. Have fun! 🙂

Heh, great!

Can I move around the world in cockpit view? I can pitch and roll just fine, but am at some fixed point in space.

bored

Heh, great!

Can I move around the world in cockpit view? I can pitch and roll just fine, but am at some fixed point in space.

Throttle! 🙂 (or whatever your controller’s Z axis maps to)

The throttle or the twist (normally rudder) controls have no effect on my Logitech Wingman Force3D. The pitch and roll is fine (well, slow … but I assume that’s deliberate for now) and the view slewing using the hat is excellent.

From what I see on the internet, the stick should work just as I expected. Could you please open Windows’ properties on it (in Windows 7, open control panel – printers and devices(?) – double-click the joystick), and go to the calibration tab? There should be a list of all axes with current input, please tell me the name of the throttle axis.

As far as I can see it just says Throttle, but I can only open Logitech’s own GUI.

Wow. Indeed there is a throttle axis, which is specificially reserved for airplane simulation devices. However, I could not find any joystick model actually using it, so any attempt to query its value would always fail.

Now that I know the value is used, I’ll make the program consider it. Give me time until this evening for it.

Just out of curiosity: What other axes are listed? I’d really like to incorporate them as well, because it’s actually cleaner than just using X, Y and Z …

This about all I see, with the various parts of the joystick lighting up as I press any button or move any control.

I have no idea how the controls would appear at a low API level.

Here’s the current build:
https://www.box.com/s/535d2c2755c7b8466890

I implemented a quick fix, and it might not work, but that’s on purpose – I want to see if the simple solution is sufficient (always trying to keep it simple 🙂 ).

If your throttle still does not react, tell me and I’ll implement another solution tomorrow. Meanwhile, you can use gravity to glide around a bit. Have fun 🙂

Just an FYI: Saitek X52 installed on winXP machine and working relatively well with appropriate input responses. Having way too much fun just flying around and appreciating the new physics code 😁

Yes, so am I … 🙂

Great! I had severe doubts the input code would work with WinXP, but hearing this makes me very glad.

Bored, your re-packed did.dat misses colours\nvgpal.col.

(I’m always working with your re-pack because it has the shortest loading time)

OK, I need to produce a new version. Now, where did I put the code?

EDIT: … but that’s going to take some time.
In the short term, I think you can just create a colours folder and put nvgpal.col in it.

Wow, it’s been a while since I last looked at anything involving the dat file. 🙁

EF2000 doesn’t work very well when using extracted files, and since we don’t know some of the important filenames, running with the previously reconstructed did.dat resulted in crashes due to missing files.

So, I’ve rewritten the unpack/pack Python scripts to, first unpack did.dat to folders containing the known and unknown files. Then all the files are repacked into a new did.dat, not just the known files.

These scripts are not very efficient, but could provide a template for implementation in VB or C.

The unpacking script only really needs to be run once, but it would be useful for modding purposes to have a fast packing method.

The new scripts are here.

https://app.box.com/s/fwpebb86qy2zz9jzkvu8

Extract the files into a clean folder, with further details in the README.

bored

since we don’t know some of the important filenames, running with the previously reconstructed did.dat resulted in crashes due to missing files.

Could we attach a debugger to EF2000, and once it crashes, look at the call stack? The file name should be somewhere in the function parameters then …

That’s how I got the known list of names in the first place. Way back in 2007, I scripted the IDA debugger to save the filenames as they passed through a breakpoint in the game. This requires playing all game modes though, and it seems I missed some TOD/weather combinations and probably other things as well.

Alright. If all we need is that it works, just repacking the unknown names is the right solution.

Hopefully, a temporary measure. I’m fairly sure that there’s very few vital files left to find.

EF2000 v2.0’s did.dat contains 8261 files, of which I’d guess about half are not used.

bored

Hopefully, a temporary measure. I’m fairly sure that there’s very few vital files left to find.

EF2000 v2.0’s did.dat contains 8261 files, of which I’d guess about half are not used.

I used some of those to construct the Lamborghini base and some are development files but still others are question marks … very inefficient.

Do you have any idea, why? Storage was expensive back in the days, and DID were no beginners … I was surprised when I found out how much unused stuff lies around in did.dat.

We can only speculate …

My theory is that they lost control of their build system, but didn’t really care as a bigger .dat file could be used as an anti-piracy measure since floppy disks were the prevalent writable media at the time.

Yes, Microsoft did something similar with Microsoft Bob in Windows XP. They did it more clever, though – the 30 megabytes of junk were not actually copied to the user’s hard drive. Well anyway, this way DID saved much of the deverlopers’ effort from being lost in time …

🤣

Also, a bigger install implies a better game.

Bored, your repacked did.dat misses samples/sfx/taxi.wav 🙁

That file doesn’t appear in the TAW2.20 files that I used to build the dat.

That did.dat must be years old. I guess I need to create a fresh one …

Do you have any idea where to find the F-22 engine sound? samples/sfx/mixeng.wav is close, but it’s with afterburner mixed in 🙁

I’ve just looked through f22.ssd and as I understand it, the only engine type sounds indexed are mixing.wav and taxi.wav.

Krishty

Bored, your repacked did.dat misses samples/sfx/taxi.wav 🙁

Here’s a repack containing that file:

https://app.box.com/s/jvropwvl8xm7rodmc7m4