Debugging Link

### Link: Error: Out of memory: unable to allocate resource buffer. (Error 27)  for largest module 
### Link: Errors prevented normal completion.

Well that’s not good. Especially since Link has previously been known to work and increasing the memory size didn’t help. But it’s repeatable and thus debuggable! The first step is running with --trace-tools to see if anything looks suspicious.

a065 StackSpace(093007f0)
     -> 32564
a065 StackSpace(093007f0)
     -> 32564
a065 StackSpace(093007f0)
     -> 32628
a11e NewPtr(00000000)
a11e NewPtr(00000001)
a11e NewPtr(00000001)
a002 Read(00f763ba)
     read(0008, 00f763ec, 00000400)
a002 Read(00f763ba)
     read(0008, 00f763ec, 00000400)
a9ee DECSTR68K(0000)
     NumToString(0000001b, 00fffc6a)
a11e NewPtr(00000064)
### Link: Error: Out of memory: unable to allocate resource buffer. (Error 27)  for largest module 
### Link: Errors prevented normal completion.

NewPtr with a size of 0. Quite suspicious indeed. Why would anyone do that? Let’s use the --debug debugger shell and investigate further.

a065 StackSpace(056577f0)
     -> 32628
$00F568CC   _NewPtr                                             ; A11E
Tool break: $a11e
$00F506E6   BEQ.B      $00F506EC                                ; 6704
$00F506EC   MOVE.B     $96D2(A5),D0                             ; 102D 96D2
$00F506F0   BEQ.B      $00F5074A                                ; 6758

$00F5074A   CLR.L      -(A7)                                    ; 42A7
            A7: 00fffff8

$00F5074C   MOVE.L     $971A(A5),-(A7)                          ; 2F2D 971A
            A7: 00fffff4

$00F50750   JSR        $6176(PC)                                ; 4EBA 6176
            A7: 00fffff0

$00F568C8   MOVEA.L    (A7)+,A1                                 ; 225F
            A1: 00f50754
            A7: 00fffff4

$00F568CA   MOVE.L     (A7)+,D0                                 ; 201F
            A7: 00fffff8

---------
$00F568CC   _NewPtr                                             ; A11E


] $00F568C8;l
$00F568C8   MOVEA.L    (A7)+,A1                                 ; 225F
$00F568CA   MOVE.L     (A7)+,D0                                 ; 201F
$00F568CC   _NewPtr                                             ; A11E
$00F568CE   MOVE.L     A0,(A7)                                  ; 2E88
$00F568D0   JMP        (A1) 

So this is glue code to call NewPtr with the pascal calling conventions (parameter passed on stack, return value returned on stack). $971A(A5) is the parameter (size), which is 0.

] $971A(A5);h
f70142:  00 00 00 00 00 00 00 00  01 00 00 00 00 00 00 00   ................
f70152:  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................

] d0
$00000000            0                                      %00000000

The NewPtr(1) calls are also based on that value – ((0 + 7) / 8) + 1 = 1

$00F5074A   CLR.L      -(A7)                                    ; 42A7
$00F5074C   MOVE.L     $971A(A5),-(A7)                          ; 2F2D 971A
$00F50750   JSR        NewPtrGlue                               ; 4EBA 6176
$00F50754   MOVE.L     (A7)+,$F084(A5)                          ; 2B5F F084
$00F50758   MOVE.L     $971A(A5),D0                             ; 202D 971A
$00F5075C   ADDQ.L     #$7,D0                                   ; 5E80
$00F5075E   DIVS.L     #$00000008,D0                            ; 4C7C 0800 00000008
$00F50766   ADDQ.L     #$1,D0                                   ; 5280
$00F50768   MOVE.L     D0,$963E(A5)                             ; 2B40 963E
$00F5076C   CLR.L      -(A7)                                    ; 42A7
$00F5076E   MOVE.L     $963E(A5),-(A7)                          ; 2F2D 963E
$00F50772   JSR        NewPtrGlue                               ; 4EBA 6154
$00F50776   MOVE.L     (A7)+,$F07E(A5)                          ; 2B5F F07E
$00F5077A   CLR.L      -(A7)                                    ; 42A7
$00F5077C   MOVE.L     $963E(A5),-(A7)                          ; 2F2D 963E
$00F50780   JSR        NewPtrGlue                               ; 4EBA 6146
$00F50784   MOVE.L     (A7)+,$96D4(A5)                          ; 2B5F 96D4
$00F50788   TST.L      $F07E(A5)                                ; 4AAD F07E
$00F5078C   SEQ.B      D0                                       ; 57C0
$00F5078E   TST.L      $96D4(A5)                                ; 4AAD 96D4
$00F50792   SEQ.B      D1                                       ; 57C1
$00F50794   OR.B       D1,D0                                    ; 8001
$00F50796   TST.L      $F084(A5)                                ; 4AAD F084
$00F5079A   SEQ.B      D1                                       ; 57C1
$00F5079C   OR.B       D1,D0                                    ; 8001
$00F5079E   BEQ.B      $00F507B0                                ; 6710
$00F507A0   MOVE.W     #$001B,-(A7)                             ; 3F3C 001B
$00F507A4   PEA        $0186(PC)                                ; 487A 0186
$00F507A8   MOVE.W     #$0004,-(A7)                             ; 3F3C 0004
$00F507AC   JSR        REPORTONEERROR                           ; 4EBA 127C

But why is it 0? I set a watchpoint and it gets set to 0…

$00F62268   MOVE.L     D0,$971A(A5)                             ; 2B40 971A

… but that’s in a routine called INITLINKER. And nobody else touches it until the NewPtr(0) call.

] pc;i
$00f6226c     16130668              %00000000111101100010001001101100
Handle $00f7c018 Pointer: $00f60000 Size: $00003854 Flags: L   R
Routine: INITLINKER+$c6

Let’s DumpCode the binary and see where else it gets set ($971A is -$68E6)

mpw DumpCode ~/mpw/Tools/Link | fgrep -A 20 -B 20 -- "-\$68E6(A5)"

000007E4: 206B 0008       ' k..'          MOVEA.L  $0008(A3),A0        
000007E8: 52A8 002E       'R...'          ADDQ.L   #$1,$002E(A0)       
000007EC: 1007            '..'            MOVE.B   D7,D0               
000007EE: 670E            'g.'            BEQ.S    *+$0010             ; 000007FE
000007F0: 42A7            'B.'            CLR.L    -(A7)               
000007F2: 2F2C 001C       '/,..'          MOVE.L   $001C(A4),-(A7)     
000007F6: 4EBA 0D90       'N...'          JSR      *+$0D92             ; 00001588
000007FA: 275F FFE4       ''_..'          MOVE.L   (A7)+,-$001C(A3)    
000007FE: 4CDF 18F0       'L...'          MOVEM.L  (A7)+,D4-D7/A3/A4   
00000802: 4E5E            'N^'            UNLK     A6                  
00000804: 4E74 0008       'Nt..'          RTD      #$0008              
00000808: 8941 4C4C       '.ALL'          PACK     D1,D4,#$4C4C        ; ALLOCENTS

0000080C: 4F43            'OC'            DC.W     $4F43               ; ????
0000080E: 454E            'EN'            DC.W     $454E               ; ????
00000810: 5453            'TS'            ADDQ.W   #$2,(A3)            
00000812: 0000 4E56       '..NV'          ORI.B    #$4E56,D0           ; 'V'
00000816: FFFC            '..'            DC.W     $FFFC               ; ????
00000818: 48E7 0018       'H...'          MOVEM.L  A3/A4,-(A7)         
0000081C: 286E 0008       '(n..'          MOVEA.L  $0008(A6),A4        
00000820: 202C 0042       ' ,.B'          MOVE.L   $0042(A4),D0        
00000824: B0AD 971A       '....'          CMP.L    -$68E6(A5),D0       
00000828: 6F06            'o.'            BLE.S    *+$0008             ; 00000830
--
00000828: 6F06            'o.'            BLE.S    *+$0008             ; 00000830
0000082A: 2B6C 0042 971A  '+l.B..'        MOVE.L   $0042(A4),-$68E6(A5) 
00000830: 296D 98D8 003A  ')m...:'        MOVE.L   -$6728(A5),$003A(A4) 
00000836: 202C 0042       ' ,.B'          MOVE.L   $0042(A4),D0        
0000083A: D1AD 98D8       '....'          ADD.L    D0,-$6728(A5)       
0000083E: 302C 0028       '0,.('          MOVE.W   $0028(A4),D0        
00000842: 41ED 9A12       'A...'          LEA      -$65EE(A5),A0       
00000846: E540            '.@'            ASL.W    #$2,D0              
00000848: 2670 0000       '&p..'          MOVEA.L  $00(A0,D0.W),A3     
0000084C: 422B 0030       'B+.0'          CLR.B    $0030(A3)           
00000850: 4CDF 1800       'L...'          MOVEM.L  (A7)+,A3/A4         
00000854: 4E5E            'N^'            UNLK     A6                  
00000856: 4E74 0004       'Nt..'          RTD      #$0004              
0000085A: 8941 4C4C       '.ALL'          PACK     D1,D4,#$4C4C        ; ALLOCDATA
0000085E: 4F43            'OC'            DC.W     $4F43               ; ????
00000860: 4441            'DA'            NEG.W    D1                  
00000862: 5441            'TA'            ADDQ.W   #$2,D1              
00000864: 0000 4E56       '..NV'          ORI.B    #$4E56,D0           ; 'V'
00000868: FFE2            '..'            DC.W     $FFE2               ; ????
0000086A: 48E7 0F18       'H...'          MOVEM.L  D4-D7/A3/A4,-(A7)   
0000086E: 422E FFE3       'B...'          CLR.B    -$001D(A6)          
00000872: 286E 0008       '(n..'          MOVEA.L  $0008(A6),A4   

So it’s set in a routine called ALLOCDATA, which is never executed. Interesting. One thing I didn’t mention is that this error came while linking a Desk Accessory, which is a driver. My previous usage was for normal applications. If we give Link a -P (for progress) flag, we learn this important fact:

Size of global data area: 0

So Link pointlessly allocates 3 pointers when there is no data. When I wrote the NewPtr code, I figured allocating a 0-length pointer was a rather silly thing to do so it just returned NULL in that case. Link was not amused. Updating NewPtr to allocate memory for 0-sized requests fixed the bug.