I am trying to decompile an executable for the 68000 processor into C code, replacing the original subroutines with C functions one by one.
The problem I faced is that I don't know how to make gcc use the calling convention that matches the one used in the original program. I need the parameters on the stack to be packed, not aligned.
Let's say we have the following function
int fun(char arg1, short arg2, int arg3) { return arg1 + arg2 + arg3;}
If we compile it with
gcc -m68000 -Os -fomit-frame-pointer -S source.c
we get the following output
fun: move.b 7(%sp),%d0 ext.w %d0 move.w 10(%sp),%a0 lea (%a0,%d0.w),%a0 move.l %a0,%d0 add.l 12(%sp),%d0 rts
As we can see, the compiler assumed that parameters have addresses 7(%sp)
, 10(%sp)
and 12(%sp)
:
but to work with the original program they need to have addresses 4(%sp)
, 5(%sp)
and 7(%sp)
:
One possible solution is to write the function in the following way (the processor is big-endian):
int fun(int bytes4to7, int bytes8to11) { char arg1 = bytes4to7>>24; short arg2 = (bytes4to7>>8)&0xffff; int arg3 = ((bytes4to7&0xff)<<24) | (bytes8to11>>8); return arg1 + arg2 + arg3;}
However, the code looks messy, and I was wondering: is there a way to both keep the code clean and achieve the desired result?