[PATCH] Add support for AVX

Hello all, I am representing a group in Sony that wishes to submit the patch below to add the described feature to AVX. I know this is somewhat unconventional, but I am not the original author of the patch, nor do I have a build or run environment for BSD. However, I have been asked to be a proxy for our BSD developer to help shepherd this patch into FreeBSD. If you have questions about the patch, or issues with it, I will relay the information to the original author and try to resolve any problems. As I am new to FreeBSD, I'm not sure the correct individual or group to direct this patch to, but hopefully someone can provide me some pointers. I am starting with freebsd-amd64, but maybe this should be sent to freebsd-hackers??? Thanks for any assistance you can provide. -- Tim Bird, Principal Software Engineer Sony Network Entertainment --------------------------- This is a patch that provides support for AVX in FreeBSD. After it is applied to FreeBSD 9.0-CURRENT, user programs on it can use AVX fully and also the kernel may use it by calling fpu_kern_enter(). AVX is the newest SIMD extension for x86 CPUs introduced from Intel's Sandy Bridge and also will be introduced by AMD's next generation processor Bulldozer. This patch can be applied to FreeBSD 9.0-CURRENT r217297. (r217297 is a revision of subversion in the repository http://svn.freebsd.org/base) You can use clang to make a kernel with it as well as gcc. The kernel runs on both amd64 processors either AVX has or not without any configurations. The strategy of the patch is the same as for FPU(MMX)/SSE. Namely, code that initializes FPU(MMX)/SSE also initializes AVX, when it saves them it also saves AVX as well, and restores it too only if the processor has AVX. If the processor doesn't have AVX, it does nothing and it only adds a test instruction and a conditional branch instruction. Actually it does the followings; - When a processor boots if the processor has AVX then enable its use. at sys/amd64/amd64/initcpu.c - Save AVX context when a thread switching occurs. at sys/amd64/amd64/cpu_switch.S - Change fxsave()/fxrstor() to xsave()/xrstor() in order to save/restore AVX status whenever it saves/restores FPU(MMX)/SSE. at sys/amd64/amd64/fpu.c - Initializes AVX. fpuinit() at sys/amd64/amd64/fpu.c - Adjust an alignment of xsave/xrstor's target addrress in order to avoid GP exception due to miss alignment. xsave/xrstor instructions require that a boundary of a target address is 64B. But the alignment of a stack is 32B. So fpugetregs()/fpusetregs() adjust it via a bounce buffer before they issue xsave/xrstor instruction. fpugetregs()/fpusetregs() at sys/amd64/amd64/fpu.c There are two issues in the patch; 1. Machine code is embedded The default assembler for building a kernel, as of 2.15 doesn't know instructions which is added for AVX. So, the patch embeds machine code for xsetbv/xsave/xrstor instruction. 2. Extra data copying If an alignment of an address of a stack is less than 64B, in order to avoid GP exception, it copies contents to/from a bounce buffer. If callers of fpugetregs()/fpusetregs() can be changed, this extra data coping is removed. Because of the callers are MI, I hesitated about changing them. Thanks, Index: sys/amd64/include/ucontext.h =================================================================== --- sys/amd64/include/ucontext.h (revision 217297) +++ sys/amd64/include/ucontext.h (working copy) @@ -87,7 +87,7 @@ /* * See <machine/fpu.h> for the internals of mc_fpstate[]. */ - long mc_fpstate[64] __aligned(16); + long mc_fpstate[104] __aligned(64); __register_t mc_fsbase; __register_t mc_gsbase; Index: sys/amd64/include/fpu.h =================================================================== --- sys/amd64/include/fpu.h (revision 217297) +++ sys/amd64/include/fpu.h (working copy) @@ -51,6 +51,11 @@ u_char xmm_bytes[16]; }; +/* Contents of each AVX extended accumulator */ +struct ymmacc { + u_char ymm_bytes[16]; +}; + struct envxmm { u_int16_t en_cw; /* control word (16bits) */ u_int16_t en_sw; /* status word (16bits) */ @@ -71,7 +76,9 @@ } sv_fp[8]; struct xmmacc sv_xmm[16]; u_char sv_pad[96]; -} __aligned(16); + u_char xsv_hd[64]; + struct ymmacc sv_ymm[16]; +} __aligned(64); #ifdef _KERNEL struct fpu_kern_ctx { Index: sys/amd64/include/signal.h =================================================================== --- sys/amd64/include/signal.h (revision 217297) +++ sys/amd64/include/signal.h (working copy) @@ -98,7 +98,7 @@ */ long sc_fpformat; long sc_ownedfp; - long sc_fpstate[64] __aligned(16); + long sc_fpstate[104] __aligned(64); long sc_fsbase; long sc_gsbase; Index: sys/amd64/include/specialreg.h =================================================================== --- sys/amd64/include/specialreg.h (revision 217297) +++ sys/amd64/include/specialreg.h (working copy) @@ -66,6 +66,7 @@ #define CR4_PCE 0x00000100 /* Performance monitoring counter enable */ #define CR4_FXSR 0x00000200 /* Fast FPU save/restore used by OS */ #define CR4_XMM 0x00000400 /* enable SIMD/MMX2 to use except 16 */ +#define CR4_XSAVE 0x00040000 /* enable XSETBV/XGETBV */ /* * Bits in AMD64 special registers. EFER is 64 bits wide. @@ -134,6 +135,9 @@ #define CPUID2_MOVBE 0x00400000 #define CPUID2_POPCNT 0x00800000 #define CPUID2_AESNI 0x02000000 +#define CPUID2_XSAVE 0x04000000 +#define CPUID2_OSXSAVE 0x08000000 +#define CPUID2_AVX 0x10000000 /* * Important bits in the Thermal and Power Management flags Index: sys/amd64/amd64/initcpu.c =================================================================== --- sys/amd64/amd64/initcpu.c (revision 217297) +++ sys/amd64/amd64/initcpu.c (working copy) @@ -142,6 +142,19 @@ } } + +static void +xsetbv(unsigned int index, unsigned long value) +{ + register_t eax = value; + register_t edx = value >> 32; + + /* xsetbv */ + __asm __volatile(".byte 0x0f,0x01,0xd1" :: + "a"(eax), "d"(edx), "c"(index)); +} + + /* * Initialize CPU control registers */ @@ -154,6 +167,11 @@ load_cr4(rcr4() | CR4_FXSR | CR4_XMM); cpu_fxsr = hw_instruction_sse = 1; } + if (cpu_feature2 & CPUID2_XSAVE) { + load_cr4(rcr4() | CR4_XSAVE); + /* enable to use xsave/xrstor */ + xsetbv(0, 7); + } if ((amd_feature & AMDID_NX) != 0) { msr = rdmsr(MSR_EFER) | EFER_NXE; wrmsr(MSR_EFER, msr); Index: sys/amd64/amd64/cpu_switch.S =================================================================== --- sys/amd64/amd64/cpu_switch.S (revision 217297) +++ sys/amd64/amd64/cpu_switch.S (working copy) @@ -115,7 +115,20 @@ jne 1f movq PCB_SAVEFPU(%r8),%r8 clts + + /* if the cpu has AVX, use xsave instead of fxsave */ + testl $CPUID2_XSAVE,cpu_feature2 + jne 2f fxsave (%r8) + jmp 3f +2: + mov $0x7, %eax + mov %rdx, %rbx + mov $0x0, %edx + /* xsave (%r8) fix me when xsave is available */ + .byte 0x41, 0x0f, 0xae, 0x20 + mov %rbx, %rdx +3: smsw %ax orb $CR0_TS,%al lmsw %ax @@ -302,8 +315,80 @@ * Update pcb, saving current processor state. */ ENTRY(savectx) + /* Fetch PCB. */ + movq %rdi,%rcx + /* Save caller's return address. */ movq (%rsp),%rax + movq %rax,PCB_RIP(%rcx) + + movq %cr3,%rax + movq %rax,PCB_CR3(%rcx) + + movq %rbx,PCB_RBX(%rcx) + movq %rsp,PCB_RSP(%rcx) + movq %rbp,PCB_RBP(%rcx) + movq %r12,PCB_R12(%rcx) + movq %r13,PCB_R13(%rcx) + movq %r14,PCB_R14(%rcx) + movq %r15,PCB_R15(%rcx) + + /* + * If fpcurthread == NULL, then the fpu h/w state is irrelevant and the + * state had better already be in the pcb. This is true for forks + * but not for dumps (the old book-keeping with FP flags in the pcb + * always lost for dumps because the dump pcb has 0 flags). + * + * If fpcurthread != NULL, then we have to save the fpu h/w state to + * fpcurthread's pcb and copy it to the requested pcb, or save to the + * requested pcb and reload. Copying is easier because we would + * have to handle h/w bugs for reloading. We used to lose the + * parent's fpu state for forks by forgetting to reload. + */ + pushfq + cli + movq PCPU(FPCURTHREAD),%rax + testq %rax,%rax + je 1f + + movq TD_PCB(%rax),%rdi + leaq PCB_SAVEFPU(%rdi),%rdi + clts + + /* if the cpu has AVX, use xsave instead of fxsave */ + testl $CPUID2_XSAVE,cpu_feature2 + jne 2f + fxsave (%rdi) + jmp 3f +2: + mov $0x7, %eax + mov $0x0, %edx + /* xsave (%rdi) fix me when xsave is available */ + .byte 0x0f, 0xae, 0x27 +3: + smsw %ax + orb $CR0_TS,%al + lmsw %ax + + movq $PCB_SAVEFPU_SIZE,%rdx /* arg 3 */ + leaq PCB_SAVEFPU(%rcx),%rsi /* arg 2 */ + /* arg 1 (%rdi) already loaded */ + call bcopy +1: + popfq + + ret +END(savectx) + +/* + * savectx2(xpcb) + * Update xpcb, saving current processor state. + */ +ENTRY(savectx2) + /* Fetch XPCB. */ + movq %rdi,%r8 + /* Save caller's return address. */ + movq (%rsp),%rax movq %rax,PCB_RIP(%rdi) movq %rbx,PCB_RBX(%rdi) @@ -355,7 +440,17 @@ str PCB_TR(%rdi) clts + /* if the cpu has AVX, use xsave instead of fxsave */ + testl $CPUID2_XSAVE,cpu_feature2 + jne 1f fxsave PCB_USERFPU(%rdi) + jmp 2f +1: + mov $0x7, %eax + mov $0x0, %edx + /* xsave PCB_USERFPU(%rdi) fix me when xsave is available */ + .byte 0x0f, 0xae, 0xa7, 0x00, 0x01, 0x00, 0x00 +2: movq %rsi,%cr0 /* The previous %cr0 is saved in %rsi. */ movl $1,%eax Index: sys/amd64/amd64/fpu.c =================================================================== --- sys/amd64/amd64/fpu.c (revision 217297) +++ sys/amd64/amd64/fpu.c (working copy) @@ -105,6 +105,40 @@ static struct savefpu fpu_initialstate; + +static void +xsave(struct savefpu *addr) +{ + if (cpu_feature2 & CPUID2_XSAVE) { + /* __asm("xsave %0" : "=m" (*(addr))) -> xsave (%rdi) */ + __asm __volatile("mov $0x7, %%eax

" + "mov $0x0, %%edx

" + ".byte 0x0f, 0xae, 0x27" + :: "D"(addr) + : "%eax", "%edx", "memory"); + } else { + fxsave(addr); + } +} + + +static void +xrstor(struct savefpu *addr) +{ + if (cpu_feature2 & CPUID2_XSAVE) { + /* __asm("xrstor %0" : : "m" (*(addr))) -> xrstor (%rdi) */ + __asm __volatile("mov $0x7, %%eax

" + "mov $0x0, %%edx

" + ".byte 0x0f, 0xae, 0x2f" + :: "D"(addr) + : "%eax", "%edx"); + } else { + fxrstor(addr); + } +} + + + /* * Initialize the floating point unit. On the boot CPU we generate a * clean state that is used to initialize the floating point unit when @@ -128,13 +162,14 @@ mxcsr = __INITIAL_MXCSR__; ldmxcsr(mxcsr); if (PCPU_GET(cpuid) == 0) { - fxsave(&fpu_initialstate); + xsave(&fpu_initialstate); if (fpu_initialstate.sv_env.en_mxcsr_mask) cpu_mxcsr_mask = fpu_initialstate.sv_env.en_mxcsr_mask; else cpu_mxcsr_mask = 0xFFBF; bzero(fpu_initialstate.sv_fp, sizeof(fpu_initialstate.sv_fp)); bzero(fpu_initialstate.sv_xmm, sizeof(fpu_initialstate.sv_xmm)); + bzero(fpu_initialstate.sv_ymm, sizeof(fpu_initialstate.sv_ymm)); } start_emulating(); intr_restore(saveintr); @@ -150,7 +185,7 @@ critical_enter(); if (curthread == PCPU_GET(fpcurthread)) { stop_emulating(); - fxsave(PCPU_GET(curpcb)->pcb_save); + xsave(PCPU_GET(curpcb)->pcb_save); start_emulating(); PCPU_SET(fpcurthread, 0); } @@ -423,7 +458,7 @@ * the PCB doesn't contain a clean FPU state. Explicitly * load an initial state. */ - fxrstor(&fpu_initialstate); + xrstor(&fpu_initialstate); if (pcb->pcb_initial_fpucw != __INITIAL_FPUCW__) fldcw(pcb->pcb_initial_fpucw); if (PCB_USER_FPU(pcb)) @@ -432,7 +467,7 @@ else set_pcb_flags(pcb, PCB_FPUINITDONE); } else - fxrstor(pcb->pcb_save); + xrstor(pcb->pcb_save); critical_exit(); } @@ -469,7 +504,26 @@ } critical_enter(); if (td == PCPU_GET(fpcurthread) && PCB_USER_FPU(pcb)) { - fxsave(&pcb->pcb_user_save); + /* + * FIXME: + * xsave requires that a boundary of a target address is 64B + * and struct savefpu has an attribute for aligned 64B. But + * gcc doesn't treat a variable on stack as 64B boundary. + * So, it uses a buffer to do xsave safely and copy them the + * target if a miss aligned address is passed. + */ + if ((unsigned long)&pcb->pcb_user_save % 64) { + char tmp[sizeof(struct savefpu) + 63]; + void *p = (void *)((unsigned long)(tmp + 63) & ~0x3f); + + bzero(((struct savefpu*)p)->xsv_hd, 64); + + xsave((struct savefpu*)p); + bcopy(p, &pcb->pcb_user_save, sizeof(struct savefpu)); + } else { + bzero(pcb->pcb_user_save.xsv_hd, 64); + xsave(&pcb->pcb_user_save); + } critical_exit(); return (_MC_FPOWNED_FPU); } else { @@ -502,7 +556,24 @@ pcb = td->td_pcb; critical_enter(); if (td == PCPU_GET(fpcurthread) && PCB_USER_FPU(pcb)) { - fxrstor(addr); + + /* In order to avoid #GP, initialize some of XSAVE.HEADER. */ + u_long *header = (u_long*)(addr->xsv_hd); + header[0] = 7; + header[1] = 0; + header[2] = 0; + + /* + * FIXME: See fpugetregs() + */ + if ((unsigned long)addr % 64) { + char tmp[sizeof(struct savefpu) + 63]; + void *p = (void *)((unsigned long)(tmp + 63) & ~0x3f); + bcopy(addr, p, sizeof(struct savefpu)); + xrstor((struct savefpu*)p); + } else { + xrstor(addr); + } critical_exit(); set_pcb_flags(pcb, PCB_FPUINITDONE | PCB_USERFPUINITDONE); } else { Index: sys/amd64/amd64/identcpu.c =================================================================== --- sys/amd64/amd64/identcpu.c (revision 217297) +++ sys/amd64/amd64/identcpu.c (working copy) @@ -275,7 +275,7 @@ "\012SSSE3" /* SSSE3 */ "\013CNXT-ID" /* L1 context ID available */ "\014<b11>" - "\015<b12>" + "\015FMA" "\016CX16" /* CMPXCHG16B Instruction */ "\017xTPR" /* Send Task Priority Messages*/ "\020PDCM" /* Perf/Debug Capability MSR */ @@ -291,7 +291,7 @@ "\032AESNI" /* AES Crypto*/ "\033XSAVE" "\034OSXSAVE" - "\035<b28>" + "\035AVX" "\036<b29>" "\037<b30>" "\040<b31>"