We assessed gene-gene and gene-physical activity interactions of polymorphisms in C-reactive protein (CRP), tumor necrosis factor-α (TNF-α), and lymphotoxin α (LTA) genes on lower extremity performance in community-dwelling elders in Taiwan. Five SNPs (rs1205, rs1130864, rs1800947, rs2794520, and rs3093059) of CRP gene, three SNPs (rs909253, rs1041981, and rs2239704) of LTA gene, and three SNPs (rs3093662, rs1800629, and rs1799964) of TNF-α gene of 472 unrelated elders were genotyped. Lower extremity performance included timed up-and-go test (TUG), walking speed, weight-adjusted leg press (waLP), and timed chair stand (TCS). We detected significant interactions between physical activity with CRP rs2794520, rs1205, and rs3093059; LTA rs909253 and rs1041981; and TNF-α rs1799964 for TCS in women after covariate adjustment (all P < 0.05). In men, significant interactions between physical activity with CRP rs2794520, rs1205, and rs3093059; and LTA rs909253 and rs1041981 for TUG; with CRP rs2794520, rs1205, rs1130864, and rs3093059; and LTA rs909253 and rs1041981 for walking speed; and with TNF-α rs3093662 for waLP after covariate adjustment (all P < 0.05). These variants also significantly interacted with physical activity on TCS in women and on walking speed in men. These results show inflammatory genes are involved in lower extremity performance, likely via gene-physical activity interactions.