Objective To explore the burden situation among caregivers of stroke survivors, and analyze the association between burden and the quality of life among caregivers. Method In this cross-sectional study, a total of 230 stroke survivor-caregivers were investigated with basic demographic information, Zarit Burden Interview (ZBI) and the World Health Organization Quality of Life Instrument-Short Form (WHOQOL-BREF) from May 2015 to November 2015. Results The mean age of the caregivers of stroke survivors was (66.7±11.7) years, and the caregiver burden was in the mild level with the mean ZBI score of 21.11±6.96. The multiple linear regression analysis showed that the influencing factors of caregiver burden were the complications, self-care ability, residence of stroke survivors, and the self-rating health status of caregivers (P<0.05). And the total ZBI score was negatively correlated with the total WHOQOL-BREF scores (P<0.01), physiological dimensional scores (P<0.01), social dimensional scores (P<0.01), and environmental dimensional scores of caregivers (P<0.01). Conclusions The caregivers of stroke survivors suffer from general caregiver burden, and the heavier caregiver burden is, the poorer the quality of life of the caregivers is. According to the different conditions between urban and rural areas of China, it is reasonable to formulate a targeted program with the consideration of requirements referring to stroke survivors and their caregivers. It should involve physiological, psychological, social, environmental factors and so on to improve the caregivers’ quality of life finally.