We developed a multiscale approach to explore the effective thermal conductivity of polycrystalline graphene sheets. By performing equilibrium molecular dynamics (EMD) simulations, the grain size effect on the thermal conductivity of ultra-fine grained polycrystalline graphene sheets is investigated. Our results reveal that the ultra-fine grained graphene structures have thermal conductivity one order of magnitude smaller than that of pristine graphene. Based on the information provided by the EMD simulations, we constructed finite element models of polycrystalline graphene sheets to probe the thermal conductivity of samples with larger grain sizes. Using the developed multiscale approach, we also investigated the effects of grain size distribution and thermal conductivity of grains on the effective thermal conductivity of polycrystalline graphene. The proposed multiscale approach on the basis of molecular dynamics and finite element methods could be used to evaluate the effective thermal conductivity of polycrystalline graphene and other 2D structures.