Yen et al. advanced a direct approach for the calculation of self-gravitational force to second-order accuracy based on uniform grid discretization. This method improves the accuracy of N-body calculation using exact integration of kernel functions and employing the Fast Fourier Transform to reduce the complexity of computation to be nearly linear. This direct approach is free of artificial boundary conditions; however, the applicability is limited by the uniform discretization of grids. We report here an advancement in the direct method with the implementation of adaptive mesh refinement and maintaining second-order accuracy, which breaks the barrier set by uniform grid discretization. The adoption of graphic process units can significantly speed up the computation and make application of this method possible for the astrophysical systems of gaseous disk galaxies and protoplanetary disks. [Full article]