Just building this image takes long enough that it might be worth pre-building it and pushing it to an image repository (e.g. DockerHub).
Part of the overhead is just downloading the base image; this can't be avoided (though will be faster if the base image has already been downloaded to the node that binder runs it on, though we have no way to control that).
Though even with the base image already downloaded, the build of this image takes several minutes (e.g. installing gap_packages requires downloading the entire GAP source tarball of several hundred megabytes, and this alone takes time).