lnd: further abstract the top-level lnd package to support external initialization of core interfaces

[Roasbeef]

With the newer versions of lnd, we'd started to progressively refactor lnd as itself so it's more generically useable. We've recently reached the point where it's possible to initialize an lnd instance within another go process, and also generate a set of bindings to support usage of lnd within mobile deployments.

The aim of this issue is to further abstract the main lnd package with the goal of being able to:

1. Perform configuration parsing externally at will.
2. Related to channeldb/kvdb: introduce new KV-store database abstraction  #3833, be able to provide initialized versions of core interfaces when "building" up an lnd node incrementally.

Once the two steps above have been completed, a user will be able to embed lnd within another Go process or package, and also provide new implementations of core interfaces such as the Signer or newly created kvdb.Backend interface.

Potential External API

I figure the best way to provide guidance on precisely how to achieve the two goals above is to provide an example using an hypothetical interface, then proceed backwards (a la top-down design) to refactor lnd to support our target use case.

Here's an example of a user creating a new lnd instance in an new package and providing a custom implementation (really just re-directs or maybe proxies to a remote server) of the kvdb.Backend interface.

type IO struct {
    Stdout io.Writer 
    Stdin io.Reader
    Stderr io.writer
}

// ConfigBuilder is a type that represents a way to provide custom 
// implementations of base lnd interfaces as parmaters to lnd 
// initialization. The logic in `chainregistry.go` (and also server.go) as it 
// exists now will conditionally create things like the database based 
// on the default-ness of values like this. 
type ConfigBuilder func(*ChainControl)

// CustomDatabaseBuilder is an example of a builder that adds in a 
// custom DB implementation. 
func CustomDatabaseBuilder(db kvdb.Backend) func(*ChainControl) {
    return func(cc *ChainControl) {
        cc.db = db
    }
}

// This parses the main config as we normally do, but doesn't 
// yet carry out any of the steps which are found 
// today in `chainregistry.go` .
baseCfg, err := lnd.ParseMainConfig(configPath)
if err != nil {
      return err
}

// When then create a new instance that provides our own 
// database implementation and uses the standard set 
// of interfaces for stdout/stderr, etc. 
lndInstance, err := lnd.New(
    baseCfg, IO{}, CustomDatabaseBuilder(myNewDb),
)
if err != nil {
    return err
}

[roeierez]

This is nice!

Being able to use external implementations for LND internal modules will make it more adaptive to specific platforms and will allow us to improve the use experience in general.
In Breez, we modified lnd to allow us to pass such interfaces to the main function and it will be great to have such functionality built in.

Perhaps this is a good place to outline our requirements:

ChainService
Passing a ChainService implementation to LND as a core interface will enable running the chain service decoupled from LND which provides the following advantages:

• In case of Neutrino, we can synchronize in the background and not worry about another ChainService that is running in parallel.
• The ChainService passed to LND is already running and the result is a faster startup time.

ChannelDB
One requirement here is to be able to detect channels that were closed in a background task (mobile) without running LND. Currently, we do that by running a standalone ChainService and querying the channelDB for the list of opened channels the daemon is aware of.
In order to do that, we needed to instantiate a channeldb.DB structure.
Passing a channeldb.DB to the daemon will enable the implementor to query the channels state without the need to start the daemon and wait for its RPC interface to be ready.

Breez can really benefit from these architectural changes and it would be great to be involved in this process.

[cfromknecht]

As of #4286 it's now possible to configure sub loggers before starting lnd.Main, so we do have some progress on this issue. However, I'm not expecting the remainder of the issue to be fulfilled by 0.11 so I'm removing it from the milestone in the meantime.

[Roasbeef]

Related to #5486

[saubyk]

Moving to discussions. Can be reopened when taken up for development.