[QUESTION] Why do process handles keep increasing.

[yjs112233]

[describe]
--version , demo download from this.
--start from bt.cli.CliClient.
--After running multiple instances[new CliClient()] in the same process and all in shutdown, the handle still exists. Maybe somewhere forgot to close stream.
--other. A large number of connections were found in CLOSE_WAIT state.

If this is a problem, how can I solve it to keep the program running consistently?
Thanks.

[pyckle]

Hey @yjs112233

CliClient is designed to be used from the command line, not programatically. The runtime shutdown is disabled:

bt/bt-cli/src/main/java/bt/cli/CliClient.java

Line 139 in 7a141ed

.disableAutomaticShutdown()

Is there a reason why you're using the CliClient class rather than directly integrating with BtClient/BtRuntime?

[yjs112233]

@pyckle

Thank you very much for paying attention to this problem that has bothered me for several months.
In fact, I've been using BitTorrent for almost a year. At first, I integrated it into my application to receive BT file parsing and download. However, after integration, I found that the same BT file cannot be received in the same BtRuntime, and it will stop running because it has been registered. Therefore, I changed the integration mode. The parsing of a BT file will start a BtRuntime and mount a unique BtClient (even though BtRuntime can attch many BtClients in design). After the BT download is completed, let it close itself.
According to this idea, on the BtRuntime configuration, do not use .disableAutomaticShutdown() so that it will shutdown automatically when it is completed. Each BtRuntime uses the same pair of port numbers 6891 and 49001.
Here is my BtRuntime configuration:

public BtRuntime getBtRuntime(){
BtRuntimeBuilder runtimeBuilder = getBtRuntimeBuilder(6891,49001);
BtRuntime runtime = runtimeBuilder.build();
runtime.startup();
return runtime;
}

private BtRuntimeBuilder getBtRuntimeBuilder(int port, int dhtPort){
    return BtRuntime.builder(this.getConfig(port))
                                 .module(this.getDHTMpdule(dhtPort))
                                 .autoLoadModules();
}
private Config getConfig(int port){
Config config = new Config() {
@OverRide
public InetAddress getAcceptorAddress() {
return super.getAcceptorAddress();
}

        @Override
        public int getAcceptorPort() {
            return port;
        }

        @Override
        public int getNumOfHashingThreads() {
            return Runtime.getRuntime().availableProcessors() * 2;
        }

        @Override
        public EncryptionPolicy getEncryptionPolicy() {
            return EncryptionPolicy.REQUIRE_ENCRYPTED;
        }
    };
    return config;
}
private Module getDHTMpdule(int dhtPort){
Module dhtModule = new DHTModule(new DHTConfig() {
@OverRide
public int getListeningPort() {
return dhtPort;
}

        @Override
        public boolean shouldUseRouterBootstrap() {
            return true;
        }
    });
    return dhtModule;
}

Here is my BtClient configuration:

public BtClient torrentClient(Torrent torrent,String id, List fileList, BtRuntime btRuntime){
if (fileList == null || fileList.isEmpty()){
throw new TorrentException(TorrentResultEnum.SELECTOR_IS_NULL);
}
String magnet = TorrentUtils.toMagnetLink(torrent);
magnet = addLatestTrackers(magnet);
magnet = TorrentUtils.magnetTrim(magnet);
BtFileSelector selector = new BtFileSelector(fileList);
BtClientBuilder builder = Bt.client(btRuntime)
.fileSelector(selector)
.magnet(magnet)
.storage(this.getStorage(id))
.randomizedRarestSelector();
return builder.build();
}

At first everything was normal. And it can support multiple tasks at the same time.
However, with the gradual increase of tasks, more and more too many open files exceptions appear in my application.
For further confirmation, I only used bt.cli.cliclient. To reproduce this problem, and found that it was the same.
Back to the above question, is it my configuration error or a bug in BitTorrent.

[yjs112233]

I use BT version 1.9 from maven repository, the self-contained org. Eclipse. Jetty is excluded. And additional integration with org.eclipse.jetty version： 8.2.0.v20160908；

[atomashpolskiy]

Sorry, it's not entirely clear from your message -- is there only one instance of runtime/client active at each given time? I.e. are you saying that the runtime leaves files/connections open after having shut down? Because in the case of multiple runtimes/clients you can easily hit the OS limit on the number of simultaneously open file descriptors and will have to increase this limit or re-think your application's pipeline.

…

On Tue, 5 Oct 2021, 14:58 yjs112233, ***@***.***> wrote: I use BT version 1.9 from maven repository, the self-contained org. Eclipse. Jetty is excluded. And additional integration with org.eclipse.jetty version： 8.2.0.v20160908； — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#185 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4TJBVP5D3LO4QNGHPBYILUFLY6FANCNFSM5CTTJWNQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[yjs112233]

is there only one instance of runtime/client active at each given time?
-- yes ! But there may be multiple instances at the same time.
eg:
Here is a file named A.torrent. My application will get a BtRuntime instance through the method .getBtRuntime(), create a BtClient with this BtRuntime instance, and start parsing and downloading the contents of A.torrent.
At the same time, B.torrent will get a new BtRuntime and a new BtClient in the same way, and start parsing and downloading the content of B.torrent.
C.torrent 、D.torrent、more and more....
Each BtRuntine is configured to allow automatic shutdown. When the download is complete, I confirm through the log that it has been closed,

End: After running some torrent files like this, and wait until all BtRuntime instance has been closed, A large number of connections were found in CLOSE_WAIT state. If more torrent files are run, the application will throw too many open files.

[pyckle]

However, after integration, I found that the same BT file cannot be received in the same BtRuntime, and it will stop running because it has been registered.

If this is the case, using the latest snapshot build should fix your issue.
This is the same bug as #146 and I believe this was fixed 1ad9d4c

Looking through the code briefly, I noticed the following

• PeerConnectionPool does not close connections for torrents that are stopped - this seems like a bug. When a torrent is stopped, we should close all connections to peers of that torrent. When I find some time, I will try to fix this.
• PeerConnectionPool creates an executor in its constructor that is never used. PR to fix this remove executor pool that was moved to ConnectionSource #193

Neither of these address your issue, but they probably should be fixed. I would suggest as a next step to better understand this issue

• See if this reproduces on the latest code in github - many bugs have been fixed since 1.9
• Enable trace logging in bt.net.SocketPeerConnection - see if close is called on the peers as the runtimes finish. If close is called, the bug is in the close code. if close is not called, then the bug is in the client shutdown behavior
• Manually make your app run GC a few time (to ensure finalizers are called) via jcmd: https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr006.html jcmd GC.run
• If the GCs make these connections go away, it's definitely an issue with not calling close somewhere.

[yjs112233]

thanks !

[yjs112233]

Hi!I am back.
I downloaded the latest code and just added the code of #193, Now the application can work persistently.
It looks as if the leak has been fixed, The number of close_waits becomes normal.
On question #146, It does solve the problem of duplicate registrations.The fly in the ointment,The same TorrentId does not support running simultaneously in the same BtRuntime. If it does, BT will be better off.

[pyckle]

Now the application can work persistently.
It looks as if the leak has been fixed, The number of close_waits becomes normal.

Excellent (:

The fly in the ointment,The same TorrentId does not support running simultaneously in the same BtRuntime. If it does, BT will be better off.

Can you explain why you have this use case? Why do you want to download a torrent multiple times in the same runtime?

From a protocol perspective, this can't be implemented because if an incoming peer connects to the torrent, the runtime won't know which instance to associate it with.

[yjs112233]

If BT is used as an service, two different people may want to download the same Torrent and it may happen at the same time.
If the protocol does not satisfy this idea, it may be implemented only through creating a new BtRuntime

[pyckle]

There are two solutions that I see for this use case.

• Create a new runtime for each user. This is the easiest solution.
• Use the same runtime, and if the torrent is already in the runtime, share the torrent between the two users, and delete it once no user wants it (reference counting).

The second solution is technically superior, but more coding, and some bt changes. If there are additional tracker(s) (announce/announce-list) in the torrent loaded second, those should be somehow added to the torrent which is downloading. Some of the work has been done for this PeerRegistry.addPeerSource(), but more work would need to be done to allow this API to be easily externally accessible.

Note, this second solution won't work properly with private trackers that track download/upload amounts because they often put a unique identifier in the announce url.

[yjs112233]

Thank you for helping me solve these lssues !

[pyckle]

Sure 👍

[atomashpolskiy]

Another way would be to implement a storage that can be re-used by concurrent users without having to create a new BtClient instance. E.g. sharing a single storage among multiple user requests.

…

On Wed, 13 Oct 2021, 09:29 pyckle, ***@***.***> wrote: Sure 👍 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#185 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4TJBXFYBSG6SALSUFSZXTUGUYOFANCNFSM5CTTJWNQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.