SGE issues

[manuelgarvan]

Hi,

I am trying to run dkfz workflow, it has been running for a few hours but the resource usage was quite low.

some investigations pointed out that SGE is not working

roddy@master:/var/spool/cwl$ qstat
error: commlib error: got select error (Connection refused)
error: unable to send message to qmaster using port 6444 on host "master": got send error

So I restarted the sge master, but the main.q is in a 'u' state

roddy@master:/var/spool/cwl$ qstat -f
queuename                      qtype resv/used/tot. load_avg arch          states
---------------------------------------------------------------------------------
main.q@master                  BIP   0/0/2          1.27     lx26-amd64    u

I was wondering if you know any workaround for this so I can run your workflow?

NOTE: I rebooted the host server but same behaviour

thank you very much

[denis-yuen]

Hi,
Could you email me (link in my profile), I should be able to get you in direct email contact with the author of the workflow. That should be quicker than going through GitHub. Thanks!

[cruknagano]

I am also receiving the same error from quay.io/pancancer/pcawg-dkfz-workflow from
$ wget -O pcawg-dkfz-variant-caller.cwl "https://github.com/ICGC-TCGA-PanCancer/dkfz_dockered_workflows/blob/2.0.2_cwl1.0/Dockstore.cwl"

$ nohup cwltool --debug --non-strict pcawg-dkfz-variant-caller.cwl pcawg-dkfz-variant-caller.job.json > pcawg-dkfz-variant-caller.log 2>&1 &

TASK: [grid-engine | SGE | Configure master] ********************************** failed: [cd972eb32e00] => {"changed": true, "cmd": ["qconf", "-as", "cd972eb32e00"], "delta": "0:00:00.015928", "end": "2018-10-15 11:27:55.300010", "rc": 1, "start": "2018-10-15 11:27:55.284082", "stdout_lines": []} stderr: error: commlib error: got select error (Connection refused) ERROR: unable to send message to qmaster using port 6444 on host "cd972eb32e00": got send error

Following fix seems to work for me.

- name: SGE | Ensure gridengine-master is started and enabled service: name=gridengine-master state=started enabled=yes
- name: Wait for port 6444 to become open on the host, don't start checking for 10 seconds wait_for: port: 6444 delay: 10
- name: SGE | Configure master command: qconf -as {{ inventory_hostname }} register: configure_master_as changed_when: not 'already exists' in configure_master_as.stderr when: docker is not defined

Please could you add this to the file roles/grid-engine/tasks/master.yml in your repository?

Thank you

[cruknagano]

I am also receiving the same error from quay.io/pancancer/pcawg-dkfz-workflow from
$ wget -O pcawg-dkfz-variant-caller.cwl "https://github.com/ICGC-TCGA-PanCancer/dkfz_dockered_workflows/blob/2.0.2_cwl1.0/Dockstore.cwl"

$ nohup cwltool --debug --non-strict pcawg-dkfz-variant-caller.cwl pcawg-dkfz-variant-caller.job.json > pcawg-dkfz-variant-caller.log 2>&1 &

TASK: [grid-engine | SGE | Configure master] ********************************** failed: [cd972eb32e00] => {"changed": true, "cmd": ["qconf", "-as", "cd972eb32e00"], "delta": "0:00:00.015928", "end": "2018-10-15 11:27:55.300010", "rc": 1, "start": "2018-10-15 11:27:55.284082", "stdout_lines": []} stderr: error: commlib error: got select error (Connection refused) ERROR: unable to send message to qmaster using port 6444 on host "cd972eb32e00": got send error

Following fix seems to work for me.

- name: SGE | Ensure gridengine-master is started and enabled service: name=gridengine-master state=started enabled=yes
- name: Wait for port 6444 to become open on the host, don't start checking for 10 seconds wait_for: port: 6444 delay: 10
- name: SGE | Configure master command: qconf -as {{ inventory_hostname }} register: configure_master_as changed_when: not 'already exists' in configure_master_as.stderr when: docker is not defined

Please could you add this to the file roles/grid-engine/tasks/master.yml in your repository?

Thank you

[junjun-zhang]

@cruknagano, thanks for reporting this and your proposed fix.

Please feel free to fork this repository and fix it on your fork and then create a Pull Request against our repository.

[cruknagano]

I am still receiving the same error from quay.io/pancancer/pcawg-dkfz-workflow
It seems the copy command
COPY roles /root/roles
at Line138 in Dockerfile is not working due to the error, "this path skips through empty directories".
Please could you add an empty file of ( ex. .gitignore ) under ./roles/ in your repository to make Git recognise the directory ./role/?
Thank you

[cruknagano]

I am still receiving the same error from quay.io/pancancer/pcawg-dkfz-workflow
Could you manually build the docker image please, which has not been automatically processed?
I confirmed Github (ICGC-TCGA-PanCancer/dkfz_dockered_workflows) is merged but it seems docker ( quay.io/pancancer/pcawg-dkfz-workflow ) is not updated since June, according to the version record in
( https://dockstore.org/containers/quay.io/pancancer/pcawg-dkfz-workflow:develop?tab=versions )
Thank you