[Bug]: Symfony Command "sh" shell_exec error spans

[orlandothoeny]

Bug report

We experience the same issue as described in #2589.
But I cannot reopen that bug, so I'm creating a new one.

We have error spans with operation=command_execution and resource=sh in our Symfony Command traces. These would be no issue, if we could just remove them with a sampling rate of 0 using tracer sampling rules. But that does not work, because we use the DD_APM_FEATURES=error_rare_sample_tracer_drop setting (Docs).

If I understand the documentation correctly, we would loose errors of traces that are removed by our custom dd-trace sampling rules if we disable the error_rare_sample_tracer_drop setting:

By default, spans dropped by tracing library rules or custom logic such as manual.drop are excluded under the error sampler.

So we cannot remove these sh traces at the moment.

---

I had a look if we can remove these sh spans manually. But as I understand, this is not possible. I'm not 100% sure though.

One thing I saw was the \dd_untrace('shell_exec'); function, but as I understand, this will just remove registered hooks. The code that adds the sh traces (ExecIntegration::init()) however, adds a hook that manually creates a span. So I'm not sure if using dd_untrace() would work.

Creating a posthook around shell_exec using \DDTrace\hook_method() and just removing all the error metadata from the sh span also does not work I think. I don't see a way how we could gain access to the span created by the ExecIntegration hook.

Here's some pseudocode of these thoughts:

\DDTrace\hook_method(
            'Symfony\Component\Console\Application',
            'doRun',
            null,
            /**
             * Remove errors from meta, added by the "shell_exec" hook created by {@see \DDTrace\Integrations\Exec\ExecIntegration}.
             * Because the {@see Symfony\Component\Console\Terminal::hasSttyAvailable()} method runs a shell command
             * that always returns null if no TTY is available (which is always the case on the server).
             * The "shell_exec" hook of the ExecIntegration treats null return values as errors.
             * Thus, an error span with the name "sh" is always added to commands that print something.
             * This causes issues with trace ingestion sampling.
             * Since these "shell_exec" spans are errors, the error sampler is triggered (https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_mechanisms/?tab=php#error-traces).
             * Which circumvents our sampling rate config.
             * And manually dropping/ignoring the "sh" resource does not work because we use DD_APM_FEATURES=error_rare_sample_tracer_drop,
             * because we want that all error traces are sampled, regardless of the sample rate of the resource.
             */
            function (): void {
                // Doesn't work as I understand
                \dd_untrace('shell_exec');

                // We don't have the hook $id
                remove_hook($id);
                
                // We don't have to stack to be able to edit the "sh" span
                switch_stack($shellExecStack);
               // Get "sh" span and clear the error keys from meta
            }
        );

---

Example sampling rule config (from php --ri=ddtrace; php --ri=datadog-profiling):

datadog.trace.sampling_rules => [{"resource":"foo_endpoint","sample_rate":0.1,"service":"api"}] => [{"resource":"foo_endpoint","sample_rate":0.1,"service":"api"}

The code that runs the stty 2> /dev/null shell command is the following: https://github.com/symfony/console/blob/e51498ea18570c062e7df29d05a7003585b19b88/Terminal.php#L131
This code is triggered when a Symfony Command runs.
We are using Symfony 5.4, so would be great if a fix would also cover that version if it is somehow tied to Symfony.

PHP version

8.2.27

Tracer or profiler version

0.99.1

Installed extensions

[PHP Modules]
amqp
apcu
ast
bcmath
bz2
calendar
Core
ctype
curl
date
ddappsec
ddtrace
dom
exif
FFI
fileinfo
filter
ftp
gd
gettext
hash
iconv
intl
json
libxml
mbstring
mongodb
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
random
readline
Reflection
session
shmop
SimpleXML
soap
sockets
sodium
SPL
sqlite3
standard
sysvmsg
sysvsem
sysvshm
tokenizer
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache
ddappsec
ddtrace

Output of phpinfo()

ddtrace


Datadog PHP tracer extension
For help, check out the documentation at https://docs.datadoghq.com/tracing/languages/php/
(c) Datadog 2020

Datadog tracing support => enabled
Version => 0.99.1
DATADOG TRACER CONFIGURATION => {
    "date": "2025-03-13T15:13:43Z",
    "os_name": "Linux admin-5975bddd7c-7749c 6.6.72+ #1 SMP PREEMPT_DYNAMIC Sat Feb  8 10:02:01 UTC 2025 x86_64",
    "os_version": "6.6.72+",
    "version": "0.99.1",
    "lang": "php",
    "lang_version": "8.2.13",
    "env": "dev",
    "enabled": true,
    "service": "admin",
    "enabled_cli": true,
    "agent_url": "http:\/\/datadog-agent.infrastructure.svc.cluster.local:8126",
    "debug": false,
    "analytics_enabled": false,
    "sample_rate": -1,
    "sampling_rules": [],
    "tags": [],
    "service_mapping": [],
    "distributed_tracing_enabled": true,
    "priority_sampling_enabled": true,
    "dd_version": "main-latest-6db5eff0",
    "architecture": "x86_64",
    "sapi": "cli",
    "datadog.trace.request_init_hook": "\/opt\/datadog\/dd-library\/0.99.1\/dd-trace-sources\/bridge\/dd_wrap_autoloader.php",
    "open_basedir_configured": false,
    "uri_fragment_regex": null,
    "uri_mapping_incoming": null,
    "uri_mapping_outgoing": null,
    "auto_flush_enabled": false,
    "generate_root_span": true,
    "http_client_split_by_domain": false,
    "measure_compile_time": true,
    "report_hostname_on_root_span": false,
    "traced_internal_functions": null,
    "auto_prepend_file_configured": false,
    "integrations_disabled": "default",
    "enabled_from_env": true,
    "opcache.file_cache": null
}

                               Diagnostics
Diagnostic checks => passed

Directive => Local Value => Master Value
ddtrace.disable => 0 => 0
ddtrace.cgroup_file => /proc/self/cgroup => /proc/self/cgroup
datadog.trace.sidecar_trace_sender => Off => Off
datadog.trace.request_init_hook => /opt/datadog/dd-library/0.99.1/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.99.1/dd-trace-sources/bridge/dd_wrap_autoloader.php
ddtrace.request_init_hook => /opt/datadog/dd-library/0.99.1/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.99.1/dd-trace-sources/bridge/dd_wrap_autoloader.php
datadog.trace.agent_url => no value => no value
datadog.agent_host => datadog-agent.infrastructure.svc.cluster.local => datadog-agent.infrastructure.svc.cluster.local
datadog.dogstatsd_url => no value => no value
datadog.api_key => no value => no value
datadog.distributed_tracing => On => On
datadog.dogstatsd_port => 8125 => 8125
datadog.env => dev => dev
datadog.autofinish_spans => Off => Off
datadog.trace.url_as_resource_names_enabled => On => On
datadog.http_server_route_based_naming => On => On
datadog.integrations_disabled => default => default
datadog.priority_sampling => On => On
datadog.service => admin => admin
datadog.service_name => admin => admin
datadog.service_mapping => no value => no value
datadog.tags => no value => no value
datadog.trace.global_tags => no value => no value
datadog.trace.agent_port => 0 => 0
datadog.trace.analytics_enabled => Off => Off
datadog.trace.append_trace_ids_to_logs => Off => Off
datadog.trace.auto_flush_enabled => Off => Off
datadog.trace.cli_enabled => On => On
datadog.trace.measure_compile_time => On => On
datadog.trace.debug => Off => Off
datadog.trace.enabled => On => On
datadog.instrumentation_telemetry_enabled => On => On
datadog.trace.health_metrics_enabled => Off => Off
datadog.trace.health_metrics_heartbeat_sample_rate => 0.001 => 0.001
datadog.trace.db_client_split_by_instance => Off => Off
datadog.trace.http_client_split_by_domain => Off => Off
datadog.trace.redis_client_split_by_host => Off => Off
datadog.trace.memory_limit => no value => no value
datadog.trace.report_hostname => Off => Off
datadog.trace.flush_collect_cycles => Off => Off
datadog.trace.laravel_queue_distributed_tracing => On => On
datadog.trace.remove_root_span_laravel_queue => On => On
datadog.trace.remove_autoinstrumentation_orphans => Off => Off
datadog.trace.resource_uri_fragment_regex => no value => no value
datadog.trace.resource_uri_mapping_incoming => no value => no value
datadog.trace.resource_uri_mapping_outgoing => no value => no value
datadog.trace.resource_uri_query_param_allowed => no value => no value
datadog.trace.http_url_query_param_allowed => * => *
datadog.trace.http_post_data_param_allowed => no value => no value
datadog.trace.rate_limit => 0 => 0
datadog.trace.sample_rate => -1 => -1
datadog.sampling_rate => -1 => -1
datadog.trace.sampling_rules => []
datadog.trace.sampling_rules_format => regex => regex
datadog.span_sampling_rules => [] => []
datadog.span_sampling_rules_file => no value => no value
datadog.trace.header_tags => no value => no value
datadog.trace.x_datadog_tags_max_length => 512 => 512
datadog.trace.peer_service_mapping => no value => no value
datadog.trace.peer_service_defaults_enabled => Off => Off
datadog.trace.remove_integration_service_names_enabled => Off => Off
datadog.trace.propagate_service => Off => Off
datadog.trace.propagation_style_extract => datadog,tracecontext,B3,B3 single header => datadog,tracecontext,B3,B3 single header
datadog.propagation_style_extract => datadog,tracecontext,B3,B3 single header => datadog,tracecontext,B3,B3 single header
datadog.trace.propagation_style_inject => datadog,tracecontext => datadog,tracecontext
datadog.propagation_style_inject => datadog,tracecontext => datadog,tracecontext
datadog.trace.propagation_style => datadog,tracecontext => datadog,tracecontext
datadog.trace.traced_internal_functions => no value => no value
datadog.trace.agent_timeout => 500 => 500
datadog.trace.agent_connect_timeout => 100 => 100
datadog.trace.debug_prng_seed => -1 => -1
datadog.log_backtrace => Off => Off
datadog.trace.generate_root_span => On => On
datadog.trace.spans_limit => 1000 => 1000
datadog.trace.128_bit_traceid_generation_enabled => On => On
datadog.trace.128_bit_traceid_logging_enabled => Off => Off
datadog.trace.agent_max_consecutive_failures => 3 => 3
datadog.trace.agent_attempt_retry_time_msec => 5000 => 5000
datadog.trace.bgs_connect_timeout => 2000 => 2000
datadog.trace.bgs_timeout => 5000 => 5000
datadog.trace.agent_flush_interval => 5000 => 5000
datadog.trace.agent_flush_after_n_requests => 10 => 10
datadog.trace.shutdown_timeout => 5000 => 5000
datadog.trace.startup_logs => On => On
datadog.trace.once_logs => On => On
datadog.trace.agent_retries => 0 => 0
datadog.trace.agent_debug_verbose_curl => Off => Off
datadog.trace.debug_curl_output => Off => Off
datadog.trace.beta_high_memory_pressure_percent => 80 => 80
datadog.trace.agentless => Off => Off
datadog.trace.warn_legacy_dd_trace => On => On
datadog.trace.retain_thread_capabilities => Off => Off
datadog.version => main-latest-6db5eff0 => main-latest-6db5eff0
datadog.trace.obfuscation_query_string_regexp => (?i)(?:(?:"|%22)?)(?:(?:old[-_]?|new[-_]?)?p(?:ass)?w(?:or)?d(?:1|2)?|pass(?:[-_]?phrase)?|secret|(?:api[-_]?|private[-_]?|public[-_]?|access[-_]?|secr
datadog.trace.client_ip_enabled => On => On
datadog.trace.client_ip_header => no value => no value
datadog.trace.forked_process => On => On
datadog.trace.hook_limit => 100 => 100
datadog.trace.agent_max_payload_size => 52428800 => 52428800
datadog.trace.agent_stack_initial_size => 131072 => 131072
datadog.trace.agent_stack_backlog => 12 => 12
datadog.trace.propagate_user_id_default => Off => Off
datadog.dbm_propagation_mode => full => full
datadog.trace.wordpress_additional_actions => no value => no value
datadog.trace.wordpress_callbacks => Off => Off
datadog.trace.wordpress_enhanced_integration => Off => Off
datadog.trace.otel_enabled => Off => Off
datadog.trace.log_file => no value => no value
datadog.trace.log_level => error => error
datadog.appsec.sca_enabled => Off => Off
datadog.trace.amqp_enabled => On => On
datadog.trace.amqp_analytics_enabled => Off => Off
datadog.amqp_analytics_enabled => Off => Off
datadog.trace.amqp_analytics_sample_rate => 1 => 1
datadog.amqp_analytics_sample_rate => 1 => 1
datadog.trace.cakephp_enabled => On => On
datadog.trace.cakephp_analytics_enabled => Off => Off
datadog.cakephp_analytics_enabled => Off => Off
datadog.trace.cakephp_analytics_sample_rate => 1 => 1
datadog.cakephp_analytics_sample_rate => 1 => 1
datadog.trace.codeigniter_enabled => On => On
datadog.trace.codeigniter_analytics_enabled => Off => Off
datadog.codeigniter_analytics_enabled => Off => Off
datadog.trace.codeigniter_analytics_sample_rate => 1 => 1
datadog.codeigniter_analytics_sample_rate => 1 => 1
datadog.trace.exec_enabled => On => On
datadog.trace.exec_analytics_enabled => Off => Off
datadog.exec_analytics_enabled => Off => Off
datadog.trace.exec_analytics_sample_rate => 1 => 1
datadog.exec_analytics_sample_rate => 1 => 1
datadog.trace.curl_enabled => On => On
datadog.trace.curl_analytics_enabled => Off => Off
datadog.curl_analytics_enabled => Off => Off
datadog.trace.curl_analytics_sample_rate => 1 => 1
datadog.curl_analytics_sample_rate => 1 => 1
datadog.trace.drupal_enabled => On => On
datadog.trace.drupal_analytics_enabled => Off => Off
datadog.drupal_analytics_enabled => Off => Off
datadog.trace.drupal_analytics_sample_rate => 1 => 1
datadog.drupal_analytics_sample_rate => 1 => 1
datadog.trace.elasticsearch_enabled => On => On
datadog.trace.elasticsearch_analytics_enabled => Off => Off
datadog.elasticsearch_analytics_enabled => Off => Off
datadog.trace.elasticsearch_analytics_sample_rate => 1 => 1
datadog.elasticsearch_analytics_sample_rate => 1 => 1
datadog.trace.eloquent_enabled => On => On
datadog.trace.eloquent_analytics_enabled => Off => Off
datadog.eloquent_analytics_enabled => Off => Off
datadog.trace.eloquent_analytics_sample_rate => 1 => 1
datadog.eloquent_analytics_sample_rate => 1 => 1
datadog.trace.frankenphp_enabled => On => On
datadog.trace.frankenphp_analytics_enabled => Off => Off
datadog.frankenphp_analytics_enabled => Off => Off
datadog.trace.frankenphp_analytics_sample_rate => 1 => 1
datadog.frankenphp_analytics_sample_rate => 1 => 1
datadog.trace.guzzle_enabled => On => On
datadog.trace.guzzle_analytics_enabled => Off => Off
datadog.guzzle_analytics_enabled => Off => Off
datadog.trace.guzzle_analytics_sample_rate => 1 => 1
datadog.guzzle_analytics_sample_rate => 1 => 1
datadog.trace.laminas_enabled => On => On
datadog.trace.laminas_analytics_enabled => Off => Off
datadog.laminas_analytics_enabled => Off => Off
datadog.trace.laminas_analytics_sample_rate => 1 => 1
datadog.laminas_analytics_sample_rate => 1 => 1
datadog.trace.laravel_enabled => On => On
datadog.trace.laravel_analytics_enabled => Off => Off
datadog.laravel_analytics_enabled => Off => Off
datadog.trace.laravel_analytics_sample_rate => 1 => 1
datadog.laravel_analytics_sample_rate => 1 => 1
datadog.trace.laravelqueue_enabled => On => On
datadog.trace.laravelqueue_analytics_enabled => Off => Off
datadog.laravelqueue_analytics_enabled => Off => Off
datadog.trace.laravelqueue_analytics_sample_rate => 1 => 1
datadog.laravelqueue_analytics_sample_rate => 1 => 1
datadog.trace.logs_enabled => Off => Off
datadog.logs_injection => Off => Off
datadog.trace.logs_analytics_enabled => Off => Off
datadog.logs_analytics_enabled => Off => Off
datadog.trace.logs_analytics_sample_rate => 1 => 1
datadog.logs_analytics_sample_rate => 1 => 1
datadog.trace.lumen_enabled => On => On
datadog.trace.lumen_analytics_enabled => Off => Off
datadog.lumen_analytics_enabled => Off => Off
datadog.trace.lumen_analytics_sample_rate => 1 => 1
datadog.lumen_analytics_sample_rate => 1 => 1
datadog.trace.magento_enabled => On => On
datadog.trace.magento_analytics_enabled => Off => Off
datadog.magento_analytics_enabled => Off => Off
datadog.trace.magento_analytics_sample_rate => 1 => 1
datadog.magento_analytics_sample_rate => 1 => 1
datadog.trace.memcache_enabled => On => On
datadog.trace.memcache_analytics_enabled => Off => Off
datadog.memcache_analytics_enabled => Off => Off
datadog.trace.memcache_analytics_sample_rate => 1 => 1
datadog.memcache_analytics_sample_rate => 1 => 1
datadog.trace.memcached_enabled => On => On
datadog.trace.memcached_analytics_enabled => Off => Off
datadog.memcached_analytics_enabled => Off => Off
datadog.trace.memcached_analytics_sample_rate => 1 => 1
datadog.memcached_analytics_sample_rate => 1 => 1
datadog.trace.mongo_enabled => On => On
datadog.trace.mongo_analytics_enabled => Off => Off
datadog.mongo_analytics_enabled => Off => Off
datadog.trace.mongo_analytics_sample_rate => 1 => 1
datadog.mongo_analytics_sample_rate => 1 => 1
datadog.trace.mongodb_enabled => On => On
datadog.trace.mongodb_analytics_enabled => Off => Off
datadog.mongodb_analytics_enabled => Off => Off
datadog.trace.mongodb_analytics_sample_rate => 1 => 1
datadog.mongodb_analytics_sample_rate => 1 => 1
datadog.trace.mysqli_enabled => On => On
datadog.trace.mysqli_analytics_enabled => Off => Off
datadog.mysqli_analytics_enabled => Off => Off
datadog.trace.mysqli_analytics_sample_rate => 1 => 1
datadog.mysqli_analytics_sample_rate => 1 => 1
datadog.trace.nette_enabled => On => On
datadog.trace.nette_analytics_enabled => Off => Off
datadog.nette_analytics_enabled => Off => Off
datadog.trace.nette_analytics_sample_rate => 1 => 1
datadog.nette_analytics_sample_rate => 1 => 1
datadog.trace.pcntl_enabled => On => On
datadog.trace.pcntl_analytics_enabled => Off => Off
datadog.pcntl_analytics_enabled => Off => Off
datadog.trace.pcntl_analytics_sample_rate => 1 => 1
datadog.pcntl_analytics_sample_rate => 1 => 1
datadog.trace.pdo_enabled => On => On
datadog.trace.pdo_analytics_enabled => Off => Off
datadog.pdo_analytics_enabled => Off => Off
datadog.trace.pdo_analytics_sample_rate => 1 => 1
datadog.pdo_analytics_sample_rate => 1 => 1
datadog.trace.phpredis_enabled => On => On
datadog.trace.phpredis_analytics_enabled => Off => Off
datadog.phpredis_analytics_enabled => Off => Off
datadog.trace.phpredis_analytics_sample_rate => 1 => 1
datadog.phpredis_analytics_sample_rate => 1 => 1
datadog.trace.predis_enabled => On => On
datadog.trace.predis_analytics_enabled => Off => Off
datadog.predis_analytics_enabled => Off => Off
datadog.trace.predis_analytics_sample_rate => 1 => 1
datadog.predis_analytics_sample_rate => 1 => 1
datadog.trace.psr18_enabled => On => On
datadog.trace.psr18_analytics_enabled => Off => Off
datadog.psr18_analytics_enabled => Off => Off
datadog.trace.psr18_analytics_sample_rate => 1 => 1
datadog.psr18_analytics_sample_rate => 1 => 1
datadog.trace.roadrunner_enabled => On => On
datadog.trace.roadrunner_analytics_enabled => Off => Off
datadog.roadrunner_analytics_enabled => Off => Off
datadog.trace.roadrunner_analytics_sample_rate => 1 => 1
datadog.roadrunner_analytics_sample_rate => 1 => 1
datadog.trace.sqlsrv_enabled => On => On
datadog.trace.sqlsrv_analytics_enabled => Off => Off
datadog.sqlsrv_analytics_enabled => Off => Off
datadog.trace.sqlsrv_analytics_sample_rate => 1 => 1
datadog.sqlsrv_analytics_sample_rate => 1 => 1
datadog.trace.slim_enabled => On => On
datadog.trace.slim_analytics_enabled => Off => Off
datadog.slim_analytics_enabled => Off => Off
datadog.trace.slim_analytics_sample_rate => 1 => 1
datadog.slim_analytics_sample_rate => 1 => 1
datadog.trace.swoole_enabled => On => On
datadog.trace.swoole_analytics_enabled => Off => Off
datadog.swoole_analytics_enabled => Off => Off
datadog.trace.swoole_analytics_sample_rate => 1 => 1
datadog.swoole_analytics_sample_rate => 1 => 1
datadog.trace.symfony_enabled => On => On
datadog.trace.symfony_analytics_enabled => Off => Off
datadog.symfony_analytics_enabled => Off => Off
datadog.trace.symfony_analytics_sample_rate => 1 => 1
datadog.symfony_analytics_sample_rate => 1 => 1
datadog.trace.web_enabled => On => On
datadog.trace.web_analytics_enabled => Off => Off
datadog.web_analytics_enabled => Off => Off
datadog.trace.web_analytics_sample_rate => 1 => 1
datadog.web_analytics_sample_rate => 1 => 1
datadog.trace.wordpress_enabled => On => On
datadog.trace.wordpress_analytics_enabled => Off => Off
datadog.wordpress_analytics_enabled => Off => Off
datadog.trace.wordpress_analytics_sample_rate => 1 => 1
datadog.wordpress_analytics_sample_rate => 1 => 1
datadog.trace.yii_enabled => On => On
datadog.trace.yii_analytics_enabled => Off => Off
datadog.yii_analytics_enabled => Off => Off
datadog.trace.yii_analytics_sample_rate => 1 => 1
datadog.yii_analytics_sample_rate => 1 => 1
datadog.trace.zendframework_enabled => On => On
datadog.trace.zendframework_analytics_enabled => Off => Off
datadog.zendframework_analytics_enabled => Off => Off
datadog.trace.zendframework_analytics_sample_rate => 1 => 1
datadog.zendframework_analytics_sample_rate => 1 => 1
Extension 'datadog-profiling' not present.

Upgrading from

No response

[bwoebi]

So, if I understand it right, the goal is to specifically not trace that specific span, right?

\DDTrace\install_hook('Symfony\Component\Console\Application::doRun', function(\DDTrace\HookData $hook) {
    $hook->data = \DDTrace\install_hook('DDTrace\Integrations\Exec\ExecIntegration::createSpan', function(DDTrace\HookData $hook) {
        $hook->suppressCall();
    });
}, function (\DDTrace\HookData $hook) {
    \DDTrace\remove_hook($hook->data);
});

Untested - it should prevent the span from being created at all during that function.

Something along these lines should suppress the span creation. This is a bit hacky though and I agree that there should be a better mechanism to selectively suppress spans from integrations.

Thinking that possibly disabling the integration via ini should temporarily pause the spans from that integration (and resume once re-enabled again).

[orlandothoeny]

@bwoebi Yes that's right. We only want to get rid of that specific span.

I can give that a try.

I also thought about disabling the integration. If possible, we would like to keep the exec integration, to have visibility into the useful things it tracks.

[bwoebi]

hook_method will also work, I'm mostly using install_hook to stuff the hook id into the $data property to remove_hook it afterwards. But suppressCall is specific to install_hook.

[orlandothoeny]

Yes that makes sense. I'll try it that way