Pivoting on target-cpu/arch

[lilith]

We have a couple common generations of CPU above the baseline x86_64 instruction - namely sandybridge and haswell, with AVX and AVX2/BMI/BMI2 respectively.

LLVM-backed languages and GGC 4.9+ all support "x86-64, sandybridge, haswell, native" for the -march/--target-cpu parameters. GCC 4.8 uses alternate identifiers corei7-avx and core-avx2 for those platforms.

These map nicely to MSVC /arch:AVX and /arch:AVX2, which is as granular as MSVC goes.

For now I'm using an extra field in .conan/settings.yml: target_cpu: [x86, x86-64, nehalem, sandybridge, haswell, native], but I need to move this down into the packages I consume as well, if I want to pivot on sandybridge/haswell support.

Has this come up before? Any convention to adopt?

[lasote]

I have no previous experience managing those alternative architectures, but, why they are not just additional arch setting values?

I'm not sure either about the native setting, from the gcc page it says This selects the CPU to generate code for at compilation time by determining the processor type of the compiling machine It sounds like it should not be a setting value, because it's variable and not determinable.

Any community feedback would be great.

[lilith]

If there is no conditional logic based on arch, and we only support the extra instructions in 64-bit mode, then 'arch' would be fine. But do we want to exclude 32-bit pointer-sized code optimized for these platforms?

[lilith]

These instructions sets are actually available for both x86 and x86_64 architectures.

I think native would need special handling - it should be source-only.

I'm bumping up against this pretty hard with libjpeg-turbo. It's 2-3x faster when compiled for haswell vs baseline x86_64, but recompiling from source pushes Travis over the edge and hits the 45-minute timeout.

[lasote]

I would like to push this for the 0.29, some other user has required the same thing and it's time to establish a convention to follow. Initially only the base settings, later we can think about the build helpers to inject some needed flag and some detection of the CPU microarchitecture (https://pypi.python.org/pypi/cpuid @fpelliccioni) to warn if a bad setting is detected. So @memsharded, lets work on it.
@nathanaeljones I think we could:

arch: 
  x86:
       microarch: [None, "nehallem", "bonnell", "sandy_bridge", "ivy_bridge", "silvermont", "haswell", "broadwell", "skylake", "goldmont", "kaby_lake", "coffee_lake"]
  x86_64:
       microarch: [None, "nehallem", "bonnell", "sandy_bridge", "ivy_bridge", "silvermont", "haswell", "broadwell", "skylake", "goldmont", "kaby_lake", "coffee_lake"]
  ppc64le:
  ppc64:
  armv6:
  armv7:
  armv7hf:
  armv8:
  sparc:
  sparcv9:
  mips:
  mips64:
  avr:

I don't like to repeat the microarchitectures, but I don't see a better approach.
The "None" allows the user to not specify the subsetting.

[lilith]

I'd also suggest that we may want to support 'native', i.e, whichever features are supported on the build machine. This value would need to disable build caching, though - all packages would need to be built from source.

Also, we may want to consider matching the gcc/llvm names as closely as possible. For GCC < 5 we'll have to map a few anyway, though.

[lilith]

Generations are also not very specific, and may not work on mobile or low-end editions.

I started with x86_64, nehalem, sandybridge, haswell, native, but skylake should probably be added for TSX support.

I'm not sure there's much value in including tick releases unless they add new instruction sets.

[lilith]

For LLVM:

llc -march=x86 -mattr=help
llc -march=x86-64 -mattr=help

I forget how to list the values for GCC.

[tru]

For arm we are shipping a couple of different architectures right now:

• armv7 hardfloat
• armv7 softfloat
• armv7 hardfloat + neon
• armv7 hardfloat + thumb + neon

I think it makes sense for the armv7 platform to contain: float=["hard", "soft"], thumb=[True, False], neon=[True, False]

On some platforms we also need to set the the specific FPU like this: -mfpu=vfpv3-d16 not sure if that is something that should be abstracted in conan though.

[fpelliccioni]

I was thinking about it, here I leave some of my conclusions:

A. Relying on micro-architecture is a breakthrough, but ... what if I do
the following?

1. g++ xxx.cpp -O3 -march=sandybridge ...

2. g++ xxx.cpp -O3 -march=haswell ...

3. g++ xxx.cpp -O3 -march=skylake ...

   It is likely that the resulting binary files of 1, 2 and 3 are exactly
   the same, in such a case, it does not make sense to differentiate them as
   they are compatible or equal binaries/packages.

B. Some Intel micro-architectures have the same extensions as others. For
example, according to the Intel tick-tock model, in theory, Sandybridge and
Ivybridge are equivalent (with respect to sets of instructions or
extensions).
Therefore, it is not worth differentiating them.

I think, in both cases, what really matters is what sets of instructions
were used.
For this, I am working on a tool that examines an executable or library
(.a, .so, .dll, etc ...) and report which sets of instructions were used.

For example:

get_extensions("a.out",...) == ['MODE64', 'SSE', 'AVX']

In this way, the micro-architecture no longer matters, but what really
matters are the sets of instructions used.

I think Conan packages can have a setting to determine which extensions
the binaries use. For example:

class HelloConan(ConanFile):
    settings = "os", "compiler", "build_type", "arch", "extensions" [0]

extensions have to be assigned after the binary was compiled, I imagine,
creating a new method (member function) in the ConanFile class, for
example:

def set_extensions(self):
    self.extensions = get_extensions(... list of binary files ...)

On the client side, when Conan looks for a package, it can identify which
instructions are available for the processor (using the cpuid python
package, for example) and in this way find the package that best fits.

I have a demo of the tool that analyzes the executables, if the idea is of
interest/utility to the community, I could invest some time on it.
The demo for now only works for x86 and Elf format, but it can be
extensible for other architectures and formats (PE, Mach-O).