Skip to content
This repository has been archived by the owner on Nov 30, 2022. It is now read-only.

Cleanup in mcap_vendor package #62

Merged
merged 6 commits into from
Oct 27, 2022
Merged

Conversation

MichaelOrlov
Copy link
Member

@MichaelOrlov MichaelOrlov commented Oct 21, 2022

  • Use ${PROJECT_NAME} instead of confusing mcap for library name and exporting target, to be consistent with all other packages in ROS2.

  • Add export for path to the mcap includes and export for mcap_vendor library. It was an issue in downstream packages that mcap/mcap.hpp was not visible since we were not exporting path to it explicitly.

  • Removed test section with linters for mcap_vendor package. Since package itself doesn't contain any tests and its own code and we shouldn't run linters on any third party code. Prevent CI failures.

Signed-off-by: Michael Orlov [email protected]

@MichaelOrlov MichaelOrlov force-pushed the morlov/cleanup-in-mcap_vendor branch from ad7d88b to 70a96c8 Compare October 21, 2022 21:36
@MichaelOrlov MichaelOrlov marked this pull request as ready for review October 21, 2022 21:43
DESTINATION
${CMAKE_INSTALL_PREFIX}/include
DIRECTORY ${_mcap_include_dir}/mcap
DESTINATION ${CMAKE_INSTALL_PREFIX}/include
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While we are in here, we may as well make this follow our new recommendations to put this in a sub-sub-directory. You can change this to:

DESTINATION include/${PROJECT_NAME}

(I believe the CMAKE_INSTALL_PREFIX is implicit, and we should add ${PROJECT_NAME} to the end)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@clalancette I am not sure that I am correctly understand how do you want it to be with suggestion to replace it with DESTINATION include/${PROJECT_NAME}.
Do you want it to be the path as install/mcap_vendor/include/mcap_vendor/ and then header files from mcap ?
If so, this is actually not what we want for this package.

Currently we have destination path as install/mcap_vendor/include/mcap/ and then header files from mcap. The rational for that is that include files not from mcap_vendor package itself but rather from external mcap C++ project. And we want it to be clear and consistent.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or may be you want the path to be as install/mcap_vendor/include/mcap_vendor/mcap/ and then header files from mcap. This is possible but looks odd and I don't see a rational for this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or may be you want the path to be as install/mcap_vendor/include/mcap_vendor/mcap/ and then header files from mcap. This is possible but looks odd and I don't see a rational for this.

This is what I am going for.

There is a good rationale for it; see ros2/ros2#1150 for the gory details. In short, if you look in /opt/ros/rolling/include, you'll see that every single one of the includes in the core packages lives in a sub-sub-directory so that we don't accidentally pull in the wrong header file during underlay/overlay processing. I think we should do the same here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@clalancette Ok I changed it t o be under the ${PROJECT_NAME} i.e. install/mcap_vendor/include/mcap_vendor/mcap/

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@clalancette Apparently it doesn't work this way. Build fails with error message that mcap/mcap.hpp not found.
I guess will need to change include to point to the mcap_vendor/mcap/mcap.hpp.
I will try to debug it locally.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. changing include to the mcap_vendor/mcap/mcap.hpp helped with compilation error.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed it as amend commit

mcap_vendor/CMakeLists.txt Outdated Show resolved Hide resolved
@MichaelOrlov
Copy link
Member Author

Gist: https://gist.githubusercontent.com/MichaelOrlov/8dd65556eb06031d211447a0b0193f65/raw/86c621126e24e948807e3c86fd2a43fd1f5d1b35/ros2.repos
BUILD args: --packages-above-and-dependencies rosbag2_storage_mcap
TEST args: --packages-above rosbag2_storage_mcap
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/11024

  • Linux Build Status
  • Linux-aarch64 Build Status
  • Windows Build Status

@MichaelOrlov MichaelOrlov force-pushed the morlov/cleanup-in-mcap_vendor branch from a88c9e4 to c67f27a Compare October 25, 2022 17:56
@MichaelOrlov
Copy link
Member Author

re-run CI after fixing compilation issue.

  • Linux Build Status
  • Linux-aarch64 Build Status
  • Windows Build Status

@MichaelOrlov
Copy link
Member Author

@clalancette I've addressed your comments. Could you please reiterate on review?

- Use `${PROJECT_NAME}` instead of confusing `mcap` for library name and
exporting target, to be consistent with all other packages in ROS2.

- Add export for path to the `mcap` includes and export for mcap_vendor
library. It was an issue in downstream packages that `mcap/mcap.hpp` was
not visible since we were not exporting path to it explicitly.

- Removed test section with linters for `mcap_vendor` package. Since
package itself doesn't contain any tests and own source code and we
shouldn't run linters on any third party code. Prevent CI failures.

Signed-off-by: Michael Orlov <[email protected]>
@MichaelOrlov MichaelOrlov force-pushed the morlov/cleanup-in-mcap_vendor branch from c67f27a to f3606f9 Compare October 26, 2022 07:00
@MichaelOrlov
Copy link
Member Author

re-run Windows build after rebasing on top of rename test_fixture_interfaces package to testdata (#64)

  • Windows Build Status

@@ -35,48 +34,37 @@ macro(build_mcap_vendor)
file(GLOB _lz4_srcs
${lz4_SOURCE_DIR}/lib/*.c)

add_library(
mcap SHARED
add_library(${PROJECT_NAME} SHARED
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I really don't think you want to change the name of the library like this.

That is, the idea behind vendor packages is that they are temporary (for some definition of temporary) stand-ins for the package in question. In some day in the future, we should be able to remove the vendor package and just get the library from the underlying OS.

Thus, all of our vendor packages actually build and export the actual library name, not a "vendored" version of it. Someday if libmcap is available in Ubuntu, then we can make this package just skip building on Ubuntu.

For those reasons, I do think you should switch back to the mcap library and target name.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@clalancette Ok, fair point about libmcap. I have reverted renames.

rosbag2_storage_mcap/src/mcap_storage.cpp Outdated Show resolved Hide resolved
@MichaelOrlov
Copy link
Member Author

re-run Windows build after renames and fixing include issue.

  • Windows Build Status

Copy link
Contributor

@clalancette clalancette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, this looks how I expect, but it is still failing to build on Windows :(.

@MichaelOrlov
Copy link
Member Author

@clalancette I see.
I don't have to much ideas why library can't be found on Windows.
Trying re-run Windows build after adding LIBRARY DESTINATION lib to install targets

  • Windows Build Status

BTW could be that

 install(TARGETS mcap EXPORT mcap)

calling before

ament_export_libraries(mcap)
ament_export_targets(mcap  HAS_LIBRARY_TARGET)

??

@clalancette
Copy link
Contributor

BTW could be that

 install(TARGETS mcap EXPORT mcap)

calling before

ament_export_libraries(mcap)
ament_export_targets(mcap  HAS_LIBRARY_TARGET)

I don't think so; that seems to be the typical way we do things. As a random example that I know works on Windows, see https://github.com/ros2/geometry2/blob/rolling/tf2/CMakeLists.txt .

@MichaelOrlov
Copy link
Member Author

@clalancette It seems the problem is that mcap.dll installing in install/bin folder

mcap.vcxproj -> C:\ci\ws\build\mcap_vendor\Release\mcap.dll

  Building Custom Rule C:/ci/ws/src/ros-tooling/rosbag2_storage_mcap/mcap_vendor/CMakeLists.txt

-- Installing: C:/ci/ws/install/include/mcap_vendor/mcap
-- Installing: C:/ci/ws/install/include/mcap_vendor/mcap/crc32.hpp
-- Installing: C:/ci/ws/install/include/mcap_vendor/mcap/errors.hpp
-- Installing: C:/ci/ws/install/include/mcap_vendor/mcap/internal.hpp
-- Installing: C:/ci/ws/install/include/mcap_vendor/mcap/intervaltree.hpp
-- Installing: C:/ci/ws/install/include/mcap_vendor/mcap/mcap.hpp
-- Installing: C:/ci/ws/install/include/mcap_vendor/mcap/reader.hpp
-- Installing: C:/ci/ws/install/include/mcap_vendor/mcap/reader.inl
-- Installing: C:/ci/ws/install/include/mcap_vendor/mcap/read_job_queue.hpp
-- Installing: C:/ci/ws/install/include/mcap_vendor/mcap/types.hpp
-- Installing: C:/ci/ws/install/include/mcap_vendor/mcap/types.inl
-- Installing: C:/ci/ws/install/include/mcap_vendor/mcap/writer.hpp
-- Installing: C:/ci/ws/install/include/mcap_vendor/mcap/writer.inl
-- Installing: C:/ci/ws/install/bin/mcap.dll
-- Installing: C:/ci/ws/install/share/ament_index/resource_index/package_run_dependencies/mcap_vendor
-- Installing: C:/ci/ws/install/share/ament_index/resource_index/parent_prefix_path/mcap_vendor
-- Installing: C:/ci/ws/install/share/mcap_vendor/environment/ament_prefix_path.bat
-- Installing: C:/ci/ws/install/share/mcap_vendor/environment/ament_prefix_path.dsv
-- Installing: C:/ci/ws/install/share/mcap_vendor/environment/path.bat
-- Installing: C:/ci/ws/install/share/mcap_vendor/environment/path.dsv
-- Installing: C:/ci/ws/install/share/mcap_vendor/local_setup.bat
-- Installing: C:/ci/ws/install/share/mcap_vendor/local_setup.dsv
-- Installing: C:/ci/ws/install/share/mcap_vendor/package.dsv
-- Installing: C:/ci/ws/install/share/ament_index/resource_index/packages/mcap_vendor
-- Installing: C:/ci/ws/install/share/mcap_vendor/cmake/mcapExport.cmake
-- Installing: C:/ci/ws/install/share/mcap_vendor/cmake/mcapExport-release.cmake
-- Installing: C:/ci/ws/install/share/mcap_vendor/cmake/ament_cmake_export_include_directories-extras.cmake
-- Installing: C:/ci/ws/install/share/mcap_vendor/cmake/ament_cmake_export_libraries-extras.cmake
-- Installing: C:/ci/ws/install/share/mcap_vendor/cmake/ament_cmake_export_targets-extras.cmake
-- Installing: C:/ci/ws/install/share/mcap_vendor/cmake/ament_cmake_export_dependencies-extras.cmake
-- Installing: C:/ci/ws/install/share/mcap_vendor/cmake/mcap_vendorConfig.cmake
-- Installing: C:/ci/ws/install/share/mcap_vendor/cmake/mcap_vendorConfig-version.cmake
-- Installing: C:/ci/ws/install/share/mcap_vendor/package.xml
---
Finished <<< mcap_vendor [48.7s]

Although ament_cmake_export_libraries-extra.cmake expecting to find it inside install/lib subfolder

     find_library(
        _lib NAMES "${_library}"
        PATHS "${mcap_vendor_DIR}/../../../lib"
        NO_DEFAULT_PATH NO_CMAKE_FIND_ROOT_PATH
      )

@clalancette Do you have any ideas what's wrong in CMake file?

@MichaelOrlov
Copy link
Member Author

Trying re-run Windows build after removing ament_export_libraries(mcap) from CMake file

  • Windows Build Status

@MichaelOrlov
Copy link
Member Author

@clalancette I have a suspicious that mcap.lib which is reported to be missed on linker stage hasn't been really built due to the warnings possibly treated as errors

Checking Build System
  Building Custom Rule C:/ci/ws/src/ros-tooling/rosbag2_storage_mcap/mcap_vendor/CMakeLists.txt
  main.cpp
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\internal.hpp(54,46): warning C4267: '+=': conversion from 'size_t' to 'uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\reader.inl(67,23): warning C4244: 'argument': conversion from 'uint64_t' to 'long', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\reader.inl(324,16): warning C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\reader.inl(548,43): warning C4267: '=': conversion from 'size_t' to 'uint16_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\reader.inl(549,45): warning C4267: '=': conversion from 'size_t' to 'uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\reader.inl(550,57): warning C4267: '=': conversion from 'size_t' to 'uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\reader.inl(551,53): warning C4267: '=': conversion from 'size_t' to 'uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\reader.inl(552,47): warning C4267: '=': conversion from 'size_t' to 'uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\reader.inl(1013,40): warning C4267: '+=': conversion from 'size_t' to 'uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\reader.inl(1020,47): warning C4267: '+=': conversion from 'size_t' to 'uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\reader.inl(1034,33): warning C4244: '+=': conversion from 'uint64_t' to 'uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\reader.inl(1068,45): warning C4267: '+=': conversion from 'size_t' to 'uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\crc32.hpp(86,44): warning C4267: 'initializing': conversion from 'size_t' to 'uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\writer.inl(45,16): warning C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\writer.inl(268,93): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\writer.inl(268,93): warning C4267: 'initializing': conversion from 'size_t' to 'const int', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\writer.inl(907,57): warning C4267: 'initializing': conversion from 'size_t' to 'uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\writer.inl(907,57): warning C4267: 'initializing': conversion from 'size_t' to 'const uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\writer.inl(924,81): warning C4267: 'initializing': conversion from 'size_t' to 'uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\writer.inl(924,81): warning C4267: 'initializing': conversion from 'size_t' to 'const uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\writer.inl(993,83): warning C4267: 'initializing': conversion from 'size_t' to 'uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
C:\ci\ws\build\mcap_vendor\_deps\mcap-src\cpp\mcap\include\mcap\writer.inl(993,83): warning C4267: 'initializing': conversion from 'size_t' to 'const uint32_t', possible loss of data [C:\ci\ws\build\mcap_vendor\mcap.vcxproj]
  lz4.c
  lz4frame.c
  lz4hc.c
  xxhash.c
  mcap.vcxproj -> C:\ci\ws\build\mcap_vendor\Release\mcap.dll

Can we somehow verify that mcap.lib has been successfully built or somehow suppress those warnings?
I am sorry I don't have a local machine with Windows setup.

@MichaelOrlov
Copy link
Member Author

Trying re-run Windows build after suppressing warnings with \w option

  • Windows Build Status

@clalancette
Copy link
Contributor

In short, I don't know. Windows is always a pain, and we don't have good ways to go into the CI containers and see what is in there. To move forward here, someone with a Windows machine is going to have to debug it. I do have a Windows machine locally, but I probably don't have time to look into it this week. Ideas:

  1. Find someone else who has a Windows machine handy to help debug.
  2. Go ahead with this PR as-is, since it fixes a couple of other issues, and then fix Windows later (though we'll obviously need to do that before merging this into rosbag2).
  3. Wait for next week when I might be able to spend some time on it.

@MichaelOrlov
Copy link
Member Author

@clalancette I would suggest to move forward with merging this PR and I will create a follow up issue about Windows build failure.
I've reverted my last commit with disabling warnings on Windows build, since it doesn't solve a problem.

Wouldn't you mind to approve this PR?

@MichaelOrlov MichaelOrlov force-pushed the morlov/cleanup-in-mcap_vendor branch from 9aa40e0 to fc5f8aa Compare October 27, 2022 15:58
Copy link
Contributor

@clalancette clalancette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left one nit inline, this otherwise looks OK to me.

I'll suggest running one more CI here, since there has been some back and forth since the last time it was run. But I'll leave it up to you.

mcap_vendor/CMakeLists.txt Outdated Show resolved Hide resolved
@MichaelOrlov
Copy link
Member Author

Re-run CI

  • Linux Build Status
  • Linux-aarch64 Build Status

Signed-off-by: Michael Orlov <[email protected]>
Co-authored-by: Chris Lalancette <[email protected]>
@MichaelOrlov MichaelOrlov force-pushed the morlov/cleanup-in-mcap_vendor branch from b898f19 to 2bcff24 Compare October 27, 2022 21:15
@MichaelOrlov MichaelOrlov merged commit 4d8cb9b into main Oct 27, 2022
james-rms pushed a commit to james-rms/rosbag2 that referenced this pull request Nov 18, 2022
* Cleanup in `mcap_vendor` package

- Use `${PROJECT_NAME}` instead of confusing `mcap` for library name and
exporting target, to be consistent with all other packages in ROS2.

- Add export for path to the `mcap` includes and export for mcap_vendor
library. It was an issue in downstream packages that `mcap/mcap.hpp` was
not visible since we were not exporting path to it explicitly.

- Removed test section with linters for `mcap_vendor` package. Since
package itself doesn't contain any tests and own source code and we
shouldn't run linters on any third party code. Prevent CI failures.

Signed-off-by: Michael Orlov <[email protected]>

* Install `mcap` headers in `include/${PROJECT_NAME}/mcap` and export them

Signed-off-by: Michael Orlov <[email protected]>

* Revert renaming `mcap` library to `mcap_vendor` and fixed include issue

Signed-off-by: Michael Orlov <[email protected]>

* Add `LIBRARY DESTINATION lib` for install targets

Signed-off-by: Michael Orlov <[email protected]>

* Attempt to fix Windows build error by removing `ament_export_libraries(mcap)`

Signed-off-by: Michael Orlov <[email protected]>

* Nitpick. remove extra space before `HAS_LIBRARY_TARGET`

Signed-off-by: Michael Orlov <[email protected]>
Co-authored-by: Chris Lalancette <[email protected]>

Signed-off-by: Michael Orlov <[email protected]>
Co-authored-by: Chris Lalancette <[email protected]>
Signed-off-by: James Smith <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants