leveldown Package Support

[reconbot]

I've come across my first popular package that can't cross compile.

leveldown@1.4.3

While I'm not sure if this repo should be the support page for such problems long term, I think it's the perfect place to deal with this particular package. It's being used by @HipsterBrown via pouchdb.

The C code looks for some platform features that might be missing from our little tessel. I don't know enough to debug it.

../deps/leveldb/leveldb-1.18.0/port/atomic_pointer.h:211:2: error: #error Please implement AtomicPointer for this platform.
 #error Please implement AtomicPointer for this platform.

My google foo find me the file in question, a forum thread with possible fixes.

I took a step back for a moment and noticed they're not using node pre-gyp. They're using something called prebuild, and they claim to support the arm chipset.

I'm investigating further but I'd love help from anyone in the know.

[reconbot]

I did a regular old npm install after setting all the cross compiling env vars. I had to manually make the tar of the package after that. I have the tar up on s3 for testing leveldown leveldown@1.4.3.

[reconbot]

@HipsterBrown is tribute to test it out

[HipsterBrown]

Smoke Test

My test project:

Setup:
mkdir tessel-test
t2 init
npm install --save pouchdb

Files:
index.js

var PouchDB = require('pouchdb');

var db = new PouchDB('./db');

db.put({
  "_id": "test",
  "desc": "This is a test doc."
});

db.info()
.then(function(info) {
  console.log(info);
});

console.log('This is a test.');

.tesselinclude

Included prebuild for compiling on the device.

node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/

Run project with t2 run index.js

CLI Patch

t2-cli/lib/tessel/deploy.js

The source of the binary was found in ~/.tessel/binaries/leveldown-1.4.3-Release/package/build/Release/leveldown.node instead of ~/.tessel/binaries/leveldown-1.4.3-Release/Release/leveldown.node. The fs.copySync also required a try/catch block to catch silent errors that were preventing continued execution of the deploy script.

actions.injectBinaryModules = function(globRoot, tempBundlePath) {
  return new Promise(resolve => {
    // For every binary module in use...
    binaryModulesUsed.forEach(details => {
    var sourceBinary = path.join(details.extractPath, 'package/build', 'Release', details.binName);
    var tempTargetModulePath = path.join(tempBundlePath, details.path);
    var tempTargetBinary = path.join(tempTargetModulePath, 'build/Release', details.binName);

    try {
      fs.copySync(sourceBinary, tempTargetBinary);
    } catch (err) {
      console.error(err);
    }
   });
   // All binary modules have been replaced, resolve.
   return resolve();
  });
};

Results:

After running with the patched CLI:

Nicks-MacBook-Pro:tessel-another-test hipsterbrown$ t2 run index.js --lan
INFO Looking for your Tessel...
INFO Connected to tessel-router.
INFO Writing project to RAM on tessel-router (8544.768 kB)...
INFO Deployed.
INFO Running index.js...
node: '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/build/Release/leveldown.node' is not an ELF executable for MIPS
This is a test.

That not ELF executable for MIPS error led me to this issue.

Then I tried compiling the binary on Tessel:

root@tessel-router:/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown# ./node_modules/prebuild/bin.js --install
prebuild info begin Prebuild version 2.8.1
prebuild info looking for local prebuild @ prebuilds/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild info npm cache directory missing, creating it...
prebuild info looking for cached prebuild @ /root/.npm/_prebuilds/https-github.com-level-leveldown-releases-download-v1.4.3-leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild http request GET https://github.com/level/leveldown/releases/download/v1.4.3/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild http 404 https://github.com/level/leveldown/releases/download/v1.4.3/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild WARN install Prebuilt binaries for node version v4.2.1 are not available
prebuild info install We will now try to compile from source.
gyp info spawn /usr/bin/python2
gyp info spawn args [ '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/root/.node-gyp/4.2.1/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/root/.node-gyp/4.2.1',
gyp info spawn args   '-Dnode_gyp_dir=/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=node.lib',
gyp info spawn args   '-Dmodule_root_dir=/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
^C
root@tessel-router:/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown# Traceback (most recent call last):
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/gyp_main.py", line 16, in <module>
    sys.exit(gyp.script_main())
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 545, in script_main
    return main(sys.argv[1:])
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 538, in main
    return gyp_main(args)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 514, in gyp_main
    options.duplicate_basename_check)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 130, in Load
    params['parallel'], params['root_targets'])
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 2775, in Load
    variables, includes, depth, check, True)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 417, in LoadTargetBuildFile
    build_file_data, PHASE_EARLY, variables, build_file_path)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1292, in ProcessVariablesAndConditionsInDict
    build_file)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1307, in ProcessVariablesAndConditionsInList
    ProcessVariablesAndConditionsInDict(item, phase, variables, build_file)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1292, in ProcessVariablesAndConditionsInDict
    build_file)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1311, in ProcessVariablesAndConditionsInList
    expanded = ExpandVariables(item, phase, variables, build_file)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 911, in ExpandVariables
    p_stdout, p_stderr = p.communicate('')
  File "/usr/lib/python2.7/subprocess.py", line 799, in communicate
    return self._communicate(input)
  File "/usr/lib/python2.7/subprocess.py", line 1409, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib/python2.7/subprocess.py", line 1463, in _communicate_with_poll
    ready = poller.poll()
KeyboardInterrupt

This ended up hanging for too long. Still looking into that issue, but glad to get a successful deploy onto Tessel.

[reconbot]

I did wierd things when making the package.
On Jan 15, 2016 1:10 AM, "Nick Hehr" notifications@github.com wrote:

Smoke Test My test project:

Setup:
mkdir tessel-test
t2 init
npm install --save pouchdb

Files:
index.js

var PouchDB = require('pouchdb');
var db = new PouchDB('./db');
db.put({
"_id": "test",
"desc": "This is a test doc."
});
db.info()
.then(function(info) {
console.log(info);
});
console.log('This is a test.');

.tesselinclude

Included prebuild for compiling on the device.

node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/

Run project with t2 run index.js
CLI Patch

t2-cli/lib/tessel/deploy.js

The source of the binary was found in
~/.tessel/binaries/leveldown-1.4.3-Release/package/build/Release/leveldown.node
instead of
~/.tessel/binaries/leveldown-1.4.3-Release/Release/leveldown.node. The
fs.copySync also required a try/catch block to catch silent errors that
were preventing continued execution of the deploy script.

actions.injectBinaryModules = function(globRoot, tempBundlePath) {
return new Promise(resolve => {
// For every binary module in use...
binaryModulesUsed.forEach(details => {
var sourceBinary = path.join(details.extractPath, 'package/build', 'Release', details.binName);
var tempTargetModulePath = path.join(tempBundlePath, details.path);
var tempTargetBinary = path.join(tempTargetModulePath, 'build/Release', details.binName);

try {
  fs.copySync(sourceBinary, tempTargetBinary);
} catch (err) {
  console.error(err);
}

});
// All binary modules have been replaced, resolve.
return resolve();
});
};

Results:

After running with the patched CLI:

Nicks-MacBook-Pro:tessel-another-test hipsterbrown$ t2 run index.js --lan
INFO Looking for your Tessel...
INFO Connected to tessel-router.
INFO Writing project to RAM on tessel-router (8544.768 kB)...
INFO Deployed.
INFO Running index.js...
node: '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/build/Release/leveldown.node' is not an ELF executable for MIPS
This is a test.

That not ELF executable for MIPS error led me to this issue
nodejs/node#1440.

Then I tried compiling the binary on Tessel:

root@tessel-router:/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown# ./node_modules/prebuild/bin.js --install
prebuild info begin Prebuild version 2.8.1
prebuild info looking for local prebuild @ prebuilds/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild info npm cache directory missing, creating it...
prebuild info looking for cached prebuild @ /root/.npm/_prebuilds/https-github.com-level-leveldown-releases-download-v1.4.3-leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild http request GET https://github.com/level/leveldown/releases/download/v1.4.3/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild http 404 https://github.com/level/leveldown/releases/download/v1.4.3/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild WARN install Prebuilt binaries for node version v4.2.1 are not available
prebuild info install We will now try to compile from source.
gyp info spawn /usr/bin/python2
gyp info spawn args [ '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args 'binding.gyp',
gyp info spawn args '-f',
gyp info spawn args 'make',
gyp info spawn args '-I',
gyp info spawn args '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/build/config.gypi',
gyp info spawn args '-I',
gyp info spawn args '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/addon.gypi',
gyp info spawn args '-I',
gyp info spawn args '/root/.node-gyp/4.2.1/include/node/common.gypi',
gyp info spawn args '-Dlibrary=shared_library',
gyp info spawn args '-Dvisibility=default',
gyp info spawn args '-Dnode_root_dir=/root/.node-gyp/4.2.1',
gyp info spawn args '-Dnode_gyp_dir=/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp',
gyp info spawn args '-Dnode_lib_file=node.lib',
gyp info spawn args '-Dmodule_root_dir=/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown',
gyp info spawn args '--depth=.',
gyp info spawn args '--no-parallel',
gyp info spawn args '--generator-output',
gyp info spawn args 'build',
gyp info spawn args '-Goutput_dir=.' ]
^C
root@tessel-router:/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown# Traceback (most recent call last):
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/gyp_main.py", line 16, in
sys.exit(gyp.script_main())
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 545, in script_main
return main(sys.argv[1:])
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 538, in main
return gyp_main(args)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 514, in gyp_main
options.duplicate_basename_check)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 130, in Load
params['parallel'], params['root_targets'])
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 2775, in Load
variables, includes, depth, check, True)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 417, in LoadTargetBuildFile
build_file_data, PHASE_EARLY, variables, build_file_path)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1292, in ProcessVariablesAndConditionsInDict
build_file)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1307, in ProcessVariablesAndConditionsInList
ProcessVariablesAndConditionsInDict(item, phase, variables, build_file)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1292, in ProcessVariablesAndConditionsInDict
build_file)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1311, in ProcessVariablesAndConditionsInList
expanded = ExpandVariables(item, phase, variables, build_file)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 911, in ExpandVariables
p_stdout, p_stderr = p.communicate('')
File "/usr/lib/python2.7/subprocess.py", line 799, in communicate
return self._communicate(input)
File "/usr/lib/python2.7/subprocess.py", line 1409, in _communicate
stdout, stderr = self._communicate_with_poll(input)
File "/usr/lib/python2.7/subprocess.py", line 1463, in _communicate_with_poll
ready = poller.poll()
KeyboardInterrupt

This ended up hanging for too long. Still looking into that issue, but
glad to get a successful deploy onto Tessel.

—
Reply to this email directly or view it on GitHub
#14 (comment).

[johnnyman727]

@reconbot not sure if I can be helpful here but I'd like to understand the issue(s). Is there a problem with how T2 handles prebuild and an issue with the underlying atomicpointer library? (Along with the two CLI issues @HipsterBrown caught)

[reconbot]

I think our issue is with prebuild and not atomicpointer. Give it a whirl
and try to build the leveldown package and see what happens.

On Mon, Jan 18, 2016 at 11:15 AM, Jon notifications@github.com wrote:

@reconbot https://github.com/reconbot not sure if I can be helpful here
but I'd like to understand the issue(s). Is there a problem with how T2
handles prebuild and an issue with the underlying atomicpointer library?
(Along with the two CLI issues @HipsterBrown
https://github.com/HipsterBrown caught)

—
Reply to this email directly or view it on GitHub
#14 (comment).

Francis Gulotta
Director of Web Connected Devices
Mobile: 917-805-3079

[johnnyman727]

Ah I see - well I probably won't be much help on this one. @tcr might have some advice.

[nolanlawson]

FWIW I also tried the sqlite3 package (PouchDB's other Node-supported backing database), but no luck so far: #27