No dependencies recognized with yarn

[Gemma Cabero]

Hi,

I’m attempting to make use of license_finder for a series of modules written in different languages. Unfortunately the ones written in javascript, and making use of yarn as a package manager, the tool does not seem to work properly.  I can run 'yarn licences list —json’ and get an output but the result is empty when just typing license_finder.  Any indications on how to make license_finder to produce an output for a valid yarn project?

Many thanks,

Gemma

$> license_finder            

LicenseFinder::NPM: is active

LicenseFinder::Yarn: is active

No dependencies recognized!

$>license_finder -p or $>license_finder

LicenseFinder::NPM: is active

LicenseFinder::Yarn: is active

No dependencies recognized!

$>yarn --version

3.1.1

$>yarn licenses list

├─ MIT

│  ├─ @types/react@npm:16.14.2 (via npm:^16.9.35)

│  │  ├─ URL: git+https://github.com/DefinitelyTyped/DefinitelyTyped.git

│  │  └─ VendorUrl: https://github.com/DefinitelyTyped/DefinitelyTyped#readme

│  ├─ @types/react-dom@npm:16.9.10 (via npm:^16.9.8)

│  │  ├─ URL: git+https://github.com/DefinitelyTyped/DefinitelyTyped.git

│  │  └─ VendorUrl: https://github.com/DefinitelyTyped/DefinitelyTyped#readme

│  ├─ @types/reactstrap@npm:8.7.2 (via npm:^8.4.2)

│  │  ├─ URL: git+https://github.com/DefinitelyTyped/DefinitelyTyped.git

│  │  └─ VendorUrl: https://github.com/DefinitelyTyped/DefinitelyTyped#readme

│  ├─ concurrently@npm:3.6.1 (via npm:^3.6.1)

│  │  ├─ URL: git+https://github.com/kimmobrunfeldt/concurrently.git

│  │  ├─ VendorName: Kimmo Brunfeldt

│  │  └─ VendorUrl: https://github.com/kimmobrunfeldt/concurrently

│  └─ enquirer@npm:2.3.6 (via npm:^2.3.6)

│     ├─ URL: git+https://github.com/enquirer/enquirer.git

│     ├─ VendorName: Jon Schlinkert

│     └─ VendorUrl: https://github.com/enquirer/enquirer

│

├─ BSD-2-Clause

│  └─ dotenv@npm:16.0.0 (via npm:^16.0.0)

│     ├─ URL: git://github.com/motdotla/dotenv.git

│     └─ VendorUrl: https://github.com/motdotla/dotenv#readme

│

└─ ISC

   └─ poll-until-promise@npm:4.0.4 (via npm:^4.0.4)

      ├─ URL: git+https://github.com/AlonMiz/poll-until-promise.git

      ├─ VendorName: Alon Mizrahi

      └─ VendorUrl: https://github.com/AlonMiz/poll-until-promise#readme

[Jason Smith]

Hi Gemma,

We believe we may have a fix for this issue, but it has not been tested in the wild yet. Yarn 3 seemed to have changed the format of its output, so the parser failed to handle it and thus returns an empty result. You can see more info on this in this thread: https://github.com/pivotal/LicenseFinder/issues/912

Feel free to try the fix in this PR https://github.com/pivotal/LicenseFinder/pull/936 and let us know if it works. If so, we will merge it in.

Regards,
Jason

[Gemma Cabero]

Thanks Jason,

How do you suggest I try it, by cloning the branch and build it locally? I'm afraid I'm not a ruby developer and I was only using the installation via brew. Is it cloning and using gem?

Many thanks,

Gemma

[Gemma Cabero]

Hi,

I've cloned the project and checkout the PR to build it locally but the execution seems to show the same results.  This is the list of steps I've used:

$> git clone https://github.com/pivotal/LicenseFinder.git

$> git fetch origin pull/936/head:yarn-parsing-fix

$> git checkout yarn-parsing-fix

$> gem install license_finder

$javascript_project>license_finder

LicenseFinder::NPM: is active

LicenseFinder::Yarn: is active

No dependencies recognized!

Is there a way to check the version of code in the PR? 

Best regards,

Gemma

[Jason Smith]

Hi Gemma,

You cloned and checked out the branch correctly. However, I think when you ran "gem install license_finder" you still installed the released version of the binary and not the one from the cloned repo.

I suggest you try the following.  First, uninstall all version of license_finder on your system with:

$> gem uninstall license_finder
Say 'y' to the prompt. It will also want to uninstall license_finder.py, and that's OK

Secondly, install the license_finder from the clone repo(make sure you are in your 'yarn-parsing-fix' branch) with :

$> bundle install
$> rake install

Lastly, verify that the correct license_finder is installed with:
$> gem list license_finder
This should output something similar to:
___

*** LOCAL GEMS ***

license_finder (7.0.1)
___
There should only be one version of license_finder listed, since all other ones should have been uninstalled with 'gem uninstall license_finder'


I hope this helps,
Jason Smith

[Gemma Cabero]

Hi,

More work on trying to get the branch installed and successfully managed after updating my Xcode version (incompatibility problem with newer MacOS versions). Unfortunately the fix may still not work as license finder still seem to find a problem on parsing. This is the error found (seems to be an edge case with empty arrays?):

license_finder

LicenseFinder::NPM: is active

LicenseFinder::Yarn: is active

/Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/package_managers/yarn.rb:101:in `block in get_yarn_packages': undefined method `[]' for nil:NilClass (NoMethodError)

   from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/package_managers/yarn.rb:94:in `each'

   from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/package_managers/yarn.rb:94:in `get_yarn_packages'

   from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/package_managers/yarn.rb:34:in `current_packages'

   from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/package_manager.rb:105:in `current_packages_with_relations'

   from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/scanner.rb:42:in `each'

   from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/scanner.rb:42:in `flat_map'

   from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/scanner.rb:42:in `active_packages'

   from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/core.rb:84:in `current_packages'

   from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/core.rb:79:in `decision_applier'

   from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/forwardable.rb:224:in `any_packages?'

   from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/license_aggregator.rb:17:in `block in any_packages?'

   from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/license_aggregator.rb:15:in `map'

   from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/license_aggregator.rb:15:in `any_packages?'

   from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/cli/main.rb:120:in `action_items'

   from /Library/Ruby/Gems/2.6.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'

   from /Library/Ruby/Gems/2.6.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'

   from /Library/Ruby/Gems/2.6.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'

   from /Library/Ruby/Gems/2.6.0/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'

   from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/bin/license_finder:6:in `<top (required)>'

   from /usr/local/bin/license_finder:23:in `load'

   from /usr/local/bin/license_finder:23:in `<main>'

Hope it helps,

Gemma

[Jason Smith]

Hi Gemma,

Thanks for that feedback.  

Is the project you are scanning in a repo? If so, could you share this repo with us so we can try LicenseFinder on it do debug it? Or if you can't, do you know which module is causing the problem? We'll try and dig deeper into that issue.

Regards,

Jason Smith

[Gemma Cabero]

Hi Jason,

I'm afraid I can't give access to the repo that generates the problem and I'm not sure how I could give you more details as that's all the output I've got from running the license_finder. May I ask what the tool does under the hood? Maybe I could run some separate commands. 

Best regards,

Gemma

[Gemma Cabero]

Build from master and now seems to work, maybe the changes in this branch are not necessary? is the master version having an official release?

[Gemma Cabero]

Sorry my mistake, still error 

[Shane Lattanzio]

Hey! For the original yarn 1 version, we would run yarn licenses list --json --no-progress --cwd <PROJECT_PATH> and the output would be something like

{
'type' => 'table',
'data' => {
  'body' => [['yn', '2.0.0', 'MIT', 'https://github.com/sindresorhus/yn.git', 'sindresorhus.com', 'Sindre Sorhus']],
  'head' => %w[Name Version License URL VendorUrl VendorName]
}
}

However, in v2 and 3, we run yarn licenses list --json and it seems the output is something like the following. Please let me know if this is an incorrect assumption because I kind of guessed at it:

{
'value' => 'MIT',
'children' => {
  'yn@npm:2.0.0' => {
    'value' => {
      'locator' => 'yn@npm:2.0.0',
      'descriptor' => 'yn@npm:2.0.0'
    },
    'children' => {
      'vendorUrl' => 'sindresorhus.com',
      'vendorName' => 'Sindre Sorhus'
    }
  }
}
}

In addition, if you run license finder with a prepare command, the internal command would be for v1:    
 yarn install && yarn plugin import https://raw.githubusercontent.com/mhassan1/yarn-plugin-licenses/v0.7.2/bundles/@yarnpkg/plugin-licenses.js"
and for v2:
 yarn install && yarn plugin import https://raw.githubusercontent.com/mhassan1/yarn-plugin-licenses/v0.6.0/bundles/@yarnpkg/plugin-licenses.js"

If you are running this on master, for a yarn2 or 3 project, licensefinder returns nothing because the json has changed in the later versions so the parsing finds nothing. My PR was trying to remedy this by doing different parsing but it may be incorrect. If I could find a good example repo to test, I could make it more robust. For now, let me know if this new json output makes sense. For reference, the ruby method that is being called to do the parsing in this case is which is different from the method previously used in yarn v1:

def get_yarn_packages(json_objects)
  packages = []
  incompatible_packages = []
  json_objects.each do |json_object|
    license = json_object['value']
    body = json_object['children']

    valid_match = /(?<name>[\w,\-]+)@(?<manager>\D*):\D*(?<version>(\d+\.?)+)/ =~ body.to_s

    if valid_match
      homepage = body["#{name}@#{manager}:#{version}"]['children']['vendorUrl']
      author = body["#{name}@#{manager}:#{version}"]['children']['vendorName']
      package = YarnPackage.new(
        name,
        version,
        spec_licenses: [license],
        homepage: homepage,
        authors: author,
        install_path: project_path.join(modules_folder, name)
      )
      packages << package
    end
    incompatible_match = /(?<name>[\w,\-]+)@[a-z]*:(?<version>(\.))/ =~ body.to_s

    if incompatible_match
      package = YarnPackage.new(name, version, spec_licenses: ['unknown'])
      incompatible_packages.push(package)
    end
  end

  packages + incompatible_packages.uniq
end

[Gemma Cabero Colmenero]

Hi,

Below there is an example of the json generated. I had to apply the following plugin to be able to run yarn licenses list --json as it was possible in v1.

yarn plugin import https://raw.githubusercontent.com/mhassan1/yarn-plugin-licenses/v0.8.1/bundles/@yarnpkg/plugin-licenses.js

$> yarn licenses list --json  

{"value":"MIT","children":{"@types/react@npm:16.14.2":{"value":{"locator":"@types/react@npm:16.14.2","descriptor":"@types/react@npm:^16.9.35"},"children":{"url":"https://github.com/DefinitelyTyped/DefinitelyTyped.git"}},"@types/react-dom@npm:16.9.10":{"value":{"locator":"@types/react-dom@npm:16.9.10","descriptor":"@types/react-dom@npm:^16.9.8"},"children":{"url":"https://github.com/DefinitelyTyped/DefinitelyTyped.git"}},"@types/reactstrap@npm:8.7.2":{"value":{"locator":"@types/reactstrap@npm:8.7.2","descriptor":"@types/reactstrap@npm:^8.4.2"},"children":{}},"cipher-frontend@workspace:.":{"value":{"locator":"cipher-frontend@workspace:.","descriptor":"cipher-frontend@workspace:."},"children":{}},"concurrently@npm:3.6.1":{"value":{"locator":"concurrently@npm:3.6.1","descriptor":"concurrently@npm:^3.6.1"},"children":{"url":"https://github.com/kimmobrunfeldt/concurrently.git","vendorName":"Kimmo Brunfeldt","vendorUrl":"https://github.com/kimmobrunfeldt/concurrently"}},"enquirer@npm:2.3.6":{"value":{"locator":"enquirer@npm:2.3.6","descriptor":"enquirer@npm:^2.3.6"},"children":{"url":"https://github.com/enquirer/enquirer","vendorName":"Jon Schlinkert","vendorUrl":"https://github.com/enquirer/enquirer"}}}}
{"value":"BSD-2-Clause","children":{"dotenv@npm:16.0.0":{"value":{"locator":"dotenv@npm:16.0.0","descriptor":"dotenv@npm:^16.0.0"},"children":{"url":"git://github.com/motdotla/dotenv.git"}}}}
{"value":"ISC","children":{"poll-until-promise@npm:4.0.4":{"value":{"locator":"poll-until-promise@npm:4.0.4","descriptor":"poll-until-promise@npm:^4.0.4"},"children":{"url":"git+https://github.com/AlonMiz/poll-until-promise.git","vendorName":"Alon Mizrahi","vendorUrl":"https://github.com/AlonMiz/poll-until-promise#readme"}}}}

Maybe the plugin contains a bug as when trying to prettify the json I think there was an error :thinking_face:

Hope it helps,

Gemma

[Shane Lattanzio]

Thanks for the output. I think I did find an issue with the parsing. I just pushed a possible fix for it. Can you try it out and let me know?

[Gemma Cabero]

Hi,

Some good news! top level now seems to work. However it doesn't go recursively and some repos can have submodules. Noticed when I saw the list there were third party libs missing and tried to run yarn promises list -R which shows more. That's not a big deal because the user can go to the relevant folders and run license_finder but when I did that the original  problem surfaced so I think there is still an issue with the json.

$> license_finder    

LicenseFinder::NPM: is active
LicenseFinder::Yarn: is active

/Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/package_managers/yarn.rb:101:in `block in get_yarn_packages': undefined method `[]' for nil:NilClass (NoMethodError)
    from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/package_managers/yarn.rb:94:in `each'
    from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/package_managers/yarn.rb:94:in `get_yarn_packages'
    from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/package_managers/yarn.rb:34:in `current_packages'
    from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/package_manager.rb:105:in `current_packages_with_relations'
    from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/scanner.rb:42:in `each'
    from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/scanner.rb:42:in `flat_map'
    from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/scanner.rb:42:in `active_packages'
    from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/core.rb:84:in `current_packages'
    from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/core.rb:79:in `decision_applier'
    from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/forwardable.rb:224:in `any_packages?'
    from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/license_aggregator.rb:17:in `block in any_packages?'
    from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/license_aggregator.rb:15:in `map'
    from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/license_aggregator.rb:15:in `any_packages?'
    from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/lib/license_finder/cli/main.rb:120:in `action_items'
    from /Library/Ruby/Gems/2.6.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
    from /Library/Ruby/Gems/2.6.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
    from /Library/Ruby/Gems/2.6.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
    from /Library/Ruby/Gems/2.6.0/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'
    from /Library/Ruby/Gems/2.6.0/gems/license_finder-7.0.1/bin/license_finder:6:in `<top (required)>'
    from /usr/local/bin/license_finder:23:in `load'
    from /usr/local/bin/license_finder:23:in `<main>'

Many thanks,

Gemma

[Shane Lattanzio]

Hi,

Thanks for the sample file. I pushed another change. This should work in the case that your dependency is in the formats like the following:

@babel/preset-typescript@virtual:47c2c5b90818fd89a723b356b0612f83df8846bb5189e6ab667353b8f51c01103e37cf9a84e7354873d06d73e684d99430d4ee035b0800aa62756e346467a8c4#npm:7.18.6

@dnd-kit/core@virtual:47c2c5b90818fd89a723b356b0612f83df8846bb5189e6ab667353b8f51c01103e37cf9a84e7354873d06d73e684d99430d4ee035b0800aa62756e346467a8c4#npm:5.0.3

@types/classnames@npm:2.3.1

autosize@npm:4.0.4

react-plotly.js@virtual:47c2c5b90818fd89a723b356b0612f83df8846bb5189e6ab667353b8f51c01103e37cf9a84e7354873d06d73e684d99430d4ee035b0800aa62756e346467a8c4#npm:2.6.0

react-router-dom@virtual:47c2c5b90818fd89a723b356b0612f83df8846bb5189e6ab667353b8f51c01103e37cf9a84e7354873d06d73e684d99430d4ee035b0800aa62756e346467a8c4#npm:5.3.3

sql-formatter@npm:2.3.4

timeme.js@npm:2.1.0


License finder also does support a recursive `-r` option so I wonder if that can help you here as well. Let me know how it goes!

[Gemma Cabero Colmenero]

Hi,

The latest fix seemed to have worked :)

Many thanks,

Gemma