ImportInliner does not use asset pipeline?

[Vyacheslav Egorov]

Hi,

I have been trying to use Polymer with SASS to tidy up my CSS and I am
being bitten by ImportInliner. I am using SASS tranformer available on
pub.

If I create hydra.scss file and I try writing in my component:

<link rel="stylesheet" type="text/css" href="hydra.css">

then build fails with the following error:

Build error:
Transform ImportInliner on irhydra|web/index.html threw error: Could
not find asset irhydra|lib/src/ui/hydra.css.
null
null
Build completed with 1 errors.

I get similar error if I try to do

<link rel="stylesheet" type="text/css"
href="../../../../packages/irhydra/src/ui/hydra.css">

However this works:

<style>
@import ('packages/irhydra/src/ui/hydra.css');
</style>

because SASS transformer correctly provides hydra.css asset.

It seems that ImportInliner is not using barback correctly or there is
something else wrong.

// Vyacheslav Egorov

[Vyacheslav Egorov]

Ok, I figured it out, apparently I have to maintain topological order
of transformers in my pubspec.yaml manually:

This works:

transformers:
- sass
- polymer:
entry_points: web/index.html

However this does not

transformers:
- polymer:
entry_points: web/index.html
- sass

Bob, is it documented that user is responsible for ordering
transformers like that?


// Vyacheslav Egorov

[Seth Ladd]

+ Sharon for docs tip

--
You received this message because you are subscribed to the Google Groups "Dart Web Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web+uns...@dartlang.org.
Visit this group at http://groups.google.com/a/dartlang.org/group/web/.

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/web/CAPD5VRv%3D1DF5%2BNt-jWtJvGMLHTvmZYANpVippRF5AotkPY-FwQ%40mail.gmail.com.

[Bob Nystrom]

On Wed, Jul 16, 2014 at 8:51 AM, Vyacheslav Egorov <veg...@google.com> wrote:

Bob, is it documented that user is responsible for ordering
transformers like that?

It is: https://www.dartlang.org/tools/pub/assets-and-transformers.html#specifying-transformers

For example, consider three transformers, specified as follows:

transformers:
- [t1, t2]
- t3

The t1 and t2 transformers run first, in parallel. The t3 transformer runs in a separate phase, after t1 and t2 are finished, and can see the outputs of t1 and t2.

Sorry it took you a while to figure this out!

- bob

[Vyacheslav Egorov]

I see, I was expecting a more classical build system which figures out
the ordering itself.

Now while we on the topic: I was trying to debug this SASS transformer
and noticed that it is called several times to transform the same
assets (like three times or so) during the same pub build invocation.
Any idea what can be the reason for this?

// Vyacheslav Egorov

[Bob Nystrom]

On Wed, Jul 16, 2014 at 9:57 AM, Vyacheslav Egorov <veg...@google.com> wrote:

I see, I was expecting a more classical build system which figures out
the ordering itself.

It figures out dependencies itself, but can't figure out ordering. This is because a transformer may output files of the same kind that it creates. If you have two transformers that modify .dart code in place, there's no way it could figure out an ordering for them on its own since either ordering is equally valid.

 

Now while we on the topic: I was trying to debug this SASS transformer
and noticed that it is called several times to transform the same
assets (like three times or so) during the same pub build invocation.
Any idea what can be the reason for this?

No idea. Can you point me to something I can use to repro?

- bob

[Vyacheslav Egorov]

I could not create a simple reproduction but cloning
https://github.com/mraleph/irhydra.git and trying to pub build branch
interesting-mode would reproduce it.

Just add print("${primaryAssetId}"); into SassTransformer.transform
method and you should see the same SCSS asset transformed multiple
times, then it should break.

The transformer itself seems to have a bug as well (though I don't
known about barback to decide for sure whether its bug or not): it
consumes SASS partials as inputs and uses hasInput() to resolve them
during processing of other files that depend on them, however in
transform it just drops them to the floor to ensure that they don't
appear in the output folder. However if partial processed (and
dropped) *before* a file that refers to the partial then that file
would be unable to resolve this partial, as hasInput would return
false for processed inputs. I don't know how to fix this problem in
the normal way. But apparently this does not trigger because partials
are processed last. (not in my case however due to multiple processing
of all SASS assets).


// Vyacheslav Egorov

[Bob Nystrom]

Hmm, it may be just that the SASS transformer is a little odd. I haven't looked at it much, though.

- bob

[Vyacheslav Egorov]

Well, that's precisely what I would like to figure out --- what
exactly is wrong with the transformer.

barbark infrastructure is invoking SassTransformer.apply with the
*same* AssetId *multiple* times.

Can you take a guess at what exactly has to be odd in the transformer
itself to cause such behavior?

// Vyacheslav Egorov

[Vyacheslav Egorov]

SASS transformer needs to inline some SCSS files into other files.
E.g. in my case I have partial "_ui.scss" that is imported into other
SCSS files.

Now transformer reads these files using readInputAsString. It seems
that barback understands this as a dependency. Now after the first
processing round it dirties Sass tranform because _ui.scss is removed.
I don't understand what exactly "removed" means in this context.
Nobody removed _ui.scss from the file system. I tried changing Sass
transormer so it does not declare _ui.scss as a primary input and does
not consume it, but that does *not* help.

Any idea what is going on here, Bob?

dirtying transform node in irhydra.2 for Sass on
irhydra|lib/src/ui/hydra.scss (applied, idle, unforced) due to
irhydra|lib/src/ui/_ui.scss (removed asset irhydra|lib/src/ui/_ui.scss
/ removed)

removed available asset irhydra|lib/src/ui/_ui.scss
#0 AssetNodeController.setRemoved
(package:barback/src/asset/asset_node.dart:215)
#1 AssetForwarder.close (package:barback/src/asset/asset_forwarder.dart:47)
#2 AssetForwarder.AssetForwarder.<anonymous closure>
(package:barback/src/asset/asset_forwarder.dart:38)
#3 _rootRunUnary (dart:async/zone.dart:846)
#4 _CustomZone.runUnary (dart:async/zone.dart:748)
#5 _CustomZone.runUnaryGuarded (dart:async/zone.dart:656)
#6 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:341)
#7 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:270)
#8 _SyncBroadcastStreamController._sendData.<anonymous closure>
(dart:async/broadcast_stream_controller.dart:354)
#9 _BroadcastStreamController._forEachListener
(dart:async/broadcast_stream_controller.dart:307)
#10 _SyncBroadcastStreamController._sendData
(dart:async/broadcast_stream_controller.dart:353)
#11 _BroadcastStreamController.add
(dart:async/broadcast_stream_controller.dart:237)
#12 AssetNodeController.setRemoved
(package:barback/src/asset/asset_node.dart:221)
#13 AssetForwarder.close (package:barback/src/asset/asset_forwarder.dart:47)
#14 PhaseOutput._watchAsset.<anonymous closure>
(package:barback/src/graph/phase_output.dart:90)
#15 AssetNode.whenRemoved.<anonymous closure>
(package:barback/src/asset/asset_node.dart:102)
#16 AssetNode._waitForState.<anonymous closure>
(package:barback/src/asset/asset_node.dart:121)
#17 _rootRunUnary (dart:async/zone.dart:842)
#18 _CustomZone.runUnary (dart:async/zone.dart:748)
#19 _Future._propagateToListeners.handleValueCallback
(dart:async/future_impl.dart:488)
#20 _Future._propagateToListeners (dart:async/future_impl.dart:571)
#21 _Future._complete (dart:async/future_impl.dart:317)
#22 _cancelAndValue (dart:async/stream_pipe.dart:44)
#23 Stream.firstWhere.<anonymous closure>.<anonymous closure>
(dart:async/stream.dart:1027)
#24 _runUserCode (dart:async/stream_pipe.dart:12)
#25 Stream.firstWhere.<anonymous closure> (dart:async/stream.dart:1023)
#26 _rootRunUnary (dart:async/zone.dart:842)
#27 _CustomZone.runUnary (dart:async/zone.dart:748)
#28 _CustomZone.runUnaryGuarded (dart:async/zone.dart:656)
#29 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:341)
#30 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:270)
#31 _SyncBroadcastStreamController._sendData.<anonymous closure>
(dart:async/broadcast_stream_controller.dart:354)
#32 _BroadcastStreamController._forEachListener
(dart:async/broadcast_stream_controller.dart:307)
#33 _SyncBroadcastStreamController._sendData
(dart:async/broadcast_stream_controller.dart:353)
#34 _BroadcastStreamController.add
(dart:async/broadcast_stream_controller.dart:237)
#35 AssetNodeController.setRemoved
(package:barback/src/asset/asset_node.dart:221)
#36 PhaseForwarder._adjustOutput
(package:barback/src/graph/phase_forwarder.dart:110)
#37 PhaseForwarder.addIntermediateAsset.<anonymous closure>
(package:barback/src/graph/phase_forwarder.dart:83)
#38 _rootRunUnary (dart:async/zone.dart:846)
#39 _CustomZone.runUnary (dart:async/zone.dart:748)
#40 _CustomZone.runUnaryGuarded (dart:async/zone.dart:656)
#41 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:341)
#42 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:270)
#43 _SyncBroadcastStreamController._sendData.<anonymous closure>
(dart:async/broadcast_stream_controller.dart:354)
#44 _BroadcastStreamController._forEachListener
(dart:async/broadcast_stream_controller.dart:307)
#45 _SyncBroadcastStreamController._sendData
(dart:async/broadcast_stream_controller.dart:353)
#46 _BroadcastStreamController.add
(dart:async/broadcast_stream_controller.dart:237)
#47 AssetNodeController.setRemoved
(package:barback/src/asset/asset_node.dart:221)
#48 AssetForwarder.close (package:barback/src/asset/asset_forwarder.dart:47)
#49 TransformerClassifier.remove
(package:barback/src/graph/transformer_classifier.dart:135)
#50 Phase.updateTransformers (package:barback/src/graph/phase.dart:246)
#51 AssetCascade.updateTransformers
(package:barback/src/graph/asset_cascade.dart:181)
#52 PackageGraph.updateTransformers.<anonymous closure>
(package:barback/src/graph/package_graph.dart:187)
#53 Future.Future.sync (dart:async/future.dart:168)
#54 syncFuture (package:barback/src/utils.dart:244)
#55 PackageGraph._inErrorZone.<anonymous closure>
(package:barback/src/graph/package_graph.dart:251)
#56 _rootRun (dart:async/zone.dart:835)
#57 _CustomZone.run (dart:async/zone.dart:740)
#58 _CustomZone.runGuarded (dart:async/zone.dart:648)
#59 runZoned (dart:async/zone.dart:1180)
#60 PackageGraph._inErrorZone
(package:barback/src/graph/package_graph.dart:250)
#61 PackageGraph.updateTransformers
(package:barback/src/graph/package_graph.dart:187)
#62 Barback.updateTransformers (package:barback/src/barback.dart:116)
#63 loadAllTransformers.<anonymous closure>.<anonymous
closure>.<anonymous closure>.<anonymous closure>
(file:///usr/local/google/home/vegorov/src/dart/dart/sdk/lib/_internal/pub/lib/src/barback/load_all_transformers.dart:98)
#64 newFuture.<anonymous closure>
(file:///usr/local/google/home/vegorov/src/dart/dart/sdk/lib/_internal/pub/lib/src/utils.dart:83)
#65 _rootRunUnary (dart:async/zone.dart:842)
#66 _CustomZone.runUnary (dart:async/zone.dart:748)
#67 _Future._propagateToListeners.handleValueCallback
(dart:async/future_impl.dart:488)
#68 _Future._propagateToListeners (dart:async/future_impl.dart:571)
#69 _Future._completeWithValue (dart:async/future_impl.dart:331)
#70 _Future._asyncComplete.<anonymous closure>
(dart:async/future_impl.dart:393)
#71 _rootRun (dart:async/zone.dart:835)
#72 _CustomZone.run (dart:async/zone.dart:740)
#73 _CustomZone.runGuarded (dart:async/zone.dart:648)
#74 _CustomZone.bindCallback.<anonymous closure> (dart:async/zone.dart:673)
#75 _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:23)
#76 _asyncRunCallback (dart:async/schedule_microtask.dart:32)
#77 _RawReceivePortImpl._handleMessage
(dart:isolate-patch/isolate_patch.dart:126)



// Vyacheslav Egorov

[Bob Nystrom]

On Tue, Jul 22, 2014 at 6:52 AM, Vyacheslav Egorov <veg...@google.com> wrote:

Any idea what is going on here, Bob?

The short answer is no. I spent a little time looking into it, but couldn't really make heads or tails of it. I think the SASS transformers are running multiple times because of the polymer transformers that get loaded later. 

Transformers are loaded one phase at a time so that the code for later transformers can import transformed Dart code. That means we imperatively modify that set of transformer phases. Barback supports that, but I think it may trigger some stuff to get retransformed. It shouldn't as far as I know since the new phases are at the end, but I don't understand it in detail.

Hopefully Natalie knows what's going on here.

- bob

[Dan Grove]

How can we make this much easier to debug?

--

You received this message because you are subscribed to the Google Groups "Dart Web Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web+uns...@dartlang.org.
Visit this group at http://groups.google.com/a/dartlang.org/group/web/.

To view this discussion on the web visit https://groups.google.com/a/dartlang.org/d/msgid/web/CAF8T8LyxQzB2KQ3ZuUg3%2BE-y%2Bg%2Bb6FxG79MGdQQRziOeQaoRYQ%40mail.gmail.com.

[Vyacheslav Egorov]

Yeah, I would like to know that as well. Right now given how async and complex this whole stack is - it's next to impossible to make heads and tails of what is going on inside without sprinkling debug prints all over the code base - one level of async stuff at a time.

Transformer itself looks completely innocent yet understanding why it does not work is harder than I imagined.

[Natalie Weizenbaum]

I can't see any context here.

[Bob Nystrom]

On Tue, Jul 22, 2014 at 11:47 AM, Natalie Weizenbaum <nw...@google.com> wrote:

I can't see any context here.

Here's the full thread: https://groups.google.com/a/dartlang.org/forum/#!topic/web/8bnbBDzyd9Q

- bob

[Natalie Weizenbaum]

Bob is right that the Sass transformer is running multiple times because of the details of the transformer load process. Specifically, "package:" imports don't work by default when loading transformers, and to work around this pub adds an internal transformer that rewrites them to something that does work. This transformer sticks around until all the transformers for a package are loaded, and is then removed. Its removal causes barback to re-run the previously loaded transformers, since they're now in a different place in the transformer graph. This shouldn't cause any practical issues, though.

The "removed available asset" error is much more serious, and indicates a bug in barback itself. Please file an issue with detailed reproduction instructions and I'll look into it as soon as I'm back at work.

- Natalie

[Bob Nystrom]

On Tue, Jul 22, 2014 at 1:29 PM, Natalie Weizenbaum <nw...@google.com> wrote:

This transformer sticks around until all the transformers for a package are loaded, and is then removed. Its removal causes barback to re-run the previously loaded transformers, since they're now in a different place in the transformer graph.

Oh, right. I forgot that the import rewriter gets pulled from the beginning of the phases.

 

This shouldn't cause any practical issues, though.

It's a performance hit to re-run a number of transformers multiple times.

Will this go away when the VM has import hooks?

- bob

[Natalie Weizenbaum]

Yes.

[Vyacheslav Egorov]

Thanks for the details, Natalie!

> The "removed available asset" error is much more serious, and indicates a bug in barback itself.

It was not an error, it was a debugging print that I added trying to understand why we are rerunning transformers. So there is nothing to file.

Though I am still not completely certain if the fact that we "dirty" a transformer because of this asset "removal" is bug or not. 

Overall to reproduce this issue you just need to check out http://github.com/mraleph/irhydra/ insteresting-mode brunch and run pub build.

// Vyacheslav Egorov

[Natalie Weizenbaum]

I'm a little confused... is there still a problem here other than running the transformer multiple times?

[Vyacheslav Egorov]

There is a problem that seems to be a bug in transformer: SCSS files depend on each other through import statement --- which means you can't do consumePrimary() once you processed one SCSS file because then this SCSS file disappears from assets and can't be loaded  from another SCSS file we are processing. 

It seems that SASS transformer should be split in two:

First: inlines imports and generates CSS outputs without consuming SCSS assets themselfes.

Second: does nothing but simple does consumePrimary on all .SCSS files it sees to prevent them from appearing in the build/ folder.

// Vyacheslav Egorov

[Natalie Weizenbaum]

All the Sass transforms should run in parallel, not in sequence—if one of them consumes its primary input, its siblings should still be able to see that input, because they're running in the same phase.

[Vyacheslav Egorov]

Ok, then there is definitely a bug. I will file it.

// Vyacheslav Egorov

[Vyacheslav Egorov]

I filled it: https://code.google.com/p/dart/issues/detail?id=20180

// Vyacheslav Egorov

[Bob Nystrom]

On Thu, Jul 24, 2014 at 4:51 AM, Vyacheslav Egorov <veg...@google.com> wrote:

I filled it: https://code.google.com/p/dart/issues/detail?id=20180

Thanks! And thanks again for putting so much time looking into this.

- bob