Feature request: use virtual path provider IO

[SteveEisner]

Right now, if its source files are accessed via a virtual path
provider, Less will fail:
at System.IO.File.ReadAllText(String path)
at dotless.Core.ExtensibleEngine.TransformToCss(String filename)
at dotless.Core.AspCacheDecorator.TransformToCss(String filename)
at dotless.Core.HandlerImpl.Execute()
at dotless.Core.LessCssHttpHandler.ProcessRequest(HttpContext
context)
at
System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute
()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step,
Boolean& completedSynchronously)

The problem is that with a virtual path provider, you can't directly
access via System.IO.File, because the data might not even be in a
file. (In our case, we're using a file-backed VPP, but the files are
not sitting in the local virtual directory.)

If it's possible for you to adapt for this case, here's an explanation
about how to do it:
http://msdn.microsoft.com/en-us/library/aa479502.aspx#vpp_vga_topic6
By default, this method of accessing files will work for both VPP and
non-VPP cases, although it's a limited set of IO operations that's
available. I haven't looked through all of the .less code but the
one "File.ReadAllText" case I hit would be easily converted.

Thanks!
Steve

[Daniel Hölbling]

Sorry to ask dumb questions: But there is a file path that could be loaded through File.ReadAllText?

Because there is an appropriate abstraction for resolving the paths in .less, we'd just have to make it configurable through XML

[SteveEisner]

Not really, it's a little difficult to explain the whole process, but
simplified: there's a system on one side of a fence that knows the
"true path" and there's my ASP.NET app on the other side, and they
can't easily communicate. In fact, for the case of static file
handling, I don't even have any opportunity to intercept on the
ASP.NET side. Plus, there's no easy solution for the general case for
VPPs, in which the contents of the files could be coming from a SQL
database, ZIP file, network or memory-backed storage, etc.

Typically the correct solution would be to wrap up your own
"ReadAllText" routine (use VPP Open method that returns a stream; read
to end of stream & return that string) But I can see how maybe this
particular function is shared with your command-line version, which
couldn't use the VPP functions. Perhaps instead you should be passing
an abstracted Path - either a FilePath or a VirtualPath or whatever -
or better yet, just passing a Stream into
ExtensibleEngine.TransformToCss?

Steve

[Daniel Hölbling]

Well.. I looked at the samples by MSDN and it's pretty clear we can do that quite easily.

The only thing that does not work is passing the stream. We have to process the whole document at once, so looking at streams doesn't really work.

There is already a spike up in the air about abstracting away the file source, so this would be very easy to implement.

I've been waiting for Chris or Erik to look at the code before pushing it into the mainline, but I guess I'll just do it anyway tomorrow. (And create a VPP along the way).

greetings Daniel

[SteveEisner]

Thanks Daniel, I appreciate the quick response. Less/.NET is already
a great product and your commitment to the community only makes it
better!

[Daniel Hölbling]

Thanks.

Just one quick question (although I could check that myself.) 

Is the VPP completely backwards compatible to System.IO.File?

If so we could enable it by default and there would be no need to add any configuration stuff..

[Chris]

Sorry Daniel, Im @ ThoughtWorks training all this week so its
difficult for me to look at the code. Your a better dev than me
anyway :-). If you haven't already just add some additional tests that
account for files sources an also that the LessFileSource (or
whatever) still handles files from absolute/relative and web root
relative paths (If HttpContext or abstraction exists).

Chris

[Daniel Hölbling]

@Chris: No problem chris. I rechecked the code and decided to merge it into main.

@Steve: I just pushed a new release that adds the requested functionality. Please give it a try and report back with suggestions. You can read about how to enable the VirtualPathSource from my blogpost about it:

http://www.tigraine.at/2010/01/06/less-now-supports-files-from-the-virtualpathprovider/

greetings Daniel

[SteveEisner]

Thanks Daniel, that was quick! I'll give it a test ASAP

(BTW I think you made the right choice doing it via a configurable
element, since VPP may not always be available in every environment
you want to support)

Steve

On Jan 6, 10:16 am, Daniel Hölbling <hoelblin...@gmail.com> wrote:
> @Chris: No problem chris. I rechecked the code and decided to merge it into
> main.
>
> @Steve: I just pushed a new release that adds the requested functionality.
> Please give it a try and report back with suggestions. You can read about

> how to enable the VirtualPathSource from my blogpost about it:http://www.tigraine.at/2010/01/06/less-now-supports-files-from-the-vi...
>
> <http://www.tigraine.at/2010/01/06/less-now-supports-files-from-the-vi...>greetings

[SteveEisner]

Hmm, seems like the VirtualPathSource is passing along absolute file
paths rather than virtual paths. Here's what I saw in a request for
[app vdir]/css/test.less

dotless.Core.DLL!dotless.Core.VirtualPathSource.GetSource(string key
= "c:\\app\\css\\test.less") + 0x52 bytes
dotless.Core.DLL!dotless.Core.HandlerImpl.Execute() + 0x5e bytes
dotless.Core.DLL!dotless.Core.LessCssHttpHandler.ProcessRequest
(System.Web.HttpContext context = {System.Web.HttpContext}) + 0x7c
bytes
System.Web.dll!
System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute
() + 0xb6 bytes

I looked at your source and it seems like HandlerImpl is doing a
conversion from virtual->real via MapPath. But VPPs are only really
able to accept virtual path requests - even if the IPathProvider
(which is not a VPP, but Server?) "knows" it maps to a local file in
the vdir, the virtual path provider may not understand the filesystem.
In this case, I could write something to un-map it back to a virtual
path, but I sometimes fall through to the default provider, which will
crash on an absolute path.

Steve

[Daniel Hölbling]

Damn i knew i overlooked smth.
I'll check today and get back to you.

07.01.2010 00:43 schrieb am "SteveEisner" <steve...@gmail.com>:

Hmm, seems like the VirtualPathSource is passing along absolute file
paths rather than virtual paths.  Here's what I saw in a request for
[app vdir]/css/test.less

       dotless.Core.DLL!dotless.Core.VirtualPathSource.GetSource(string key
= "c:\\app\\css\\test.less") + 0x52 bytes
       dotless.Core.DLL!dotless.Core.HandlerImpl.Execute() + 0x5e bytes
       dotless.Core.DLL!dotless.Core.LessCssHttpHandler.ProcessRequest
(System.Web.HttpContext context = {System.Web.HttpContext}) + 0x7c
bytes
       System.Web.dll!

System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execut...

() + 0xb6 bytes

I looked at your source and it seems like HandlerImpl is doing a
conversion from virtual->real via MapPath.  But VPPs are only really
able to accept virtual path requests - even if the IPathProvider
(which is not a VPP, but Server?) "knows" it maps to a local file in
the vdir, the virtual path provider may not understand the filesystem.
In this case, I could write something to un-map it back to a virtual
path, but I sometimes fall through to the default provider, which will
crash on an absolute path.

Steve

On Jan 6, 1:21 pm, SteveEisner <steveeis...@gmail.com> wrote: > Thanks Daniel, that was quick!  I'l...

[Daniel Hölbling]

Ok, Was a busy day so sorry I didn't have time to check the issue.

I think it's related to the Server.MapPath abstraction that could be easily disabled if I provide another configuration parameter for it.

I'll think about a good way to solve the issue tomorrow.

greetings Daniel

[Daniel Hölbling]

Soo..

I was pretty stupid announcing the problem as solved last time without really investigating why the VPP failed when supplied with paths not prefixed with a ~... 

While coding I had this distinct feeling that I was doing something that was already there, but I didn't realize it until I looked at the code today and found out that we already have an abstraction to Server.Mappath (the IPathProvider interface).

I removed that one and sticked to the more complete ILessSource abstraction (IPathProvider was only translating paths from virtual to physical, ILessSource translates also loads the contents of files).

There are no changes required (besides the one from the blogpost), once you define a different class in your DotlessConfiguration that implements ILessSource it will load stuff from there, thus working with the VirtualPathProvider.

I am now also investigating how much the VirtualPathProvider differs from the Server.Mappath. From what I read IIS uses VirtualPathProvider by default to serve up files inside a web, and since I removed the Server.Mappath abstraction and no absolute paths get passed into the PathProvider there should not be any real issues with just setting VirtualPathProvider as default.

Anyway, the newest code from TeamCity solves the problem (this time for real). 

I'll think about removing the Server.MapPath source alltogether. 

I am also not quite content with how caching is handled at the moment. Some sources (as the VPP one) are not cacheable at the moment, and you may see an Exception telling you that. I am currently looking at ways of tying the cache method used with the Source, so every source uses the appropriate way of caching (making it difficult to invalidate it).

So long, please let me know if anything is broken. I tested pretty extensively this time, but maybe I missed something.

The code I mean is:

http://www.dotlesscss.com:8081/viewLog.html?buildId=133&tab=buildResultsDiv&buildTypeId=bt3

greetings Daniel

[SteveEisner]

Hi Daniel. With the latest source, I'm getting this:

ArgumentException: Absolute path information is required.
at
System.Security.Util.StringExpressionSet.CreateListFromExpressions
(String[] str, Boolean needFullPath)
at System.Security.Permissions.FileIOPermission.AddPathList
(FileIOPermissionAccess access, AccessControlActions control, String[]
pathListOrig, Boolean checkForDuplicates, Boolean needFullPath,
Boolean copyPathList)
at System.Security.Permissions.FileIOPermission..ctor
(FileIOPermissionAccess access, String path)
at System.Web.InternalSecurityPermissions.PathDiscovery(String
path)
at System.Web.Caching.CacheDependency.Init(Boolean isPublic, String
[] filenamesArg, String[] cachekeysArg, CacheDependency dependency,
DateTime utcStart)
at System.Web.Caching.CacheDependency..ctor(String filename,
DateTime start)
at System.Web.Caching.CacheDependency..ctor(String filename)
at dotless.Core.Abstractions.CssCache.Insert(String fileName,
String css)
at dotless.Core.AspCacheDecorator.TransformToCss(LessSourceObject
source)

at dotless.Core.HandlerImpl.Execute()
at dotless.Core.LessCssHttpHandler.ProcessRequest(HttpContext
context)
at
System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute
()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step,
Boolean& completedSynchronously)

which may be the "not cacheable" exception you were talking about?
I can try to help you debug the issue if you'd like, though I'm not
doing anything other than what you described in your blog post.

If it helps, I have similar caching problems with our own VPP. I
ended up using

public override CacheDependency GetCacheDependency(string
virtualPath, IEnumerable virtualPathDependencies, DateTime utcStart)
{
var coll = new System.Collections.Generic.List<string>();
foreach (string v in virtualPathDependencies)
{
if (FileExists(v)) // <- a call to its own FileExists
override
coll.Add(HttpRuntime.AppDomainAppPath + v);
}

return new System.Web.Caching.CacheDependency(coll.ToArray
(), utcStart);
}

which isn't pretty... it ignores dependencies on anything that doesn't
exist as a physical file. But it gets past the problem!

Steve

> The code I mean is:http://www.dotlesscss.com:8081/viewLog.html?buildId=133&tab=buildResu...
>
> greetings Daniel

>
> On Fri, Jan 8, 2010 at 12:11 AM, Daniel Hölbling <tigra...@tigraine.at>wrote:
>
>
>
> > Ok, Was a busy day so sorry I didn't have time to check the issue.
> > I think it's related to the Server.MapPath abstraction that could be easily
> > disabled if I provide another configuration parameter for it.
> > I'll think about a good way to solve the issue tomorrow.
>
> > greetings Daniel
>

> > On Thu, Jan 7, 2010 at 10:16 AM, Daniel Hölbling <tigra...@tigraine.at>wrote:
>
> >> Damn i knew i overlooked smth.
> >> I'll check today and get back to you.
>

[Daniel Hölbling]

Hi Steve,

yes. That is the problem with the VirtualPathProvider I was talking about. You can't cache it since there is no absolute path present.

You solution looks good, but I already had a safeguard in place that should be throwing a custom exception that is far more descriptive than the one you are getting. I'll recheck and fix that.

For now, disable the cache through your web.config and you should be fine.

I'm also looking for ways on how to cache a VPP stream.. (possible just in-memory, but let's see)

greetings Daniel

[Daniel Hölbling]

Oh, and while at it.

I found that the sampleweb web.config had a bug in it.

The proper attribute for disabling the cache is 

cache="false" 

not cacheEnabled="false".

Since the setting was set to true all the time that mistake got overlooked. So to fix your problem, just set cache="false". I'll now fix the exception.

[Daniel Hölbling]

Also, while at it.

I forgot to mention that the new default is not FileSource but AspServerPathSource. FileSource is only used by the Console runner while AspServerPathSource is used by the HttpHandler. 

[Daniel Hölbling]

The latest pinned build should now politely tell you that your PathProvider does not support caching and you need to disable it. 

Here the direct link:

http://www.dotlesscss.com:8081/viewLog.html?buildId=134&buildTypeId=bt3&tab=artifacts

greetings Daniel

[Wayne]

You could just combine them statically in your build process or at
design time :-)

http://blog.waynebrantley.com/2009/12/ultimate-automatic-stylesheet-combining.html

On Jan 14, 6:24 am, Daniel Hölbling <hoelblin...@gmail.com> wrote:
> The latest pinned build should now politely tell you that your PathProvider
> does not support caching and you need to disable it.

> Here the direct link:http://www.dotlesscss.com:8081/viewLog.html?buildId=134&buildTypeId=b...
>
> <http://www.dotlesscss.com:8081/viewLog.html?buildId=134&buildTypeId=b...>greetings
> Daniel

>
> On Thu, Jan 14, 2010 at 12:20 PM, Daniel Hölbling <hoelblin...@gmail.com>wrote:
>
> > Also, while at it.
> > I forgot to mention that the new default is not FileSource but
> > AspServerPathSource. FileSource is only used by the Console runner while
> > AspServerPathSource is used by the HttpHandler.
>

> > On Thu, Jan 14, 2010 at 12:13 PM, Daniel Hölbling <hoelblin...@gmail.com>wrote:
>
> >> Oh, and while at it.
> >> I found that the sampleweb web.config had a bug in it.
> >> The proper attribute for disabling the cache is
>
> >> cache="false"
> >> not cacheEnabled="false".
>
> >> Since the setting was set to true all the time that mistake got
> >> overlooked. So to fix your problem, just set cache="false". I'll now fix the
> >> exception.
>

> >> On Thu, Jan 14, 2010 at 12:07 PM, Daniel Hölbling <hoelblin...@gmail.com>wrote:
>
> >>> Hi Steve,
> >>> yes. That is the problem with the VirtualPathProvider I was talking
> >>> about. You can't cache it since there is no absolute path present.
> >>> You solution looks good, but I already had a safeguard in place that
> >>> should be throwing a custom exception that is far more descriptive than the
> >>> one you are getting. I'll recheck and fix that.
> >>> For now, disable the cache through your web.config and you should be
> >>> fine.
> >>> I'm also looking for ways on how to cache a VPP stream.. (possible just
> >>> in-memory, but let's see)
>
> >>> greetings Daniel
>

[SteveEisner]

Just to follow up - with cache="false", and the latest code, it
appears that the virtual path provider is working fine!

Thanks for all the work on this. I can't wait to get my team switched
over to .less

Steve

On Jan 14, 3:24 am, Daniel Hölbling <hoelblin...@gmail.com> wrote:
> The latest pinned build should now politely tell you that your PathProvider
> does not support caching and you need to disable it.

> Here the direct link:http://www.dotlesscss.com:8081/viewLog.html?buildId=134&buildTypeId=b...
>
> <http://www.dotlesscss.com:8081/viewLog.html?buildId=134&buildTypeId=b...>greetings
> Daniel

>
> On Thu, Jan 14, 2010 at 12:20 PM, Daniel Hölbling <hoelblin...@gmail.com>wrote:
>
>
>
> > Also, while at it.
> > I forgot to mention that the new default is not FileSource but
> > AspServerPathSource. FileSource is only used by the Console runner while
> > AspServerPathSource is used by the HttpHandler.
>

> > On Thu, Jan 14, 2010 at 12:13 PM, Daniel Hölbling <hoelblin...@gmail.com>wrote:
>
> >> Oh, and while at it.
> >> I found that the sampleweb web.config had a bug in it.
> >> The proper attribute for disabling the cache is
>
> >> cache="false"
> >> not cacheEnabled="false".
>
> >> Since the setting was set to true all the time that mistake got
> >> overlooked. So to fix your problem, just set cache="false". I'll now fix the
> >> exception.
>

> >> On Thu, Jan 14, 2010 at 12:07 PM, Daniel Hölbling <hoelblin...@gmail.com>wrote:
>
> >>> Hi Steve,
> >>> yes. That is the problem with the VirtualPathProvider I was talking
> >>> about. You can't cache it since there is no absolute path present.
> >>> You solution looks good, but I already had a safeguard in place that
> >>> should be throwing a custom exception that is far more descriptive than the
> >>> one you are getting. I'll recheck and fix that.
> >>> For now, disable the cache through your web.config and you should be
> >>> fine.
> >>> I'm also looking for ways on how to cache a VPP stream.. (possible just
> >>> in-memory, but let's see)
>
> >>> greetings Daniel
>