Datafiles with xml-special characters loaded via pre

25 views
Skip to first unread message

Charilaos Skiadas

unread,
Dec 28, 2025, 4:23:07 PM12/28/25
to prete...@googlegroups.com
I made an issue about it  https://github.com/PreTeXtBook/pretext/issues/2754

But thought I’d post here as well. My situation is that I have a C-header file that naturally contains a bunch of < symbols, and I would like to be able to #include it, so I thought the datafile + pre route was the way to go. Even if there is an alternative, the problem would still remain in the case of any text files containing < or other similarly special characters.

Currently during the asset generation process this file’s code is read as is and plucked into a <pi:text-file xmlns:pi="http://pretextbook.org/2020/pretext/internal">{}</pi:text-file>
container which would result in invalid xml, and in my case spurious runestone errors about not being able to locate the file.

Hoping folks can verify that this is a bug or point me in the right direction.

Charilaos Skiadas
Department of Mathematics
Hanover College

Andrew Scholer

unread,
Dec 28, 2025, 4:38:57 PM12/28/25
to prete...@googlegroups.com
On fone, so brief...

IMO datafiles are for data. Code should be in a program or pf.

Runestone programs that run server side can include other programs, even ones that are not activecodes.

See Listing 5.2.18: add.h (version 1)

For data file/pre I recommend you not use @source. Instead do an xi:include inside the pre.


Andrew Scholer


On Sun, Dec 28, 2025, 1:23 PM Charilaos Skiadas <cski...@gmail.com> wrote:
But thought I’d post here as well. My situation is that I ha ve a C-header file that naturally contains a bunch of < symbols, and I would like to be able to #include it, so I thought the datafile + pre route was the way to go. Even if there is an alternative, the problem would still remain in the case of any text files containing < or other similarly special characters.

Currently during the asset generation process this file’s code is read as is and plucked into a <pi:text-file xmlns:pi="http://pretextbook.org/2020/pretext/internal">{}</pi:text-file>
container which would result in invalid xml, and in my case spurious runestone errors about not being able to locate the file.

Hoping folks can verify that this is a bug or point me in the right direction.

Charilaos Skiadas
Department of Mathematics
Hanover College

--
You received this message because you are subscribed to the Google Groups "PreTeXt development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/F56AB760-BBC7-46BA-9926-F06B2E7BEA40%40gmail.com.

Charilaos Skiadas

unread,
Dec 28, 2025, 5:05:34 PM12/28/25
to prete...@googlegroups.com
Ah ok and I see in that discussion that the pre-source thing should be getting deprecated, which sounds good. I wonder why I didn’t find that when I searched in the issues for datafile, I guess it’s because it’s actually a pull request instead.

So my problem is that I don’t want these files to be visible anywhere in the book, and I think that having them as listings/programs would make them visible in some way, am I wrong in that? They are meant to be solely there so that #include in the exercise program preamble can find them. So they are simply extra source files that the jobe server should have around when trying to compile and run the student code, and should not be visible anywhere else.

I’ll try the xi:include approach and see if that fixes my problems at least in terms of using datafile.

Charilaos Skiadas
Department of Mathematics
Hanover College

Charilaos Skiadas

unread,
Dec 29, 2025, 11:02:01 AM12/29/25
to prete...@googlegroups.com
Update: xi:include made the examples work on my end (a tad annoying that I had to change the path a little bit as xi:include is relative to the processed file, but that makes sense).   
I would still like to know if there is a better approach to having some code that lives in a separate file, that is available for #include in a C-program, and which is not otherwise visible to the book reader, than using the datafile @hide=“yes” option. But I can live with the current approach, even if it’s misusing datafile a bit.

But in the meantime I set off to adjust the sample book and guide so they don’t suggest pre@source as the appropriate approach, and ran into a snag. When I changed https://github.com/PreTeXtBook/pretext/blob/0340cc4228e0a978abc8942dc7a25e2af9b5642a/examples/sample-book/rune.xml#L877
to instead use:
                <pre>
                    <xi:include parse="text" href="./ext/datafiles/stackoverflow.csv" />
                </pre>

then tried to build with: 
 ./pretext/pretext -v -c all -f html -d output/newSample2 -p examples/sample-book/publication-noparts.xml examples/sample-book/sample-book.xml
(which worked just fine before I made that change)

I got an error:

PTX:ERROR   : processing with /Users/haris/Downloads/pretext-master/xsl/pretext-html.xsl has failed

Traceback (most recent call last):
  File "/Users/haris/Downloads/pretext-master/./pretext/pretext", line 829, in <module>
    main()
  File "/Users/haris/Downloads/pretext-master/./pretext/pretext", line 718, in main
    ptx.html(
  File "/Users/haris/Downloads/pretext-master/pretext/pretext.py", line 3615, in html
    xsltproc(extraction_xslt, xml, None, tmp_dir, stringparams)
  File "/Users/haris/Downloads/pretext-master/pretext/pretext.py", line 3880, in xsltproc
    raise (e)
  File "/Users/haris/Downloads/pretext-master/pretext/pretext.py", line 3876, in xsltproc
    raise (texc)
  File "/Users/haris/Downloads/pretext-master/pretext/pretext.py", line 3839, in transform
    result_tree = xslt(src_tree, **stringparams)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/lxml/xslt.pxi", line 596, in lxml.etree.XSLT.__call__
lxml.etree.XSLTApplyError: margin

I’m attaching the full error log. Any ideas?
log.txt

Andrew Scholer

unread,
Dec 29, 2025, 11:32:40 AM12/29/25
to prete...@googlegroups.com
Just focusing on the datafile question (and not the Sample Book) for now.

I wanted to do the same invisible "files" thing for a while. Eventually, I decided to just make an appendix with the helper files from the book. It is not that I need readers not to see them, I just don't care that they do and I don't want to engage in discussing their contents where we make use of that code. If a reader really wants to go check out the code (possibly for a good reason, like running something locally) they can do so.

Also worth noting: hidden data files are still in the book. They are just hidden from view. An enterprising reader can view the source of the page and see the contents of the datafile.

If you are building an exercise and you want to hide some code from the user, there are other ways to do that (program/preamble/@visible="no" or program/tests). And you can use the xi:include trick in them.

Andrew

--
You received this message because you are subscribed to the Google Groups "PreTeXt development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.
On Dec 28, 2025, at 5:05 PM, Charilaos Skiadas <cski...@gmail.com> wrote:

Ah ok and I see in that discussion that the pre-source thing should be getting deprecated, which sounds good. I wonder why I didn’t find that when I searched in the issues for datafile, I guess it’s because it’s actually a pull request instead.

So my problem is that I don’t want these files to be visible anywhere in the book, and I think that having them as listings/programs would make them visible in some way, am I wrong in that? They are meant to be solely there so that #include in the exercise program preamble can find them. So they are simply extra source files that the jobe server should have around when trying to compile and run the student code, and should not be visible anywhere else.

I’ll try the xi:include approach and see if that fixes my problems at least in terms of using datafile.
Charilaos Skiadas
Department of Mathematics
Hanover College
On Dec 28, 2025, at 4:38 PM, Andrew Scholer <asch...@chemeketa.edu> wrote:

On fone, so brief...

IMO datafiles are for data. Code should be in a program or pf.

Runestone programs that run server side can include other programs, even ones that are not activecodes.

See Listing 5.2.18: add.h (version 1)

For data file/pre I recommend you not use @source. Instead do an xi:include inside the pre.


Andrew Scholer


On Sun, Dec 28, 2025, 1:23 PM Charilaos Skiadas <cski...@gmail.com> wrote:
I made an issue about it  https://github.com/PreTeXtBook/pretext/issues/2754

But thought I’d post here as well. My situation is that I ha ve a C-header file that naturally contains a bunch of < symbols, and I would like to be able to #include it, so I thought the datafile + pre route was the way to go. Even if there is an alternative, the problem would still remain in the case of any text files containing < or other similarly special characters.

Currently during the asset generation process this file’s code is read as is and plucked into a <pi:text-file xmlns:pi="http://pretextbook.org/2020/pretext/internal">{}</pi:text-file>
container which would result in invalid xml, and in my case spurious runestone errors about not being able to locate the file.

Hoping folks can verify that this is a bug or point me in the right direction.
Charilaos Skiadas
Department of Mathematics
Hanover College

--
You received this message because you are subscribed to the Google Groups "PreTeXt development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/F56AB760-BBC7-46BA-9926-F06B2E7BEA40%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "PreTeXt development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/CACm44N9f%3D0yXWBBbsH6Yn9qCTVmg_tAbub1Mu2hWZhWde_vnPQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "PreTeXt development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.

Andrew Scholer

unread,
Dec 29, 2025, 12:08:04 PM12/29/25
to prete...@googlegroups.com
re: Sample Book:

I can't reproduce what you are seeing.

You could try running xsltproc directly. Sometimes it spits out an error message that is more meaningful than that from the python script. From the SB dir, something like:

xsltproc --xinclude -o output/ --stringparam publisher publication-noparts.xml ../../xsl/pretext-html.xsl sample-book.xml

If you get it figured out and make a PR, please reference this (new) issue:
https://github.com/PreTeXtBook/pretext/issues/2755

Andrew

On Mon, Dec 29, 2025 at 8:02 AM Charilaos Skiadas <cski...@gmail.com> wrote:
--
You received this message because you are subscribed to the Google Groups "PreTeXt development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.
On Dec 28, 2025, at 5:05 PM, Charilaos Skiadas <cski...@gmail.com> wrote:

Ah ok and I see in that discussion that the pre-source thing should be getting deprecated, which sounds good. I wonder why I didn’t find that when I searched in the issues for datafile, I guess it’s because it’s actually a pull request instead.

So my problem is that I don’t want these files to be visible anywhere in the book, and I think that having them as listings/programs would make them visible in some way, am I wrong in that? They are meant to be solely there so that #include in the exercise program preamble can find them. So they are simply extra source files that the jobe server should have around when trying to compile and run the student code, and should not be visible anywhere else.

I’ll try the xi:include approach and see if that fixes my problems at least in terms of using datafile.
Charilaos Skiadas
Department of Mathematics
Hanover College
On Dec 28, 2025, at 4:38 PM, Andrew Scholer <asch...@chemeketa.edu> wrote:

On fone, so brief...

IMO datafiles are for data. Code should be in a program or pf.

Runestone programs that run server side can include other programs, even ones that are not activecodes.

See Listing 5.2.18: add.h (version 1)

For data file/pre I recommend you not use @source. Instead do an xi:include inside the pre.


Andrew Scholer


On Sun, Dec 28, 2025, 1:23 PM Charilaos Skiadas <cski...@gmail.com> wrote:
I made an issue about it  https://github.com/PreTeXtBook/pretext/issues/2754

But thought I’d post here as well. My situation is that I ha ve a C-header file that naturally contains a bunch of < symbols, and I would like to be able to #include it, so I thought the datafile + pre route was the way to go. Even if there is an alternative, the problem would still remain in the case of any text files containing < or other similarly special characters.

Currently during the asset generation process this file’s code is read as is and plucked into a <pi:text-file xmlns:pi="http://pretextbook.org/2020/pretext/internal">{}</pi:text-file>
container which would result in invalid xml, and in my case spurious runestone errors about not being able to locate the file.

Hoping folks can verify that this is a bug or point me in the right direction.
Charilaos Skiadas
Department of Mathematics
Hanover College

--
You received this message because you are subscribed to the Google Groups "PreTeXt development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/F56AB760-BBC7-46BA-9926-F06B2E7BEA40%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "PreTeXt development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/CACm44N9f%3D0yXWBBbsH6Yn9qCTVmg_tAbub1Mu2hWZhWde_vnPQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "PreTeXt development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.

Charilaos Skiadas

unread,
Dec 29, 2025, 1:05:28 PM12/29/25
to prete...@googlegroups.com
Turns out I was being an idiot and running off some old version of pretext. Apparently the fact that the directory was called “pretext-master” convinced me I was in the right place, when in fact that wasn’t even actively tracked as a git repository.

Anyway, switching to the real pretext and building worked just fine. I did run into the following “snag”: If you use something like:

<pre>
<xi:include parse="text" href="./ext/datafiles/stackoverflow.csv" />
</pre>

Then all the spaces before the <xi part are actually included in the final produced file, so we end up with a csv file whose top looks like this:

PastedGraphic-1.png

Perhaps this is completely to be expected, but it’s certainly not what someone trying to use this pattern would expect to have happen, and it could certainly end in problems if loading Python code this way (yes not the intended use-case). To avoid it one would have to write instead:
<pre><xi:include parse="text" href="./ext/datafiles/stackoverflow.csv" /></pre>

(or else indent the xi all the way to the left, which feels wrong.)

I’ll prepare the PR with that formulation in mind, though I really wish there was a more intuitive way around this.

Charilaos Skiadas
Department of Mathematics
Hanover College

Charilaos Skiadas

unread,
Dec 29, 2025, 2:08:21 PM12/29/25
to prete...@googlegroups.com
On Dec 29, 2025, at 1:05 PM, Charilaos Skiadas <cski...@gmail.com> wrote:

Turns out I was being an idiot and running off some old version of pretext. Apparently the fact that the directory was called “pretext-master” convinced me I was in the right place, when in fact that wasn’t even actively tracked as a git repository.

Anyway, switching to the real pretext and building worked just fine. I did run into the following “snag”: If you use something like:

<pre>
<xi:include parse="text" href="./ext/datafiles/stackoverflow.csv" />
</pre>

Then all the spaces before the <xi part are actually included in the final produced file, so we end up with a csv file whose top looks like this:

Andrew Scholer

unread,
Dec 30, 2025, 12:07:29 PM12/30/25
to prete...@googlegroups.com
Yes, I use the 

        <pre>
<xi:include parse="text" href="./ext/datafiles/stackoverflow.csv" />
        </pre>

construct a lot for inserting code.

It does look like odd formatting of the XML, but once you realize that xi:include is "paste that file's contents right here", it makes sense. You would not write:

        <pre>
            First line
Second line
Third line
        </pre>

And expect the lines to be evenly indented. And that is exactly what you have written if you indent the xi:include.

We could hit datafile/pre with a template that forcibly trims whitespace from the first line. But as soon as we do that, someone is going to have a reason they need to start a datafile with some spaces. (I can think of programming exercises I have used where a student is reading text that starts with whitespace.)

It might work acceptably well to de-indent the first line and remove up to one blank line. That way if there is some cruft space it is taken care of. But if you really want to start with whitespace you can add a blank line before the whitespace you want to keep.

That said, after lots of experience wrestling with whitespace handling in codelike elements as both an author and a developer, I lean more towards giving the author exactly what they wrote as opposed to a cleaned up version of what they wrote. Most of the time we can eliminate spacing that comes from XML nesting without violating author intent. But as an author, I would much rather need to learn to not indent my xi:includes than to realize PreTeXt is stripping the indentation from something in a way that I can't overcome.


Rob Beezer

unread,
Dec 31, 2025, 11:18:00 AM12/31/25
to prete...@googlegroups.com
Just taking a look at this PR now, with an eye to suggesting a "repair" of the
deprecation by the pre-processor.

Edit to the sample article now hard-codes the directory named "ext" (which is
the name of the "external" directory), which is purposely *not* named "external"
to test the relocatable (or replaceable for a version) nature of this directory.
So that cannot stand as-is, not at all correct to have "ext" hard-coded into
source. And suggests a "repair" is going to be difficult.

I'll admit to not having followed this discussion closely, given preparations
for JMM, and will not soon have time to dig deeper. But it seems there is a
philosophical bent to this:

A. Is a datafile part of your source, like a #p or a #theorem? Then it lives
with your source files (near or below main.ptx) and it is appropriate to
xi:include it, or just author it right inside datafile/pre.

B. Or is a datafile an external file that you bring to your project, like a
JPEG image? Then it lives int eh "external" directory and it is consistent with
other markup to point at it with @source.

Perhaps I should have looked closer at the original complaint about handling
special characters, but it feels like this has moved off into other directions.

Rob
>>> https://github.com/PreTeXtBook/pretext/issues/2755 <https://github.com/
>>> PreTeXtBook/pretext/issues/2755>
>>>
>>> Andrew
>>>
>>> On Mon, Dec 29, 2025 at 8:02 AM Charilaos Skiadas <cski...@gmail.com
>>> <mailto:cski...@gmail.com>> wrote:
>>>
>>> Update: xi:include made the examples work on my end (a tad annoying that
>>> I had to change the path a little bit as xi:include is relative to the
>>> processed file, but that makes sense).
>>> I would still like to know if there is a better approach to having some
>>> code that lives in a separate file, that is available for #include in a
>>> C-program, and which is not otherwise visible to the book reader, than
>>> using the datafile @hide=“yes” option. But I can live with the current
>>> approach, even if it’s misusing datafile a bit.
>>>
>>> But in the meantime I set off to adjust the sample book and guide so they
>>> don’t suggest pre@source as the appropriate approach, and ran into a
>>> snag. When I changed https://github.com/PreTeXtBook/pretext/
>>> blob/0340cc4228e0a978abc8942dc7a25e2af9b5642a/examples/sample-book/
>>> rune.xml#L877 <https://github.com/PreTeXtBook/pretext/
>>> blob/0340cc4228e0a978abc8942dc7a25e2af9b5642a/examples/sample-book/
>>> rune.xml#L877>
>>> email to pretext-dev...@googlegroups.com <mailto:pretext-
>>> dev+uns...@googlegroups.com>.
>>> To view this discussion visit https://groups.google.com/d/msgid/pretext-
>>> dev/8E9BC386-0CC6-4674-8C2A-69B38A9DE9D9%40gmail.com <https://
>>> groups.google.com/d/msgid/pretext-
>>> dev/8E9BC386-0CC6-4674-8C2A-69B38A9DE9D9%40gmail.com?
>>> utm_medium=email&utm_source=footer>.
>>>
>>>> On Dec 28, 2025, at 5:05 PM, Charilaos Skiadas <cski...@gmail.com
>>>> <mailto:cski...@gmail.com>> wrote:
>>>>
>>>> Ah ok and I see in that discussion that the pre-source thing should be
>>>> getting deprecated, which sounds good. I wonder why I didn’t find that
>>>> when I searched in the issues for datafile, I guess it’s because it’s
>>>> actually a pull request instead.
>>>>
>>>> So my problem is that I don’t want these files to be visible anywhere in
>>>> the book, and I think that having them as listings/programs would make
>>>> them visible in some way, am I wrong in that? They are meant to be
>>>> solely there so that #include in the exercise program preamble can find
>>>> them. So they are simply extra source files that the jobe server should
>>>> have around when trying to compile and run the student code, and should
>>>> not be visible anywhere else.
>>>>
>>>> I’ll try the xi:include approach and see if that fixes my problems at
>>>> least in terms of using datafile.
>>>>
>>>> Charilaos Skiadas
>>>> Department of Mathematics
>>>> Hanover College
>>>>
>>>>
>>>>> On Dec 28, 2025, at 4:38 PM, Andrew Scholer <asch...@chemeketa.edu
>>>>> <mailto:asch...@chemeketa.edu>> wrote:
>>>>>
>>>>> On fone, so brief...
>>>>>
>>>>> IMO datafiles are for data. Code should be in a program or pf.
>>>>>
>>>>> Runestone programs that run server side can include other programs,
>>>>> even ones that are not activecodes.
>>>>>
>>>>> See Listing 5.2.18: add.h (version 1)
>>>>> https://runestone.academy/ns/books/published/PTXSB/
>>>>> activecode.html#addh-v1-listing <https://runestone.academy/ns/books/
>>>>> published/PTXSB/activecode.html#addh-v1-listing>
>>>>>
>>>>> For data file/pre I recommend you not use @source. Instead do an
>>>>> xi:include inside the pre.
>>>>>
>>>>> See https://github.com/PreTeXtBook/pretext/pull/2615 <https://
>>>>> github.com/PreTeXtBook/pretext/pull/2615>
>>>>>
>>>>> Andrew Scholer
>>>>>
>>>>>
>>>>> On Sun, Dec 28, 2025, 1:23 PM Charilaos Skiadas <cski...@gmail.com
>>>>> <mailto:cski...@gmail.com>> wrote:
>>>>>
>>>>> I made an issue about it https://github.com/PreTeXtBook/pretext/
>>>>> issues/2754 <https://github.com/PreTeXtBook/pretext/issues/2754>
>>>>>
>>>>> But thought I’d post here as well. My situation is that I ha ve a
>>>>> C-header file that naturally contains a bunch of < symbols, and I
>>>>> would like to be able to #include it, so I thought the datafile +
>>>>> pre route was the way to go. Even if there is an alternative, the
>>>>> problem would still remain in the case of any text files containing
>>>>> < or other similarly special characters.
>>>>>
>>>>> Currently during the asset generation process this file’s code is
>>>>> read as is and plucked into a <pi:text-file xmlns:pi="http://
>>>>> pretextbook.org/2020/pretext/internal <http://pretextbook.org/2020/
>>>>> pretext/internal>">{}</pi:text-file>
>>>>> container which would result in invalid xml, and in my case
>>>>> spurious runestone errors about not being able to locate the file.
>>>>>
>>>>> Hoping folks can verify that this is a bug or point me in the right
>>>>> direction.
>>>>>
>>>>> Charilaos Skiadas
>>>>> Department of Mathematics
>>>>> Hanover College
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "PreTeXt development" group.
>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>> send an email to pretext-dev...@googlegroups.com
>>>>> <mailto:pretext-dev...@googlegroups.com>.
>>>>> To view this discussion visit https://groups.google.com/d/msgid/
>>>>> pretext-dev/F56AB760-BBC7-46BA-9926-F06B2E7BEA40%40gmail.com
>>>>> <https://groups.google.com/d/msgid/pretext-dev/F56AB760-
>>>>> BBC7-46BA-9926-F06B2E7BEA40%40gmail.com?
>>>>> utm_medium=email&utm_source=footer>.
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "PreTeXt development" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to pretext-dev...@googlegroups.com <mailto:pretext-
>>>>> dev+uns...@googlegroups.com>.
>>>>> To view this discussion visit https://groups.google.com/d/msgid/
>>>>> pretext-dev/
>>>>> CACm44N9f%3D0yXWBBbsH6Yn9qCTVmg_tAbub1Mu2hWZhWde_vnPQ%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/pretext-dev/
>>>>> CACm44N9f%3D0yXWBBbsH6Yn9qCTVmg_tAbub1Mu2hWZhWde_vnPQ%40mail.gmail.com?
>>>>> utm_medium=email&utm_source=footer>.
>>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "PreTeXt development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to pretext-dev...@googlegroups.com <mailto:pretext-
>>> dev+uns...@googlegroups.com>.
>>> To view this discussion visit https://groups.google.com/d/msgid/pretext-
>>> dev/8E9BC386-0CC6-4674-8C2A-69B38A9DE9D9%40gmail.com <https://
>>> groups.google.com/d/msgid/pretext-
>>> dev/8E9BC386-0CC6-4674-8C2A-69B38A9DE9D9%40gmail.com?
>>> utm_medium=email&utm_source=footer>.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "PreTeXt development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to pretext-dev...@googlegroups.com <mailto:pretext-
>>> dev+uns...@googlegroups.com>.
>>> To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/
>>> CACm44N8yKuarXvCDUjp%2BTE-N6XXBw%3DMTXn1jDBxEfsoD%3Dy69dw%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/pretext-dev/CACm44N8yKuarXvCDUjp%2BTE-
>>> N6XXBw%3DMTXn1jDBxEfsoD%3Dy69dw%40mail.gmail.com?
>>> utm_medium=email&utm_source=footer>.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "PreTeXt development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to pretext-dev...@googlegroups.com <mailto:pretext-
> dev+uns...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/
> C0BCFC35-E6B2-4A5D-AF80-A65BFE27B2E4%40gmail.com <https://groups.google.com/d/
> msgid/pretext-dev/C0BCFC35-E6B2-4A5D-AF80-A65BFE27B2E4%40gmail.com?
> utm_medium=email&utm_source=footer>.

Charilaos Skiadas

unread,
Dec 31, 2025, 12:55:06 PM12/31/25
to prete...@googlegroups.com
No urgency on this discussion, it can comfortably wait until after JMM (or in my case the first week of a new semester with a new course).

That's a great question Rob. To me it raises some issues about the current structure of the datafile spec.

I tend to think of datafiles being external to the code. For me (admittedly limited usage at this point) their purpose is to provide the content of some file that a program needs when it runs, for one reason or another. This could be a text file, an image file, some code to be #include'd, a zip file representing a directory, etc. I don't think non-image binary files are really supported at the moment though. But I think they should.

In my view this suggest a key distinction with the current behavior: That the visual inclusion of this file in the view page should NOT be a default, or even expected or allowed, behavior. The files are there to be read by the program, not be seen by the user.

At the same time we do want a situation where a user can edit a text file that will then be processed by a file, in some way. Any initial text for this text file feels to me more part of the book source, not unlike the initial text in a program code block.

So in my view I think I would rather see a world where a datafile tag has two ways to use it:

1. To specify an external file, without any visual representation of it. In this case it is an empty tag, and the file is linked to this via a @source attribute to the datafile tag (not an included pre tag). If the file is an image and the user wants to have it appear visually, then they can use a separate, sibling, image tag. There will be no need for the pretext processor to interpret this file in any way.
2. To specify a file as text that the user is meant to play with and possibly edit, or at the very least visually inspect. In this case this text is very much part of the book "source", and a datafile with no @source but with a pre and possibly xi:include sounds fine to me, although I personally would prefer it without a "pre" tag at all, i.e. the contents of the datafile tag are simply the contents that the user gets to play with, and they can be provided as either text directly in the tag, or via xi:include.

This way the datafile tag will have either a @source attribute or contents, but not both, and it will not need to have a @visible attribute. Not sure how "repairable" all that is though, and/or what use cases I might be missing here.

In terms of the PR the example in the sample book is currently my case 2 above, so I would advocate relocating/duplicating the file to the source, so that xi:include links work with it.

Charilaos Skiadas
Department of Mathematics
Hanover College
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/MTAwMDAzNC5iZWV6ZXI.1767197877%40pnsh.

Rob Beezer

unread,
Jan 2, 2026, 1:55:13 PMJan 2
to prete...@googlegroups.com
Thanks, Haris, for the discussion, which makes sense to me.

> This way the datafile tag will have either a @source attribute or contents,
but not both

Just as a technical matter, this distinction is easy to recognize and handle.

I'd love to hear more from Andrew and Brad about their hopes and dreams for
#datafile. I never felt too comfortable with what I was doing when they were
first implemented as PreText markup.

If I remember right, (binary) image files get a base64 encoding and then live on
the page. A bit of a mess. I do think supporting other binary files
(non-visible, non-editable) would make sense, but then perhaps they coud just be
their own files. But I'm not on top of the details here on this one.

Rob
>> On Dec 31, 2025, at 11:17 AM, 'Rob Beezer' via PreTeXt development <pretext-
> DA3933D8-CB2C-4E02-835C-A64CC3DC9BEC%40gmail.com <https://groups.google.com/d/
> msgid/pretext-dev/DA3933D8-CB2C-4E02-835C-A64CC3DC9BEC%40gmail.com?
> utm_medium=email&utm_source=footer>.

Andrew Scholer

unread,
Jan 2, 2026, 2:52:23 PMJan 2
to prete...@googlegroups.com
Agree that it would make sense to revisit datafile from a fundamental level. The technical needs of RS and the Jobe server that runs server side programs, the semantics of "data", and differing wants of authors have conspired to make it a bit of a catchall element. And at some level, every file is data, so it is not clear what a "datafile" is.

Root goals data file currently satisfies:
1) Authors want to provide programs that interact with files. This requires that RS get the "file" into the database in some way so that it works in places like assignment pages. For server side programs, it also needs to be uploaded from RS to the Jobe server as part of running the program.

2) Authors want to create programs that utilize code other than what is in the current <program>. That code needs to end up in the database. For server side code, it also needs to be uploaded to the Jobe server. In that case, the code might be included (C/C++ #include) passively or actively built (C++ .cpp file, Java code that is pretending to be a jar) as part of the compile command. Sometimes, authors want to link to visible code, other times they want to have some "invisible" libraries.

I think semantically it is worth distinguishing between the two cases. The first is clearly a "data" file. The second could be treated as data, but we would normally think of it as "code". It is part of the source of the program.

Is program source always book source??? I would be willing to say that it is. Regardless, I think that if you look at the book source, and there is an element with content that needs to be in RS DB and the Jobe server, it should be semantically clear if that content is intended to be part of the progam's source or whether it is "just data". (No opinion on what to do with quines).

The main reason people head down the road of putting code in a datafile seems to be: a) in RST books that was the best way to make the code available; b) it lets you make it invisible.

Other than supporting conversions somewhat gracefully, "a" should not be a concern. If "b" is a valid concern, then instead of improving the way datafile handles program contents, we should make it possible to make an "invisible" program element that does not get rendered in the book but does end up in the Runestone manifest for DB/Jobe purposes.



Also, I don't agree with "not intended to be seen by user" as a general statement. If I am writing a program to modify an image, it is often nice to see what I am starting with. If I am reading from a text file, it is very useful to see what is in the text file, even if I can't modify the content. Clearly a non-image binary file probably doesn't make sense to display, but for text/image I think the default should be to display the contents.




Rob Beezer

unread,
Jan 2, 2026, 3:15:17 PMJan 2
to prete...@googlegroups.com
Thanks, Andrew. That is very helpful, and I'm in general agreement on your
points. Whatever the *default* for display might be, perhaps we want the author
to have an option, or the interface might have an option to show/hide.

I meant to say that I thought you had done some recent work to make things like
setup-code, or shared libraries, or include'd code, more obvious in the markup,
but maybe I'm not fully on top of all the recent improvements.

Rob
> >>> PR prepared: https://github.com/PreTeXtBook/pretext/pull/2756 <https://
> github.com/PreTeXtBook/pretext/pull/2756>
> >>>> On Dec 29, 2025, at 1:05 PM, Charilaos Skiadas <cski...@gmail.com
> <mailto:asch...@chemeketa.edu>> wrote:
> >>>>>
> >>>>> re: Sample Book:
> >>>>>
> >>>>> I can't reproduce what you are seeing.
> >>>>>
> >>>>> You could try running xsltproc directly. Sometimes it spits out an error
> >>>>> message that is more meaningful than that from the python script.
> From the
> >>>>> SB dir, something like:
> >>>>>
> >>>>> xsltproc --xinclude -o output/ --stringparam publisher publication-
> >>>>> noparts.xml ../../xsl/pretext-html.xsl sample-book.xml
> >>>>>
> >>>>> If you get it figured out and make a PR, please reference this (new)
> issue:
> >>>>> https://github.com/PreTeXtBook/pretext/issues/2755 <https://
> github.com/PreTeXtBook/pretext/issues/2755> <https://github.com/ <https://
> github.com/>
> >>>>> PreTeXtBook/pretext/issues/2755>
> >>>>>
> >>>>> Andrew
> >>>>>
> >>>>> On Mon, Dec 29, 2025 at 8:02 AM Charilaos Skiadas <cski...@gmail.com
> <mailto:cski...@gmail.com>
> >>>>> <mailto:cski...@gmail.com <mailto:cski...@gmail.com>>> wrote:
> >>>>>
> >>>>>    Update: xi:include made the examples work on my end (a tad
> annoying that
> >>>>>    I had to change the path a little bit as xi:include is relative to the
> >>>>>    processed file, but that makes sense).
> >>>>>    I would still like to know if there is a better approach to having
> some
> >>>>>    code that lives in a separate file, that is available for #include
> in a
> >>>>>    C-program, and which is not otherwise visible to the book reader, than
> >>>>>    using the datafile @hide=“yes” option. But I can live with the current
> >>>>>    approach, even if it’s misusing datafile a bit.
> >>>>>
> >>>>>    But in the meantime I set off to adjust the sample book and guide
> so they
> >>>>>    don’t suggest pre@source as the appropriate approach, and ran into a
> >>>>>    snag. When I changed https://github.com/PreTeXtBook/pretext/
> <https://github.com/PreTeXtBook/pretext/>
> >>>>>    blob/0340cc4228e0a978abc8942dc7a25e2af9b5642a/examples/sample-book/
> >>>>>    rune.xml#L877 <https://github.com/PreTeXtBook/pretext/ <https://
> github.com/PreTeXtBook/pretext/>
> dev%2Bunsu...@googlegroups.com> <mailto:pretext- <mailto:pretext->
> >>>>> dev+uns...@googlegroups.com
> <mailto:dev%2Bunsu...@googlegroups.com>>.
> >>>>>    To view this discussion visit https://groups.google.com/d/msgid/
> pretext- <https://groups.google.com/d/msgid/pretext->
> >>>>>    dev/8E9BC386-0CC6-4674-8C2A-69B38A9DE9D9%40gmail.com
> <http://40gmail.com> <https://
> >>>>> groups.google.com/d/msgid/pretext- <http://groups.google.com/d/msgid/
> pretext->
> >>>>>    dev/8E9BC386-0CC6-4674-8C2A-69B38A9DE9D9%40gmail.com
> <http://40gmail.com>?
> >>>>>    utm_medium=email&utm_source=footer>.
> >>>>>
> >>>>>>    On Dec 28, 2025, at 5:05 PM, Charilaos Skiadas
> <cski...@gmail.com <mailto:cski...@gmail.com>
> >>>>>>>    <mailto:asch...@chemeketa.edu <mailto:asch...@chemeketa.edu>>>
> wrote:
> >>>>>>>
> >>>>>>>    On fone, so brief...
> >>>>>>>
> >>>>>>>    IMO datafiles are for data. Code should be in a program or pf.
> >>>>>>>
> >>>>>>>    Runestone programs that run server side can include other programs,
> >>>>>>>    even ones that are not activecodes.
> >>>>>>>
> >>>>>>>    See Listing 5.2.18: add.h (version 1)
> >>>>>>> https://runestone.academy/ns/books/published/PTXSB/ <https://
> runestone.academy/ns/books/published/PTXSB/>
> >>>>>>>    activecode.html#addh-v1-listing <https://runestone.academy/ns/
> books/ <https://runestone.academy/ns/books/>
> >>>>>>>    published/PTXSB/activecode.html#addh-v1-listing>
> >>>>>>>
> >>>>>>>    For data file/pre I recommend you not use @source. Instead do an
> >>>>>>>    xi:include inside the pre.
> >>>>>>>
> >>>>>>>    See https://github.com/PreTeXtBook/pretext/pull/2615 <https://
> github.com/PreTeXtBook/pretext/pull/2615> <https://
> >>>>>>> github.com/PreTeXtBook/pretext/pull/2615 <http://github.com/
> PreTeXtBook/pretext/pull/2615>>
> >>>>>>>
> >>>>>>>    Andrew Scholer
> >>>>>>>
> >>>>>>>
> >>>>>>>    On Sun, Dec 28, 2025, 1:23 PM Charilaos Skiadas
> <cski...@gmail.com <mailto:cski...@gmail.com>
> >>>>>>>    <mailto:cski...@gmail.com <mailto:cski...@gmail.com>>> wrote:
> >>>>>>>
> >>>>>>>        I made an issue about it https://github.com/PreTeXtBook/
> pretext/ <https://github.com/PreTeXtBook/pretext/>
> >>>>>>>        issues/2754 <https://github.com/PreTeXtBook/pretext/
> issues/2754 <https://github.com/PreTeXtBook/pretext/issues/2754>>
> >>>>>>>
> >>>>>>>        But thought I’d post here as well. My situation is that I ha
> ve a
> >>>>>>>        C-header file that naturally contains a bunch of < symbols,
> and I
> >>>>>>>        would like to be able to #include it, so I thought the
> datafile +
> >>>>>>>        pre route was the way to go. Even if there is an
> alternative, the
> >>>>>>>        problem would still remain in the case of any text files
> containing
> >>>>>>>        < or other similarly special characters.
> >>>>>>>
> >>>>>>>        Currently during the asset generation process this file’s
> code is
> >>>>>>>        read as is and plucked into a <pi:text-file xmlns:pi="http://
> >>>>>>> pretextbook.org/2020/pretext/internal <http://pretextbook.org/2020/
> pretext/internal> <http://pretextbook.org/2020/ <http://pretextbook.org/2020/>
> >>>>>>>        pretext/internal>">{}</pi:text-file>
> >>>>>>>        container which would result in invalid xml, and in my case
> >>>>>>>        spurious runestone errors about not being able to locate the
> file.
> >>>>>>>
> >>>>>>>        Hoping folks can verify that this is a bug or point me in
> the right
> >>>>>>>        direction.
> >>>>>>>
> >>>>>>>        Charilaos Skiadas
> >>>>>>>        Department of Mathematics
> >>>>>>>        Hanover College
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>        --         You received this message because you are
> subscribed to
> >>>>>>> the Google
> >>>>>>>        Groups "PreTeXt development" group.
> >>>>>>>        To unsubscribe from this group and stop receiving emails
> from it,
> >>>>>>>        send an email to pretext-dev...@googlegroups.com
> <mailto:pretext-dev%2Bunsu...@googlegroups.com>
> >>>>>>>        <mailto:pretext-dev...@googlegroups.com
> <mailto:pretext-dev%2Bunsu...@googlegroups.com>>.
> >>>>>>>        To view this discussion visit https://groups.google.com/d/
> msgid/ <https://groups.google.com/d/msgid/>
> >>>>>>>        pretext-dev/F56AB760-BBC7-46BA-9926-F06B2E7BEA40%40gmail.com
> <http://40gmail.com>
> >>>>>>>        <https://groups.google.com/d/msgid/pretext-dev/F56AB760-
> <https://groups.google.com/d/msgid/pretext-dev/F56AB760->
> >>>>>>>        BBC7-46BA-9926-F06B2E7BEA40%40gmail.com <http://40gmail.com>?
> >>>>>>>        utm_medium=email&utm_source=footer>.
> >>>>>>>
> >>>>>>>
> >>>>>>>    --     You received this message because you are subscribed to
> the Google
> >>>>>>>    Groups "PreTeXt development" group.
> >>>>>>>    To unsubscribe from this group and stop receiving emails from
> it, send
> >>>>>>>    an email to pretext-dev...@googlegroups.com
> <mailto:pretext-dev%2Bunsu...@googlegroups.com> <mailto:pretext-
> <mailto:pretext->
> >>>>>>> dev+uns...@googlegroups.com
> <mailto:dev%2Bunsu...@googlegroups.com>>.
> >>>>>>>    To view this discussion visit https://groups.google.com/d/msgid/
> <https://groups.google.com/d/msgid/>
> >>>>>>>    pretext-dev/
> >>>>>>>
>    CACm44N9f%3D0yXWBBbsH6Yn9qCTVmg_tAbub1Mu2hWZhWde_vnPQ%40mail.gmail.com
> <http://40mail.gmail.com>
> >>>>>>>    <https://groups.google.com/d/msgid/pretext-dev/ <https://
> groups.google.com/d/msgid/pretext-dev/>
> >>>>>>>
>    CACm44N9f%3D0yXWBBbsH6Yn9qCTVmg_tAbub1Mu2hWZhWde_vnPQ%40mail.gmail.com
> <http://40mail.gmail.com>?
> >>>>>>>    utm_medium=email&utm_source=footer>.
> >>>>>>
> >>>>>
> >>>>>
> >>>>>    --     You received this message because you are subscribed to the
> >>>>> Google Groups
> >>>>>    "PreTeXt development" group.
> >>>>>    To unsubscribe from this group and stop receiving emails from it,
> send an
> >>>>>    email to pretext-dev...@googlegroups.com <mailto:pretext-
> dev%2Bunsu...@googlegroups.com> <mailto:pretext- <mailto:pretext->
> >>>>> dev+uns...@googlegroups.com
> <mailto:dev%2Bunsu...@googlegroups.com>>.
> >>>>>    To view this discussion visit https://groups.google.com/d/msgid/
> pretext- <https://groups.google.com/d/msgid/pretext->
> >>>>>    dev/8E9BC386-0CC6-4674-8C2A-69B38A9DE9D9%40gmail.com
> <http://40gmail.com> <https://
> >>>>> groups.google.com/d/msgid/pretext- <http://groups.google.com/d/msgid/
> pretext->
> >>>>>    dev/8E9BC386-0CC6-4674-8C2A-69B38A9DE9D9%40gmail.com
> <http://40gmail.com>?
> >>>>>    utm_medium=email&utm_source=footer>.
> >>>>>
> >>>>>
> >>>>> --
> >>>>> You received this message because you are subscribed to the Google
> Groups
> >>>>> "PreTeXt development" group.
> >>>>> To unsubscribe from this group and stop receiving emails from it,
> send an
> >>>>> email to pretext-dev...@googlegroups.com <mailto:pretext-
> dev%2Bunsu...@googlegroups.com> <mailto:pretext- <mailto:pretext->
> >>>>> dev+uns...@googlegroups.com
> <mailto:dev%2Bunsu...@googlegroups.com>>.
> >>>>> To view this discussion visit https://groups.google.com/d/msgid/
> pretext- <https://groups.google.com/d/msgid/pretext->
> >>>>> dev/ CACm44N8yKuarXvCDUjp%2BTE-
> >>>>> N6XXBw%3DMTXn1jDBxEfsoD%3Dy69dw%40mail.gmail.com
> <http://40mail.gmail.com> <https://
> >>>>> groups.google.com/d/msgid/pretext-dev/CACm44N8yKuarXvCDUjp%2BTE-
> <http://groups.google.com/d/msgid/pretext-dev/CACm44N8yKuarXvCDUjp%2BTE->
> >>>>> N6XXBw%3DMTXn1jDBxEfsoD%3Dy69dw%40mail.gmail.com
> <http://40mail.gmail.com>?
> >>>>> utm_medium=email&utm_source=footer>.
> >>>>
> >>> --
> >>> You received this message because you are subscribed to the Google Groups
> >>> "PreTeXt development" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send an
> >>> email to pretext-dev...@googlegroups.com <mailto:pretext-
> dev%2Bunsu...@googlegroups.com> <mailto:pretext- <mailto:pretext->
> >>> dev+uns...@googlegroups.com
> <mailto:dev%2Bunsu...@googlegroups.com>>.
> >>> To view this discussion visit https://groups.google.com/d/msgid/
> pretext-dev/ <https://groups.google.com/d/msgid/pretext-dev/>
> >>> C0BCFC35-E6B2-4A5D-AF80-A65BFE27B2E4%40gmail.com <http://40gmail.com>
> <https://groups.google.com/ <https://groups.google.com/>
> >>> d/ msgid/pretext-dev/C0BCFC35-E6B2-4A5D-AF80-A65BFE27B2E4%40gmail.com
> <http://40gmail.com>?
> >>> utm_medium=email&utm_source=footer>.
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "PreTeXt development" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an
> >> email to pretext-dev...@googlegroups.com <mailto:pretext-
> dev%2Bunsu...@googlegroups.com>.
> >> To view this discussion visit https://groups.google.com/d/msgid/pretext-
> dev/ <https://groups.google.com/d/msgid/pretext-dev/>
> >> MTAwMDAzNC5iZWV6ZXI.1767197877%40pnsh.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "PreTeXt development" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> email
> > to pretext-dev...@googlegroups.com <mailto:pretext-
> dev%2Bunsu...@googlegroups.com> <mailto:pretext- <mailto:pretext->
> > dev+uns...@googlegroups.com
> <mailto:dev%2Bunsu...@googlegroups.com>>.
> > To view this discussion visit https://groups.google.com/d/msgid/pretext-
> dev/ <https://groups.google.com/d/msgid/pretext-dev/>
> > DA3933D8-CB2C-4E02-835C-A64CC3DC9BEC%40gmail.com <http://40gmail.com>
> <https://groups.google.com/d/ <https://groups.google.com/d/>
> > msgid/pretext-dev/DA3933D8-CB2C-4E02-835C-A64CC3DC9BEC%40gmail.com
> <http://40gmail.com>?
> > utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to the Google Groups
> "PreTeXt development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pretext-dev...@googlegroups.com <mailto:pretext-
> dev%2Bunsu...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/
> MTAwMDAzMi5iZWV6ZXI.1767380109%40pnsh <https://groups.google.com/d/msgid/
> pretext-dev/MTAwMDAzMi5iZWV6ZXI.1767380109%40pnsh>.
>
> --
> You received this message because you are subscribed to the Google Groups
> "PreTeXt development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to pretext-dev...@googlegroups.com <mailto:pretext-
> dev+uns...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/
> CACm44N8x1ndwiq3giC90k7BN0qdRuGG3PQONMk5n6HuQVTxJyg%40mail.gmail.com <https://
> groups.google.com/d/msgid/pretext-dev/
> CACm44N8x1ndwiq3giC90k7BN0qdRuGG3PQONMk5n6HuQVTxJyg%40mail.gmail.com?
> utm_medium=email&utm_source=footer>.

Andrew Scholer

unread,
Jan 2, 2026, 3:27:53 PMJan 2
to prete...@googlegroups.com
The semi recent work on <program> is documented here:
https://pretextbook.org/doc/guide/html/topic-program.html#interactive-program-activecode-8-4-11

In <program>
@add-files says "make sure to copy this file into program directory" it works with datafiles or programs

@compile-also says "copy this file into the program directory AND include this in the build recipe for the program". In theory it should be restricted to programs. But I think you could probably point at a datafile and have that work (to whatever extent compiling that file made sense).

So from a consumer side, we are currently agnostic as to what is being added when the program is run. Whatever it is just needs an xml:id and an @filename.


To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/MTAwMDAyNC5iZWV6ZXI.1767384915%40pnsh.

Charilaos Skiadas

unread,
Jun 2, 2026, 2:01:25 PM (4 days ago) Jun 2
to prete...@googlegroups.com
Alright coming back to this after some hiatus, and here are my thoughts and proposal for the #datafile element:

- As I see it the purpose of the element is to provide some files in place when programs run. As such I propose first of all renaming it to #file. Though I find some references to #file in the context of SCORM I believe, not sure if this will cause problems. Perhaps #filecontents or #filesource ?
- For the same reason, it should by default be invisible. Getting its contents to appear in some form should be a very deliberate action of the author. This would allow the file format to be whatever (e.g. a zip or jar file), without worrying about any physical appearance.
- With that in mind, it seems much more straightforward that we can specify the source of the file via a @source attribute to the #file tag itself, rather than having to include #image or #pre as child elements.
- So with that approach either the #file tag has a @source attribute linking to an external file, or it has its contents (possibly wrapped in a #pre) directly written out (or included vis xi:include text). This way I believe the #pre has no need for a @source, unless we need that in other parts. I didn't find any examples of it used that way outside of datafile.
- If the file is an image that the author wants to have show as part of the book, then a separate image tag should be used.
- What remains is the desire for a textfile to be editable by the user, which is currently specified by the @editable attribute. And the choice to make it visible or not. I think that choice is meant to be controlled by include-source. (Parenthetical -- I am surprised that searching for @include-source brings up some examples with an exercise tag getting @include-source, even though the spec doesn't seem to specify it in any other context other than data-file). And obviously editable="yes" only makes sense if include-source="yes" so there is some restrictions in the flags there.
- I also like the idea of allowing the user to upload a file to act as the source. So I propose to replace @editable+@include-source with a @user-interaction attribute with four possible values: "none" (default, file is hidden), "view" (file visible but uneditable), "edit" (file visible and editable), and "upload" (user can upload a file but contents otherwise invisible). The view/edit options would only make sense for text-content files of course, while the others make sense regardless. It is up to the author to use those appropriately.
- For non-HTML versions, I think it is a good question what to do, if anything. Might make sense to include the contents if user-interaction=view/edit but to completely hide otherwise. Or maybe always hide?

In terms of transition, I think we can:

- Preprocess convert  #datafile>#image to #file@source and #datafile>#pre@source to #file@source if @include-source is "no" and to #datafile@source + #image@source if @include-source is "yes"
- Preprocess convert @editable@include-source to @user-interaction (only using the none/view/edit cases)
- Add the new #file tag to the spec
- Deprecate #datafile

The current PM would probably be closed if the above plan is to be implemented.

Thoughts/objections? Or should I go ahead and implement this?

Haris

Charilaos Skiadas
Department of Mathematics
Hanover College

Andrew Scholer

unread,
Jun 2, 2026, 3:11:13 PM (3 days ago) Jun 2
to prete...@googlegroups.com
Most of those proposals make sense to me (or at least do not not make sense).  I presume everything Runesone facing would remain the same? i.e. you might now have an #file, but as far as Runestone is concerned, it is still a "datafile"?

Changing the default visibility of something in existing books seems dangerous.  I agree that separating the presentation of "here is a file you can work with" from providing the file itself (mostly to Runestone) might be a useful separation of concerns.  (Another possible use case - I might want to show an image in a format like JPEG on the page as "here is the image", but provide the code with a simpler file format (BMP) to parse.)

But if the default for @visbile is going to change, the preprocessor remapping should make sure #datafile ends up with @visible="yes" on the file unless it was specifically already set to "no".
 
- What remains is the desire for a textfile to be editable by the user, which is currently specified by the @editable attribute. And the choice to make it visible or not. I think that choice is meant to be controlled by include-source. (Parenthetical -- I am surprised that searching for @include-source brings up some examples with an exercise tag getting @include-source, even though the spec doesn't seem to specify it in any other context other than data-file). And obviously editable="yes" only makes sense if include-source="yes" so there is some restrictions in the flags there.

@include-source was a temporary workaround that has been removed:
https://groups.google.com/g/pretext-dev/c/YRU6O_VMKJ0/m/jIlNapsrCgAJ 

Editable is controlled by @editable:  
<datafile label="datafile-numbers2" xml:id="datafile-numbers2" filename="Numbers2.txt" rows="2" cols="20" editable="yes">
<pre>
10 20 30
40 50 60
</pre>
</datafile>

- For non-HTML versions, I think it is a good question what to do, if anything. Might make sense to include the contents if user-interaction=view/edit but to completely hide otherwise. Or maybe always hide?

I think displaying in view/edit makes sense. If the task is "write a regex to match each phone number in the text" it would be useful while solving by hand to see the data being used.

Andrew

To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/F0C72AAD-5772-4D3E-BEA5-69849336A890%40gmail.com.

Charilaos Skiadas

unread,
Jun 3, 2026, 11:45:35 AM (3 days ago) Jun 3
to prete...@googlegroups.com
Thanks Andrew,

1. Yes I believe Runestone-facing side would stay the same. At least if/until we decide to implement the “user can upload a file” part of it. But I am thinking that is a placeholder in the first pass.
2. My plan was to effectively maintain the current visibility: Current datafiles will be converted on the back-end to mirror the existing behavior. For example image datafiles would be converted to a combination of #file and #image siblings. It would only be the behavior of the new #file to be by default non-visible.
3. OK so @include-source is out which makes it simpler. So datafiles never had a visible switch then, as far as I can tell.  So we convert #datafiles+#pre with @editable=“no”  or missing to #files with @user-interaction=“view”, and those with @editable=“yes” to @user-interaction=“edit”, and all #datafiles+#image to @user-interaction=“none” (and also add a stand-alone image file). I think that captures existing behavior.

Charilaos Skiadas
Department of Mathematics
Hanover College
Reply all
Reply to author
Forward
0 new messages