Git submodules

[Rudi]

Hi folks,

recently I had a conversation with tom about the following problem:


RI> BTW, there is a somewhat unusual thing with one of the git
RI> repos. Take a look here:
RI>
RI> https://github.com/linux-sunxi/sunxi-mali/tree/master/lib
RI>
RI> Notice the green folder "mali" ? This is a link to another
RI> repo. The readme says, that in order to retrieve it one
RI> should enter:
RI>
RI> git clone https://github.com/linux-sunxi/sunxi-mali.git
RI> <chdir to working copy>
RI> git submodule init
RI> git submodule update
RI>
RI> I have not found a way to tell our scripts to do that. Any
RI> ideas ?


TG> got the same problem with others packages (gstreamer & co in mind)
TG> Iso i have packaged them by hand
TG>
TG> in scripts/get :
TG>
TG> git)
TG> GITREV=HEAD
TG> [ -n "$PKG_URL_REV" ] && GITREV="$PKG_URL_REV"
TG> git clone --recursive $PKG_URL $SOURCES/$1/$REPO_BASENAME
TG> git --git-dir=$SOURCES/$1/$REPO_BASENAME/.git
--work-tree=$SOURCES/$1/$REPO_BASENAME checkout $GITREV
TG> (cd $SOURCES/$1/$REPO_BASENAME; $ROOT/$SCRIPTS/git-archive-all --prefix
$REPO_BASENAME/ --rev $GITREV | bzip2 > ../$REPO_ARCHIVE )
TG>
TG> maybe adding an extra value in meta like
TG> GIT_SUBMODULE="true"
TG> and in last line :
TG>
TG> (cd $SOURCES/$1/$REPO_BASENAME; if [ "$GIT_SUBMODULE" = true ] ; then git submodule
init && git submodule update ; fi ; $ROOT/$SCRIPTS/git-archive-all --prefix
$REPO_BASENAME/ --rev $GITREV | bzip2 > ../$REPO_ARCHIVE )
TG>
TG> but not tested ....
TG> or add the previous check in git-archive-all script



I did not yet try this. However maybe some other guys here can comment...


Besides that, I got another problem with an (obviously broken) Git repo. It also seems
to have an invalid reference to another one. Therefore the option "-recursive" causes
the clone operation to fail. Waht do you think about a way to disable recursive operation ?


--

Ruediger "Rudi" Ihle

[Thomas Genty]

Le 05/06/2013 07:57, Rudi a �crit :

:-)

>
> Besides that, I got another problem with an (obviously broken) Git repo. It also seems
> to have an invalid reference to another one. Therefore the option "-recursive" causes
> the clone operation to fail. Waht do you think about a way to disable recursive operation ?
>
>

same as my first proposition, add a value in meta, then if true, modify
"git clone --recursive ..." by "git clone ..."

Tom

[Rudi]

Hi,

>> I did not yet try this. However maybe some other guys here can comment...

According to my short tests it looks like there is also a modification in
git-archive-all necessary as well. I put that aside for now and package the
files manually as I want to make some progress on the A10 front...

> same as my first proposition, add a value in meta, then if true, modify
> "git clone --recursive ..." by "git clone ..."

I may do so. At the moment I have not yet included the repo in question
(https://github.com/empatzero/xbmca10.git). The master plan is to extract
the interesting patches and apply them to out main XBMC anyway...



--

Ruediger "Rudi" Ihle

[Thomas Genty]

Le 09/06/2013 16:36, Rudi a �crit :

> Hi,
>
>>> I did not yet try this. However maybe some other guys here can
>>> comment...
>
> According to my short tests it looks like there is also a modification in
> git-archive-all necessary as well. I put that aside for now and
> package the
> files manually as I want to make some progress on the A10 front...

I can look at this if you want

>
>
>> same as my first proposition, add a value in meta, then if true, modify
> > "git clone --recursive ..." by "git clone ..."
>
> I may do so. At the moment I have not yet included the repo in question
> (https://github.com/empatzero/xbmca10.git). The master plan is to extract
> the interesting patches and apply them to out main XBMC anyway...
>
>
>

Tom

[Rudi]

Hi Tom,

>> According to my short tests it looks like there is also a modification in
>> git-archive-all necessary as well. I put that aside for now and package the
>> files manually as I want to make some progress on the A10 front...
> I can look at this if you want

Finally I've found the reason why submodules will not be included. However,
I'm afraid that fixing this unconditionally might introduce inconsistent
behaviour. I.e. fetching from existing tarballs vs. from original repo or
new components that where never built before and therefore fail.

So I tend to add an additional keyword in "meta" to enable submodule
fetching on per package basis. What do you think ?



--

Ruediger "Rudi" Ihle

[Thomas Genty]

Le 14/12/2013 08:46, Rudi a �crit :
> Hi Tom,
Hello Rudi,

I guess that's the more elegant way.
please, do it ;-)

Tom
>
>
>

[Rudi]

On 14.12.2013 08:55, Thomas Genty wrote:

>> So I tend to add an additional keyword in "meta" to enable submodule
>> fetching on per package basis. What do you think ?
> I guess that's the more elegant way.
> please, do it ;-)

Done in r16617. I'm now using this feature in the A10 mali package.

Enabling this for XBMC (which I have not yet done) causes the skin
'touched' to be built. I'm not so sure that we really need that on
the platforms we are building now...



--

Ruediger "Rudi" Ihle

[stefan fiala]

In trying to build the last published cubietruck build from openpctv
I beleive I ran into the above issue.
git clone openpctv
git checkout ae89d1 [by date the build I want circa 26/nov/2013]
make menuconfig
use alternate config from config/defconfigs/openpctv-xbmc-a20-cubietruck.conf
exit
cp config/defconfigs/openpctv-xbmc-a20-cubietruck.conf  build/build.host/kconfig-frontends-3.8.0.0/.config
[Missing patch that looks in more places]
make
.....
--2014-02-24 16:19:58--  http://sources.openbricks.org/devel/xbmc-frodo-12.2-re7d9567.tar.bz2
Resolving sources.openbricks.org (sources.openbricks.org)... 67.203.12.194
Connecting to sources.openbricks.org (sources.openbricks.org)|67.203.12.194|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2014-02-24 16:19:58 ERROR 404: Not Found.
[repeats]
[This version no longer there]

So it tries git
[using config/platforms/arm/a20/packages/xbmc/meta and xbmc-pvr-addons/meta]
Can't get xbmc sources, falling back to repository
Cloning into 'sources/xbmc/xbmc-frodo-12.2-re7d9567'...
remote: Counting objects: 263980, done.
remote: Compressing objects: 100% (67467/67467), done.
remote: Total 263980 (delta 188634), reused 263975 (delta 188629)
Receiving objects: 100% (263980/263980), 389.78 MiB | 1.04 MiB/s, done.
Resolving deltas: 100% (188634/188634), done.
Submodule 'touched' (git://github.com/xbmc/skin.touched.git) registered for path 'addons/skin.touched'
Submodule 'pvr-addons' (git-at-github.com:rellla/xbmc-pvr-addons.git) registered for path 'pvr-addons'
Cloning into 'addons/skin.touched'...
remote: Counting objects: 2330, done.
remote: Compressing objects: 100% (645/645), done.
remote: Total 2330 (delta 1428), reused 2330 (delta 1428)
Receiving objects: 100% (2330/2330), 9.28 MiB | 1.24 MiB/s, done.
Resolving deltas: 100% (1428/1428), done.
Submodule path 'addons/skin.touched': checked out '932a728725889c4220bc4ac42f18b082e4361f86'
Cloning into 'pvr-addons'...
Permission denied (publickey).
fatal: The remote end hung up unexpectedly
Clone of 'git-at-github.com:rellla/xbmc-pvr-addons.git' into submodule path 'pvr-addons' failed

[Manual clones of these pieces work !]

Then I found:-
# HG changeset patch
# User Ricky Xian <xxxxxx-at-gmail.com>
# Date 1385453291 -28800
# Node ID 4441fc70a0e1ac1b486064665681396c4deef694
# Parent  f5b3eb46f6847c600444e9a84cc82d90351e4b23
scripts/get: detect exist packages in sources folder

Which I manually applied.

So I git'd/zipped/tarred the appropriate pieces [with appropriate names] into .tar.bz2's
and put them in sources/xbmc/
scripts/get xbmc
never gets to this chunk..

I could live with manually getting/adding the tar.bz2's if this can be made to work.
[Is there more to the patch than the piece I found? or some further changes reguired to get it to work ? ].
Also each time scripts/get xbmc runs it deletes the tar.bz2.
Or if the relevant xbmc,xbmc-pvr-addons could be put back into http://sources.openbricks.org/devel/
Or if there's a recipe to get it all to build, I'd use that. [I tried the existing xbmc-frodo's 12.2,12.3 but they
run into a variety of other issues].
Or if there's a way of doing a manual git clone I'll use that, I've never worked with submodules..
Or if there's a better candidate a20/cubieboard/cubietruck  build from another project I'll try that.
Thanks...

[Rudi]

Hi stefan,

> In trying to build the last published cubietruck build from openpctv
> I beleive I ran into the above issue.

I already replied to you in the other thread. Did you read that ?


> Cloning into 'pvr-addons'...
> Permission denied (publickey). <---------!!!!!!!!!

> fatal: The remote end hung up unexpectedly
> Clone of 'git-at-github.com:rellla/xbmc-pvr-addons.git' into submodule path 'pvr-addons'
> failed

As I wrote, it's a problem of the repo. Both geexbox and openpctv have a separate
package named "xbmc-pvr-addons". Therefore they will not use the sources from that
submodule. In my other reply I suggested to disable submodule fetching by editing
"scripts/get" (actuall it's line 125). However, a better solution would be to apply:

http://hg.openbricks.org/openbricks/rev/ad1ed08c5842

This patch enables submodule fetching only if explicitely specified in the "meta" (via
PKG_RECURSIVE=yes). Also it corrects a bug in "scripts/git-archive-all", which was the
reason why submodule fetching - even from a good repo (!) - *NEVER* really worked.

> Or if the relevant xbmc,xbmc-pvr-addons could be put back into
> http://sources.openbricks.org/devel/

We are no longer using these files. They were removed to preserve disk space. I'm not so
much a friend of keeping stuff on our server just because someone else might reference it.

> Or if there's a better candidate a20/cubieboard/cubietruck build from another project
> I'll try that.

I spent a good part of the previous weekend to add a configuration named
"geexbox-xbmc-a20-cubieboard2". Quote from my other post:

RI> It should build, but is completely untested. Hardware video decoding will most
RI> likely not work due to issues with cedarx, but it's questionable if this initial
RI> version will get you to that point...

I don't know if it's better since I don't have any a20 hardware. But I'd be interrested
in feedback. If you want cubietruck, you can make a copy of
"geexbox-xbmc-a20-cubieboard2.conf" and replace any occurance of "cubieboard2" with
"cubietruck".



--

Ruediger "Rudi" Ihle

[Andreas Baierl]

Hi Rudi,

if i understand correctly, you're not using my repo anymore but patching
upstream frodo with allwinner patches? Maybe i should do that as well in
my branch.
And for completeness: I think there is something messed up with pvr
submodule in my repo :p
FYI: I successfully got VDR working with OSD. Needs some improvements
though. As I read, it's part of openbricks, too.

rellla

[stefan fiala]

I'm afraid I've spaced the other thread, could you remind me [point me at it], sorry.
Thanks much for the reply, I'm afraid I dug a bit of a hole, but it was a learning
experience... I'll try any a20 type builds you have.....
I found an unrelated issue with the packages/vdr/meta, version 2.x.x (2.1.1) comes from
a /Developer tree. I can send a patch if interested [or already unfixed..].

[Rudi]

Hi rellla,

> if i understand correctly, you're not using my repo anymore but patching upstream frodo
> with allwinner patches?

Yes. For geexbox we are checking out 9ed3e580ea7c000541273cbac116ec9e314124ca
(Dec 12, 2013) from https://github.com/xbmc/xbmc/commits/Frodo

Then we apply the two patch sets found here:

http://hg.openbricks.org/openbricks/file/17f81151aabe/packages/xbmc/patches

and here:

http://hg.openbricks.org/openbricks/file/17f81151aabe/config/platforms/arm/a10/packages/xbmc/patches

The first one is pretty much specific to geexbox, but the texturepacker and libjpeg
patches might be useful in general. From the other one, the most interresting things the
patches that contain "a10" in their names. Note that since we are using our own
cross-compile environment, the build system setup stuff might not work for the
standalone compile you seem to be using by default. But it's probably not much that
needs to be tweaked.

Another change that I did (not directly related to the above) is that I created a shared
libary named libcedarx.so. Our XBMC is linked against this shared lib, instead of
libcedarv and libvecore. So the cedarx stuff is easier to replace. The patches are here:

http://hg.openbricks.org/openbricks/file/17f81151aabe/packages/libcedarx/patches

Just recently I modified the Makefile(s) as well as:
http://hg.openbricks.org/openbricks/file/17f81151aabe/packages/libcedarx/build

to be able to create libcedarx.so for A10 and A20 from the armhf2 subdir. They build,
but don't work. The first reason for that is that the new code expects a device named
"/dev/sunxi_mem" to handle physical memory allocations. It looks like the kernels for
Android supplied by AllWinner have this device, while our kernels do not. Do you have
any idea, where the source code for this device driver can be found ? It might also be
possible to switch back to the old "avheap", but I have not tried this...


P.S.: I'm aware of the fact that the latest and greatest is the attempt to port VDPAU,
but if the newer blobs could be made working I would go that route for a short term
solution.


P.S.2: Also recently switched to kernel 3.4.79 for the A10. But I experienced more audio
hiccups. Especially in HDMI passthrough mode. Furthermore I had some crashes that I
don't remeber having seen before. But that might also be related to the fact the I
started using a USB WLAN adpater at the same time. Need to rule that out...


Cheers !



--

Ruediger "Rudi" Ihle

[Andreas Baierl]

Hi Rudi,

Am 25.02.2014 18:29, schrieb Rudi:
> Hi rellla,
>
>> if i understand correctly, you're not using my repo anymore but patching upstream frodo
>> with allwinner patches?
> Yes. For geexbox we are checking out 9ed3e580ea7c000541273cbac116ec9e314124ca
> (Dec 12, 2013) from https://github.com/xbmc/xbmc/commits/Frodo
>
> Then we apply the two patch sets found here:
>
> http://hg.openbricks.org/openbricks/file/17f81151aabe/packages/xbmc/patches
>
> and here:
>
> http://hg.openbricks.org/openbricks/file/17f81151aabe/config/platforms/arm/a10/packages/xbmc/patches
>
> The first one is pretty much specific to geexbox, but the texturepacker and libjpeg
> patches might be useful in general. From the other one, the most interresting things the
> patches that contain "a10" in their names. Note that since we are using our own
> cross-compile environment, the build system setup stuff might not work for the
> standalone compile you seem to be using by default. But it's probably not much that
> needs to be tweaked.
>
> Another change that I did (not directly related to the above) is that I created a shared
> libary named libcedarx.so. Our XBMC is linked against this shared lib, instead of
> libcedarv and libvecore. So the cedarx stuff is easier to replace. The patches are here:
>
> http://hg.openbricks.org/openbricks/file/17f81151aabe/packages/libcedarx/patches

Good to know. I try to go this way as well. With a build system to build
a "standalone" xbmc.

> Just recently I modified the Makefile(s) as well as:
> http://hg.openbricks.org/openbricks/file/17f81151aabe/packages/libcedarx/build
>
> to be able to create libcedarx.so for A10 and A20 from the armhf2 subdir. They build,
> but don't work. The first reason for that is that the new code expects a device named
> "/dev/sunxi_mem" to handle physical memory allocations. It looks like the kernels for
> Android supplied by AllWinner have this device, while our kernels do not. Do you have
> any idea, where the source code for this device driver can be found ? It might also be
> possible to switch back to the old "avheap", but I have not tried this...

I have tried to use this stuff with sunxi_mem a time ago but never
finished it. Iirc I borrowed the code from
https://github.com/cubieboard2/linux-sunxi/tree/sunxi-3.3-cb2/drivers/char
But i did have to do a few tweaks with #defines. I had to take a look in
all deps for sunxi_mem as well, e.g.
https://github.com/cubieboard2/linux-sunxi/blob/sunxi-3.3-cb2/include/linux/sunxi_physmem.h
After some hacking i ended up i a /dev/sunxi_mem but then i stopped
trying it with the armhf2 blobs. If you want try using sunxi_mem it
shouldn't be that hard for you to make it at least compile with 3.4 kernel.

>
> P.S.: I'm aware of the fact that the latest and greatest is the attempt to port VDPAU,
> but if the newer blobs could be made working I would go that route for a short term
> solution.

Good way, because i don't know anyone that is working on the libvdpau
way atm. I'm pausing working with XBMC and set my focus on getting VDR
work with vdpau to use it as a client. It's working basically including
OSD. Some issues though. And next big goal would be to add the
deinterlacer, that should be included in the disp scaler. At least the
3.3 code speaks of it. E.g. here
https://github.com/cubieboard2/linux-sunxi/blob/sunxi-3.3-cb2/drivers/video/sun7i/disp/de_bsp/de/ebios/de_fe.c#L912

It would be a great goal to get this also running. I don't know of any
tiny device, that can do hardware based deinterlacing atm. So maybe it
would be worth to take a look at it...

>
> P.S.2: Also recently switched to kernel 3.4.79 for the A10. But I experienced more audio
> hiccups. Especially in HDMI passthrough mode. Furthermore I had some crashes that I
> don't remeber having seen before. But that might also be related to the fact the I
> started using a USB WLAN adpater at the same time. Need to rule that out...
>

Remember to patch XBMC to use "sunxi" instead of "sun4i" in the audio
regarding files, as the device name has changed. Maybe you did this already.
> Cheers !
>
>
>
Happy coding!

rellla

[Rudi]

Hi rellla,

> I have tried to use this stuff with sunxi_mem a time ago but never finished it. Iirc I
> borrowed the code from
> https://github.com/cubieboard2/linux-sunxi/tree/sunxi-3.3-cb2/drivers/char
> But i did have to do a few tweaks with #defines. I had to take a look in all deps for
> sunxi_mem as well, e.g.
> https://github.com/cubieboard2/linux-sunxi/blob/sunxi-3.3-cb2/include/linux/sunxi_physmem.h
> After some hacking i ended up i a /dev/sunxi_mem but then i stopped trying it with the
> armhf2 blobs. If you want try using sunxi_mem it shouldn't be that hard for you to make
> it at least compile with 3.4 kernel.

Thanks for the pointer ! I'll give it a try when I find the time. But I'm going to be
rather busy during the next 2..3 weeks...

> I'm pausing working with XBMC and set my focus on getting VDR work with vdpau to use it
> as a client. It's working basically including OSD. Some issues though. And next big goal
> would be to add the deinterlacer, that should be included in the disp scaler. At least
> the 3.3 code speaks of it. E.g. here
> https://github.com/cubieboard2/linux-sunxi/blob/sunxi-3.3-cb2/drivers/video/sun7i/disp/de_bsp/de/ebios/de_fe.c#L912

O.K, I see... Even though we have the VDR package our repo, I have never used it. I have
to confess, that I'm not so terribly interrested in doing live TV or TV recordings. But
a lot of people are.

> It would be a great goal to get this also running. I don't know of any tiny device, that
> can do hardware based deinterlacing atm. So maybe it would be worth to take a look at it...

Our latest "holy cow", the Freescale iMX.6 as found in CuBox-i and Utilite can do that
as well.

> Remember to patch XBMC to use "sunxi" instead of "sun4i" in the audio regarding files,
> as the device name has changed. Maybe you did this already.

Yes. There was a question about this on your repo's issue tracker. That prevented me
from having to search myself ;-).




--

Ruediger "Rudi" Ihle