Cannot get video streams working in demos

[Joe Palmer]

I have compiled Janus on my Mac (10.11.5) and am trying to get the video room demo working but I cannot successfully share the video between 2 sessions. When someone joins the room, Janus constantly logs lines like the following (about 3 per second):

[WARN] Not video and not audio? dropping (SSRC 2147483667)...

I see the local video fine but when a second user joins the room, the remote video shows the username and a spinner but the video never appears. There are no errors in the JavaScript console.

I have tried the other demos and the echo test work fine (although it also outputs the above warnings) but all the others suffer from the same endless spinner. I cant even watch the Lorenzo says hello! video in the playout demo (it always produces exactly the following warnings every time I try and watch it which is interesting):

[WARN] Not video and not audio? dropping (SSRC 797440060)...

[WARN] Not video and not audio? dropping (SSRC 797440060)...

[WARN] Not video and not audio? dropping (SSRC 2147483649)...

[WARN] Not video and not audio? dropping (SSRC 2147483649)...

[WARN] Not video and not audio? dropping (SSRC 2147483650)...

[WARN] Not video and not audio? dropping (SSRC 2147483651)...

[WARN] Not video and not audio? dropping (SSRC 2147483652)...

[WARN] Not video and not audio? dropping (SSRC 2147483653)...

[WARN] Not video and not audio? dropping (SSRC 2147483654)...

[WARN] Not video and not audio? dropping (SSRC 2147483650)...

[WARN] Not video and not audio? dropping (SSRC 2147483655)...

[WARN] Not video and not audio? dropping (SSRC 2147483656)...

[WARN] Not video and not audio? dropping (SSRC 2147483657)...

[WARN] Not video and not audio? dropping (SSRC 2147483658)...

[WARN] Not video and not audio? dropping (SSRC 2147483651)...

Any idea why this is happening? Are there any more logs of debug info I can look at or provide that would help?

Many thanks,

Joe

[Lorenzo Miniero]

Are you using the latest master, or an older tag? We had a similar issue before, but it has been fixed a long time ago.

L.

[Joe Palmer]

Thanks for you reply. I pulled from your master 2 days ago and have hardly changed anything:

https://github.com/softforge/janus-gateway

I did find this thread but couldn't work out how the problem was resolved. Do you think this could be related to running everything locally?

Many thanks,

Joe

[Lorenzo Miniero]

The link to the thread doesn't seem to be working. Running locally is not an issue, I test locally all the time. Is this happening with Chrome, Firefox, both? Which versions?

L.

[Joe Palmer]

Sorry, this is the link I was referring to:

https://groups.google.com/forum/#!msg/meetecho-janus/1ZV5hQfcNmA/OjOd2ft9FwAJ

I have just tested again on Chrome v50.0.2661.102 (64-bit) and Firefox v46.0.1. I recorded a short video so you can see what I'm seeing:

http://screencast.com/t/ZBHhFQeb

Thanks for you help with this.

Joe

[Lorenzo Miniero]

You should check the admin API for that handle, in order to see which SSRCs were negotiated and if that unknown SSRC appears anywhere in the SDP.

L.

[Joe Palmer]

OK, this is what I see in the Admin/Monitor when I try and play the video:

http://pastebin.com/Hs3fbpP5

Does this give any clues? I'm relatively new to streaming with WebRTC so still getting to grips with the exact process (so this is a good learning experience :)

[Lorenzo Miniero]

The remote SDP provided by the browser doesn't include any SSRC related info. Since we don't know which SSRCs were advertized, we don't know if an incoming packet is audio or video related, and we discard it. Not sure why it's not there, and why it's not there on both Chrome and Firefox (maybe a Mac OS weirdness? wouldn't be the first :-) ). We probably could try and "guess" and thus assign the right SSRC anyway by looking at the payload type, but that's not in the code yet.

L.

[Joe Palmer]

OK, interesting. I just tested the online demo and I'm able to record and stream from the same versions of Chrome and Firefox on the same Mac so it is working with that server setup. Not sure if there is an easy way we can compare the differences between the browser handshaking on you server and my Mac?

So maybe the OSX weirdness is effecting the server rather then the browser. We would eventually like to deploy Janus on our Centos 7 servers so do you think it is worth me setting up a Cenos 7 VM and trying the same tests on that?

Joe

[Lorenzo Miniero]

If it works with our online demos, it means your browser is setting the SSRCs in the SDP when sending it there. For some reason this isn't happening in your own setup. Maybe when opening a localhost page on Mac OS, the browser there fails to add them for some weird reason? When I get back I'll try to make SSRC detection smarter for cases like that.

As to CentOS, I do remember there were serious issues with CentOS 6.5, mostly due to dependencies being too old. That shouldn't be a problem with CentOS 7 but I personally never tried that: it works fine on my Fedora, so that should be cool.

L.

[Joe Palmer]

OK thanks. Let me know if there is anything I can do to help.

Joe

[Lorenzo Miniero]

Try the demos on your local address instead of localhost, e.g., https://192.168.0.1/echotest.html or whatever (you'll need an HTTPS webserver once you use something different from localhost), just to make sure that's the cause.

L.

[Joe Palmer]

Good idea! It turns out Firefox allows camera access on non-localhost with http so I tested with my IP address but get exactly the same issue :(

I can try setting up SSL if you think that will make difference?

Joe

[Lorenzo Miniero]

It may help just to see if Chrome does the same thing. Anyway, it's weird that both browsers are behaving the same way (unless they're sharing more code than I thought?)

L.

[Josh Dickson]

I'm also seeing these ([WARN] Not video and not audio? dropping) errors... don't have a firm idea on why yet. Seem to be random. Same two connections don't work, then reboot chrome and they do work, then some time passes and they don't work. Going to update to latest Janus build - when you say this has been fixed for a while, how long is a while out of curiosity?

[Lorenzo Miniero]

When that happens, check the admin API: if you don't see any peer SSRC in there, that's the cause. As anticipated, I'll try to work on a workaround when I get back home (currently abroad).

L.

[Lorenzo Miniero]

This commit should fix the issue:

https://github.com/meetecho/janus-gateway/commit/c67d7b43c6d72862b36dbb0681d11874d7712697

L.

[Josh Dickson]

FWIW on a new install with latest Janus (including this) I still see these turning up in logs. This is with OpenSSL 1.0.2g (which was giving errors for other things).

Creating new session: 2928163645

Creating new handle in session 2928163645: 3368184594

[3368184594] Creating ICE agent (ICE Full mode, controlled)

[WARN] [3368184594]     Missing valid SRTP session (packet arrived too early?), skipping...

[3368184594] The DTLS handshake has been completed

WebRTC media is now available

[WARN] [3368184594] Not video and not audio? dropping (SSRC 1456451976)...

[WARN] [3368184594] Not video and not audio? dropping (SSRC 1456451976)...

[WARN] [3368184594] Not video and not audio? dropping (SSRC 2016085669)...

Could you perhaps explain a bit on what you mean by ‘check the admin API’? I see the admin api docs, etc, I’m sure I can connect to it - I just have no idea what I should be doing in order to better figure out what’s going on. You’ve commented that (check the admin api) a few times and I would love to be able to dig into it - just not sure what I should be looking for! Thanks :)

--
You received this message because you are subscribed to a topic in the Google Groups "meetecho-janus" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/meetecho-janus/9hP72hTTKk0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to meetecho-janu...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Josh Dickson]

Oops - I see what you mean with getting more info via the Admin API. I was reading the wrong thing. Will try this out. 

[Josh Dickson]

First attempt to get more info on what’s going on here, a dump at log level 7.

http://pastebin.com/GwYnsxiv

[Lorenzo Miniero]

Are you sure you updated Janus? I definitely managed to reproduce the issue, and fixed that by autodetecting the media type in case of unknown SSRCs.

Anyway, sorry but I won't go through tons of logs, especially at level 7 ;-)

By Admin API I mean the API we have to poll info from active handles. There's an admin.html page that provides a nice interface to that. You can learn more about what it is. how to enable it and how to get that info here: http://www.meetecho.com/blog/understanding-the-janus-admin-api/

L.

To unsubscribe from this group and all its topics, send an email to meetecho-janus+unsubscribe@googlegroups.com.

[Joe Palmer]

Thanks for your work on this! I've definitely updated Janus but am also still getting the same issue. To prove I was running the new code, I adding (test) into the log statement which was then outputted. So for some reason, this line is being run:

https://github.com/meetecho/janus-gateway/blob/master/ice.c#L1712

I took a screenshot to show you what I did:

Anything else I can do to help diagnose the problem?

Many thanks.

Joe

To unsubscribe from this group and all its topics, send an email to meetecho-janu...@googlegroups.com.

[Lorenzo Miniero]

Il giorno martedì 24 maggio 2016 11:12:16 UTC+2, Joe Palmer ha scritto:

Anything else I can do to help diagnose the problem?

Yes --- the Admin API info,

L. 

[Joe Palmer]

Here you go:

http://pastebin.com/s7tJDxcH

This is a snapshot from mid way though trying to play "Lorenzo says hello!" - I'm determined to watch this video eventually! :)

[Lorenzo Miniero]

Mh, peer SSRCs still missing, which means the autodetection is not working for you, not sure why.

Can you try adding this log line after the "Apparently" here:

   https://github.com/meetecho/janus-gateway/blob/master/ice.c#L1679

JANUS_LOG(LOG_FATAL, "%p, %p\n", stream->audio_payload_types, stream->video_payload_types);

and this line before both ifs in the loop:

   https://github.com/meetecho/janus-gateway/blob/master/ice.c#L1686

   https://github.com/meetecho/janus-gateway/blob/master/ice.c#L1700

JANUS_LOG(LOG_FATAL, "Comparing %"SCNu16" and %"SCNu16\n", pt, audio_pt);

recompile, and try again? This will print much more debug for yout test, and should hopefully help us understand why it's not working for you.

L.

[Joe Palmer]

OK, I have done this (the second log line needed an extra quote before \n). Here is what is logged when trying to play that video:

Creating new session: 2826401335
Creating new handle in session 2826401335: 1681162657
This is an audio recording, assuming Opus
This is a video recording, assuming VP8
[1681162657] Creating ICE agent (ICE Full mode, controlling)
(process:46373): GLib-WARNING **: poll(2) failed due to: Bad file descriptor.
(process:46373): GLib-WARNING **: poll(2) failed due to: Bad file descriptor.
(process:46373): GLib-WARNING **: poll(2) failed due to: Bad file descriptor.
[WARN] [1681162657] ICE failed for component 1 in stream 1, but we're still waiting for some info so we don't care... (trickle pending, answer received, alert not set)
[1681162657] The DTLS handshake has been completed

WebRTC media is now available

Joining playout thread
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fd1fb457360, 0x0
[WARN] [1681162657] Not video and not audio? dropping (SSRC 2147483649)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fd1fb457360, 0x0
[WARN] [1681162657] Not video and not audio? dropping (SSRC 2147483649)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fd1fb457360, 0x0
[WARN] [1681162657] Not video and not audio? dropping (SSRC 2147483650)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fd1fb457360, 0x0
[WARN] [1681162657] Not video and not audio? dropping (SSRC 2147483651)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fd1fb457360, 0x0
[WARN] [1681162657] Not video and not audio? dropping (SSRC 2147483652)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fd1fb457360, 0x0
[WARN] [1681162657] Not video and not audio? dropping (SSRC 2147483653)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fd1fb457360, 0x0
[WARN] [1681162657] Not video and not audio? dropping (SSRC 2147483654)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fd1fb457360, 0x0
[WARN] [1681162657] Not video and not audio? dropping (SSRC 2147483650)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fd1fb457360, 0x0
[WARN] [1681162657] Not video and not audio? dropping (SSRC 2147483655)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fd1fb457360, 0x0
[WARN] [1681162657] Not video and not audio? dropping (SSRC 2147483656)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fd1fb457360, 0x0
[WARN] [1681162657] Not video and not audio? dropping (SSRC 2147483657)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fd1fb457360, 0x0
[WARN] [1681162657] Not video and not audio? dropping (SSRC 2147483651)...
Leaving playout thread
No WebRTC media anymore
[1681162657] WebRTC resources freed

Very minor points, shouldn't the comment be Apparently we were not told the peer SSRCs? And in the second if, wouldn't it make more sense if the variable was called video_pt? E.g:

https://github.com/softforge/janus-gateway/commit/2d8fae64e9c4b46a447aeb608cd59345790a4f5a

Thanks again, happy to try/test any other suggestions.

Joe

[Lorenzo Miniero]

The "comparing" stuff should be OUTSIDE the if... if you put it in, it means those payload types are equal, which is not happening in your case.

Anyway, it's already weird that the video payload types list is empty in your case (0x0), especially considering they're listed in the SDP.

As to the typos, I just commited a fix for those, thanks for noticing.

L.

[Joe Palmer]

Whoops! The logs made no sense being inside the if, sorry I misread you request without thinking. Here are the new logs:

Creating new session: 2448414595
Creating new handle in session 2448414595: 2755596412

This is an audio recording, assuming Opus
This is a video recording, assuming VP8

[2755596412] Creating ICE agent (ICE Full mode, controlling)
(process:50512): GLib-WARNING **: poll(2) failed due to: Bad file descriptor.
(process:50512): GLib-WARNING **: poll(2) failed due to: Bad file descriptor.
[WARN] [2755596412] ICE failed for component 1 in stream 1, but we're still waiting for some info so we don't care... (trickle pending, answer received, alert not set)
[2755596412] The DTLS handshake has been completed

WebRTC media is now available
Joining playout thread

[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fb051713160, 0x0
[FATAL] [ice.c:janus_ice_cb_nice_recv:1687] Comparing 100 and 111
[WARN] [2755596412] Not video and not audio? dropping (SSRC 2147483649)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fb051713160, 0x0
[FATAL] [ice.c:janus_ice_cb_nice_recv:1687] Comparing 100 and 111
[WARN] [2755596412] Not video and not audio? dropping (SSRC 2147483650)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fb051713160, 0x0
[FATAL] [ice.c:janus_ice_cb_nice_recv:1687] Comparing 100 and 111
[WARN] [2755596412] Not video and not audio? dropping (SSRC 2147483651)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fb051713160, 0x0
[FATAL] [ice.c:janus_ice_cb_nice_recv:1687] Comparing 100 and 111
[WARN] [2755596412] Not video and not audio? dropping (SSRC 2147483652)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fb051713160, 0x0
[FATAL] [ice.c:janus_ice_cb_nice_recv:1687] Comparing 100 and 111
[WARN] [2755596412] Not video and not audio? dropping (SSRC 2147483649)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fb051713160, 0x0
[FATAL] [ice.c:janus_ice_cb_nice_recv:1687] Comparing 100 and 111
[WARN] [2755596412] Not video and not audio? dropping (SSRC 2147483653)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fb051713160, 0x0
[FATAL] [ice.c:janus_ice_cb_nice_recv:1687] Comparing 100 and 111
[WARN] [2755596412] Not video and not audio? dropping (SSRC 2147483654)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fb051713160, 0x0
[FATAL] [ice.c:janus_ice_cb_nice_recv:1687] Comparing 100 and 111
[WARN] [2755596412] Not video and not audio? dropping (SSRC 2147483655)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fb051713160, 0x0
[FATAL] [ice.c:janus_ice_cb_nice_recv:1687] Comparing 100 and 111
[WARN] [2755596412] Not video and not audio? dropping (SSRC 2147483656)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fb051713160, 0x0
[FATAL] [ice.c:janus_ice_cb_nice_recv:1687] Comparing 100 and 111
[WARN] [2755596412] Not video and not audio? dropping (SSRC 2147483650)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fb051713160, 0x0
[FATAL] [ice.c:janus_ice_cb_nice_recv:1687] Comparing 100 and 111
[WARN] [2755596412] Not video and not audio? dropping (SSRC 2147483657)...
[FATAL] [ice.c:janus_ice_cb_nice_recv:1680] 0x7fb051713160, 0x0
[FATAL] [ice.c:janus_ice_cb_nice_recv:1687] Comparing 100 and 111
[WARN] [2755596412] Not video and not audio? dropping (SSRC 2147483658)...

Leaving playout thread
No WebRTC media anymore

[2755596412] WebRTC resources freed

Thanks updating the typos. :)

[Lorenzo Miniero]

The check is failing because it's comparing the video payload type (100) only with the audio one it knows (111). Since the video_payload_types list is empty for some reason, it doesn't check there. I don't see any attempt for audio instead.

Anyway, the dump still doesn't make any sense, as I count 6 different SSRCs, there, and all sequential: 2147483649, 2147483650, 2147483651, 2147483652, 2147483653, 2147483654.

This should never happen: you should have one SSRC for audio and one for video, and maybe one more for video retransmission. The fact that you're getting so many different SSRCs for the same media type indicates that something's wrong on your end (maybe update the browser?)

That said, playing the "Lorenzo says hello" demo doesn't require the browser to send anything: only Janus would send media (the browser woul go in "receive only" mode), which means it should work already. If you want to make sure the feature works, try something bidirectional like the EchoTest.

L.

[WARN]<span style="color: #000;" class="styled-by-prett

[Joe Palmer]

OK, so the the Echo Test does actually work but still does produce logs like this:

Creating new session: 3674402604
Creating new handle in session 3674402604: 3918262724
[3918262724] Creating ICE agent (ICE Full mode, controlled)
[WARN] [3918262724] Skipping disabled/unsupported media line...
[WARN] [3918262724] Skipping disabled/unsupported media line...
[WARN] [3918262724] ICE failed for component 1 in stream 1, but we're still waiting for some info so we don't care... (trickle pending, answer received, alert not set)
[3918262724] The DTLS handshake has been completed

WebRTC media is now available

[WARN] [3918262724] Not video and not audio? dropping (SSRC 2036569383)...
[WARN] [3918262724] Not video and not audio? dropping (SSRC 1030055544)...
[WARN] [3918262724] Not video and not audio? dropping (SSRC 2896104696)...
[WARN] [3918262724] Not video and not audio? dropping (SSRC 389529023)...

Here are the details from the admin API:

http://pastebin.com/nz8Bs3iw

However, the Video Room demo has the same issue. It shows the warning and then the new fatal errors when a second person joins the room. What I would like to do is set up RTP forwarding which I believe is only a feature of the video room. I've got RTP forwarding working but haven't been able to decode the RTP stream. I assume that's because of this problem as the same browser is unable to decode the stream. Or should RTP forwarding be unaffected by this issue?

Do you think the problem could be related to these logs?

(process:50512): GLib-WARNING **: poll(2) failed due to: Bad file descriptor.
(process:50512): GLib-WARNING **: poll(2) failed due to: Bad file descriptor.
(process:50512): GLib-WARNING **: poll(2) failed due to: Bad file descriptor.

I noticed I always get them before the other logs on all demos except the Echo Test which is the only one that works for me.

Do you think this problem is specific to OSX? I can try deploying Janus on one our our dev servers running Centos7 if you think that would help. I did try compiling Janus on a Centos7 VM but it couldn't find 'sofia-sip-ua' (yum install sofia-sip-devel says no package available).

Any ideas you can suggest to get the Janus Video Room demo running in some form will be very welcome!

[Josh Dickson]

The issues are still present for me and I do not see any of the errors you're highlighting Joe. So I would guess that is not related. 

This definitely still happens with the latest Janus build across latest Chrome 50 (stable) and 52. 

--

[Joe Palmer]

So I have managed to compile Janus on a Centos 7 VM and all of the demos work (once I set the IP address in the JavaScript)! This is using the same browsers on my Mac that have the problem with the local version so it is definitely something Mac specific and related to the server rather then the browser.

I'd really like to get the local version working in the same was as it will be much easier to develop with so if there is anything more I can do to help get this running, let me know.

Thanks for the work you have done on it so far.

Joe

</s

[Josh Dickson]

FWIW I was never able to get it running on Mac. 

Do you still see the "not audio and not video" errors?

--

[Joe Palmer]

OK, which OS were you getting those errors on? I'm not getting them on CentOS 7, but had some fun compiling it as sofia-sip is not available in the main repos.

[Josh Dickson]

Getting them on Ubuntu 14.04 and 16.04 running Janus with latest Chrome for Mac ( 50.0.2661.102)

[Lorenzo Miniero]

Wondering if this can have anything to do with endianness? My guess is this is very unlikely, but maybe it's just parsing RTP wrong for you because of that.

L.

You received this message because you are subscribed to the Google Groups "meetecho-janus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.

[Joe Palmer]

That's an interesting idea. Do you know how we could test that?

[Josh Dickson]

FYI here is the admin dump you were asking for - apologies, took me a little while to figure this out but the instructions were in fact plenty clear enough.

http://pastebin.com/c956xqXH

& this is what I see in the (level 4) logs re: the “Not video and not audio?” warnings:

[WARN] [1845864325] No stream, queueing this trickle as it got here before the SDP...

[1845864325] Creating ICE agent (ICE Full mode, controlled)

[WARN] [1845864325]     Missing valid SRTP session (packet arrived too early?), skipping...

[1845864325] The DTLS handshake has been completed

WebRTC media is now available

[WARN] [1845864325] Not video and not audio? dropping (SSRC 561265344)...

[WARN] [1845864325] Not video and not audio? dropping (SSRC 1180642930)...

On Tue, May 24, 2016 at 1:09 PM Joe Palmer <Joe Palmer > wrote:

That's an interesting idea. Do you know how we could test that?

[Josh Dickson]

More info on the ‘Not video and not audio’ - seems to definitely be Chrome only. I do not see these issues developing in Firefox & unsure if present on Edge as I don’t have a testing machine handy.

[Lorenzo Miniero]

Try to use the post-processing tool (janus-pp-rec) to convert the recording we ship in the repo. It will do the same RTP parsing to order packets, and so in case something weird happens there it may end up creating an invalid video file (just guessing though).

https://janus.conf.meetecho.com/docs/janus-pp-rec_8c.html

L.

[Akil]

Hi,

I also have the same problem, video stream don't seem to work, i installed and enabled BoringSSL and the echotest now works, streaming demo doesn't work for the opus/v8 live stream locally.

[Lorenzo Miniero]

Have you started the test_gstreamer.sh script? Without anything feeding that mountpoint, you'll get no media obviously.

L.

[Akil]

Thanks, i missed that.

[Дмитрий Тютерев]

 I think I can help here.
After testing i have found that browser wondered
Chrome and Firefox can't return microphone resourse. And Chrome don't worry about it, he just send empty stream
You can check it on Firefox - he more clever and don't create handle without microphone accessed.

I think it can be fixed (on nix) by modifying asound. It should be dmix resourse in deafult


Sorry for my English

вторник, 17 мая 2016 г., 18:32:32 UTC+3 пользователь Joe Palmer написал: