Curl helper mixes up logs, when website does a lot of curl requests

7 views
Skip to first unread message

Alexander Obuhovich

unread,
Dec 13, 2010, 4:34:17 AM12/13/10
to In-Portal Bugs
Curl helper has nice logging feature, that allows to trace all curl requests made over time.

Problem is, that it's not very easy to handle such amount of information, when it is written into a single file. Also when website has large amount of concurrent curl requests, then curl request data is all mixed up together and there is no way to detect what actually happens.

To solve it I propose to log curl requests to database.

Here is table structure, that I've used:

CREATE TABLE IF NOT EXISTS `ptn_CurlLog` (
  `LogId` int(11) NOT NULL AUTO_INCREMENT,
  `Message` varchar(255) NOT NULL,
  `PageUrl` varchar(255) NOT NULL,
  `RequestUrl` varchar(255) NOT NULL,
  `PortalUserId` int(11) NOT NULL,
  `SessionKey` int(11) NOT NULL,
  `IsAdmin` tinyint(4) NOT NULL,
  `PageData` text,
  `RequestData` text,
  `ResponseData` text,
  `RequestDate` int(11) DEFAULT NULL,
  `ResponseDate` int(11) DEFAULT NULL,
  `ResponseHttpCode` int(11) NOT NULL,
  `CurlError` varchar(255) NOT NULL,
  PRIMARY KEY (`LogId`),
  KEY `Message` (`Message`),
  KEY `PageUrl` (`PageUrl`),
  KEY `RequestUrl` (`RequestUrl`),
  KEY `PortalUserId` (`PortalUserId`),
  KEY `SessionKey` (`SessionKey`),
  KEY `IsAdmin` (`IsAdmin`),
  KEY `RequestDate` (`RequestDate`),
  KEY `ResponseDate` (`ResponseDate`),
  KEY `ResponseHttpCode` (`ResponseHttpCode`),
  KEY `CurlError` (`CurlError`)
);

I've also attached kCurlHelper extended class, that does all that.


As you can see, there is not much code to do the stuff, but difference is huge, since browsing an organized table is far more easier, then a huge log file.


e_curl_helper.php

Phil -- wbtc.fr --

unread,
Dec 13, 2010, 4:41:00 AM12/13/10
to in-port...@googlegroups.com
just for me and any other reader information, when does we use Curl in
standard install, and which purpose?

2010/12/13 Alexander Obuhovich <aik....@gmail.com>:

> --
> You received this message because you are subscribed to the Google Groups
> "In-Portal Bugs Team" group.
> To post to this group, send email to in-port...@googlegroups.com.
> To unsubscribe from this group, send email to
> in-portal-bug...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/in-portal-bugs?hl=en.
>

Alexander Obuhovich

unread,
Dec 13, 2010, 4:46:13 AM12/13/10
to in-port...@googlegroups.com
Curl allows to retrieve data from other websites. We are only using it for RSS retrieval in In-News.

However, when website, based on In-Portal uses a lot of different API, e.g. QuickBooks, weather services, google calendar, then more curl requests are made.

In case if something breaks down on live webasite, then such logs would come in handy.

Phil -- wbtc.fr --

unread,
Dec 13, 2010, 5:27:41 AM12/13/10
to in-port...@googlegroups.com
thanks for details :)

Dmitry A.

unread,
Dec 13, 2010, 11:38:01 AM12/13/10
to in-port...@googlegroups.com
Hi Alex,


You have read my mind, my friend!


I was struggling to working some sort of Curl logging in the files with no much success and here is your solution. I would love to ask you to put this in 5.2.0 or may be even 5.1.x release if it's not too much to ask!

Again thanks for sharing this code and please create a task.


Cheers!


DA

Alexander Obuhovich

unread,
Dec 13, 2010, 1:35:35 PM12/13/10
to in-port...@googlegroups.com
Here is the task: http://tracker.in-portal.org/view.php?id=950

--
You received this message because you are subscribed to the Google Groups "In-Portal Bugs Team" group.
To post to this group, send email to in-port...@googlegroups.com.
To unsubscribe from this group, send email to in-portal-bug...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/in-portal-bugs?hl=en.
Reply all
Reply to author
Forward
0 new messages