Compile is using too much memory, causing signal 9

813 views
Skip to first unread message

Felix Sun

unread,
Mar 14, 2013, 7:19:40 AM3/14/13
to golang-nuts
Some of our dev server having 1G memory, and when compiling the whole application, It's causing signal 9, Which is using too much memory. Is there a way to control it not using that much memory when compiling?

Dave Cheney

unread,
Mar 14, 2013, 7:38:49 AM3/14/13
to Felix Sun, golang-nuts
No, there are no controls for compiler memory usage. The memory usage is in proportion to the number of expressions in your package. 

How many lines of source are there in the package that is causing the OOM killer to fire?

Can you post the output of go build -x on the package which will not compile. 



On 14/03/2013, at 10:19 PM, Felix Sun <sun...@gmail.com> wrote:

Some of our dev server having 1G memory, and when compiling the whole application, It's causing signal 9, Which is using too much memory. Is there a way to control it not using that much memory when compiling?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

chris dollin

unread,
Mar 14, 2013, 7:40:16 AM3/14/13
to Felix Sun, golang-nuts
I think you need to tell us more. Which version of Go?
What machine/OS? How big is your source code and
how is it structured? What's it doing when it runs out
of memory?

Chris

--
Chris "allusive" Dollin

Felix Sun

unread,
Mar 14, 2013, 9:44:50 AM3/14/13
to golan...@googlegroups.com
The environment is Ubuntu running on Amazon EC2

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu precise (development branch)"

Ubuntu precise (development branch) (GNU/Linux 3.2.0-20-virtual x86_64)


/usr/local/gopkg/src/github.com/theplant/qortex# go build -x
WORK=/tmp/go-build072308373
mkdir -p $WORK/github.com/theplant/qortex/configs/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/configs
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/configs/_obj/_go_.6 -p github.com/theplant/qortex/configs -D _/usr/local/gopkg/src/github.com/theplant/qortex/configs -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./config.go ./tips.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/configs.a $WORK/github.com/theplant/qortex/configs/_obj/_go_.6
mkdir -p $WORK/github.com/sunfmin/tenpu/_obj/
cd /usr/local/gopkg/src/github.com/sunfmin/tenpu
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/sunfmin/tenpu/_obj/_go_.6 -p github.com/sunfmin/tenpu -D _/usr/local/gopkg/src/github.com/sunfmin/tenpu -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./handlers.go ./tenpu.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/sunfmin/tenpu.a $WORK/github.com/sunfmin/tenpu/_obj/_go_.6
mkdir -p $WORK/github.com/sunfmin/tenpu/gridfs/_obj/
cd /usr/local/gopkg/src/github.com/sunfmin/tenpu/gridfs
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/sunfmin/tenpu/gridfs/_obj/_go_.6 -p github.com/sunfmin/tenpu/gridfs -D _/usr/local/gopkg/src/github.com/sunfmin/tenpu/gridfs -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./fs.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/sunfmin/tenpu/gridfs.a $WORK/github.com/sunfmin/tenpu/gridfs/_obj/_go_.6
mkdir -p $WORK/github.com/sunfmin/tenpu/mgometa/_obj/
cd /usr/local/gopkg/src/github.com/sunfmin/tenpu/mgometa
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/sunfmin/tenpu/mgometa/_obj/_go_.6 -p github.com/sunfmin/tenpu/mgometa -D _/usr/local/gopkg/src/github.com/sunfmin/tenpu/mgometa -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./storage.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/sunfmin/tenpu/mgometa.a $WORK/github.com/sunfmin/tenpu/mgometa/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/utils/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/utils
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/utils/_obj/_go_.6 -p github.com/theplant/qortex/utils -D _/usr/local/gopkg/src/github.com/theplant/qortex/utils -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./date.go ./debug.go ./set.go ./strings.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/utils.a $WORK/github.com/theplant/qortex/utils/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortexapi/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortexapi
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortexapi/_obj/_go_.6 -p github.com/theplant/qortexapi -D _/usr/local/gopkg/src/github.com/theplant/qortexapi -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./api.go ./data.go ./errors.go ./input.go ./ws_reply_data.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortexapi.a $WORK/github.com/theplant/qortexapi/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/organizations/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/organizations
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/organizations/_obj/_go_.6 -p github.com/theplant/qortex/organizations -D _/usr/local/gopkg/src/github.com/theplant/qortex/organizations -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./modify.go ./organization.go ./validation.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/organizations.a $WORK/github.com/theplant/qortex/organizations/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/mails/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/mails
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/mails/_obj/_go_.6 -p github.com/theplant/qortex/mails -D _/usr/local/gopkg/src/github.com/theplant/qortex/mails -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./mail_sender.go ./mail_updates.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/mails.a $WORK/github.com/theplant/qortex/mails/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/users/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/users
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/users/_obj/_go_.6 -p github.com/theplant/qortex/users -D _/usr/local/gopkg/src/github.com/theplant/qortex/users -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./curd_user.go ./follow.go ./inline_help.go ./sort.go ./user.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/users.a $WORK/github.com/theplant/qortex/users/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/groups/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/groups
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/groups/_obj/_go_.6 -p github.com/theplant/qortex/groups -D _/usr/local/gopkg/src/github.com/theplant/qortex/groups -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./associate.go ./curd.go ./filter.go ./group.go ./index.go ./modify.go ./my_group.go ./query.go ./sort.go ./validation.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/groups.a $WORK/github.com/theplant/qortex/groups/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/chats/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/chats
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/chats/_obj/_go_.6 -p github.com/theplant/qortex/chats -D _/usr/local/gopkg/src/github.com/theplant/qortex/chats -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./conversation.go ./ejabberd_message.go ./message.go ./room.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/chats.a $WORK/github.com/theplant/qortex/chats/_obj/_go_.6
mkdir -p $WORK/github.com/sunfmin/tenpu/thumbnails/_obj/
cd /usr/local/gopkg/src/github.com/sunfmin/tenpu/thumbnails
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/sunfmin/tenpu/thumbnails/_obj/_go_.6 -p github.com/sunfmin/tenpu/thumbnails -D _/usr/local/gopkg/src/github.com/sunfmin/tenpu/thumbnails -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./handlers.go ./storage.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/sunfmin/tenpu/thumbnails.a $WORK/github.com/sunfmin/tenpu/thumbnails/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/history/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/history
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/history/_obj/_go_.6 -p github.com/theplant/qortex/history -D _/usr/local/gopkg/src/github.com/theplant/qortex/history -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./backup.go ./delete.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/history.a $WORK/github.com/theplant/qortex/history/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/sharings/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/sharings
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/sharings/_obj/_go_.6 -p github.com/theplant/qortex/sharings -D _/usr/local/gopkg/src/github.com/theplant/qortex/sharings -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./notify.go ./query.go ./request.go ./sort.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/sharings.a $WORK/github.com/theplant/qortex/sharings/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/entries/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/entries
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/entries/_obj/_go_.6 -p github.com/theplant/qortex/entries -D _/usr/local/gopkg/src/github.com/theplant/qortex/entries -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./attachment_makers.go ./draft.go ./entries_reader.go ./entry.go ./entry_version.go ./followed_item.go ./index.go ./like.go ./modify.go ./my_item.go ./query.go ./sort.go ./system_message.go ./task.go ./validation.go ./watch_list.go ./wiki_section.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/entries.a $WORK/github.com/theplant/qortex/entries/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/files/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/files
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/files/_obj/_go_.6 -p github.com/theplant/qortex/files -D _/usr/local/gopkg/src/github.com/theplant/qortex/files -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./files.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/files.a $WORK/github.com/theplant/qortex/files/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/helpings/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/helpings
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/helpings/_obj/_go_.6 -p github.com/theplant/qortex/helpings -D _/usr/local/gopkg/src/github.com/theplant/qortex/helpings -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./helping.go ./validations.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/helpings.a $WORK/github.com/theplant/qortex/helpings/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/invitations/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/invitations
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/invitations/_obj/_go_.6 -p github.com/theplant/qortex/invitations -D _/usr/local/gopkg/src/github.com/theplant/qortex/invitations -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./invitation.go ./modify.go ./query.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/invitations.a $WORK/github.com/theplant/qortex/invitations/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/access/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/access
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/access/_obj/_go_.6 -p github.com/theplant/qortex/access -D _/usr/local/gopkg/src/github.com/theplant/qortex/access -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./access.go ./modify.go ./query.go ./validation.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/access.a $WORK/github.com/theplant/qortex/access/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/members/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/members
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/members/_obj/_go_.6 -p github.com/theplant/qortex/members -D _/usr/local/gopkg/src/github.com/theplant/qortex/members -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./associate.go ./curd_member.go ./forget_password.go ./login.go ./member.go ./registeration.go ./validation.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/members.a $WORK/github.com/theplant/qortex/members/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/notifications/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/notifications
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/notifications/_obj/_go_.6 -p github.com/theplant/qortex/notifications -D _/usr/local/gopkg/src/github.com/theplant/qortex/notifications -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./count.go ./curd_notification.go ./entity.go ./event.go ./notification_item.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/notifications.a $WORK/github.com/theplant/qortex/notifications/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/mails_hub/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/mails_hub
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/mails_hub/_obj/_go_.6 -p github.com/theplant/qortex/mails_hub -D _/usr/local/gopkg/src/github.com/theplant/qortex/mails_hub -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./attachment.go ./broadcast.go ./change_email.go ./daily_offline_updates.go ./daily_summary.go ./domain_confirmation.go ./entry_base.go ./entry_notified_mail.go ./entry_notify.go ./find_password.go ./invite_access_code.go ./invite_user.go ./like.go ./like_mail.go ./mail_Base.go ./mail_entry_item.go ./multiple_entries.go ./notify_admin.go ./password_changed.go ./shared_group_request.go ./sharing_invitation.go ./signup_confirmation.go ./task_mail.go ./welcome.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/mails_hub.a $WORK/github.com/theplant/qortex/mails_hub/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/newsletters/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/newsletters
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/newsletters/_obj/_go_.6 -p github.com/theplant/qortex/newsletters -D _/usr/local/gopkg/src/github.com/theplant/qortex/newsletters -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./newsletter.go ./validation.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/newsletters.a $WORK/github.com/theplant/qortex/newsletters/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/stats/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/stats
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/stats/_obj/_go_.6 -p github.com/theplant/qortex/stats -D _/usr/local/gopkg/src/github.com/theplant/qortex/stats -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./triggers.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/stats.a $WORK/github.com/theplant/qortex/stats/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/services/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/services
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/services/_obj/_go_.6 -p github.com/theplant/qortex/services -D _/usr/local/gopkg/src/github.com/theplant/qortex/services -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./backup_services.go ./blog_services.go ./broadcast_services.go ./chat_services.go ./deletion_services.go ./drafts_services.go ./entries_services.go ./files_services.go ./groups_services.go ./help_services.go ./invitations_services.go ./join_organization.go ./mail_services.go ./member_organization_services.go ./member_services.go ./member_user_services.go ./model_transfor.go ./newsletter_services.go ./notification_services.go ./organizations_services.go ./requests_services.go ./services.go ./settings_services.go ./setup_organization.go ./sharing_flow.go ./sharings_services.go ./sort.go ./task_services.go ./user_actions_services.go ./user_specific.go ./users_services.go ./watchlist_services.go ./wiki_services.go ./ws_actived_organization_services.go ./ws_counter_services.go ./ws_draft_services.go ./ws_global_services.go ./ws_online_user_services.go ./ws_rpc_init_services.go ./xmpp_services.go
go build github.com/theplant/qortex/services: signal 9
mkdir -p $WORK/github.com/theplant/qortex/smtpd/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/smtpd
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/smtpd/_obj/_go_.6 -p github.com/theplant/qortex/smtpd -D _/usr/local/gopkg/src/github.com/theplant/qortex/smtpd -I $WORK ./processor.go ./smtp.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/smtpd.a $WORK/github.com/theplant/qortex/smtpd/_obj/_go_.6
mkdir -p $WORK/github.com/theplant/qortex/pendingitems/_obj/
cd /usr/local/gopkg/src/github.com/theplant/qortex/pendingitems
/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/pendingitems/_obj/_go_.6 -p github.com/theplant/qortex/pendingitems -D _/usr/local/gopkg/src/github.com/theplant/qortex/pendingitems -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./pendingitems.go
/usr/local/go/pkg/tool/linux_amd64/pack grc $WORK/github.com/theplant/qortex/pendingitems.a $WORK/github.com/theplant/qortex/pendingitems/_obj/_go_.6


Felix Sun

unread,
Mar 14, 2013, 9:49:26 AM3/14/13
to golan...@googlegroups.com
Total is about 46815 line number of code.

Felix Sun

unread,
Mar 14, 2013, 9:52:22 AM3/14/13
to golan...@googlegroups.com
$ go version
go version go1.0.3

Rob Pike

unread,
Mar 14, 2013, 1:55:01 PM3/14/13
to Felix Sun, golan...@googlegroups.com
Which program is running when it runs out of memory? Is it the pack
command listed at the end of your trace?

Dave Cheney

unread,
Mar 14, 2013, 5:05:55 PM3/14/13
to Felix Sun, golan...@googlegroups.com
/usr/local/go/pkg/tool/linux_amd64/6g -o
$WORK/github.com/theplant/qortex/services/_obj/_go_.6 -p
github.com/theplant/qortex/services -D
_/usr/local/gopkg/src/github.com/theplant/qortex/services -I $WORK -I
/usr/local/gopkg/pkg/linux_amd64 ./backup_services.go
./blog_services.go ./broadcast_services.go ./chat_services.go
./deletion_services.go ./drafts_services.go ./entries_services.go
./files_services.go ./groups_services.go ./help_services.go
./invitations_services.go ./join_organization.go ./mail_services.go
./member_organization_services.go ./member_services.go
./member_user_services.go ./model_transfor.go ./newsletter_services.go
./notification_services.go ./organizations_services.go
./requests_services.go ./services.go ./settings_services.go
./setup_organization.go ./sharing_flow.go ./sharings_services.go
./sort.go ./task_services.go ./user_actions_services.go
./user_specific.go ./users_services.go ./watchlist_services.go
./wiki_services.go ./ws_actived_organization_services.go
./ws_counter_services.go ./ws_draft_services.go
./ws_global_services.go ./ws_online_user_services.go
./ws_rpc_init_services.go ./xmpp_services.go
go build github.com/theplant/qortex/services: signal 9

Can you provide more details, check /var/log/syslog or dmesg for
evidence of the OOM killer.

On Fri, Mar 15, 2013 at 12:49 AM, Felix Sun <sun...@gmail.com> wrote:
> Total is about 46815 line number of code.
>

Felix Sun

unread,
Mar 14, 2013, 10:06:55 PM3/14/13
to golan...@googlegroups.com
Here is from syslog, seems mongod did it?

Mar 15 02:04:34 ip-10-156-103-7 kernel: [6922370.723870] mongod invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723877] mongod cpuset=/ mems_allowed=0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723881] Pid: 29828, comm: mongod Not tainted 3.2.0-20-virtual #33-Ubuntu
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723884] Call Trace:
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723898] [<ffffffff810bdb3d>] ? cpuset_print_task_mems_allowed+0x9d/0xb0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723907] [<ffffffff81118221>] dump_header+0x91/0xe0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723911] [<ffffffff811185a5>] oom_kill_process+0x85/0xb0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723914] [<ffffffff8111894a>] out_of_memory+0xfa/0x220
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723919] [<ffffffff8111e37a>] __alloc_pages_nodemask+0x7ea/0x800
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723927] [<ffffffff81154d53>] alloc_pages_current+0xa3/0x110
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723931] [<ffffffff81114f8f>] __page_cache_alloc+0x8f/0xa0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723935] [<ffffffff8111528e>] ? find_get_page+0x1e/0x90
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723939] [<ffffffff81117112>] filemap_fault+0x212/0x3c0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723943] [<ffffffff81137332>] __do_fault+0x72/0x550
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723948] [<ffffffff8113abba>] handle_pte_fault+0xfa/0x200
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723954] [<ffffffff810067be>] ? xen_pmd_val+0xe/0x10
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723959] [<ffffffff81005209>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723962] [<ffffffff8113b078>] handle_mm_fault+0x1f8/0x350
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723969] [<ffffffff81654f8e>] ? _raw_spin_unlock_irqrestore+0x1e/0x30
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723975] [<ffffffff8165889b>] do_page_fault+0x14b/0x520
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723979] [<ffffffff81653e04>] ? do_nanosleep+0xa4/0xd0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723985] [<ffffffff8108dca8>] ? hrtimer_nanosleep+0xb8/0x180
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723989] [<ffffffff8108c8d0>] ? update_rmtp+0x70/0x70
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723993] [<ffffffff8108d6c4>] ? hrtimer_start_range_ns+0x14/0x20
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723997] [<ffffffff816554f5>] page_fault+0x25/0x30
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.724000] Mem-Info:
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.724002] Node 0 DMA per-cpu:
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.724005] CPU 0: hi: 0, btch: 1 usd: 0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.724007] Node 0 DMA32 per-cpu:
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.724010] CPU 0: hi: 186, btch: 31 usd: 44
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.724015] active_anon:140280 inactive_anon:16 isolated_anon:0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.724016] active_file:69 inactive_file:127 isolated_file:0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.724017] unevictable:0 dirty:0 writeback:0 unstable:0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.724018] free:1372 slab_reclaimable:2015 slab_unreclaimable:1918
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.724019] mapped:56 shmem:48 pagetables:1751 bounce:0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.724021] Node 0 DMA free:2468kB min:72kB low:88kB high:108kB active_anon:11860kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14524kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:232kB slab_unreclaimable:36kB kernel_stack:0kB pagetables:148kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.760049] lowmem_reserve[]: 0 597 597 597
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.760059] Node 0 DMA32 free:3020kB min:3088kB low:3860kB high:4632kB active_anon:549260kB inactive_anon:64kB active_file:276kB inactive_file:508kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:611856kB mlocked:0kB dirty:0kB writeback:0kB mapped:224kB shmem:192kB slab_reclaimable:7828kB slab_unreclaimable:7636kB kernel_stack:984kB pagetables:6856kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1070 all_unreclaimable? yes
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.760073] lowmem_reserve[]: 0 0 0 0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.760081] Node 0 DMA: 109*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2468kB
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.760100] Node 0 DMA32: 261*4kB 5*8kB 1*16kB 2*32kB 3*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3020kB
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.760119] 256 total pagecache pages
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.760121] 0 pages in swap cache
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.760123] Swap cache stats: add 0, delete 0, find 0/0
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.760125] Free swap = 0kB
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.760127] Total swap = 0kB
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762398] 159472 pages RAM
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762403] 8381 pages reserved
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762405] 607 pages shared
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762406] 149095 pages non-shared
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762409] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762421] [ 224] 0 224 4306 51 0 0 0 upstart-udev-br
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762425] [ 232] 0 232 5365 94 0 -17 -1000 udevd
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762429] [ 296] 0 296 5364 94 0 -17 -1000 udevd
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762433] [ 306] 0 306 5364 95 0 -17 -1000 udevd
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762438] [ 325] 0 325 3795 50 0 0 0 upstart-socket-
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762442] [ 387] 0 387 1814 125 0 0 0 dhclient3
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762446] [ 545] 0 545 12487 156 0 -17 -1000 sshd
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762450] [ 551] 101 551 63524 774 0 0 0 rsyslogd
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762454] [ 557] 102 557 5952 73 0 0 0 dbus-daemon
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762459] [ 628] 0 628 3624 39 0 0 0 getty
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762462] [ 633] 0 633 3624 41 0 0 0 getty
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762467] [ 637] 0 637 3624 40 0 0 0 getty
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762471] [ 640] 0 640 3624 40 0 0 0 getty
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762474] [ 642] 0 642 3624 41 0 0 0 getty
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762479] [ 652] 0 652 1080 35 0 0 0 acpid
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762482] [ 659] 0 659 4225 40 0 0 0 atd
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762486] [ 660] 0 660 4776 53 0 0 0 cron
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762490] [ 683] 105 683 27915 292 0 0 0 whoopsie
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762494] [ 687] 109 687 2750 57 0 0 0 beanstalkd
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762498] [ 724] 107 724 1850 37 0 0 0 epmd
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762502] [ 771] 108 771 17011 14541 0 0 0 redis-server
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762506] [ 829] 0 829 3624 41 0 0 0 getty
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762510] [ 7429] 107 7429 27655 15141 0 0 0 beam
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762514] [ 7444] 107 7444 1841 24 0 0 0 inet_gethost
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762518] [ 7445] 107 7445 3426 41 0 0 0 inet_gethost
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762523] [29824] 106 29824 2230750 10679 0 0 0 mongod
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762526] [ 1118] 0 1118 8694 171 0 0 0 monit
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762530] [27567] 107 27567 3426 44 0 0 0 inet_gethost
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762534] [27568] 107 27568 3426 41 0 0 0 inet_gethost
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762539] [27569] 107 27569 3426 37 0 0 0 inet_gethost
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762543] [ 8280] 0 8280 79121 849 0 0 0 batchbuy
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762547] [16154] 0 16154 193001 5431 0 0 0 qortex_dev
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762551] [18888] 0 18888 18338 209 0 0 0 sshd
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762555] [19014] 1000 19014 18338 202 0 0 0 sshd
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762558] [19015] 1000 19015 6713 1925 0 0 0 bash
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762562] [19120] 0 19120 18338 208 0 0 0 sshd
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762566] [19204] 1000 19204 18338 201 0 0 0 sshd
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762569] [19205] 1000 19205 6714 1926 0 0 0 bash
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762573] [19346] 1000 19346 1085 20 0 0 0 tail
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762577] [19388] 1000 19388 17388 1520 0 0 0 go
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762581] [19440] 1000 19440 86829 84801 0 0 0 6g
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762585] Out of memory: Kill process 19440 (6g) score 562 or sacrifice child
Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.762603] Killed process 19440 (6g) total-vm:347316kB, anon-rss:339076kB, file-rss:128kB

Dave Cheney

unread,
Mar 14, 2013, 10:17:59 PM3/14/13
to Felix Sun, golan...@googlegroups.com
> Mar 15 02:04:34 ip-10-156-103-7 kernel: [6922370.723870] mongod invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
> Mar 15 02:04:35 ip-10-156-103-7 kernel: [6922370.723877] mongod cpuset=/ mems_allowed=0

Mongo ate all your ram. This isn't 6g, just your machine is overcommitted.

Felix Sun

unread,
Mar 14, 2013, 10:25:18 PM3/14/13
to golan...@googlegroups.com
The thing is, whenever I invoke the build, and the 6g gets killed. Not mongod. That's normal?

Felix Sun

unread,
Mar 14, 2013, 10:28:02 PM3/14/13
to golan...@googlegroups.com
Ok, I stopped mongod, and did the same, Now the syslog is:


Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127391] 6g invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127397] 6g cpuset=/ mems_allowed=0
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127402] Pid: 19845, comm: 6g Not tainted 3.2.0-20-virtual #33-Ubuntu
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127404] Call Trace:
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127418] [<ffffffff810bdb3d>] ? cpuset_print_task_mems_allowed+0x9d/0xb0
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127426] [<ffffffff81118221>] dump_header+0x91/0xe0
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127430] [<ffffffff811185a5>] oom_kill_process+0x85/0xb0
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127434] [<ffffffff8111894a>] out_of_memory+0xfa/0x220
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127438] [<ffffffff8111e37a>] __alloc_pages_nodemask+0x7ea/0x800
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127446] [<ffffffff81154d53>] alloc_pages_current+0xa3/0x110
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127451] [<ffffffff81114f8f>] __page_cache_alloc+0x8f/0xa0
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127454] [<ffffffff8111528e>] ? find_get_page+0x1e/0x90
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127458] [<ffffffff81117112>] filemap_fault+0x212/0x3c0
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127463] [<ffffffff81137332>] __do_fault+0x72/0x550
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127467] [<ffffffff81121b9d>] ? lru_cache_add_lru+0x2d/0x50
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127471] [<ffffffff8113abba>] handle_pte_fault+0xfa/0x200
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127478] [<ffffffff810067be>] ? xen_pmd_val+0xe/0x10
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127481] [<ffffffff81005209>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127485] [<ffffffff8113b078>] handle_mm_fault+0x1f8/0x350
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127490] [<ffffffff8100a25d>] ? xen_force_evtchn_callback+0xd/0x10
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127496] [<ffffffff8165889b>] do_page_fault+0x14b/0x520
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127500] [<ffffffff8100aa32>] ? check_events+0x12/0x20
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127504] [<ffffffff8100aa1f>] ? xen_restore_fl_direct_reloc+0x4/0x4
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127508] [<ffffffff8100465d>] ? xen_clts+0x8d/0x190
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127512] [<ffffffff816554f5>] page_fault+0x25/0x30
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127515] Mem-Info:
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127517] Node 0 DMA per-cpu:
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127520] CPU 0: hi: 0, btch: 1 usd: 0
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127522] Node 0 DMA32 per-cpu:
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127526] CPU 0: hi: 186, btch: 31 usd: 99
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127530] active_anon:141160 inactive_anon:17 isolated_anon:0
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127531] active_file:24 inactive_file:33 isolated_file:0
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127532] unevictable:0 dirty:0 writeback:0 unstable:0
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127533] free:1366 slab_reclaimable:1809 slab_unreclaimable:1935
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127534] mapped:16 shmem:49 pagetables:1162 bounce:0
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127537] Node 0 DMA free:2460kB min:72kB low:88kB high:108kB active_anon:12140kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14524kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:112kB slab_unreclaimable:40kB kernel_stack:0kB pagetables:24kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127546] lowmem_reserve[]: 0 597 597 597
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127552] Node 0 DMA32 free:3004kB min:3088kB low:3860kB high:4632kB active_anon:552500kB inactive_anon:68kB active_file:96kB inactive_file:132kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:611856kB mlocked:0kB dirty:0kB writeback:0kB mapped:64kB shmem:196kB slab_reclaimable:7124kB slab_unreclaimable:7700kB kernel_stack:896kB pagetables:4624kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:363 all_unreclaimable? yes
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127562] lowmem_reserve[]: 0 0 0 0
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127568] Node 0 DMA: 65*4kB 21*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2460kB
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127583] Node 0 DMA32: 295*4kB 0*8kB 0*16kB 1*32kB 2*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3004kB
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127598] 112 total pagecache pages
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127600] 0 pages in swap cache
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127602] Swap cache stats: add 0, delete 0, find 0/0
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127604] Free swap = 0kB
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.127606] Total swap = 0kB
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.129965] 159472 pages RAM
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.129970] 8381 pages reserved
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.129972] 623 pages shared
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.129973] 148963 pages non-shared
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.129976] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.129989] [ 224] 0 224 4306 51 0 0 0 upstart-udev-br
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.129996] [ 232] 0 232 5365 94 0 -17 -1000 udevd
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130000] [ 296] 0 296 5364 94 0 -17 -1000 udevd
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130004] [ 306] 0 306 5364 95 0 -17 -1000 udevd
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130009] [ 325] 0 325 3795 50 0 0 0 upstart-socket-
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130013] [ 387] 0 387 1814 125 0 0 0 dhclient3
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130017] [ 545] 0 545 12487 153 0 -17 -1000 sshd
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130021] [ 551] 101 551 63524 774 0 0 0 rsyslogd
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130025] [ 557] 102 557 5952 73 0 0 0 dbus-daemon
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130030] [ 628] 0 628 3624 39 0 0 0 getty
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130034] [ 633] 0 633 3624 41 0 0 0 getty
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130037] [ 637] 0 637 3624 40 0 0 0 getty
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130041] [ 640] 0 640 3624 40 0 0 0 getty
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130045] [ 642] 0 642 3624 41 0 0 0 getty
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130050] [ 652] 0 652 1080 35 0 0 0 acpid
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130054] [ 659] 0 659 4225 40 0 0 0 atd
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130057] [ 660] 0 660 4776 51 0 0 0 cron
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130061] [ 683] 105 683 27915 292 0 0 0 whoopsie
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130065] [ 687] 109 687 2750 57 0 0 0 beanstalkd
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130068] [ 724] 107 724 1850 39 0 0 0 epmd
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130072] [ 771] 108 771 17011 14541 0 0 0 redis-server
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130076] [ 829] 0 829 3624 41 0 0 0 getty
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130080] [ 7429] 107 7429 26707 14188 0 0 0 beam
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130083] [ 7444] 107 7444 1841 24 0 0 0 inet_gethost
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130087] [ 7445] 107 7445 3426 41 0 0 0 inet_gethost
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130090] [ 1118] 0 1118 8694 171 0 0 0 monit
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130094] [27567] 107 27567 3426 40 0 0 0 inet_gethost
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130098] [27568] 107 27568 3426 41 0 0 0 inet_gethost
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130102] [27569] 107 27569 3426 37 0 0 0 inet_gethost
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130106] [ 8280] 0 8280 95505 896 0 0 0 batchbuy
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130109] [16154] 0 16154 193001 12575 0 0 0 qortex_dev
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130113] [18888] 0 18888 18338 204 0 0 0 sshd
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130117] [19014] 1000 19014 18338 202 0 0 0 sshd
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130121] [19015] 1000 19015 6713 1921 0 0 0 bash
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130125] [19120] 0 19120 18338 203 0 0 0 sshd
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130129] [19204] 1000 19204 18338 204 0 0 0 sshd
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130133] [19205] 1000 19205 6714 1923 0 0 0 bash
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130137] [19465] 1000 19465 9565 88 0 0 0 sudo
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130140] [19466] 0 19466 5228 837 0 0 0 bash
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130144] [19579] 0 19579 5844 296 0 0 0 htop
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130148] [19584] 0 19584 18338 202 0 0 0 sshd
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130151] [19667] 1000 19667 18338 200 0 0 0 sshd
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130155] [19668] 1000 19668 6713 1924 0 0 0 bash
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130159] [19793] 1000 19793 1085 20 0 0 0 tail
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130162] [19794] 1000 19794 17388 1656 0 0 0 go
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130166] [19845] 1000 19845 88674 86579 0 0 0 6g
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130170] Out of memory: Kill process 19845 (6g) score 574 or sacrifice child
Mar 15 02:27:19 ip-10-156-103-7 kernel: [6923735.130188] Killed process 19845 (6g) total-vm:354696kB, anon-rss:346316kB, file-rss:0kB


Dave Cheney

unread,
Mar 14, 2013, 10:29:28 PM3/14/13
to Felix Sun, golan...@googlegroups.com
Add some swap.

Felix Sun

unread,
Mar 14, 2013, 10:34:21 PM3/14/13
to golan...@googlegroups.com

Ok, the 6g quickly using more than 350M of memory, It's expected? I understand the memory usage is not big, and my machine has too few memories, and can easily work around this issue, But Just thought that maybe something to improve for the memory usage of the compiling.

PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
19903 ubuntu 20 0 343M 336M 800 R 89.0 57.0 0:17.57 /usr/local/go/pkg/tool/linux_amd64/6g -o /tmp/go-build625065795/github.com/theplant/qortex/servic

Russ Cox

unread,
Mar 14, 2013, 11:59:21 PM3/14/13
to Felix Sun, golang-nuts
Is the machine has multiple virtual cores, try invoking the go command with -p=1 -x. That will limit the number of subcommands being executed to 1 and print the commands being executed. Then you can be sure that the last command is the one that used all the RAM. I'd be interested to know what package is causing the problems and anything about it that makes it unusual.

From your ps listing I looked for but could not find any packages named github.com/theplant/qortex. I found github.com/theplant/qortexapi, but that looks like a fairly small package.

Russ

Felix Sun

unread,
Mar 15, 2013, 12:21:57 AM3/15/13
to Russ Cox, golang-nuts
From looking at the htop, and tried `go build -p=1 -x`, I am sure this command causing the kill

/usr/local/go/pkg/tool/linux_amd64/6g -o $WORK/github.com/theplant/qortex/services/_obj/_go_.6 -p github.com/theplant/qortex/services -D _/usr/local/gopkg/src/github.com/theplant/qortex/services -I $WORK -I /usr/local/gopkg/pkg/linux_amd64 ./backup_services.go ./blog_services.go ./broadcast_services.go ./chat_services.go ./deletion_services.go ./drafts_services.go ./entries_services.go ./files_services.go ./groups_services.go ./help_services.go ./invitations_services.go ./join_organization.go ./mail_services.go ./member_organization_services.go ./member_services.go ./member_user_services.go ./model_transfor.go ./newsletter_services.go ./notification_services.go ./organizations_services.go ./requests_services.go ./services.go ./settings_services.go ./setup_organization.go ./sharing_flow.go ./sharings_services.go ./sort.go ./task_services.go ./user_actions_services.go ./user_specific.go ./users_services.go ./watchlist_services.go ./wiki_services.go ./ws_actived_organization_services.go ./ws_counter_services.go ./ws_draft_services.go ./ws_global_services.go ./ws_online_user_services.go ./ws_rpc_init_services.go ./xmpp_services.go


The htop output
22785 root       20   0  831M  512M   288 D 11.0 86.8  0:17.01 /usr/local/go/pkg/tool/linux_amd64/6g -o /tmp/go-build478772641/github.com/theplant/qortex/servic


the package github.com/theplant/qortex/services (which is not public on github) is a big monster and has:

10927 lines of source code
130 dependency packages
40 .go files

FelixMBPR:services sunfmin$ go install -v -a
runtime
errors
sync/atomic
unicode
unicode/utf8
math
crypto/subtle
unicode/utf16
container/list
sync
image/color
io
syscall
hash
crypto/cipher
crypto
crypto/hmac
hash/crc32
hash/adler32
crypto/md5
crypto/sha1
bytes
strings
path
bufio
html
time
os
strconv
math/rand
sort
path/filepath
crypto/aes
reflect
crypto/rc4
encoding/base64
io/ioutil
net/url
regexp/syntax
image
encoding/pem
image/jpeg
regexp
encoding/binary
fmt
crypto/des
math/big
runtime/cgo
encoding/json
compress/flate
log
mime
runtime/debug
encoding/gob
text/template/parse
text/template
encoding/hex
crypto/rand
crypto/elliptic
crypto/dsa
crypto/rsa
encoding/asn1
html/template
compress/lzw
crypto/x509/pkix
image/gif
compress/gzip
archive/zip
compress/zlib
image/png
crypto/x509
net
crypto/tls
net/textproto
mime/multipart
net/http
net/smtp
net/rpc
net/rpc/jsonrpc

Russ Cox

unread,
Mar 15, 2013, 12:21:02 PM3/15/13
to Felix Sun, golang-nuts
It sounds like maybe you should build on a server with more memory, or split up the package. 6g's memory footprint can probably be reduced but nothing significant will happen there before Go 1.1.

Russ

Dave Cheney

unread,
Mar 15, 2013, 8:33:07 PM3/15/13
to Russ Cox, Felix Sun, golang-nuts
Felix,

Are you using a t1.micro ? The OOM killer says you have 159472 pages
RAM, a page is 4096 bytes, and that is just under 640mb of ram, and no
swap. And you are sharing that machine with a running copy of redis,
an erlang beam and a running copy of your application.

rsc is correct that the compilers should use less memory, but I think
you need to be reasonable about the environment you compile your
application in.

Cheers

Dave

Felix Sun

unread,
Mar 15, 2013, 9:30:43 PM3/15/13
to Dave Cheney, Russ Cox, golang-nuts
Yes, you are right! We are running ejabberd, redis, mongodb, beanstalk, and our go app on a t1.micro, and it runs quite smoothly except compiling ( but we already doing cross compiling), I am quite proud of Go when talking with people about this! Haha.

John Nagle

unread,
Mar 20, 2013, 4:08:11 PM3/20/13
to golan...@googlegroups.com
On 3/14/2013 6:49 AM, Felix Sun wrote:
> Some of our dev server having 1G memory, and when compiling the whole
application, It's causing signal 9,
...
> Total is about 46815 line number of code.

So the compiler needs about 20K of RAM per line of source
code?

John Nagle




minux

unread,
Mar 20, 2013, 4:15:43 PM3/20/13
to Felix Sun, golan...@googlegroups.com
a common problem for excessive 6g memory usage is using a lot of small integers
in slice/array initialization.

Go 1.0.x used to do badly for deeply nested if-then-else chain, but Go
tip has fixed
that, so probably you could try go tip and see if anything changes.

Rémy Oudompheng

unread,
Mar 20, 2013, 4:53:48 PM3/20/13
to minux, Felix Sun, golan...@googlegroups.com
The cost of importing packages could also be excessive, Go tip fixes
that, and the difference is visible.

Rémy.


2013/3/20, minux <minu...@gmail.com>:
Reply all
Reply to author
Forward
0 new messages