Suggestions and Improvements

[MassConfusion]

I probably should have done this a while ago.

* adding a Last_Successful_Run and/or Last_Error_Reason columns to the
my_fleets view to make it easier to determine why a fleet script
failed without access to the NOTIFY channel. Also helpful when a
script runs longer than 1 minute because through the web interface it
seems to time out after 60 seconds regardless of how long the fleet's
runtime entry is.

[Abstrct]

Thanks again for the suggestion MC!

I have no issue with implementing this into the the fleet system. I
have a bit of a backlog right now to get through regarding system
changes but I will definitely work this in when I can. Hopefully
sometime this week.

-Abstrct

[derpfish]

One other thought: it'd be nice if there was something like an extra
column on my_fleets which holds how long the fleet script took the
last time it ran. That way there would be an easier way to tell if you
should make your script a bit more efficient or else upgrade your
fleet script time.

[MassConfusion]

This, definately. With only access to the system through the web
interface its increadibly difficult to pinpoint trouble spots. A last
run length field would be great, this is something I was trying to do
with variables that just never quite worked right. Super excited to
play with the new move system this week!

[Abstrct]

Executed time on the successful scripts will be a bit of an estimate but at least it will be something. I will add this change as well when adding last_successful_run and last_error_msg. 

On a similar note, what do you guys think of the run_time system in fleets currently? A part of me worries about performance once numerous players are involved with large scripts and I wonder if the tic will start to take way too long. As many have brought up, it is also slightly unfair when somebody doesn't use the fleet system and has their own external script polling and running fleet_script() functions manually as this gives them an extra huge pile of cash of the start to do as they wish. 

If every player had only one minute of time each tic would that make writing scripts for the game too annoying or would it add an interesting new element to it? Right now, if you have a huge amount of money you can buy more fleet time and keep controlling every aspect of your fleet. Would it be a positive gameplay change if you could only strategically command portions of the fleet instead?

Earlier in the games life cycle I don't think this would have been an option but now that ships have a state you can manage (action and action_target_id) and a proper move system in place, it seems like it may be a positive change.

Thanks for your input on this. Please don't hesitate to tell me this is an awful idea if you feel that way. 

-Abstrct

[Christopher Browne]

On Wednesday, 7 March 2012 10:46:15 UTC-5, Abstrct wrote:

Executed time on the successful scripts will be a bit of an estimate but at least it will be something. I will add this change as well when adding last_successful_run and last_error_msg. 

clock_timestamp(), called at start and end, should give reasonable results. 

On a similar note, what do you guys think of the run_time system in fleets currently? A part of me worries about performance once numerous players are involved with large scripts and I wonder if the tic will start to take way too long. As many have brought up, it is also slightly unfair when somebody doesn't use the fleet system and has their own external script polling and running fleet_script() functions manually as this gives them an extra huge pile of cash of the start to do as they wish. 

Maybe the first minute is free?
 

If every player had only one minute of time each tic would that make writing scripts for the game too annoying or would it add an interesting new element to it? Right now, if you have a huge amount of money you can buy more fleet time and keep controlling every aspect of your fleet. Would it be a positive gameplay change if you could only strategically command portions of the fleet instead?

If there's more about the ship's individual activities that are *fairly* autonomous, e.g., burning fuel to accelerate them towards targets,  that allows the "strategic portion" to be limited, and sure, that seems pretty plausible.

[Tigereye .]

Good idea. Ditch the first 10m credits at round start and give everyone 1min on their startscript. 

Sent from my mobile device

[MassConfusion]

I'm not 100% opposed to limiting the scripts to 1 minute, but having a longer running script myself it would definitely be a bit aggravating to try and rewrite it to accomplish everything that it does now in a shorter period of time. I think limiting it to 1 minute would cause a lot of strategy changes (or forced limitations to those strategies)

If we're concerned about performance, and this is probably going to be just as irritating as the scripts being limited, how about a ship cap for each player? I could probably live with a few thousand ships max.

The only problem with automatically adding 1 min to a player's startscript would be if that player doesn't have one specified, but I guess in those cases you could leave them the 10mil so they could use it wherever, or create a default script for each player and give it 1 minute regardless.

To keep the tic alive, would it not be possible to have something run every few minutes and when it sees that the tic hasn't advanced in 30+ minutes have it kill the process, dump the entire tic, and start over?

[Abstrct]

Ya, it would certainly mean a re-write of most script logic if we were to limit each player to a minute per tic. I don't think it is an idea we want to tackle yet but I appreciate the discussion around the idea. 

To change it so that everybody get's a free minute, I will just need to change round_control() and upgrade().  round_control() won't deduct money for the starting script and upgrade will just give the first upgrade for free by checking the fleet table for other scripts. Not a huge amount of effort to implement.

I have considered limiting ship creation a couple times as well. I think it would help performance a lot but it would also change everyone's script logic quite a bit. Limiting ship creation is likely inevitable for the game but I want to put a bit more thought into it before moving forward with the change. 

I assume your comment about the TIC being dead after thirty minutes is in regards to your tweet last night about the system being down. Usually if there is an issue that has caused the tic to crash, simply rolling everything back and retrying the tic would potentially, at best, just delay another crash for one or two tics. Normally, it is happening because of a value outgrowing it's column data type (integer to bigint), some stale data in either planet_miners or ship_flight_recorder that requires removal or other strange bugs. Repairing this always means fixing a bug in one of the many functions or deciding to increase the size of a column and/or create a limit on the value. Except for big issues like that, the tic rarely just crashes in a way that can be restarted with ease. I would certainly love to get to a point where there are no more large issues like this stalling tic.pl now and then, and it will get there, but it is still a relatively young repository of code and I'm no master dev. 

As more players start joining the game and more weird corner cases are discovered, the game will hopefully stabilize :) 

-Abstrct

[derpfish]

On Mar 9, 11:34 am, Abstrct <j...@saucetel.com> wrote:
> Ya, it would certainly mean a re-write of most script logic if we were to
> limit each player to a minute per tic. I don't think it is an idea we want
> to tackle yet but I appreciate the discussion around the idea.
>

It can be a bit annoying later in the game when it slows down to 10+
minutes per tic, so I can see the appeal of wanting to limit script
time. Although 1 minute per tic is usually enough for me earlier in
the game, once I run into enemy space and have to start going through
ships_in_range it seems to quickly become insufficient. Since I'd like
to see battle as a more critical component (despite not yet having
written a good battle script) I think we ought to allow higher
runtimes.

> To change it so that everybody get's a free minute, I will just need to
> change round_control() and upgrade().  round_control() won't deduct money
> for the starting script and upgrade will just give the first upgrade for
> free by checking the fleet table for other scripts. Not a huge amount of
> effort to implement.
>

This seems like a good solution to me.

> I have considered limiting ship creation a couple times as well. I think it
> would help performance a lot but it would also change everyone's script
> logic quite a bit. Limiting ship creation is likely inevitable for the game
> but I want to put a bit more thought into it before moving forward with the
> change.
>

Is the number of ships what is actually bottlenecking runtime? Thus
far I haven't really found a good way to figure out what part of my
script is the slow bit, so I feel like I can't give a lot of good
input. In my current script incarnation, miners make up at least 90%
of my fleet, and since I pretty much just set their action and ignore
them I don't think they effect the runtime of my script. so it doesn't
seem like capping the number of ships you get would do much to change
the speed of my fleet script. However, I can see that the size of the
ship table would regardless effect querying ships_in_range, so if
that's the case, what sort of cap would we be looking at?

> If there's more about the ship's individual activities that are *fairly*
> autonomous, e.g., burning fuel to accelerate them towards targets, that
> allows the "strategic portion" to be limited, and sure, that seems pretty
> plausible.

Besides figuring out if there are ships_in_range that need to be
attacked, there's still a little bit of ship behavior that I handle
manually in my scripts with regard to movement. But I think this gets
into the realm of additional refinements to movement which have been
hinted at on github ( https://github.com/Abstrct/Schemaverse/issues/11
).

[Abstrct]

It can be a bit annoying later in the game when it slows down to 10+
minutes per tic, so I can see the appeal of wanting to limit script
time. Although 1 minute per tic is usually enough for me earlier in
the game, once I run into enemy space and have to start going through
ships_in_range it seems to quickly become insufficient. Since I'd like
to see battle as a more critical component (despite not yet having
written a good battle script) I think we ought to allow higher
runtimes.

Hopefully, now that ships_in_range is based off cached data, this complete slow down shouldn't be as obvious. I am at least happy to hear that, for the most, more run time then a minute isn't needed but I since that is not always the case I will likely leave the run-time system alone until we can optimize it enough that one minute is all you actually need. It is quite possible that this won't be achieved but we shall see what the future brings us :)

I definitely want to see the battle become a larger aspect of the game too. This was the reasoning behind a lot of changes lately (new planet starting points, removal of the middle spawn zone, and new trophy system). Keep us all posted on your journey into the ATTACK() function. 

Is the number of ships what is actually bottlenecking runtime? Thus
far I haven't really found a good way to figure out what part of my
script is the slow bit, so I feel like I can't give a lot of good
input. In my current script incarnation, miners make up at least 90%
of my fleet, and since I pretty much just set their action and ignore
them I don't think they effect the runtime of my script. so it doesn't
seem like capping the number of ships you get would do much to change
the speed of my fleet script. However, I can see that the size of the
ship table would regardless effect querying ships_in_range, so if
that's the case, what sort of cap would we be looking at?

I do believe that the number of ships in the system is proportionate to the performance that players notice. More ships means more upgrades, more actions and more movement. All of that means more queries overall, longer time to build the ships_in_range cache (or previously, to view the ships_in_range in general), perform move_ships and perform perform_minute  and, most noticeably, far more action logs in the event table.

 I guess the real question though is, would limiting ship creation be the best solution to put in place. Maybe changing to the cached ships_in_range will make this mostly a non-issue. If the event table is getting too big then maybe changing what constitutes an event needs to be looked at again. Similarly, perform_mining() hasn't really been looked at in a while, maybe there is some way to optimize it or any of the ship actions for that matter. 

[Mike N.]

think the tic is down again chief
no worries, looks like I started close enough to get to derpfish before he spread to far

unfortunately the new script also started too close.. and got mangled yesterday

[Mike N.]

nevermind :) I guess it's just you playing with the security

[derpfish]

> > think the tic is down again chief
> > no worries, looks like I started close enough to get to derpfish before he
> > spread to far

grrr, maybe it's about time I figured out and implemented a good
strategy for hostile takeovers.

> nevermind :) I guess it's just you playing with the security

Yea, a couple weeks ago when I was poking around at the source I
noticed a potential vulnerability given carefully crafted usernames,
but as it turns out the account creation does plenty of input
sanitizing so it was a non-starter.

[Abstrct]

Yea, a couple weeks ago when I was poking around at the source I
noticed a potential vulnerability given carefully crafted usernames,
but as it turns out the account creation does plenty of input
sanitizing so it was a non-starter.

I noticed some of your attempts on the leader board before the latest rounds stats kicked in ;) Glad to hear that was locked down though. It is a pretty high risk spot due to all the user input and high level credentials being used.  

[MassConfusion]

Sorry Derp :) I got lucky on this one.
The script I put together is pretty robust for combat now, but taking
over a planet with a bunch of defenders is really a drain on
resources, not to mention that this situation gets priority for my
fleet, so if I don't get lucky and take you down early you manage to
run off and capture twice as many planets while I'm busy securing just
a few.
I'm still not sure how you move so quick! it may just be the sheer
amount of stuff my script is trying to accomplish without timing out,
but up until the recent speed improvements I just haven't been able to
keep up.
I've even gone so far as to start working up a different kind of
script that may be able to keep up, but naturally it died by my own
hand long before I got far, and besides the tic is down again this
morning XD maybe I'll have better luck next round.

[Abstrct]

The TIC is actually running fine, it looks like it is something wrong with ref.pl instead. 

For some reason fleet script 142 (which belongs to you MC) is getting exclusive use of some tables that ref.pl needs to use which is preventing it (ref.pl) from killing the long running script 142 :P

Bit of a catch-22. I may need to move some data around to prevent this but Im not exactly sure what yet....

[MassConfusion]

heh.. oops

I'm totally going to get labeled a trouble-maker now..

I turned off 142, if you can let me know what it was caught on I'll
write around it in the script *facepalm*

[Abstrct]

Feel free to leave it on, it may help me diagnose the problem. I think the issue is to do with the fact that it is trying to update the fleet table, does script 142 do any updates there?

Damn trouble maker D:

[MassConfusion]

Update the fleet table? no not anymore, there are several EXECUTE
calls to call the other bits of the script and one "SELECT
EXTRACT(MINUTE from runtime)::integer from my_fleets WHERE id =
this_fleet_id" to get the current runtime. The only section that
updates anything to do with fleets is increasing the runtime, but
since it's already reached its maximum value it shouldn't be doing
that either.

could this be causes from two fleet scripts running concurrently?
becuase I did have 222 enabled as well

[Abstrct]

Just a quick little update on this request. You will now see two new actions showing up in your my_events view: FLEET_SUCCESS and FLEET_FAIL

SELECT * FROM my_events WHERE action IN ('FLEET_SUCCESS','FLEET_FAIL');

FLEET_SUCCESS will include how long the script took to ran as the descriptor_string column (the actual value is an interval but it will have to be cast back into this from text)

This will NOT start showing up until you update your fleet scripts. The timer and event creation code will be injected into your fleet script functions automatically from now on.

FLEET_FAIL will include the error message that occurred on failure in the descriptor_string column. There is nothing you need to do to enable this, it has already begun logging error messages for you.

As with all the events, you can write a script to read these programmatically, or you can format them nicely with read_event(id)

I know the event table can get a bit slow, but I also did some improvements to speed it up so if give it a way of narrowing down your results, you will (almost) definitely get your answer back within the time query time limit.

Let me know what you think. If you think something will improve the new events just say the word. 

-Abstrct

[Christopher Browne]

On Thursday, 29 March 2012 12:00:07 UTC-4, Abstrct wrote:

Just a quick little update on this request. You will now see two new actions showing up in your my_events view: FLEET_SUCCESS and FLEET_FAIL

SELECT * FROM my_events WHERE action IN ('FLEET_SUCCESS','FLEET_FAIL');

FLEET_SUCCESS will include how long the script took to ran as the descriptor_string column (the actual value is an interval but it will have to be cast back into this from text)

This will NOT start showing up until you update your fleet scripts. The timer and event creation code will be injected into your fleet script functions automatically from now on.

FLEET_FAIL will include the error message that occurred on failure in the descriptor_string column. There is nothing you need to do to enable this, it has already begun logging error messages for you.

As with all the events, you can write a script to read these programmatically, or you can format them nicely with read_event(id)

I know the event table can get a bit slow, but I also did some improvements to speed it up so if give it a way of narrowing down your results, you will (almost) definitely get your answer back within the time query time limit.

Pretty awesome for debugging.

 schem...@db.schemaverse.com->  select action, descriptor_string from my_events where player_id_1 = XXXX and action like 'FLEET_FAI%' order by id desc limit 5;
             action             |                                                     descriptor_string
--------------------------------+---------------------------------------------------------------------------------------------------------------------------
 FLEET_FAIL                     | DBD::Pg::db do failed: ERROR:  function convert_resource(unknown, bigint) does not exist                                 +
                                | LINE 1: SELECT convert_resource('FUEL', (select count(*) * (select c...                                                  +
                                |                ^                                                                                                         +
                                | HINT:  No function matches the given name and argument types. You might need to add explicit type casts.                 +
                                | QUERY:  SELECT convert_resource('FUEL', (select count(*) * (select cost from price_list where code = 'PROSPECTING') * 5))+
                                | CONTEXT:  PL/pgSQL function "fleet_script_XXX" line 267 at PERFORM at tic.pl line 79.                                    +
                                |
 FLEET_FAIL                     | DBD::Pg::db do failed: ERROR:  function convert_resource(unknown, bigint) does not exist                                 +
                                | LINE 1: SELECT convert_resource('FUEL', (select count(*) * (select c...                                                  +
                                |                ^                                                                                                         +
                                | HINT:  No function matches the given name and argument types. You might need to add explicit type casts.                 +
                                | QUERY:  SELECT convert_resource('FUEL', (select count(*) * (select cost from price_list where code = 'PROSPECTING') * 5))+
                                | CONTEXT:  PL/pgSQL function "fleet_script_XXX" line 267 at PERFORM at tic.pl line 79.                                    +
                                |
 FLEET_FAIL                     | DBD::Pg::db do failed: ERROR:  function convert_resource(unknown, bigint) does not exist                                 +
                                | LINE 1: SELECT convert_resource('FUEL', (select count(*) * (select c...                                                  +
                                |                ^                                                                                                         +
                                | HINT:  No function matches the given name and argument types. You might need to add explicit type casts.                 +
                                | QUERY:  SELECT convert_resource('FUEL', (select count(*) * (select cost from price_list where code = 'PROSPECTING') * 5))+
                                | CONTEXT:  PL/pgSQL function "fleet_script_XXX" line 267 at PERFORM at tic.pl line 79.                                    +
                                |
(3 rows)

[MassConfusion]

awesome job Ab!

Is it just me or are things running a bit slower today? Yesterday the
script was running with 80 planets or so in just over 30 seconds, this
afternoon it's taking closer to 7 minutes for some reason with just a
few extra planets.

It did inspire me, however, to add a timer to see how long each
section was running, and then add a bypass so that if it comes within
15 seconds of timing out it'll stop what it's doing and end the script
*cackle* problem solved.