On Tuesday, August 7, 2012 9:49:48 AM UTC-4, Ben Dyer wrote:
> Hi Ian,
> Sure. Sorry about the delay on the pull request, I've now given up trying > to get qemu running properly on OS X 10.8 and am setting it up in a Windows > VM instead.
> Regards,
> Ben
> On 07/08/2012, at 23:40 , Ian Seyler <ise...@gmail.com <javascript:>> > wrote:
> I have updated the BMFS spec based on your suggestion about removing the > free block bitmap. Very clever idea! Is it ok if I include your name the > doc?
> Still working on the callback layout...
> -Ian
> On Tuesday, July 24, 2012 7:37:34 AM UTC-4, Ben Dyer wrote:
>> On 24/07/2012, at 10:11 , Ian Seyler wrote:
>> > If you want to merge the AHCI and BMFS code into BMOS as well that >> would be great!
>> Will do.
>> > Callbacks should be used for disk access as well. I have removed the >> ring buffer support in the 'networkcallback' branch of BareMetal OS. I have >> not implemented the network callback yet as I'm still figuring out the best >> way to implement it.
>> Fair enough — I was going to try to copy the approach used by the network >> interface code as far as possible :)
>> I guess the main question is how I/O scheduling and concurrency will fit >> into the os_smp_* concurrency procedures; in the short term, I'll set >> everything up as blocking, and we can split it up as required once >> scheduling/concurrency is pinned down.
Keep in mind that everything runs in ring 0 so there is no context switching like in other operating systems.
The problem with the process queue is that there is no way to pre-empt something. If all cores are busy then the new job in the queue will not execute.
I have completed some stack fudging features in the past for another project (Calling an OS function from an interrupt). Looking into that.
On Wednesday, August 8, 2012 12:09:57 AM UTC-4, Ben Dyer wrote:
> Hi Ian,
> On 07/08/2012, at 23:40 , Ian Seyler <ise...@gmail.com <javascript:>> > wrote:
> > Still working on the callback layout...
> Are you planning on running the callbacks from I/O interrupts, or will > they be run at user level like the process itself?
> I've been giving the issue some thought and if it's the latter, it seems > to me that process suspend/resume around blocking I/O operations is > probably going to be the easiest to work with. Relatively few changes to > the existing process queue would be required, and it wouldn't complicate > the concurrency model by introducing new concepts.
On 09/08/2012, at 00:17 , Ian Seyler <isey...@gmail.com> wrote:
> The problem with the process queue is that there is no way to pre-empt something. If all cores are busy then the new job in the queue will not execute.
If the expected time required for a process to execute is small enough, that won't be an issue, but effectively that's just co-operative multitasking across the process set. For the jobs I have in mind, that's probably not a problem, but it'd make coding for HPC applications or long-running tasks unnecessarily complex.
> I have completed some stack fudging features in the past for another project (Calling an OS function from an interrupt). Looking into that.
So are you thinking of adding some sort of pre-emptively scheduled I/O tasks to the concurrency model, or would the user-developed/application code be expected to register I/O handlers that would be called from the ISRs?
I found my stack fudge code. My reasoning for going with this was to not have a crash within the interrupt call.
Current setup: Application is running Network interrupt is triggered, executes, and finishes Application execution is returned to where it left off
Putting the callback within the interrupt handler will cause issues if the callback is faulty. For instance if the callback never finishes.
Future setup: Application installs network callback Application is running Network interrupt is triggered, executes, adjusts the stack, and finishes Application callback is executed, and finishes Application execution is returned to where it left off
On Wednesday, August 8, 2012 10:17:43 AM UTC-4, Ian Seyler wrote:
> Keep in mind that everything runs in ring 0 so there is no context > switching like in other operating systems.
> The problem with the process queue is that there is no way to pre-empt > something. If all cores are busy then the new job in the queue will not > execute.
> I have completed some stack fudging features in the past for another > project (Calling an OS function from an interrupt). Looking into that.
> -Ian
> On Wednesday, August 8, 2012 12:09:57 AM UTC-4, Ben Dyer wrote:
>> Hi Ian,
>> On 07/08/2012, at 23:40 , Ian Seyler <ise...@gmail.com> wrote:
>> > Still working on the callback layout...
>> Are you planning on running the callbacks from I/O interrupts, or will >> they be run at user level like the process itself?
>> I've been giving the issue some thought and if it's the latter, it seems >> to me that process suspend/resume around blocking I/O operations is >> probably going to be the easiest to work with. Relatively few changes to >> the existing process queue would be required, and it wouldn't complicate >> the concurrency model by introducing new concepts.
On 09/08/2012, at 01:29 , Ian Seyler <isey...@gmail.com> wrote:
> Future setup:
> Application installs network callback
> Application is running
> Network interrupt is triggered, executes, adjusts the stack, and finishes
> Application callback is executed, and finishes
> Application execution is returned to where it left off
Ah, that makes sense. So although the network interrupts are initially serviced by the OS, the application is ultimately responsible for handling everything.
Are callbacks going to be for raw packets, so applications link with/include the IP stack themselves, or will the callbacks be defined on higher-level events like TCP connect, retransmit etc?
Yes, but it depends on the type of packet that is received (or perhaps how the callback is installed, example: promiscuous mode).
The IP stack will live in the network interrupt handler. I would not want the application dealing with re-transmits, etc. For instance the network handler would deal with ARP and PING replies directly and just pass the data on to the application.
Since we don't have IP yet the network handler will just pass all received packets to the callback handler.
On Wednesday, August 8, 2012 11:45:10 AM UTC-4, Ben Dyer wrote:
> On 09/08/2012, at 01:29 , Ian Seyler <ise...@gmail.com <javascript:>> > wrote:
> > Future setup: > > Application installs network callback > > Application is running > > Network interrupt is triggered, executes, adjusts the stack, and > finishes > > Application callback is executed, and finishes > > Application execution is returned to where it left off
> Ah, that makes sense. So although the network interrupts are initially > serviced by the OS, the application is ultimately responsible for handling > everything.
> Are callbacks going to be for raw packets, so applications link > with/include the IP stack themselves, or will the callbacks be defined on > higher-level events like TCP connect, retransmit etc?
Sounds great — I've just been looking at some other IP stacks (uIP and lwIP) to get a sense of the various programming models that can be used, and the approach you're describing sounds like it'll have a good balance of architectural flexibility and power/performance.
On 09/08/2012, at 04:21 , Ian Seyler <isey...@gmail.com> wrote:
> Yes, but it depends on the type of packet that is received (or perhaps how the callback is installed, example: promiscuous mode).
> The IP stack will live in the network interrupt handler. I would not want the application dealing with re-transmits, etc. For instance the network handler would deal with ARP and PING replies directly and just pass the data on to the application.
> Since we don't have IP yet the network handler will just pass all received packets to the callback handler.
From: "Ben Dyer" <ben_d...@mac.com> To: baremetal-os@googlegroups.com Sent: Thursday, 9 August, 2012 8:12:33 AM Subject: Re: Pure64 filesystem support
Sounds great — I've just been looking at some other IP stacks (uIP and lwIP) to get a sense of the various programming models that can be used, and the approach you're describing sounds like it'll have a good balance of architectural flexibility and power/performance.
On 09/08/2012, at 04:21 , Ian Seyler <isey...@gmail.com> wrote:
> Yes, but it depends on the type of packet that is received (or perhaps how the callback is installed, example: promiscuous mode).
> The IP stack will live in the network interrupt handler. I would not want the application dealing with re-transmits, etc. For instance the network handler would deal with ARP and PING replies directly and just pass the data on to the application.
> Since we don't have IP yet the network handler will just pass all received packets to the callback handler.
The coder I hired to build the stack dropped out of the project halfway through and only implemented ARP, ICMP, and IP. TCP and UDP were not completed. I ended up removing his code from GitHub since it wasn't complete. You can find it in an older revision: GitHub<https://github.com/ReturnInfinity/BareMetal-OS/commit/1164e6dbe8aa911...>
As for current progress I may have more to report next week. I was approached by someone about a month ago about writing a stack for BareMetal OS. He has experience with it before and I was informed yesterday that he was a few days from completion. Once I know more I will announce it.
> Sounds great — I've just been looking at some other IP stacks (uIP and > lwIP) to get a sense of the various programming models that can be used, > and the approach you're describing sounds like it'll have a good balance of > architectural flexibility and power/performance.
> On 09/08/2012, at 04:21 , Ian Seyler <ise...@gmail.com <javascript:>> > wrote:
> > Yes, but it depends on the type of packet that is received (or perhaps > how the callback is installed, example: promiscuous mode).
> > The IP stack will live in the network interrupt handler. I would not > want the application dealing with re-transmits, etc. For instance the > network handler would deal with ARP and PING replies directly and just pass > the data on to the application.
> > Since we don't have IP yet the network handler will just pass all > received packets to the callback handler.
From: "Ian Seyler" <isey...@gmail.com> To: baremetal-os@googlegroups.com Sent: Thursday, 9 August, 2012 4:35:52 PM Subject: Re: Pure64 filesystem support
Hi Adrien,
The coder I hired to build the stack dropped out of the project halfway through and only implemented ARP, ICMP, and IP. TCP and UDP were not completed. I ended up removing his code from GitHub since it wasn't complete. You can find it in an older revision: GitHub
As for current progress I may have more to report next week. I was approached by someone about a month ago about writing a stack for BareMetal OS. He has experience with it before and I was informed yesterday that he was a few days from completion. Once I know more I will announce it.
-Ian
On Thursday, August 9, 2012 5:36:24 AM UTC-4, Adrien wrote:
Ian,
talking about IP stack, you know I would love to work on a HTTP Server for BMOS.
I still can't find the (initial) IP stack in the sources on Github. Are there yet?
Adrien
From: "Ben Dyer" < ben_...@mac.com > To: bareme...@googlegroups.com Sent: Thursday, 9 August, 2012 8:12:33 AM Subject: Re: Pure64 filesystem support
Sounds great — I've just been looking at some other IP stacks (uIP and lwIP) to get a sense of the various programming models that can be used, and the approach you're describing sounds like it'll have a good balance of architectural flexibility and power/performance.
On 09/08/2012, at 04:21 , Ian Seyler < ise...@gmail.com > wrote:
> Yes, but it depends on the type of packet that is received (or perhaps how the callback is installed, example: promiscuous mode).
> The IP stack will live in the network interrupt handler. I would not want the application dealing with re-transmits, etc. For instance the network handler would deal with ARP and PING replies directly and just pass the data on to the application.
> Since we don't have IP yet the network handler will just pass all received packets to the callback handler.
I'm interested in the same thing — I've been planning something similar to Python's Tornado framework, but with request handlers written in C and able to be deployed/updated on the fly, without restarting or interrupting requests.
The rationale behind building such a thing on BMOS, rather than a general-purpose OS, is that I think crash-only approaches [1] are going to be the only way to ensure high availability in increasingly complex distributed systems. I've had a lot of success with applications designed around this approach running under Linux, however after 9 months in production the operating system and related services are now the most common failure mode — obscure log files consuming all disk space, and various other misconfigurations that only crop up after considerable uptime. By using something which I understand from the ground up, I can ensure that there are no edge cases or components that behave unexpectedly, and can design everything to be terminated at any time and recover gracefully.
In addition, by stripping away unnecessary dependencies and complexity, it would be possible to do automated deployment and on-line software upgrades directly from version control, without the need for multiple layers of package management and automation solutions (for an application using the Tornado framework, an automated deploy solution would involve pip/easy_install, yum/apt-get, plus puppet/chef).
Anyway, if you're interested we should compare notes once the IP stack is in place, and see if our HTTP server needs are in alignment.
On Thursday, August 9, 2012 2:12:33 AM UTC-4, Ben Dyer wrote:
> Sounds great — I've just been looking at some other IP stacks (uIP and > lwIP) to get a sense of the various programming models that can be used, > and the approach you're describing sounds like it'll have a good balance of > architectural flexibility and power/performance.
> On 09/08/2012, at 04:21 , Ian Seyler <ise...@gmail.com <javascript:>> > wrote:
> > Yes, but it depends on the type of packet that is received (or perhaps > how the callback is installed, example: promiscuous mode).
> > The IP stack will live in the network interrupt handler. I would not > want the application dealing with re-transmits, etc. For instance the > network handler would deal with ARP and PING replies directly and just pass > the data on to the application.
> > Since we don't have IP yet the network handler will just pass all > received packets to the callback handler.
I follow this wonderful project for a very similar reason. If you understand what you are doing from end to end, you have a chance to succeed, doomed otherwise.
What I like with BareMetal OS is that, even with a low level of x64 understanding (I am more a RISC person) I can write a program (using C) without complex limitations. For instance, I abandoned a project of web server well on its way just because you cannot transfer an open connection from a thread to another. You just cannot give the descriptor to something else. So I couldn't do what I wanted, i.e. on a N core machine, have 1 listener put requests in a local queue and N-1 workers extract from the queue, process and answer. Come on! That is the only design you can think of to maximise the throughput of a server whatever you are serving.
The only problem I have really with BareMetal OS is that, not knowing ASM, I need to use C and C's memory management is just disastrous for software quality. I need to remove rust on this.
It is not a surprise I "meet" you on this mailing list. Where else? Not on the "Bloat OS" mailing-list.
Ian,
you guest it. We are two guys waiting on this stack to be ready to work from it.
From: "Ben Dyer" <ben_d...@mac.com> To: baremetal-os@googlegroups.com Sent: Thursday, 9 August, 2012 4:43:43 PM Subject: Re: Pure64 filesystem support
Adrien,
I'm interested in the same thing — I've been planning something similar to Python's Tornado framework, but with request handlers written in C and able to be deployed/updated on the fly, without restarting or interrupting requests.
The rationale behind building such a thing on BMOS, rather than a general-purpose OS, is that I think crash-only approaches [1] are going to be the only way to ensure high availability in increasingly complex distributed systems. I've had a lot of success with applications designed around this approach running under Linux, however after 9 months in production the operating system and related services are now the most common failure mode — obscure log files consuming all disk space, and various other misconfigurations that only crop up after considerable uptime. By using something which I understand from the ground up, I can ensure that there are no edge cases or components that behave unexpectedly, and can design everything to be terminated at any time and recover gracefully.
In addition, by stripping away unnecessary dependencies and complexity, it would be possible to do automated deployment and on-line software upgrades directly from version control, without the need for multiple layers of package management and automation solutions (for an application using the Tornado framework, an automated deploy solution would involve pip/easy_install, yum/apt-get, plus puppet/chef).
Anyway, if you're interested we should compare notes once the IP stack is in place, and see if our HTTP server needs are in alignment.
In 2012, network is a core service, there should be interrupts for it. It reminds me of a version of the Z80 in the early '00s called eZ80 where four 8 bits registers for IP addresses were added.
From: "Ian Seyler" <isey...@gmail.com> To: baremetal-os@googlegroups.com Sent: Thursday, 9 August, 2012 4:52:14 PM Subject: Re: Pure64 filesystem support
In regards to uIP I have had some success with porting version 0.9 to BareMetal OS (It was responding to pings).
I had issues compiling the more advanced apps like the HTTP server and decided that it would be best to have the stack in the OS instead of the app.
-Ian
On Thursday, August 9, 2012 2:12:33 AM UTC-4, Ben Dyer wrote:
Sounds great — I've just been looking at some other IP stacks (uIP and lwIP) to get a sense of the various programming models that can be used, and the approach you're describing sounds like it'll have a good balance of architectural flexibility and power/performance.
On 09/08/2012, at 04:21 , Ian Seyler < ise...@gmail.com > wrote:
> Yes, but it depends on the type of packet that is received (or perhaps how the callback is installed, example: promiscuous mode).
> The IP stack will live in the network interrupt handler. I would not want the application dealing with re-transmits, etc. For instance the network handler would deal with ARP and PING replies directly and just pass the data on to the application.
> Since we don't have IP yet the network handler will just pass all received packets to the callback handler.
On the topic of memory management, I found JPL's C standards [1] interesting.
They strictly prohibit dynamic memory allocation except at application startup, which of course gets rid of a huge number of potential errors. The high-performance web server nginx also uses a similar approach in that it permits only a certain number of buffers of a fixed size to be used by a request or response before it writes the data to a temporary file.
Another interesting restriction was "only one unbounded loop per process", i.e. you are permitted to use a main event loop, but all others must have a statically-verifiable upper bound. This prevents you from getting stuck in an infinite loop, similar to the crash-only principle of having timeouts for everything. Very easy to enforce when you have complete transparency in the OS, very difficult otherwise.
> I follow this wonderful project for a very similar reason. If you understand what you are doing from end to end, you have a chance to succeed, doomed otherwise.
> What I like with BareMetal OS is that, even with a low level of x64 understanding (I am more a RISC person) I can write a program (using C) without complex limitations. For instance, I abandoned a project of web server well on its way just because you cannot transfer an open connection from a thread to another. You just cannot give the descriptor to something else. So I couldn't do what I wanted, i.e. on a N core machine, have 1 listener put requests in a local queue and N-1 workers extract from the queue, process and answer. Come on! That is the only design you can think of to maximise the throughput of a server whatever you are serving.
> The only problem I have really with BareMetal OS is that, not knowing ASM, I need to use C and C's memory management is just disastrous for software quality. I need to remove rust on this.
> It is not a surprise I "meet" you on this mailing list. Where else? Not on the "Bloat OS" mailing-list.
> Ian,
> you guest it. We are two guys waiting on this stack to be ready to work from it.
> I'm interested in the same thing — I've been planning something similar to Python's Tornado framework, but with request handlers written in C and able to be deployed/updated on the fly, without restarting or interrupting requests.
> The rationale behind building such a thing on BMOS, rather than a general-purpose OS, is that I think crash-only approaches [1] are going to be the only way to ensure high availability in increasingly complex distributed systems. I've had a lot of success with applications designed around this approach running under Linux, however after 9 months in production the operating system and related services are now the most common failure mode — obscure log files consuming all disk space, and various other misconfigurations that only crop up after considerable uptime. By using something which I understand from the ground up, I can ensure that there are no edge cases or components that behave unexpectedly, and can design everything to be terminated at any time and recover gracefully.
> In addition, by stripping away unnecessary dependencies and complexity, it would be possible to do automated deployment and on-line software upgrades directly from version control, without the need for multiple layers of package management and automation solutions (for an application using the Tornado framework, an automated deploy solution would involve pip/easy_install, yum/apt-get, plus puppet/chef).
> Anyway, if you're interested we should compare notes once the IP stack is in place, and see if our HTTP server needs are in alignment.
ok but this only works in a preemptive OS or in an environment where the compiler is patched. BareMetal OS is not the first and I think should not have the latter.
But keeping your point, you suggest to allocate memory in block at start and use it up to the point where it is freed automatically at the end.
That is remove malloc, realloc and free and do this instead :
void main() { char memory[1024 * 1024 *1024]; char *memory_pointer; /*use 1G of memory as you like*/ ...
}
Is that what you suggest? Why not, I like the idea not to deal with malloc, but it seems that malloc and free, as bad as they can be were invented for some case where this is not available. A virtue I can see is that you know before hand if you are going to fail for a memory reason.
You could set this as a good practice but how could you set this as a development rule under BareMetal OS?
From: "Ben Dyer" <ben_d...@mac.com> To: baremetal-os@googlegroups.com Sent: Thursday, 9 August, 2012 5:28:44 PM Subject: Re: Pure64 filesystem support
On the topic of memory management, I found JPL's C standards [1] interesting.
They strictly prohibit dynamic memory allocation except at application startup, which of course gets rid of a huge number of potential errors. The high-performance web server nginx also uses a similar approach in that it permits only a certain number of buffers of a fixed size to be used by a request or response before it writes the data to a temporary file.
Another interesting restriction was "only one unbounded loop per process", i.e. you are permitted to use a main event loop, but all others must have a statically-verifiable upper bound. This prevents you from getting stuck in an infinite loop, similar to the crash-only principle of having timeouts for everything. Very easy to enforce when you have complete transparency in the OS, very difficult otherwise.
On 10/08/2012, at 01:04 , Adrien Arculeo <aarcu...@1024degres.com> wrote:
> Ben,
> I follow this wonderful project for a very similar reason. If you understand what you are doing from end to end, you have a chance to succeed, doomed otherwise.
> What I like with BareMetal OS is that, even with a low level of x64 understanding (I am more a RISC person) I can write a program (using C) without complex limitations. For instance, I abandoned a project of web server well on its way just because you cannot transfer an open connection from a thread to another. You just cannot give the descriptor to something else. So I couldn't do what I wanted, i.e. on a N core machine, have 1 listener put requests in a local queue and N-1 workers extract from the queue, process and answer. Come on! That is the only design you can think of to maximise the throughput of a server whatever you are serving.
> The only problem I have really with BareMetal OS is that, not knowing ASM, I need to use C and C's memory management is just disastrous for software quality. I need to remove rust on this.
> It is not a surprise I "meet" you on this mailing list. Where else? Not on the "Bloat OS" mailing-list.
> Ian,
> you guest it. We are two guys waiting on this stack to be ready to work from it.
> I'm interested in the same thing — I've been planning something similar to Python's Tornado framework, but with request handlers written in C and able to be deployed/updated on the fly, without restarting or interrupting requests.
> The rationale behind building such a thing on BMOS, rather than a general-purpose OS, is that I think crash-only approaches [1] are going to be the only way to ensure high availability in increasingly complex distributed systems. I've had a lot of success with applications designed around this approach running under Linux, however after 9 months in production the operating system and related services are now the most common failure mode — obscure log files consuming all disk space, and various other misconfigurations that only crop up after considerable uptime. By using something which I understand from the ground up, I can ensure that there are no edge cases or components that behave unexpectedly, and can design everything to be terminated at any time and recover gracefully.
> In addition, by stripping away unnecessary dependencies and complexity, it would be possible to do automated deployment and on-line software upgrades directly from version control, without the need for multiple layers of package management and automation solutions (for an application using the Tornado framework, an automated deploy solution would involve pip/easy_install, yum/apt-get, plus puppet/chef).
> Anyway, if you're interested we should compare notes once the IP stack is in place, and see if our HTTP server needs are in alignment.
On 10/08/2012, at 04:34 , Adrien Arculeo <aarcu...@1024degres.com> wrote:
> Is that what you suggest? Why not, I like the idea not to deal with malloc, but it seems that malloc and free, as bad as they can be were invented for some case where this is not available. A virtue I can see is that you know before hand if you are going to fail for a memory reason.
Not exactly — you'd use a bunch of statically-allocated buffers for various things, rather than just one huge one. And this would be on a per-request basis, so it'd mean you'd know exactly how many concurrent requests you could deal with. It'd also mitigate the impact of a whole class of DoS attacks since you're forced to set sane upper limits on the sizes of requests, headers, and bodies.
As far as security is concerned, fixed-size buffers are only a problem when using the C string library, which should be avoided anyway because it's too difficult to use safely.
And yes, it's a bit more difficult to write code like that but I suspect if you look at it in terms of [time taken to write code without malloc] vs [time taken to write code with malloc + time taken to debug memory leaks + time taken to deal with memory being accessed after being freed + time taken to deal with out-of-memory conditions + time taken to consider the impact of heap fragmentation] it's probably quicker to spend more time writing the initial code, and much less time debugging.
> You could set this as a good practice but how could you set this as a development rule under BareMetal OS?
I don't think avoidance of dynamic memory allocation makes sense as a general development rule, but for my specific interest — high-concurrency, fault-tolerant network services — it seems to be a common approach and I think would be worth enforcing within individual request handlers. I'd also like to use memory protection on a per-request basis to ensure they're completely isolated from each other.
What you suggest is rather more "think of the memory you need beforehand for each variable, allocate them statically and work with that constraint as if you were programming for an embedded system in which the memory limit is very hard."
I started with embedded systems so I feel comfortable about that and yes, in a server, you need to know some fact like "n concurrent user per amount of RAM per user < Max memory of the server".
I am going to read the crash only software document this weekend.
From: "Ben Dyer" <ben_d...@mac.com> To: baremetal-os@googlegroups.com Sent: Friday, 10 August, 2012 5:01:31 AM Subject: Re: Pure64 filesystem support
On 10/08/2012, at 04:34 , Adrien Arculeo <aarcu...@1024degres.com> wrote:
> Is that what you suggest? Why not, I like the idea not to deal with malloc, but it seems that malloc and free, as bad as they can be were invented for some case where this is not available. A virtue I can see is that you know before hand if you are going to fail for a memory reason.
Not exactly — you'd use a bunch of statically-allocated buffers for various things, rather than just one huge one. And this would be on a per-request basis, so it'd mean you'd know exactly how many concurrent requests you could deal with. It'd also mitigate the impact of a whole class of DoS attacks since you're forced to set sane upper limits on the sizes of requests, headers, and bodies.
As far as security is concerned, fixed-size buffers are only a problem when using the C string library, which should be avoided anyway because it's too difficult to use safely.
And yes, it's a bit more difficult to write code like that but I suspect if you look at it in terms of [time taken to write code without malloc] vs [time taken to write code with malloc + time taken to debug memory leaks + time taken to deal with memory being accessed after being freed + time taken to deal with out-of-memory conditions + time taken to consider the impact of heap fragmentation] it's probably quicker to spend more time writing the initial code, and much less time debugging.
> You could set this as a good practice but how could you set this as a development rule under BareMetal OS?
I don't think avoidance of dynamic memory allocation makes sense as a general development rule, but for my specific interest — high-concurrency, fault-tolerant network services — it seems to be a common approach and I think would be worth enforcing within individual request handlers. I'd also like to use memory protection on a per-request basis to ensure they're completely isolated from each other.
Thanks for sharing this document! My thoughts were similar in regards to memory allocation at application startup. This also leads to a much more simplified memory manager at the OS level.
On Thursday, August 9, 2012 11:28:44 AM UTC-4, Ben Dyer wrote:
> On the topic of memory management, I found JPL's C standards [1] > interesting.
> They strictly prohibit dynamic memory allocation except at application > startup, which of course gets rid of a huge number of potential errors. The > high-performance web server nginx also uses a similar approach in that it > permits only a certain number of buffers of a fixed size to be used by a > request or response before it writes the data to a temporary file.
> Another interesting restriction was "only one unbounded loop per process", > i.e. you are permitted to use a main event loop, but all others must have a > statically-verifiable upper bound. This prevents you from getting stuck in > an infinite loop, similar to the crash-only principle of having timeouts > for everything. Very easy to enforce when you have complete transparency in > the OS, very difficult otherwise.
> On 10/08/2012, at 01:04 , Adrien Arculeo <aarc...@1024degres.com<javascript:>> > wrote:
> > Ben,
> > I follow this wonderful project for a very similar reason. If you > understand what you are doing from end to end, you have a chance to > succeed, doomed otherwise.
> > What I like with BareMetal OS is that, even with a low level of x64 > understanding (I am more a RISC person) I can write a program (using C) > without complex limitations. For instance, I abandoned a project of web > server well on its way just because you cannot transfer an open connection > from a thread to another. You just cannot give the descriptor to something > else. So I couldn't do what I wanted, i.e. on a N core machine, have 1 > listener put requests in a local queue and N-1 workers extract from the > queue, process and answer. Come on! That is the only design you can think > of to maximise the throughput of a server whatever you are serving.
> > The only problem I have really with BareMetal OS is that, not knowing > ASM, I need to use C and C's memory management is just disastrous for > software quality. I need to remove rust on this.
> > It is not a surprise I "meet" you on this mailing list. Where else? Not > on the "Bloat OS" mailing-list.
> > Ian,
> > you guest it. We are two guys waiting on this stack to be ready to work > from it.
> > I'm interested in the same thing — I've been planning something similar > to Python's Tornado framework, but with request handlers written in C and > able to be deployed/updated on the fly, without restarting or interrupting > requests.
> > The rationale behind building such a thing on BMOS, rather than a > general-purpose OS, is that I think crash-only approaches [1] are going to > be the only way to ensure high availability in increasingly complex > distributed systems. I've had a lot of success with applications designed > around this approach running under Linux, however after 9 months in > production the operating system and related services are now the most > common failure mode — obscure log files consuming all disk space, and > various other misconfigurations that only crop up after considerable > uptime. By using something which I understand from the ground up, I can > ensure that there are no edge cases or components that behave unexpectedly, > and can design everything to be terminated at any time and recover > gracefully.
> > In addition, by stripping away unnecessary dependencies and complexity, > it would be possible to do automated deployment and on-line software > upgrades directly from version control, without the need for multiple > layers of package management and automation solutions (for an application > using the Tornado framework, an automated deploy solution would involve > pip/easy_install, yum/apt-get, plus puppet/chef).
> > Anyway, if you're interested we should compare notes once the IP stack > is in place, and see if our HTTP server needs are in alignment.
> The coder I hired to build the stack dropped out of the project halfway > through and only implemented ARP, ICMP, and IP. TCP and UDP were not > completed. I ended up removing his code from GitHub since it wasn't > complete. You can find it in an older revision: GitHub<https://github.com/ReturnInfinity/BareMetal-OS/commit/1164e6dbe8aa911...>
> As for current progress I may have more to report next week. I was > approached by someone about a month ago about writing a stack for BareMetal > OS. He has experience with it before and I was informed yesterday that he > was a few days from completion. Once I know more I will announce it.
On Friday, November 2, 2012 6:31:45 AM UTC-4, Daniel Cegiełka wrote:
> Hi,
> Are there any news on the implementation of the IP stack in BMOS?
> Best regards,
> Daniel
> W dniu czwartek, 9 sierpnia 2012 16:35:52 UTC+2 użytkownik Ian Seyler > napisał:
>> Hi Adrien,
>> The coder I hired to build the stack dropped out of the project halfway >> through and only implemented ARP, ICMP, and IP. TCP and UDP were not >> completed. I ended up removing his code from GitHub since it wasn't >> complete. You can find it in an older revision: GitHub<https://github.com/ReturnInfinity/BareMetal-OS/commit/1164e6dbe8aa911...>
>> As for current progress I may have more to report next week. I was >> approached by someone about a month ago about writing a stack for BareMetal >> OS. He has experience with it before and I was informed yesterday that he >> was a few days from completion. Once I know more I will announce it.
On Friday, February 17, 2012 4:40:21 PM UTC+1, Ian Seyler wrote:
> Here are the current specs for the new file system. These are the > initial specs so work is still being done. Once it is complete the > coding for the driver will be started.
It is nice to see the current file system is flat. Hierarchical file system really sucks. What users need is a tagged file system, with fast indexing and searching. I wonder what is the evolution direction of BMFS. Thanks.
