A subnet mask refers to the practice of subnetting, which is, from my understanding of it, a way of dividing one network into smaller logical networks. A subnet mask is used to tell the network devices (whether it be the computer or any routers, modems, etc.) what addresses are local (belongs to this network), and what addresses are remote (belongs to that network).
Subnetting a network is not really useful in a home network, such as yours. It is mainly used on larger networks (with 255+ computers) to reduce unnecessary network activity. MarkM provided this in the comments:
One of the biggest advantages of subnetting in a large network is to reduce the broadcast traffic. If you have thousands of hosts on a single subnet, your switches will be choking on ARP, DHCP, and other broadcasts. In a home network usually there isn't much of a need unless you want something like a guest wireless LAN that doesn't have any routing available to your main home network.
In more technical terms, the subnet mask helps machines identify the network number associated with a given IP address. The 255 represents all 1's for that byte of the address and when you perform a bit-wise AND between the address and the mask, you get the network number back. What's left is the machine number
This second configuration leaves 9 bits for the host number (in an instance where you would have more than 255 hosts). Adjusting the subnet mask gives you more host addresses and less networks or vice versa depending on whether you add or remove 1's
Since torbengb's ISP gave him a router (or told him to configure his router) with the address 10.0.0.138, it leads me to suspect that this ISPs customers share the 10.0.0.0 address space; otherwise most poeple would configure the router at 10.0.0.1 or 192.168.0.1 or similar.
The netmask DOES separate your addresses into the network portion and the host portion, but the practical reason is so your router, and the routing code in your hosts, can know which destination addresses are on your local network and which need to have their packets sent through the router's uplink.
Since you are probably not using a "real" DCHP server, you should use the subnet mask given out by you DHCP server for all systems, including the static addresses. You should also make sure your static numbers are not going to be in the space that your DHCP server allocates out of.
c80f0f1006