MD5 Magics Hashes
Avery Blaschko
TL;DR: Magic hashes are well known specific hashes used to exploit Type Juggling attacks in PHP. Combined with bcrypt limitations, we propose the concept of Super Magic Hashes. These hashes can detect 3 different vulnerabilities: type juggling, weak password storage and incorrect Bcrypt usage. A Go PoC found some MD5, SHA1 and SHA224 super magic hashes.
If you do not care about maths, you can just skip this section.Finding super magic hashes is not trivial for hashing functions with a long output. The hashes need to start by at least two 0 followed by the letter e and then only numbers.
The probability to get a SHA224 super magic hash for a random password is: 4.12e-15. Here is the probability to get at least one super magic hash depending on the number of hashes calculated (logarithmic scale):
For a quick PoC I decided to use the Go language. To be as efficient as possible we will use the nice benchmark functionality of Golang. We will also use the testing function to be sure that the hash parsing function we create does not miss (super) magic hashes.
Carl Löndahl has done what was expected for many years: tweak hashcat to find magic hashes. This is a game changer as it is probably the fastest way to compute hashes nowadays. @Chick3nman512 and @hops_ch are also working on a hashcat fork to find magic hashes. Stay tuned!
(Not sure why this is exciting? Consider how much value we get from finger prints, which behave much like hashes. They're not perfect, but they do a great job of uniquely identifing and verifying a person. The same can be said about hashes and data.)
*Note: MD5 isn't as secure as the I'm making it out to be. There are known issues. There are stronger hashes currently available. I expect those will be obsolete one day, too. MD5 is good enough for our purposes.
Since there are 1428/4900 distinct attack sets for the bishop/rook attacks, we can use 11(2048 > 1428)/13(8196 > 4900) index bits for all bishop/rook attack bitboards. The table size will be 16KB for the bishop attacks and 64KB for rook attacks. To make it possible, we must find a group of magics (one per square and dependent each other). All the magics in the group must hash its all relevant occupancies into the same 11/13 bit index without any collision with others (anyhow for each square, different occupancies can hash to the same attack sets).
But the idea seems like a wishing dream. Can we find ONE of the magic group of magics? Does it exist in theory? If not, can we use more bits to reach the condition? If yes, can we find a good magic group? A good magic group means that all the hash value
Fancy Magic Bitboards is the main stream implementation as proposed by Pradu Kannan with individual table sizes for each square. It requires an individual shift, to leave that many index bits remaining as determined by the population of relevant occupancies. As mentioned, if one tries hard enough to maximize constructive collisions while looking for magics, one may half the size for several individual squares, see Best Magics so far.
A fancy fixed shift variation was introduced by Onno Garms, looking for magics producing factors with appropriate reduced value ranges. That is, leave 12 bit rook indices for all squares, but with upper bit(s) clear for all possible occupancies for all the squares with less than 12 relevant bits, so that it efficiently becomes a less 12 bit index. Onno believes that it is possible to find magic factors with that property for most of those squares, to come close to the fancy table size with the advantage of a fixed shift [11] . Even an "overlapping" of square arrays is feasible, if they contain accordant unused slot gaps on both ends. Volker Annuss came up with a fixed shift solution of only 800 KiB and below [12] [13], and more recently even innovated with Black Magic Bitboards.
A similar idea as in Incorporating the Shift in the domain fixed shift magics was proposed by Eugene Kotlov concerning the indirection to the stored attack array, using the 16 lower bits of the magic factor as offset [22], later even combined with incorporating the shift in the domain of Fancy [23].
You may try to find your own magics - to possibly contribute to the record table for most dense fancy or plain tables. So far, there were no magics found for the expensive rook squares a1 and h1 (0, 7) with less than 12 bits, which would allow to half the plain table size. Is there any prove they don't exist - even with Black Magic?
A hash list is a JSON file that contains the necessary information to list your NFT collection on Magic Eden. It includes all the unique identifiers (hashes) of your NFTs and is essential for showcasing your collection on our marketplace. The hash list provides potential buyers with a detailed overview of the NFTs available for purchase.
git uses 160 bit SHA1 hashes for commit IDs so there are \(2^160\) possible hashes.I want to find any commit that has a magic string at the beginning ofit's hash. The typical length of the strings is about 7 (hex) characters = 3.5 bytes = 28 bits so foreach magic string, there are \(2^160-28\) possible commit hashes that startwith that string.
Alright so now where am I going to find around 10 million git commit hashes?I thought about cloning a few large git repos, like the linuxkernel, but that would require downloadinga ton of actual diff data that I don't care about, and even then the linuxkernel only has 722,647 commits at the time of writing, so I'd need to find 10or so different repositories of a similar scale.
Then I ran git ls-remote on every remote and collected all thereferences (with their commit hashes) in a file.git_all_hashes.shchains all these steps together and generates a hashes file with all theresults. The search API only allows 30 requests per minute which seriouslyrate limits the entire process, but at least its fully automated. I let thescript run for several days, and eventually collected 57.8M unique commithashes.
Ok enough explanation - here's a rundown of the commits that I found. Afterrunning a GitHub API search for every word in my system's dictionaryI collected 57.8M unique commit hashes. Out of those, I found 12 that beginwith a magic string:
During the last months, the problem of finding magic hashes for various hash functions has had a renaissance. After this tweet, the over-a-decade-old problem received new attention from a few people in the security community and, as a consequence, a flurry of new magical hashes were found.
Magical hashes have their roots in PHP. In particular, it all revolves around the improper use of the equality operator ==. PHP uses implicit typing, e.g., if a string has the format 0e[digits], then it is interpreted as the scientific notation for . This has some undesired side effects: when comparing hashes from password input it causes a small but non-negligible subset of elements that are all equal under the == operation.
--keep-guessing causes hashcat to continue looking for hashes, even after one has been found. The --self-test-disable is needed, since we have modified how SHA-224 functions, and the testcases defined in hashcat will consequently fail.
This post will cover two vulnerabilities in rConfig 3.9.6 when chained together could potentially lead to unauthenticated code execution. The first vulnerability involves type juggling using magic hashes to bypass authentication, and the second involves command injection via its cron job functionality.
For example, if any file within the object database is corrupted by a diskerror, then its hash will no longer match, alerting us to the problem. Byhashing hashes of other objects, we maintain integrity at all levels. Commitsare atomic, that is, a commit can never only partially record changes: we canonly compute the hash of a commit and store it in the database after we alreadyhave stored all relevant trees, blobs and parent commits. The objectdatabase is immune to unexpected interruptions such as power outages.
The basic identification scheme is simple: calculate the hash for each possible card, then calculate the hash for the art we're identifying. These hashes are converted to ASCII strings and stored. For each hash in the collection, calculate the hamming distance (essentially how many characters in the hash string are dissimilar), and that number describes how different they are. The process of searching through a collection of hashes to find the best match in a reasonable amount of time will be the subject of the next post in this series (hint: it involves VP trees.) Obtaining hashes for all the possible card-arts is an exercise in web scrapping and loops, and isn't something I need to dive into here.
Usually, when a hash is computed within bitcoin, it is computed twice. Most of the time SHA-256 hashes are used, however RIPEMD-160 is also used when a shorter hash is desirable (for example when creating a bitcoin address).
The locator hashes are processed by a node in the order as they appear in the message. If a block hash is found in the node's main chain, the list of its children is returned back via the inv message and the remaining locators are ignored, no matter if the requested limit was reached, or not.
To receive the next blocks hashes, one needs to issue getblocks again with a new block locator object. Keep in mind that some clients may provide blocks which are invalid if the block locator object contains a hash on the invalid branch.
Note that it is allowed to send in fewer known hashes down to a minimum of just one hash. However, the purpose of the block locator object is to detect a wrong branch in the caller's main chain. If the peer detects that you are off the main chain, it will send in block hashes which are earlier than your last known block. So if you just send in your last known hash and it is off the main chain, the peer starts over at block #1.
The mempool message sends a request to a node asking for information about transactions it has verified but which have not yet confirmed. The response to receiving this message is an inv message containing the transaction hashes for all the transactions in the node's mempool.
e2b47a7662