Calculating SHA256 DB Validation SHA256

[Derek Conniffe]

Hi all,

Does anyone know exactly how to generate the local DB SHA256 checksum?  I've tried reading all local DB prefixes (i.e. 4 byte prefix, 5 byte prefix, etc) then sorting the list and then concatenating them together as a big string and finally producing an SHA256 of the big string [of hex values].  However this doesn't match the SHA256 checksum in the returned JSON data from a full update on a specific list (e.g. MALWARE - ALL_PLATFORMS).  I'm testing with a full update (empty state variable) so the checksum.sha256 (decoded to HEX) should match up if I had it right.

thanks v much for any info - I'm unclear on what the correct code / process for this is (text on API v4 website below)

Derek


On https://developers.google.com/safe-browsing/v4/local-databases it says: -

The client first updates the lists in the local database (applying the removals before the additions). The client then computes the SHA256 hash of the (lexicographically sorted) local list and compares it to the checksum returned in the response. If the two values are equal, the Safe Browsing list is considered "correct."

If the two values are not equal, the Safe Browsing list is considered "corrupt." The client must clear the list from the database and reissue a second update with the state field set to the empty string; this will force a full update and return a brand new list and state.

[Nikolay Garbuzov]

Hi Derek,

I've recently implemented this validation for our internal library. 

So, following works for me during full/partial update:

1. Read all local prefixes

2. Sort them (hint: you can apply your sort function to full update and verify that result is the same, because full update is always sorted)

3. Remove prefixes by indecies if they are presented in partial update

4. Add new prefixes if they are presented in partial update

5. Sort prefixes

6. Concatenate prefixes as byte arrays to one big byte array

7, Calculate sha256

I use Java, so byte array is natural there. Note, that if you will work with Hex string representation then your sha256 will be really different.

BR,

Nikolay

[Derek Conniffe]

Hi Nickoley,

It's working perfectly for me now - thanks !

I was doing something stupid: converting the string hex concatenation of the sorted prefixes into bytes and then Base64 encoding that (Base64 because that's the way the list is delivered RAW) and then getting the SHA256 of that.

Derek

[Santhosh Kumar]

Hi Derek,

We are also facing the same issue. Can you please explain what was wrong in computing hash for you earlier?

[Priyanka Tanvani]

Hi Derek and Nikolay, 

Can either of you please share your java implementation for using the api ?
I'm unable to decode the raw hashes returned in the api response for https://safebrowsing.googleapis.com/v4/threatListUpdates:fetch .
What I'm currently doing is :

import com.sun.org.apache.xerces.internal.impl.dv.util.Base64;

import java.nio.charset.Charset;
import java.util.Arrays;

public class Main {

    public static void main(String[] args) {
        final Charset UTF8_CHARSET = Charset.forName("UTF-8");

        String rawHashes = "AACxPwAxumAAPlhFAGRhvACGFTIAyLDmARa91QFPIvoBpFZzAaRkoAJgQKQCYEf8AmFXqwKbo+kCw/quAs+JAQLcggEDMcVZAz+8EgO17q0DttS+A93ugAPvMPcEKUJqBDr2lAQ8kDIEPNTdBFdXwARZjB8EjLLyBK7LtgTGV6sEx6NABOPz5QT6beEFDAH3BRjFFQUei6cFLfERBTTEXgXStZYGKWA8BlB8DwZRR1kGZk/qBmrP2wZrApQGmOi/BsGemwbHa1gGzq54Buk24QcIXkgHMhG4BzY2Rgc/rCIHe50ZB3yrTweiiEoH3LfACA+t/AgfTlQIK0BRCFTfWghxji8Io3HRCMCBAwjO4DYI3r4gCO7A6wlFc0kJUCl8CbYY0Am5CrgJ2PLICff9Kgn9er4KGWMjCiTh9goqwAAKLnTtClF2PQplG68KisNFCqRG7gsAYyULQ6xkC0hZ7gtLUksLUiEqC2yRlQuEQhgLlOsMC+zUqwwes10MZM3wDHZ1OgyOKsgMkDJYDJympgydIDMMo3pGDK+gxwywBjsNant1DXbd3A2iTx8N21cEDf4Vug4DoOsOEi4wDjWE6A5DzkgOSGZ0DuYIVQ7yL8wPFdxhDyPTdw9RyE4Ps7sfD8pU1g/ZYyUP5NThD/fo/Q/++XkQJl07ECh+YBA2siEQRQscEFdk9BDDJAwQ4wfrEXXuPxGxXq8R0NkXEdQe/hHmmFAR7vUpEhkDsRIdJGgSH8fWEikzGhJyHdgSfqLIEpVxERKoNQ4SuPDpEsfECRLI800S5EmJEycY0hN79HQTg8TBE4Rn0xOguvQTzJILE9XXphQQiHsUH+R9FCK2rxQuJvgUMxXsFDd3DhRsVQ4UukY6FL0qgxTM2bYU5ZqCFVHFohVxfwAVic/DFYvRdhXLhG4WIji0Fi4suBbwdx8W/HWqFwaISxc5wigXQNSwF0F9Xhd2lzAX5LLYGBVcaxgtL48YOADFGGxI9xiVyhEYn5ceGKxeuhi8NgkYxw5kGN6fWxjgXM8ZGICCGRkAPBknergZWAvAGWtyQhlznZoZ0bsyGdO+IxnVSDoaEhT/GhXinxodjWUaSaG8Gk2CNRpYhfgauLB0GvWJwRsNKNEbFr6QGzqz9htEbNobZWrPG2lQWxtwbTobcROGG3KmWBupn4obxCsjG8VNGRv6rh4cJZM+HFKhdBxaBMccc7czHIFKdxy+Zg4c4CBiHUg2zx1XUoIdYpvbHYhNGx204M8dxrADHdEIwx33eTEeAcTSHhgopB4oD+oeYFLIHnJCyR6m5Zket2ohHrh6ih7OSLUez6R2Hv6uyx8hUqcfWj7yH1zi2B9nXK4fgM4SH5vT2R/kx7sf+W9tIAL7gSAIRMMgZI+xIH9adSCN9RIgjtYEIJuEgSCtg5YgyU8QIRDgfiEZgG0hKdmfIStCJiEzY3Yhl22jIaNejyGpwMAhvgBFIfDQuiI4CA4iT+DCImISHSKNE6Mij/bsIuBfkSLv7s4i80OpIvVKFiL5Z/sjCJ3eI2MbFiNssbIjfMaTI47LWCOeqEkj1qW2I+j9CiRHY/MkeqVnJIA9OCSWhYYkvS8/JOwM1yT9vlolXNDJJXBp3CV8qRclhGoTJadPeCWxP6cltEYuJedeRyXzVokmLI7zJkplAiZLyMUmfKL0JpYJ2CbL0w0nHL2nJzMXNydSt/wnn7lcJ+VxgSf0SzcoIuJoKEValyh8l1Yo0VKsKNn7bSjdoT0o3y7TKOiJDyjrSx0o+jRAKQMr7CkIU14pLuwIKW/k1CmF094pwVxfKkZHtCpidUYqfFBzKo7z/SqSaEQqnLIjKsZlIyrMWDAqz0qEKvGVvir8y4orTkfrK6JjwSvJ4oAr+GfLK/kL3Sv9CdIsHYhpLJegtSy4T/ktGpFpLU/sRC1XePgtag2mLaEIny27NAAt0OnTLe6/HC4HCW8uDA2GLjT08C41ECcuP/6GLkbJQS5hTcwuwSiKLsy1cC8Oc9UvHYyKL1Lqsy9d6jUv/xj2MF+TUTChffUwrGw9MMz97TDVJu4w9XFxMSlqiDFE0AYxTrPyMVxGPTGONmsxs/9PMcBKFjHDuYwx34i4MgyspDI8H4Uy84HTMwLDJzM899YzSwxbM1M98jNix9MzZTbSM5TuHjOWRTkz4lA7M+O82TP1VZI0Bg0ANDiaTTRMAaU0cRIKNHRcyjR34bA0pr6+NLP5wzTf7QA05laENTvqcjWJxPg1jmOaNZRsIDWWOQI1mSLaNbV0/TXEjto1+Jr5Nj/j6DZXp7I21ji4NtzFJjb7LWs3TauJN5wGDzfF2og3zXFXOAAjUjgW0N04J0m1OG32SziD+Kc4h666OK6zUDi4iRc43Zh9OO4m7Tj/wxE5AGNcORqB+zkwv8E5Ue5FOaM9GzmokEU5/ew5OgH82DptTts7GEmOOzWEeDs+Wa47SNWkO5tqAzum6e08I7S0PEXA1DxUSAQ8iwJDPJbHQTyquXk85FllPXQVjj2o3i49uN8QPbpnUT3Bsos99J9yPkXxqj5lqyM+wK8FPtSSvT7bvuY+9Xa6Pw5GeD9NW2w/V9jSP1tRij+nzx8/4c2CP/gtnUA56TRAlJA+QJU3p0DCwiZA/Hk4QQZIo0ESc2BBSJRyQU7p0UFQPoBBU8KAQWbGPEF2XDNBiBMvQYouFUG29pNB691IQgGNyEJBp3lCX01TQsIc00L1T+5DPa5HQ1nuuENaYHJDbMatQ3E7NEOiashDtjvoQ8yQbEPi2DJD7pp8RAI88EQK3H9EObGoRE5L7kROdo1Ee980RH6vfkSEL3dExfkXRNrScETbzZJE8tb0RShGbUUtD0JFRc/SRW8j6UXUFZ5F3WOeRf5LkUYEdcRGJS70RkTP20Zg31ZGgGiaRqWZQ0b2SzVHNdraR2KM5keKH7pHlU4KSAOF7UgRcfRIMSgtSDwB9kg98IpIPq6xSESIiUhf/itIbAlNSG4W4EhyyzFIkWQ/SLOF1Ei4txJIwuXiSOeW7kjzRH1I/p2eSRXgDUlKFQNJePcASdlft0nZZetKFfcESid8sUo6dxVKQICnSleezkqTHMlK1V87SwH4fEsP0ENLRsKxS1W4FkvjRdlL6t78TApwI0xoCKdMnUbBTNhv8kzdNjNNMfthTVW2wU2LUUFNi5FBTbyIF03BUxVNxeFtTccJxE3/4j5OZmpXTpNBF061HWlO5glVT6ppwU/PvnBP4JpsT+j29lAQuWtQJLA5UCrohlAv8O5QjPROULHNwFCyb3pQ7BisUQ8Ge1EtL3FROKFBUT55AFFKOhJRkbCUUaMzXFH1rXVR/TTzUhD9wVIf9lNSK58CUjSkyFJyVgBSm9hNUq1kjlLoNEpS8RjeUve7FVMgEShTO/hTUz24u1NC9ltTQ5UQU3je71PShh5T3YDaU94VA1QQq4lUNSI3VDbe9lQ265hUaIVCVG5QblTjuM5U+S+nVPz2TVUeBgRVVdO2VWL5W1WgHqVVqJx8VdVL71Xkrb5WBs05VhmFCFZCFHNWS6spVm2apVaNxntWrt2lVruBa1bsmEVXJ8WPVzT8XldBdsJXk/1yV6arq1eoSjBX0IYvWBCQalhSU29YZwYGWGwJ+FiZ6AZYt6gNWMpLaljgdUBZARaoWS9471k5AyNZO1RJWU3pwVlz+oNZhSaJWZEdS1nLt31Z4WQIWfwmQ1oIkRdaoVf0WspQlVrt/CVa8z7OWwuJdVtKbiRbfKLpW5mdmFusq2Bbrw/NW73gilvDSEhbysPvW9WHH1vo+ntcOGvqXENFwlydN/xcqKk+XMESCVzGVIlcydH0XNBH0FzayH9c8WdlXPwCVF00kZtdsMcuXdJJNl3kWY9d6EILXfUKHV481fteRA7MXojZmV6/TmVe1sGyXunm6l8Rfv1fJK4yX2eCLl+eXPRfr9k8X7mWcmAek0dgSQKNYJu8XWDWo3Vg1qOUYR7ITmEnJwNhh3DnYbJdlWH/3TViEbtEYm777mJyg95ipqvKYqfgRmK87GxizeLhYurP1GMqm9VjM3/1Yzx72mNAuz1jTqw2Y141zWO+Pc1kGQdtZB75JWQrlmFkNV+DZJRXzWSf4mtk2kEcZQRyg2UwNzFlU76TZVsyVGWIQBFl5J4nZkbGW2ZaymdmbgNVZrkL8mbt1h5nCNqAZ1Fs02dbv/1nXDOsZ5I+J2enWWVoIaDfaCqSzWhB8z1oSFFwaF8TgGhq8PxokqXiaLodvmjiIetpAf63aU50HWla+fZpYy7taZUPg2mpW5hpr7vXadB/M2oLR9FqGuREalBB12puO89ql+3FaqIBN2qytQVq2mcravNDbWr7p2Zq/vB8av/anmsJ+2FrOLHdaz7/8Gs/Mq1rVAapa4toT2uuXIhrvJeCa/xwSWwUMHVsMLfybEiSzGyJ1ZZsra0ibLHX0Wy5iWds0SxdbPZJuGz8QqVtFCnmbR9tEm0wXlhtPIO9bWX/C23dE9JuAdT/bgOeWW4aEDduHh9jbjM7j24/iGxuRl9ObkaMF25HYyNua+gkbqdoq26/6Qdu3pNUbunlwG8gFJNvMMoBbzJ5T2/AwnVv9ghocCWWDXCDxC5wtiZAcMte0nDlHGdxJonjcTajpnFf/rpxZFeLcXsOdXGOpiNxyd48cc3L3XHkPE1yNmj/cnpabHJ/XM9yp3/Ycs2VYXLR/q1y1Z9fcvK+X3M/221zWjzac2ZfmnPV905z5fHPc+3hq3P3uW9z/1YJdAg68HQMfBF0DLBwdBkbWnQqnGR0bpuhdJueEHS9Wrh1BxxJdRCeYHUZ/a11GjcNdUf+gnWooJ51ssd4dbNUNHXAiUt11l/NdgcgqXYUkGB2MRQddjR2NHZAT2F2RR4edkVuzna7UTx21t1GdwS183cWlXJ3Pk6Rd0Osr3dHMMh3XQJ3d3wx+3eby+B3nx9gd8YTT3fTn/939cMOd/qy0nf/Eex4B7xTeBSy2XhGv+R4S6HFeIfgCninhtp4xYmheNKNcXlDWvt5Z1RLeXXN7nmTvM553upfegiypnoR6OV6FhvLeiaDYXpop/166PUyexI5SnsvQ8B7Q0Oce4I3VHuN0Ld76DTMfCA1QnxydqR8jo7NfMazIHz8ICd9OISSfVm0O318ucB9hXQLfb8bB33Gax59y/rcfdCruH3h/0R+CwW8fhejdH4Zobh+HBt5fkSHTH5q7vR+cCPafqOYWX7uXRl/aYjFf5tLcH+vpYx/u8w+f9h9Y3/zRUyAU3wFgJq1ioCbBH+Ao2nDgL92CID8q2+BMZHsgViEg4FvKcKB2+BegkHT84J3YTqCr3tBgrdRLoLWPFaC65kWgxDE4IMfKuWDNak7g1UVjINXBJSDZdT3g2m6e4NvheSDgGfWg5Lo+oOqmzCDzSQUg9UqdoPg8jqD+1snhAxHNIQhbDSERP+ahFnPPIRacrGEYB/MhHkPzITPxZyE54WJhbBxD4W/2sGFx6l4hh8rBIY9DXiGrsPRhs/qQobkFmqG/hMChyKsnIc/NH2HYP90h3/FMoedC/qHopTvh6lH1of1Zr2IAbwniDOUWYhR7hiIXp3riGz2eoim8FGIuVapiOFkEYkka7mJPTLiiUFslImfyeuJ2NJzig8EIIouiJGKdy8UindOZYqGtgaKlZHbiqidaIqpTd6KspKmivV1V4r98MuLOVpli0dd54tOBRKLdXPfi430V4vQ1xOL5PPgjCFVaowxRpyMZmikjH6AhYx+oH+MiHFljM+fyIzTbkyM9lpWjPhZ0I0mJ5+NKn7yjVVrV41d712NkhoRjbxVmY3GhbWN+bqhjh6Eg448ziiOcxIajsAcgY7Fz+CPRcWfj5BkAY/Y3gKQJ9mZkCurAZAvprOQOCISkF0aR5BgCcGQZ20bkHKdJZCrUZKQu/yfkOt2PJEB31+RFISHkS60QZFNRr2RZ50HkZg9h5GgOCiRrTskkgP5YJINbH2SISCOkiE3HZIlLjGSM7+pkjRdipLWyIeS4yZdkzIbKZNG83KTR/Rnk1zaF5N24juTeBoGk34FCJODf1GTjeWlk45UxZOdD3yTqVp1k7zg5JP8Br2UEQ8BlDSqPpSCwMSUlPPYlLOQZJTQyiyU5LI8lOrt25UDhaSVCQ6SlVHGUpWbW3WVo3dalbBfNJWw7maVs+cRldVFBpXusrqV+h6llif2jpYtfduWL2rOllJY+JZSY2yWVvDjlqTheZbXVguW598VlvBHT5b4/gKXNwm3l0dL55dUHx2XaHkLl8M2jJgM8imYZpxWmHn9r5iOwqqYsmQ1mNlnsJkEWJ+ZJ4ArmS5jWJlGy7GZTmAemWSOv5l4wauZu2fqmdrFWZn5pvKaBbqymhvXBppUuRiaVn8ummAZH5pvzY+ac7Fjmp+JtJqmrS2axpE6mtOFVZrXPa6a/Shymw2nZJsVI7abGZpBmyYuHZtM7MybeGIKm8j295vLlFmb2c4pm/3rjpwDK2OcBxXEnAiQepwfZI+cJdbWnEfBk5xOX8WcckmXnHf9np0JnNadFACBnUxzF514G0OdueO6nhW7e54X26SeNqNbnk+D6J59CVOefjkJnoBVtJ6tPBGewFyVns8a+Z8oLDKfKnBGnzw0mZ9gv42fa+/5n3Ffo59+zlCfx3jpn92o46AfvfCgNnMvoDgFLaBjhqqgaVu1oG9nJKC89/ag4Do2oQ6FP6EjmM+hLvyAoUS+PKFHF1GhXDaXoX9t8KGsstKh2ABfoeqQbaHrmLaiJGMFoi+FlKIxipOiiBdjoo1zg6KW3nSixJ1kotQKNaLb5oyi3nXCovTBwKL6YN6jBSNUow5Ml6MXlYSjI5K1o4tC0aPdRVyj6OXto/GX2aP+0NOkJ/JlpD+gsaRPE/GkZlhCpG1USaR3zZOklfALpMtCQ6TNjw2k8dVspRed5aU44GulRJuepUtfJ6VeE+WlatBopZzcoaWuNRCl3EbNprdlTabI4/mmzgHBptDqJabfArinEeNup2GjIqdkBRmnsMECqBzKaKjB34Go4fn9qSN57Kk8iBKpQGcuqXBdManDz+Op63RJqfZ7gKoKj/yqGz7wqiLG0qopLKeqQsnUqkfQZaqK+xaqts74qsZk1qsEXCmrLK2Jq0wvRatMxOeraeyMq2087quIOcGrpyToq6r6UKvi6bar7giirIOqgqyfbSWspbTbrQVk8K0NnE2tIrnnrUTrDq1FTr+tdQHdrZVOhK2V/tGtrlu7rcevQK3aHriuPE/DrkHuaq6DhZ+uvnCwrsiN1a7ewIyu884jryqq7q85oaqvWWKHr7HvZ6++rYOv1ea3sBH1UrAnecmwN/6nsFvMT7Bc+EGwZpE2sJeOWLCbLh6wnEhSsKojbrDxrHGxDZKgsSDKhbEvBKSxTku4sWAMqrG55+SxwYmVscW+UrHooyKx+HQqsgFB37JSWrSyW2sgsmd6TLKKdAayiw/Wso9v7LKWdyey9gEWswLVmbNPZkWzpRezs8CoarQW5dq0Ii9ttDmIrrR28Se0qpWdtLzQVrTbIG21AEHbtQouUbUjRoq1O+4EtUi39rWXn4K1uE1rtcAEVbXzKAe2zLGRtwy5S7ctjVa3x0AsuACGurgV59W4HVykuGTkJriD6Ui4yZFbuOwC0bkRgIa5GsZWuRs45Ll/pgO5pqFaudq0Pbo6Ara6RhZ3umGnUrqTlRy6rxBAuuWMBrr4QHm6/QRYu1Dt1ruGsSq7vl4Ou8BtgLvAePS7x7Fgu9ffMbviUYi75i7kvAI/prwf7DO8Jhd7vCscZLxzpRG8hBXhvKsZk7zb19m85MW4vOZZerzyyKK89KsEvQVF9b0Pmp+9MBXuvVXcHb1d5xi9u7/0vebfZb4VB9++FlsUvjoz6L5NQbC+ir8KvpC4LL6bDmO+0YyNvyh2U79JhK6/h/n3v7TYk7/pO3m/82dywDi16cClcpnA9/RcwQo7i8ESh77BJck4wSbfBcEuk3TBR/1nwbteMsG+wCPB1O5lweOfs8JBS37CSysdwoL/msK7kX/C/m5AwwYndsN04J7DlKRrw8GFNsQ3e5zES08txH4ee8SY1hjErT5BxNxkW8Tw0HTFBrXcxRTg3sVHYxPFVhjvxV+rW8WVpa7Fnc29xagYUsW302fFwRDZxfiWBsYzfl3GdpjGxpbYLcaeRfDGraHdxtNX88b437XG/hWPx1rXUcdo6evHgYMIx4XdTMe58AXHyfVfx99wQMgcKu3IHeyRyCgOLsg73HrIPkjoyFtOM8iinl7JAEklyRF9tckT1hTJNw1lyTlqwclCxSzJVE7oyV/XP8mWhbvJmRHnydhp78oCB7bKD6ItyjmUt8pS3m7KXaKsymKK7spqNyTKb/NUynN05sp4ITHKe6cKyp3xQMrEUXXKzk5dys9RFsrk50XK8fnRyzWby8s5YGTLOhtSy0fCh8uNAd3LlqcWy6fCEsuuoTXLtTQMy8i6B8vZqkbL6MVGy+t8U8vz7AfL/i5Ry/902swDGR3MH+MnzFt+PMxfBm7Mc67AzHqvQ80Wi8rNH1o2zUOIp81kL7jNdDKRzZaWIc33qazN+H68zfowes4VjyvOFgKXzjLWtM5N7kfOWT4pzrjQO86+CtLOxeImztBpc87iQ3LO5cxVzv0tq88KGyzPC5bZzxAgAc8T3cnPF0/bz0YBaM9SFhHPU1Iez2mlCs+BjrPPwzcKz+yA8tAcqWzQQ/ch0I/vgdCpR5vQu3Rn0L3XkNDahl7RECmM0Shnp9Ex9TTRNiBv0UiE7tFiPKvRndud0iWwz9J+EfLSlPmC0szE4tLdRubS65sc0yZhmdMoyFPTN9EO00njNtOJfTbTttZZ0+gaRtQZNOnUULV41FUHUdRxsTbUmoJX1KXBZdTQuarU8PQ31T08c9VsiJLVmcIE1a2E4dWxrcDV0Wm61f+xf9YsgDLWSo4U1k1hwdZS6RfWboS21oAVaNaE9jHWoApR1qT8INa8oGXWyzOQ1wl8d9cUjtDXGPof1x/nbNcwDnPXhojL16eVIdfFRK/XyfCB18tOw9fnXiDX7Qq91//wRNg/pRLYZ4rq2MIa7djQ+DjY02gB2N+wjdji83HY7sXq2QOj5dkNgZ3ZKkcj2TI25NlLP9PZXfpK2WSzXdmGoQbZhxXv2cAY89ncDpnZ55iM2eehjtoOafbaFC7n2ima19o6uMraRRo/2mUg6dqR3TLa5TFk2unqdtrrl7jbG1jS2yDza9tGUPDbnl+a27S67dvLBX7bzpaf297eJdvibMDcN14j3Fy5VNyWoyLcnRuA3J5Pe9ykEi7c1K0X3NzTVN0l64vdLSpy3WMr1d1st2PdccMp3YARMd2D6TDdqGRS3cINmt3SYK3eF+V23lWhHd59fKLekfGF3qNaX98bMyDfN48b30JFo9+Ef2PfneG037ECaN/b9vDgDmOa4B6kqOCIg/fgoKTh4K/uIuCz3bbgzEN24STLYOEw++fhNCek4UdYxuFt227hrGPa4a3eduGxYAjhxOs14fMA0+H1NU7iFmSU4hzEO+IrYL7iWT6/4nhzRuKCMfjik1i24pzOuuKpgyTi2sv64xAZDONJoZfjaLGb42/WWeOI1xnjk+374/LgL+QYBavkxP4t5QEeU+UZjTblMrWR5TN2luU/tBXlTtzw5VM8ROVncJ7lcY+y5ZNysuWYj5nlq0DA5emdPeaIerTmkPnA5pJ76OalUfzmyHaY5tbUjucoy5DnN6EM50vYJud2ZfDnk0J955y7Z+e3R9PnvY8E59zRougBPbXoDQC+6BXDJehH7onoZXc76GmS2+iLpUDon8Pi6LcOc+i5/hHo+m1J6Ps+YekC13/pBz5k6Q9qpOlELQ7paKos6W+9o+mf3ufpp9sW6fHKKun73TTqCzBZ6jItwup2ZvrqlCV96qn6bOq2gETq1cgI6uRAuOrnre/rTTUd61sIouuR/gTrn7ba6+VqbevsICHsqSgS7NwHNu0eQPftUmgF7Ypk7e3Chk7t6Jta7e7jnO3wLx3t/ovw7g7peu4apJHuN/2c7l3dA+52ktnujWmR7rb8qO64g+Lu6FRz7wvKpe8kOpLvMTq671PemO9tN5/vipA+79wJBu/u743v8DUw8Bqj7PAurerwa/co8LxzsPDQV9Lw10yM8T3bfPFa1vjxYcCF8XoXTvF/qMPxpdQF8bbOs/HC9TbxzAhw8gdIQPIZchXyIs958kroRvJPcSHyYAI+8oWc4fKPEvvymg/F8qUrvPKqBYfyxnj58wS5xPMv08LzNerm8zedpfM5L/DzQH9681BAlPNcDyjzYAxU88HzwPPIMUXz1tQD8+KVCfQIL070MJnh9I8V0/Sm3XT0sa1m9LgDj/TCVi70xZ+I9Zr8BPWcwwD1xNR39eCAffXjVnj158re9hz2IfYdiiP2ISha9k9TH/Z5Sxr2hkBw9qgcDPaw3A72wyDQ9sYbePb0WeP3DNKX9ynLfvc/fGD3Snqx94K8B/eFsDD3olpa96WC9fequKf3s44Z98yzhPfgna735u7A9/Y4p/gOaPP4Ltp3+DUhDvg/tAb4TC1f+FqQYfhdxWX4kF81+JOL2/jHqBH459QJ+OpsEvkEYHL5DmvL+RENtvkyZ/z5P4bN+XNzJPmw7/7517Rs+gNjlfojK6P6Kcgz+jeqWPpIA0r6asoV+nkx3/qU6vf6z3we+vIsivr24Kz7G85A+2Q+PvtoTpf7jSM1+5RFx/uZJvj7t397+83+QfwH6Bz8ZHiv/HFz3vya3bP83hcV/WYLHf1rdhz9qC+P/dN44f3+YNf+FIMi/iXll/5bhu3+Zhj8/pe2KP6qKPv+xfky/upxcP7sd//+902U/ydwgf962gT/iLK5/5UrZf/yTHc=";
        int prefixSize = 4;

        byte[] base64Decoded = Base64.decode(rawHashes);
        System.out.println(base64Decoded.length);

        for (int i = 0; i < base64Decoded.length; i = i + prefixSize) {
            byte[] array = Arrays.copyOfRange(base64Decoded, i, i + prefixSize);
            System.out.println(new String(array, UTF8_CHARSET));
        }
    }
}

Can you share your code or point out the mistake I'm making in the above approach to print out the decoded hash prefixes ?

On Thursday, August 18, 2016 at 1:30:50 AM UTC+5:30, Derek Conniffe wrote:

[Nikolay Garbuzov]

Hi Priyanka,

Unfortunately I cannot share full code source because it's integrated with our internal solution, but I don't see any issues in your code - it seems that you do it correctly: decode base 64 and get bytes with length of prefixSize.

One thing which is not clear for me - why you use (new String(array, UTF8_CHARSET) ? If you want to print results you should convert byte array to hex string using, for example, org.apache.commons.codec.binary.Hex#encodeHexString.

BR,
Nikolay

--
You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/Z5fF1pGNbuo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-safe-browsing-api+unsub...@googlegroups.com.
To post to this group, send email to google-safe-browsing-api@googlegroups.com.
Visit this group at https://groups.google.com/group/google-safe-browsing-api.
For more options, visit https://groups.google.com/d/optout.

[Priyanka Tanvani]

Hello Nikolay, 

Thanks for your quick response. 
I still didn't understand why should i convert the byte array to hex string if i want to view the results ? 

Thanks.
- Priyanka.

[Nikolay Garbuzov]

Hi Priyanka,

If you don't convert it to hex string - you will not see actuall content of byte array, for example

Hex.encodeHexString(new byte[]{0xAB, 0x12}) will give you string "ab12"

and

new String(new byte[]{(byte)0xab, 0x12}, Charsets.UTF_8) will give you unreadabe string like "� "

BR,

Nikolay

--

[Priyanka Tanvani]

Hi Nikolay, 

You mentioned : 

6. Concatenate prefixes as byte arrays to one big byte array

7. Calculate sha256

So, i wanted to ask what will be the size of this big array ? Won't it be too big and does it fit in memory ? 
Also what database are you using to store the hash prefixes?

Thanks!
Priyanka

[Nikolay Garbuzov]

Hi Priyanka,

We don't use DB and store hash prefixes in memory. They take less than 10 Mb for each threat type.

[Priyanka Tanvani]

Hi Nikolay, 

Thanks, this helps a lot :)

Also, if i want to store full hashes for all the prefixes (for say 5 threat lists) at all times (to decrease latency- so that when i actually want to query for a url, i do not have to call the google safebrowsing api). What will be the size for storing all full hashes for each threat type ?

[Nikolay Garbuzov]

We don't store all hash prefixes, so I cannot estimate how much memory it will take. We use caching approach recommended by GSB https://developers.google.com/safe-browsing/v4/caching

--

[Nikolay Garbuzov]

Sorry, there was a typo in prev response. We don't store all full hashes.

On Thu, Dec 29, 2016 at 1:16 AM, Nikolay Garbuzov <nikolay...@cloud.upwork.com> wrote:

We don't store all hash prefixes, so I cannot estimate how much memory it will take. We use caching approach recommended by GSB https://developers.google.com/safe-browsing/v4/caching

On Wed, Dec 28, 2016 at 7:39 PM, Priyanka Tanvani <priyank...@gmail.com> wrote:

Hi Nikolay, 

Thanks, this helps a lot :)

Also, if i want to store full hashes for all the prefixes (for say 5 threat lists) at all times (to decrease latency- so that when i actually want to query for a url, i do not have to call the google safebrowsing api). What will be the size for storing all full hashes for each threat type ?

--
You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/Z5fF1pGNbuo/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-safe-browsing-api+unsubscr...@googlegroups.com.

[Priyanka Tanvani]

Okay, Thanks :) 

[Priyanka Tanvani]

Hi Nikolay, 

Can you help understand the usage limitations.

The doc says : "A single API key can make requests for up to 10,000 clients per 24-hour period" 
So does it mean that my api key can make a max of 10,000 api requests per day ?

Thanks, 
Priyanka 

[Alex Wozniak]

Hi Priyanka, that's correct. If you need additional quota, please file a request through the Google Developer Console.

--
You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.
To post to this group, send email to google-safe-...@googlegroups.com.

[Priyanka Tanvani]

Thanks Alex. :)

[Priyanka Tanvani]

Are we allowed to make concurrent requests to the safebrowsing apis with the same api key and client id credentials?
(in specific to the find fullHashes api)

On trying to do so I'm getting the following error : 

java.io.IOException: insufficient data written
 at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.close(HttpURLConnection.java:3501)
 at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:81)
 at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
 at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
 at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
 at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
 at net.media.safeBrowsing.request.FindFullHashesApiRequest.makeRequest(FindFullHashesApiRequest.java:35)
 at net.media.safeBrowsing.util.FindFullHashesCallable.retryRequest(FindFullHashesCallable.java:35)
 at net.media.safeBrowsing.util.FindFullHashesCallable.call(FindFullHashesCallable.java:29)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)

[Nikolay Garbuzov]

Hi Priyanka,

I've never faced this issue and probably it's not related to GSB but related to how you do request itself. 

May be this topic http://stackoverflow.com/questions/36320179/httpurlconnection-throws-java-io-ioexception-insufficient-data-written can help you

BR,

Nikolay

--

You received this message because you are subscribed to a topic in the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-safe-browsing-api/Z5fF1pGNbuo/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-safe-browsing-api+unsub...@googlegroups.com.
To post to this group, send email to google-safe-browsing-api@googlegroups.com.

[Priyanka Tanvani]

Hi Nikolay, 

I'm using the google  java client for google safe browsing : https://developers.google.com/resources/api-libraries/documentation/safebrowsing/v4/java/latest/ 
So, i'm not handling the http requests and responses at my end. 

In your application are you using some other java client or handling the http requests yourself ?

-Priyanka 

[Nikolay Garbuzov]

Yup, we are using GSB client also,

<groupId>com.google.apis</groupId>
<artifactId>google-api-services-safebrowsing</artifactId>
<version>v4-rev6-1.22.0</version>

--

[Priyanka Tanvani]

Hi Nikolay, 

Thanks.
Yes, there was a bug in my application code, and it is working fine now.

-Priyanka

[Priyanka Tanvani]

Hi Nikolay, 

There is a constraint for region while querying for prefixes in the fetchUpdate api. Can you tell me the significance of that constraint?

example : 

"constraints": {

      "maxUpdateEntries":      2048,

      "maxDatabaseEntries":    4096,

      "region":                "US",

      "supportedCompressions": ["RAW"]

    }

-Priyanka 

[Alex Wozniak]

Please see the docs for a description of the constraints field: https://developers.google.com/safe-browsing/v4/reference/rest/v4/threatListUpdates/fetch#constraints

--

You received this message because you are subscribed to the Google Groups "Google Safe Browsing API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-safe-browsi...@googlegroups.com.

To post to this group, send email to google-safe-...@googlegroups.com.

[Priyanka Tanvani]

Hi Alex, 

It says : 
"Requests the list for a specific geographic location. If not set the server may pick that value based on the user's IP address. Expects ISO 3166-1 alpha-2 format."
So does it mean that the prefixes retrieved will correspond to urls banned in that region? or the urls of that region ?

Thanks.

-Priyanka

[Priyanka Tanvani]

Hi Alex, 

Can you also please address either of these queries : 

http://stackoverflow.com/questions/41672886/google-safebrowsing-api-response-deadline-expired-before-operation-could-compl 

or 

https://groups.google.com/d/msg/google-safe-browsing-api/8Vs7WWawmaI/KVB_rq9VAQAJ 

Thanks.

-Priyanka

[Wayne Xin]

I have the same problem. I can't match the checksum with google's return on full update and the logic seems to be correct with following code.  The logic is:

1. read the "rawHashes" I captured in a file.

2. base64-decode to true hashes (byte array).

3. since I know the prefix size is 4, so I use it to extract individual prefixes.

4. sort the prefix (the 4-byte array string, not the base64-encoded string). I used Collections.sort() which I assume it uses lexical ascending order.

5. print the sha256 hash string.

However, the printed checksum doesn't match what google passed back. Any help is appreciated. 

private static final void testChecksum(String file) {

byte[] hashes = GsbUpdateUtilV4.decodeBase64(file);

System.out.println("Total bytes: " + hashes.length);

List<String> prefixes = new ArrayList<String>();

for (int i=0; i<hashes.length; i+=4) {

byte[] rawPrefix = Arrays.copyOfRange(hashes, i, i+4);

prefixes.add(new String(rawPrefix));

}

// sort the prefixes

Collections.sort(prefixes);

StringBuilder sb = new StringBuilder();

// concatenate the prefixes to a big string (not printable)

for (String aPref: prefixes) {

sb.append(aPref);

}

// System.out.println("Sorted Hash: " + sb.toString());

System.out.println("Checksum: " + GsbLookupUtilV4.getSha256(sb.toString()));

[Wayne Xin]

Never mind. I figured out my problem. I was not handling the string (for sorting) using "ISO_8859_1" charset. This causes the sha256 value to be off.

[Sergey Volkov]

private static final void testChecksum(String file) {

//I use a Base64 class

byte[] hashes = GsbUpdateUtilV4.decodeBase64(file);

System.out.println("Total bytes: " + hashes.length);

 

List<String> prefixes = new ArrayList<String>();

for (int i=0; i<hashes.length; i+=4) {

byte[] rawPrefix = Arrays.copyOfRange(hashes, i, i+4);

//I change this String

prefixes.add(new String(rawPrefix, Charset.forName("ISO-8859-1")));

}

 

// sort the prefixes

Collections.sort(prefixes);

StringBuilder sb = new StringBuilder();

 

// concatenate the prefixes to a big string (not printable)

for (String aPref: prefixes) {

sb.append(aPref);

}

// System.out.println("Sorted Hash: " + sb.toString());

System.out.println("Checksum: " +

 

// I use directly a MessageDigest class

GsbLookupUtilV4.getSha256(sb.toString()));

}

 

Wayne Xin, I'm trying to do so, as described in your the last message

But checksum is not equal to the specified JSON server

 

Can you put the version of the function that passes the checksum of the prefix?