Control ManifoldCF via command line (curl + JSON api) not working

[wilhel...@gmail.com]

Hi,

I would like to invoke MCF job starts externally via cronjobs, as I would like to get rescans of jobs in (serial) line with other external processes. This should be technically possible due to:

https://manifoldcf.apache.org/release/release-2.5/en_US/programmatic-operation.html#Job+objects

Basically, I could start a job by pointing a curl command to a specific URL.

Apparently, there is no URL/mcf-api-service/json provided by datafari, but there is http://localhost:8080/datafari-mcf-api-service/json

However, all commands return plain '{}'. I tried to login with:

curl -X POST -H "Content-Type: application/json" -d '{"userID":admin, "password":XXXXXX}' http://localhost:8080/datfari-mcf-api-service/json/login (which gives: {} as it should be)

However, any other follow-up GET command does not return anything (only  {}), e.g.

curl -X GET  -H "Content-Type: application/json" http://localhost:8080/datafari-mcf-api-service/json/jobs
curl -X GET  http://localhost:8080/datafari-mcf-api-service/json/jobs
curl -G  http://localhost:8080/datafari-mcf-api-service/json/jobs

Is the MCF api provided via this URL? I tried to connect to other ports owned by java processes, but none seems to provide any JSON API except the kibana process. Am I doing anything wrong using curl?

Thanks in advance!

[wilhel...@gmail.com]

So I verified that I cannot login due to a 401 error (as logged in localhost_access logfile).

manifoldcf.log throws:

FATAL 2017-03-22 13:35:45,640 (http-nio-8080-exec-6) - MCF|MCF-agent|apache.manifoldcf.misc|Exception logging in: Auth implementation class org.apache.manifoldcf.core.auth.DefaultAuthenticator could not be instantiated: null
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Auth implementation class org.apache.manifoldcf.core.auth.DefaultAuthenticator could not be instantiated: null
        at org.apache.manifoldcf.core.interfaces.AuthFactory.make(AuthFactory.java:71)
        at org.apache.manifoldcf.ui.beans.APIProfile.login(APIProfile.java:88)
        at org.apache.manifoldcf.apiservlet.APIServlet.getAPISession(APIServlet.java:67)
        at org.apache.manifoldcf.apiservlet.APIServlet.doPost(APIServlet.java:169)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:522)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:1110)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:785)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1425)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.manifoldcf.core.interfaces.AuthFactory.make(AuthFactory.java:48)
        ... 27 more
Caused by: org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected end of base64 input
        at org.apache.manifoldcf.core.common.Base64.decodeNextWord(Base64.java:94)
        at org.apache.manifoldcf.core.common.Base64.decodeStream(Base64.java:148)
        at org.apache.manifoldcf.core.common.Base64.decodeString(Base64.java:171)
        at org.apache.manifoldcf.core.system.ManifoldCF.decrypt(ManifoldCF.java:819)
        at org.apache.manifoldcf.core.system.ManifoldCF.deobfuscate(ManifoldCF.java:760)
        at org.apache.manifoldcf.core.interfaces.ManifoldCFConfiguration.getPossiblyObfuscatedStringProperty(ManifoldCFConfiguration.java:85)
        at org.apache.manifoldcf.core.interfaces.LockManagerFactory.getPossiblyObfuscatedStringProperty(LockManagerFactory.java:92)
        at org.apache.manifoldcf.core.auth.DefaultAuthenticator.<init>(DefaultAuthenticator.java:50)
        ... 32 more

Any idea?

[wilhel...@gmail.com]

Forget the above error message. It was due to some password changing error I introduced myself. However, logging in still doesn't work. 401 Errors are thrown even though the correct username/password combination is given.

[wilhel...@gmail.com]

Ok. Got it working. It is not 'userID', but plain 'user'. So the documentation of manifoldcf is errornous.

[Olivier Tavard]

Hi,

I created a new documentation page about the MCF API on the Datafari wiki following your question : https://datafari.atlassian.net/wiki/display/DATAFARI/Use+the+MCF+API

The official MCF documentation is correct about the parameters names. If you want to use curl, you need to store the JSESSIONID into a cookie in the first API call to datafari-mcf-api-service/json/LOGIN. And for the next calls, you have to use the cookie to be authenticated and avoid the 401 error.

So in your example it will be :

curl -c "cookie" -XPOST 'http://localhost:8080/datafari-mcf-api-service/json/LOGIN' -d @/root/mcf/user.json (response:  {} http code : 200)

curl -b "cookie" -XGET 'http://localhost:8080/datafari-mcf-api-service/json/jobs' (response: {"job":{"id":"1490641934221","description":"test","repository_connection":"Files .....} http code : 200)

user.json is :

{

"userID":"",

"password":"xxx"

}

Best,

Olivier

[wilhel...@gmail.com]

Hi Olvier,

just saw your post here. Thanks for the new page and the information provided. I am still a bit puzzled about the 'userID' <-> 'user' thing. Hence, I would like to share my (working) shell script, which cycles over a list of specific jobs and executes the refreshes (still with 'user'):


#/bin/bash
AWK=$(which awk)
CURL="/usr/bin/curl -b /tmp/cookies.txt -c /tmp/cookies.txt -H"
TYPE="\"Content-Type: application/json\""
POST="-X POST -s"
GET="-X GET -s"
PUT="-X PUT -s"
URL="http://localhost:8080/datafari-mcf-api-service/json"
LOGIN="{ user: admin, password: XXXXX }"
REDIRECT="> /dev/null 2>&1"
DATE="$(which date)"
LOG="$HOME/log/datafari.log"
if [ "$1" = "minimal" ]; then
        START="startminimal"
else
        START="start"
fi

JOBS="  1490194448568
        1490195083592
        1490282191612
        1492506656198
        1490177300785
        1490177459915
        1486820638564
        1486840669411
        1490177651005
        1486820638502" # List of job numbers

# Start clean
rm /tmp/cookies.txt > /dev/null 2>&1
rm $LOG > /dev/null 2>&1

# Cycle over jobs
for JOB in $JOBS; do
        echo "[$($DATE)]: Starting Job $JOB ($START)" >> "$LOG"
        # LOGIN
        $CURL "$TYPE" $POST --data "$LOGIN" $URL/LOGIN > /dev/null
        # Start JOB
        $CURL "$TYPE" $PUT "$URL/$START/$JOB" > /dev/null
        # Loop for output
        STATUS=running
        while [ "$STATUS" != "done" ]; do
                sleep 5
                # Refresh LOGIN
                $CURL "$TYPE" $POST --data "$LOGIN" $URL/LOGIN > /dev/null
                # Check JOB status
                STATUS=$($CURL "$TYPE" $GET "$URL/jobstatuses/$JOB" | $AWK -F'"status":"' '{ print $2 }' | $AWK -F\" ' { print $1 }')
        done
        rm /tmp/cookies.txt
        echo "[$($DATE)]: finished"  >> "$LOG"
done

exit 0