[Pipelines] pipeline with nif-version=2.1 fails

[jnehring]

This curl

curl -X POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" -H "Postman-Token: cc799c16-5d39-accf-b81d-1aa4a48fb5c9" -d '[
{
  "method": "POST",
  "endpoint": "https://api-dev.freme-project.eu/current/e-entity/freme-ner/documents",
  "parameters": {
    "language": "en",
    "dataset": "dbpedia",
    "nif-version": "2.1"
  },
  "headers": {
    "content-type": "text/html",
    "accept": "text/turtle"
  },
  "body": "<p>This summer there is the Zomerbar in Antwerp, one of the most beautiful cities in Belgium.</p>"
},
{
  "method": "POST",
  "endpoint": "https://api-dev.freme-project.eu/current/e-terminology/tilde",
  "parameters": {
    "source-lang": "en",
    "target-lang": "de",
    "nif-version": "2.1"
  },
  "headers": {
    "content-type": "text/turtle",
    "accept": "text/html"
  }
}
]
' "http://api-dev.freme-project.eu/current/pipelining/chain"

fails with error message

{
  "exception": "eu.freme.common.exception.InternalServerErrorException",
  "path": "/pipelining/chain",
  "message": "For input string: \"//freme-project.eu/#offset_38_45\"",
  "error": "Internal Server Error",
  "status": 500,
  "timestamp": 1475653048652
}

It works when I remove nif-version=2.1 from both API calls.

[jnehring]

The error seems to originate from within internationalization.

Error log:

ERROR   2016-10-05 11:06:28,805 [http-nio-8089-exec-1] eu.freme.bservices.controllers.pipelines.PipelinesController  - For input string: "//freme-project.eu/#offset_38_45"
java.lang.NumberFormatException: For input string: "//freme-project.eu/#offset_38_45"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:569)
        at java.lang.Integer.valueOf(Integer.java:766)
        at eu.freme.bservices.internationalization.okapi.nif.converter.HTMLBackConverter$TextUnitResource.<init>(HTMLBackConverter.java:450)
        at eu.freme.bservices.internationalization.okapi.nif.converter.HTMLBackConverter.listTextUnitResources(HTMLBackConverter.java:359)
        at eu.freme.bservices.internationalization.okapi.nif.converter.HTMLBackConverter.convertBack(HTMLBackConverter.java:165)
        at eu.freme.bservices.internationalization.okapi.nif.converter.HTMLBackConverter.convertBack(HTMLBackConverter.java:115)
        at eu.freme.bservices.internationalization.okapi.nif.converter.HTMLBackConverter.convertBack(HTMLBackConverter.java:82)
        at eu.freme.bservices.internationalization.api.InternationalizationAPI.convertBack(InternationalizationAPI.java:131)
        at eu.freme.bservices.controllers.pipelines.core.Conversion.convertBack(Conversion.java:62)
        at eu.freme.bservices.controllers.pipelines.core.PipelineService.chain(PipelineService.java:115)
        at eu.freme.bservices.controllers.pipelines.PipelinesController.pipeline(PipelinesController.java:92)
        at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvok

and

ERROR   2016-10-05 11:06:28,807 [http-nio-8089-exec-1] eu.freme.common.exception.ExceptionHandlerService  - Request: http://rv1443.1blu.de:8089/pipelining/chain raised
eu.freme.common.exception.InternalServerErrorException: For input string: "//freme-project.eu/#offset_38_45"
        at eu.freme.bservices.controllers.pipelines.PipelinesController.pipeline(PipelinesController.java:120)
        at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)

[katia-vistatec]

Hi, I tested using these two requests in sequence:

curl -X POST --header "Content-Type: text/html" --header "Accept: text/html" --header "Cache-Control: no-cache" --data "@input.txt" "http://api-dev.freme-project.eu/current/e-entity/freme-ner/documents?language=en&dataset=dbpedia&mode=spot%2Clink&nif-version=2.1" > output.txt

curl -X POST --header "Content-Type: text/html" --header "Accept: text/html" --header "Cache-Control: no-cache" --data "@output.txt" "http://api-dev.freme-project.eu/current/e-terminology/tilde?source-lang=en&target-lang=de&nif-version=2.1" > out-output.txt

where input.txt is a file whose content type is text/html and output.txt is a file with content-type text/html (the output of the first request) and it is sent as input to the second request.
The files are attached below.
I don't have the error. Can you try again now?

[katia-vistatec]

input.txt
output.txt
out-output.txt

[jnehring]

The error happens when executing the pipeline. I could not reproduce it using individual curl commands. The pipeline does not convert from html -> turtle -> html in every step. The pipeline converts from html -> turtle in the beginning, then it performs all pipeline steps with turtle and in the end it converts back to html. So the CURL commands are

curl -X POST --header "Content-Type: text/html" --header "Accept: text/turtle" --header "Cache-Control: no-cache" --data "@input.txt" "http://api-dev.freme-project.eu/current/e-entity/freme-ner/documents?language=en&dataset=dbpedia&mode=spot%2Clink&nif-version=2.1" > output.txt

curl -X POST --header "Content-Type: text/turtle" --header "Accept: text/turtle" --header "Cache-Control: no-cache" --data "@output.txt" "http://api-dev.freme-project.eu/current/e-terminology/tilde?source-lang=en&target-lang=de&nif-version=2.1"

Executing the two API requests one after another, it works. So I think the problem happens when the output HTML is created because we cannot reproduce this behaviour using separate curl requests.

[katia-vistatec]

Basing on the Log it is java.lang.NumberFormatException that occurs in HTMLBackConverter.java so in the second step of the pipeline when calling terminology and in particular when getting the begin index of "//freme-project.eu/#offset_38_45".
I think there's some problem with the nif-version. So even if it is nif-version = 2.1, the parameter it is not received correctly and it defaults to version 2.0. So when it parses a nif version 2.1 thinking it's nif 2.0 it fails with number format exception when trying to get the begin index because it uses the wrong identifier. Maybe it is possible to add some log to verify the nif version.

[katia-vistatec]

I debugged locally using this curl:

curl -X POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" -H "Postman-Token: cc799c16-5d39-accf-b81d-1aa4a48fb5c9" --data "@json.txt" "http://localhost:8080/pipelining/chain"

and with the json.txt attached (see below the attachement) in which I use http://localhost:8080/e-terminology/tilde as the endpoint.
I found that the nif version parameter that arrives to the InternationalizationAPI.java method Reader convertBack(InputStream markupsFile, InputStream enrichedFile, String nifVersion) is null. This creates the above described problem since a nif 2.1 is handled as it were a nif 2.0 (when no value is set for nif-version parameter, the version defaults to 2.0 ), so the string freme-project.eu/#offset_38_45 is not parsed correctly causing the errors.

[katia-vistatec]

json.txt

[jnehring]

Thanks for the investigation. This is a tough bug. The pipeline itself has no idea of the nif version. We can only guess the nif version by analyzing the nif content. I chose another solution. I scan all pipeline requests and if one of the requests contains a parameter "nif-version" then I submit this nif version to e-internationalization. This implementation does not fix the bug currently, I need to debug it once again. Will do it on monday.

But I do not like this solution. Guessing the nif version from the content might be better. @m1ci do you know of an implementation that guesses the nif version that we can reuse here?

[m1ci]

@m1ci do you know of an implementation that guesses the nif version that we can reuse here?

yes, see
https://api-dev.freme-project.eu/current/e-entity/freme-ner/documents?language=en&dataset=dbpedia&mode=all&outformat=turtle&informat=text&input=Diego%20Maradona%20is%20from%20Argentina.&nif-version=2.1

In the RDF you can see

<http://freme-project.eu/#collection>
        a               nif:ContextCollection ;
        nif:hasContext  <http://freme-project.eu/#offset_0_33> ;
        <http://purl.org/dc/terms/conformsTo>
                <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/nif-core/2.1> .

which says that the context http://freme-project.eu/#offset_0_33 conforms to NIF 2.1. This should help.
Also, we agreed that default version is 2.0, and by using the nif-version parameter one can set the version to 2.0 or 2.1.