GET /formList from Docker image fails with error from openros

[hblanks]

I've built a local Docker image from current master and the latest tag (v0.11.0) with docker build -t omk ., and then run it with:

mkdir -p /tmp/data/submissions /tmp/data/forms /tmp/data/deployments
docker run --rm -p 8080:3210 -v /tmp/data:/app/data --name omk omk

but although I'm able to upload the OMK sample XLS form (https://docs.google.com/spreadsheets/d/11H4-mGYTS61GLjSbVoTbmhoI5DjlF5fcBwNwQcvd2Go/edit?usp=sharing) or start from a fresh data dir mirroring that of data/ from the repo, in all cases I'm getting a 500 for HTTP GET /formList?json=true with the following traceback:

Error: not well-formed (invalid token)           
    at /app/api/odk/controllers/get-formlist.js:38:23                                             
    at buildXml (/app/node_modules/openrosa-formlist/index.js:77:21)                              
    at /app/node_modules/openrosa-formlist/node_modules/async/lib/async.js:251:17                 
    at done (/app/node_modules/openrosa-formlist/node_modules/async/lib/async.js:126:15)          
    at /app/node_modules/openrosa-formlist/node_modules/async/lib/async.js:32:16                  
    at Parser.<anonymous> (/app/node_modules/openrosa-formlist/node_modules/async/lib/async.js:248:21)
    at emitOne (events.js:101:20)                
    at Parser.emit (events.js:188:7)             
    at Parser.write (/app/node_modules/node-expat/lib/node-expat.js:68:10)                        
    at Stream.ondata (internal/streams/legacy.js:16:26) 

Is this a version mismatch perhaps with openrosa?

Or, more generally, might you all have any advice on how to proceed? (And sorry to post on issues; couldn't find a mailing list / slack channel / IRC)

[sethfitz]

Nah, this is a great issue.

It looks like a port problem; I get this error instead (do you have other services running?):

Error: connect ECONNREFUSED 127.0.0.1:8080
    at Object.exports._errnoException (util.js:1020:11)
    at exports._exceptionWithHostPort (util.js:1043:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1099:14)

The way /formList is implemented, it makes an HTTP request to itself based on the URL provided in the request (this has caused problems in other contexts, but is generally convenient). This means that it's making a request to http://localhost:8080/... but OMK Server is listening on :3210 in that context.

You should be able to work around the problem by starting OMK with:

docker run -e PORT=8080 --rm -p 8080:8080 -v /tmp/data:/app/data --name omk omk

[hblanks]

Thanks for the note! lsof -i tcp | grep LIST shows no other services running on this machine except SSH and local SMTP. I haven't seen the traceback you've got there, just the one above.

FWIW, I get the same traceback with-e PORT 8080:

~$ docker run --rm -e PORT=8080 -p 8080:8080 -v /tmp/data:/app/data --name omk omk                                                    
OpenMapKit Server is listening on port 8080.                             
cError: not well-formed (invalid token)          
    at /app/api/odk/controllers/get-formlist.js:38:23                                             
    at buildXml (/app/node_modules/openrosa-formlist/index.js:77:21)                              
    at /app/node_modules/openrosa-formlist/node_modules/async/lib/async.js:251:17                 
    at done (/app/node_modules/openrosa-formlist/node_modules/async/lib/async.js:126:15)          
    at /app/node_modules/openrosa-formlist/node_modules/async/lib/async.js:32:16                  
    at Parser.<anonymous> (/app/node_modules/openrosa-formlist/node_modules/async/lib/async.js:248:21)                    
    at emitOne (events.js:101:20)                
    at Parser.emit (events.js:188:7)             
    at Parser.write (/app/node_modules/node-expat/lib/node-expat.js:68:10)                        
    at Stream.ondata (internal/streams/legacy.js:16:26)                                           

and when using the default port:

$ docker run --rm -p 3210:3210 -v /tmp/data:/app/data --name omk omk
OpenMapKit Server is listening on port 3210.
Error: not well-formed (invalid token)
    at /app/api/odk/controllers/get-formlist.js:38:23
    at buildXml (/app/node_modules/openrosa-formlist/index.js:77:21)
    at /app/node_modules/openrosa-formlist/node_modules/async/lib/async.js:251:17
    at done (/app/node_modules/openrosa-formlist/node_modules/async/lib/async.js:126:15)
    at /app/node_modules/openrosa-formlist/node_modules/async/lib/async.js:32:16
    at Parser.<anonymous> (/app/node_modules/openrosa-formlist/node_modules/async/lib/async.js:248:21)
    at emitOne (events.js:101:20)
    at Parser.emit (events.js:188:7)
    at Parser.write (/app/node_modules/node-expat/lib/node-expat.js:68:10)
    at Stream.ondata (internal/streams/legacy.js:16:26)

To reproduce this, I:

1. Start the container
2. Upload the buildings_example.xlsx file linked above.
3. Click to view the list of forms.

It also seems to reproduce if I:

1. Copy the data/ directory from the repo into the docker volume directory.
2. Start the container
3. Click to view the list of forms.

I'll try building from a VM next and let you know if it replicates there, too. Thanks again for the help!

[sethfitz]

Can you paste the output of this (when using the default port):

curl -H "Accept: application/json" http://localhost:3210/omk/data/forms/

(These are the requests made internally to produce the form list.)

It should look like

["buildings_example.xlsx","buildings_example.xml"]

Ditto for

curl http://localhost:3210/omk/data/forms/buildings_example.xml

and

curl http://localhost:3210/omk/odk/manifest/buildings_example.xml

[hblanks]

For this reply, the command, run on host leaf, was:

docker run -p 3210:3210 --rm -v /tmp/data:/app/data --name omk omk

GET /omk/data/forms works locally:

omkserver@cadd07fc7452:/app$ curl -H "Accept: application/json" http://localhost:3210/omk/data/forms/
["buildings_example.xlsx","buildings_example.xml"]
omkserver@cadd07fc7452:/app$ exit
$ curl -H "Accept: application/json" http://localhost:3210/omk/data/forms/
["buildings_example.xlsx","buildings_example.xml"]

And remotely, too:

$ curl -H "Accept: application/json" http://leaf:3210/omk/data/forms/
["buildings_example.xlsx","buildings_example.xml"]

So does GET /omk/data/forms/:form_name:

$ curl http://localhost:3210/omk/data/forms/buildings_example.xml -s | head
<?xml version="1.0"?>
<h:html xmlns="http://www.w3.org/2002/xforms" xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:h="http://www.w3.org/1999/xhtml" xmlns:jr="http://openrosa.org/javarosa" xmlns:orx="http://openrosa.org/xforms" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <h:head>
    <h:title>buildings_example</h:title>
    <model>
      <instance>
        <buildings_example id="buildings_example">
          <start/>
          <end/>
          <today/>

and GET /omk/odk/manifest/:form_name:

$ curl http://localhost:3210/omk/odk/manifest/buildings_example.xml  -s | head
<?xml version="1.0" encoding="UTF-8"?>
<manifest xmlns="http://openrosa.org/xforms/xformsManifest">
  <#list/>

and indeed, locally, the formlist works!:

curl 'http://localhost:3210/formList?json=true'
{"xforms":{"attributes":{"xmlns":"http://openrosa.org/xforms/xformsList"},"xform":[{"name":"buildings_example","formID":"buildings_example","hash":"md5:53083d7a1356f9903e560006c526c7a2","downloadUrl":"http://localhost:3210/omk/data/forms/buildings_example.xml","totalSubmissions":0}]}}

but, still locally, not with the machine's hostname:

$ curl 'http://leaf:3210/formList?json=true'
{"status":500,"msg":"not well-formed (invalid token)","err":{}}

I did verify that the endpoint is accessible by hostname within the
container:

$ docker exec omk
omkserver@cadd07fc7452:/app$ curl 'http://leaf:3210/formList?json=true'
{"status":500,"msg":"not well-formed (invalid token)","err":{}}

Then, I thought, maybe this has something to do with the HTTP host
header? So I ran:

$ curl -H "Host: localhost" 'http://leaf:3210/formList?json=true'
{"status":500,"msg":"connect ECONNREFUSED 127.0.0.1:80","err":{"code":"ECONNREFUSED","errno":"ECONNREFUSED","syscall":"connect","address":"127.0.0.1","port":80}}

and finally got the error you were talking about. But, this doesn't seem
like the issue, since with `Host: localhost:3210" we're back to a
working response:

$ curl -H "Host: localhost:3210" 'http://leaf:3210/formList?json=true'
{"xforms":{"attributes":{"xmlns":"http://openrosa.org/xforms/xformsList"},"xform":[{"name":"buildings_example","formID":"buildings_example","hash":"md5:53083d7a1356f9903e560006c526c7a2","downloadUrl":"http://localhost:3210/omk/data/forms/buildings_example.xml","totalSubmissions":0}]}}

So, this seems to have something to do not with connectivity back to the
server in the container, but with the hostname being different than
expected after the connection's been made. Just my guess, though.

[sethfitz]

Can you make the same requests for OMK forms + manifests using the leaf hostname?

I've had moderate success in other contexts starting Docker w/ something similar to --add-host leaf:127.0.0.1, which could help with resolution.