Could not load compilationdatabase - Unknown key: ""output""

[kanihal]

I am trying to use woboq on postgres (https://github.com/postgres/postgres).
I have generated compile_commands.json using bear

cd postgres
bear -- ./configure
bear -- make 

Now when I run codebrowser_generator, I get the below-mentioned error

OUTPUT_DIRECTORY=~/public_html/postgres
DATA_DIRECTORY=$OUTPUT_DIRECTORY/../data
mkdir -p $OUTPUT_DIRECTORY
mkdir -p $DATA_DIRECTORY


BUILD_DIRECTORY=$PWD
SOURCE_DIRECTORY=$PWD
VERSION=`git describe --always --tags`

codebrowser_generator -b $BUILD_DIRECTORY -a \
-o $OUTPUT_DIRECTORY \
-p postgres:$SOURCE_DIRECTORY:$VERSION

json-compilation-database: Unknown key: ""output""

Could not load compilationdatabase. Please use the -b option to a path containing a compile_commands.json, or use '--' followed by the compilation commands.

Here is a sample entry from compile_commands.json,

  {
    "arguments": [
      "/usr/bin/x86_64-linux-gnu-gcc-10",
      "-Wall",
      "-Wmissing-prototypes",
      "-Wpointer-arith",
      "-Wdeclaration-after-statement",
      "-Werror=vla",
      "-Wendif-labels",
      "-Wmissing-format-attribute",
      "-Wimplicit-fallthrough=3",
      "-Wcast-function-type",
      "-Wformat-security",
      "-fno-strict-aliasing",
      "-fwrapv",
      "-fexcess-precision=standard",
      "-Wno-format-truncation",
      "-Wno-stringop-truncation",
      "-O2",
      "-fPIC",
      "-I.",
      "-I/home/jagadeesha/Downloads/postgres-git/src/pl/plpgsql/src",
      "-I../../../../src/include",
      "-I/home/jagadeesha/Downloads/postgres-git/src/include",
      "-D_GNU_SOURCE",
      "-c",
      "-o",
      "pl_scanner.o",
      "/home/jagadeesha/Downloads/postgres-git/src/pl/plpgsql/src/pl_scanner.c"
    ],
    "directory": "/home/jagadeesha/Downloads/postgres-git/build/src/pl/plpgsql/src",
    "file": "/home/jagadeesha/Downloads/postgres-git/src/pl/plpgsql/src/pl_scanner.c",
    "output": "/home/jagadeesha/Downloads/postgres-git/build/src/pl/plpgsql/src/pl_scanner.o"
  }

and codebrowser version details:

$ codebrowser_generator -version
LLVM (http://llvm.org/):
  LLVM version 3.8.1
  
  Optimized build.
  Built Jun  2 2017 (13:11:29).
  Default target: x86_64-pc-linux-gnu
  Host CPU: westmere

is the generated entries in compile_commands.json not supposed to contain output field?
How do I resolve this issue?

[kanihal]

@PiotrZSL @gavinchou - Any idea on how to resolve this issue?

[ogoffart]

How did you generate that compilation database?
It looks like clang's compilation database parser don't support extra key.

Also your version of LLVM is quite old, perhaps output is a recent addition. (I have no idea about that, just a guess. Upgrading clang might or might not help)

[kanihal]

@ogoffart

How did you generate that compilation database?

Since postgres is not a cmake project, I generated compile_commands.json using bear(https://github.com/rizsotto/Bear).

$ cd postgres
$ bear -- ./configure
$ bear -- make

Also your version of LLVM is quite old, ...

I had installed codebrowser_generator using .deb file linked in the woboq website: https://woboq.com/codebrowser-download.html. This is a very old build as you mentioned.

I also tried building codebrowser with LLVM-13(latest version). I get the following error while compiling,

$ cd codebrowser
$ cmake . -DCMAKE_BUILD_TYPE=Release
$ make 

[  8%] Building CXX object generator/CMakeFiles/codebrowser_generator.dir/main.cpp.o
In file included from /home/jagadeesha/Downloads/codebrowser/generator/main.cpp:39:
/home/jagadeesha/Downloads/codebrowser/generator/browserastvisitor.h: In member function ‘bool BrowserASTVisitor::VisitEnumConstantDecl(clang::EnumConstantDecl*)’:
/home/jagadeesha/Downloads/codebrowser/generator/browserastvisitor.h:132:130: error: no matching function for call to ‘llvm::APSInt::toString(int) const’
  132 | otator.registerReference(d, d->getLocation(), Annotator::EnumDecl, Annotator::Declaration, d->getInitVal().toString(10));
      |                                                                                                                       ^

In file included from /usr/lib/llvm-13/include/clang/AST/Type.h:34,
                 from /usr/lib/llvm-13/include/clang/AST/CanonicalType.h:17,
                 from /usr/lib/llvm-13/include/clang/AST/ASTContext.h:19,
                 from /usr/lib/llvm-13/include/clang/Frontend/ASTUnit.h:17,
                 from /usr/lib/llvm-13/include/clang/Frontend/FrontendAction.h:23,
                 from /usr/lib/llvm-13/include/clang/Frontend/FrontendActions.h:12,
                 from /home/jagadeesha/Downloads/codebrowser/generator/main.cpp:24:
/usr/lib/llvm-13/include/llvm/ADT/APSInt.h:82:8: note: candidate: ‘void llvm::APSInt::toString(llvm::SmallVectorImpl<char>&, unsigned int) const’
   82 |   void toString(SmallVectorImpl<char> &Str, unsigned Radix = 10) const {
      |        ^~~~~~~~
/usr/lib/llvm-13/include/llvm/ADT/APSInt.h:82:40: note:   no known conversion for argument 1 from ‘int’ to ‘llvm::SmallVectorImpl<char>&’
   82 |   void toString(SmallVectorImpl<char> &Str, unsigned Radix = 10) const {
      |                 ~~~~~~~~~~~~~~~~~~~~~~~^~~
In file included from /usr/lib/llvm-13/include/clang/AST/Type.h:33,
                 from /usr/lib/llvm-13/include/clang/AST/CanonicalType.h:17,
                 from /usr/lib/llvm-13/include/clang/AST/ASTContext.h:19,
                 from /usr/lib/llvm-13/include/clang/Frontend/ASTUnit.h:17,
                 from /usr/lib/llvm-13/include/clang/Frontend/FrontendAction.h:23,
                 from /usr/lib/llvm-13/include/clang/Frontend/FrontendActions.h:12,
                 from /home/jagadeesha/Downloads/codebrowser/generator/main.cpp:24:
/usr/lib/llvm-13/include/llvm/ADT/APInt.h:1741:8: note: candidate: ‘void llvm::APInt::toString(llvm::SmallVectorImpl<char>&, unsigned int, bool, bool) const’
 1741 |   void toString(SmallVectorImpl<char> &Str, unsigned Radix, bool Signed,
      |        ^~~~~~~~
/usr/lib/llvm-13/include/llvm/ADT/APInt.h:1741:8: note:   candidate expects 4 arguments, 1 provided
/home/jagadeesha/Downloads/codebrowser/generator/main.cpp: In function ‘bool proceedCommand(std::vector<std::__cxx11::basic_string<char> >, llvm::StringRef, llvm::StringRef, clang::FileManager*, DatabaseType)’:
/home/jagadeesha/Downloads/codebrowser/generator/main.cpp:303:15: error: ‘class clang::tooling::ToolInvocation’ has no member named ‘mapVirtualFile’
  303 |           Inv.mapVirtualFile(f->filename, {f->content , f->size } );
      |               ^~~~~~~~~~~~~~
make[2]: *** [generator/CMakeFiles/codebrowser_generator.dir/build.make:82: generator/CMakeFiles/codebrowser_generator.dir/main.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:138: generator/CMakeFiles/codebrowser_generator.dir/all] Error 2
make: *** [Makefile:149: all] Error 2

The 2 errors are:

codebrowser/generator/browserastvisitor.h:132:130: error: no matching function for call to ‘llvm::APSInt::toString(int) const’

codebrowser/generator/main.cpp:303:15: error: ‘class clang::tooling::ToolInvocation’ has no member named ‘mapVirtualFile’
303 | Inv.mapVirtualFile(f->filename, {f->content , f->size } );

[kanihal]

Workaround Fix:
I followed the instructions given in rizsotto/Bear#196 (comment) to generate compile_commands.json in the older format (i.e. without output field). codebrowser_generator worked fine with the older format.

Adding config.json that I used for reference here.

{
    "compilation": {},
    "output": {
        "content": {
            "include_only_existing_source": true
        },
        "format": {
            "command_as_array": false,
            "drop_output_field": true
        }
    }
}

As for the codebrowser build failure with LLVM-13, I have logged a new issue (#95).

[gavinchou]

@kanihal Since you've solved the db problem, I would like to share some tips on generating compilation.json. Hope it may help in the future.

For any project I'd like to compile with codebrowser, I would like to write another new CMakeList.txt for it to get full control of the source code and dependencies needed by codebrowser, even though it already has one.
Of course, I need some script to help, here is the example
https://github.com/gavinchou/codebrowser/tree/master/examples/build-project-with-template