Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Codeql database create fails when building mozilla #16001

Open
mies47 opened this issue Mar 21, 2024 · 4 comments
Open

Codeql database create fails when building mozilla #16001

mies47 opened this issue Mar 21, 2024 · 4 comments
Labels
question Further information is requested

Comments

@mies47
Copy link

mies47 commented Mar 21, 2024

I'm trying to build mozilla from this repo and create a cpp codeql database.

In order to that, I first run ./mach configure and after it's done, I run the following command to create the database:

~/codeql/codeql database create mozilla4 --language=c-cpp --command="./mach build"

Which builds mozilla successfully, but it fails in the importing trap file stage. I have included the logs for creating the database and importing dataset.

A glance of the errors is like:

[ERROR] 14135855_0.trap.br, 1: java.io.IOException: Brotli stream decoding failed
                              org.brotli.dec.BrotliInputStream.read(BrotliInputStream.java:167)
                              com.semmle.inmemory.trap.TrapInputStream.read(TrapInputStream.java:60)
                              com.semmle.inmemory.trap.TrapScanner.fill(TrapScanner.java:451)
                              com.semmle.inmemory.trap.TrapScanner.ensureNext(TrapScanner.java:428)
                              com.semmle.inmemory.trap.TrapScanner.nextToken(TrapScanner.java:61)
                              com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:493)
                              com.semmle.inmemory.trap.TRAPLinker$TrapLinkDirectiveScanner.scanTuplesAndLabels(TRAPLinker.java:311)
                              com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:414)
                              com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:400)
                              com.semmle.inmemory.trap.TRAPLinker.lambda$getTasks$2(TRAPLinker.java:215)
                              com.semmle.util.concurrent.FutureUtils.lambda$mapAsync_$8(FutureUtils.java:161)
                              java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
                              java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
                              java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                              java.base/java.lang.Thread.run(Unknown Source)

                               ... caused by:

                              org.brotli.dec.BrotliRuntimeException: Corrupted Huffman code histogram
                              org.brotli.dec.Decode.readComplexHuffmanCode(Decode.java:591)
                              org.brotli.dec.Decode.readHuffmanCode(Decode.java:613)
                              org.brotli.dec.Decode.readMetablockPartition(Decode.java:783)
                              org.brotli.dec.Decode.readMetablockHuffmanCodesAndContextMaps(Decode.java:825)
                              org.brotli.dec.Decode.decompress(Decode.java:1110)
                              org.brotli.dec.BrotliInputStream.read(BrotliInputStream.java:162)
                              com.semmle.inmemory.trap.TrapInputStream.read(TrapInputStream.java:60)
                              com.semmle.inmemory.trap.TrapScanner.fill(TrapScanner.java:451)
                              com.semmle.inmemory.trap.TrapScanner.ensureNext(TrapScanner.java:428)
                              com.semmle.inmemory.trap.TrapScanner.nextToken(TrapScanner.java:61)
                              com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:493)
                              com.semmle.inmemory.trap.TRAPLinker$TrapLinkDirectiveScanner.scanTuplesAndLabels(TRAPLinker.java:311)
                              com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:414)
                              com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:400)
                              com.semmle.inmemory.trap.TRAPLinker.lambda$getTasks$2(TRAPLinker.java:215)
                              com.semmle.util.concurrent.FutureUtils.lambda$mapAsync_$8(FutureUtils.java:161)
                              java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
                              java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
                              java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                              java.base/java.lang.Thread.run(Unknown Source)
                              at (start of line)

I can build mozilla separately following similar steps, but when I try to create a database it fails. The database is created in the end, but some of the files are missing and are not added to the database.

Can someone please point me to the right direction on how to solve this issue?
@mies47 mies47 added the question Further information is requested label Mar 21, 2024
@smowton
Copy link
Contributor

smowton commented Mar 21, 2024

Could you zip up and upload the database directory? It will contain the logs of what is probably a crash midway through writing a compressed trap file.

@mies47
Copy link
Author

mies47 commented Mar 21, 2024

Could you zip up and upload the database directory? It will contain the logs of what is probably a crash midway through writing a compressed trap file.

Of course. Here's the zip of database directory. Thanks for your help

@jketema
Copy link
Contributor

jketema commented Mar 21, 2024 •
edited

It seems that there are indeed around 360 files on which we crash, as @smowton already suspected. These seem internal errors in the C/C++ frontend that we use. You might want to re-try with CodeQL 2.16.4 or later, as that includes some C/C++ frontend improvements, which may solve your issues. Note that this concerns less than 10% of source files being compiled, so you still should have a fairly complete database.

@mies47
Copy link
Author

mies47 commented Mar 22, 2024

It seems that there are indeed around 360 files on which we crash, as @smowton already suspected. These seem internal errors in the C/C++ frontend that we use. You might want to re-try with CodeQL 2.16.4 or later, as that includes some C/C++ frontend improvements, which may solve your issues. Note that this concerns less than 10% of source files being compiled, so you still should have a fairly complete database.

Thanks for your suggestion. I tried it again with CodeQL 2.16.5 and I think it worked better but the issue persists.
I agree that it compiles most of the code base, but I'm trying to run a query on the IPC related files and those are the files that are missing.
To confirm my guess that IPC related files are not getting completely imported I tried running a query to find the files with names that match "%Child.cpp" or "%Parent.cpp" and this is the results:

"PFetchParent.cpp"
"PBackgroundIDBCursorChild.cpp"
"PBackgroundIDBCursorParent.cpp"
"PBackgroundIDBDatabaseChild.cpp"
"PBackgroundIDBDatabaseFileChild.cpp"
"PBackgroundIDBDatabaseFileParent.cpp"
"PBackgroundIDBDatabaseParent.cpp"
"SessionStoreChild.cpp"
"SessionStoreParent.cpp"
"PSessionStoreChild.cpp"
"PSessionStoreParent.cpp"
"GPUParent.cpp"
"VRLayerParent.cpp"
"ContentChild.cpp"
"RemoteDecoderManagerChild.cpp"
"ActorsParent.cpp"
"HeapSnapshotTempFileHelperParent.cpp"
"TestShellChild.cpp"
"VsyncParent.cpp"
"VsyncMainChild.cpp"
"RemoteDecoderManagerParent.cpp"
"VsyncWorkerChild.cpp"
"RDDParent.cpp"
"RemoteDecoderChild.cpp"
"TestShellParent.cpp"
"ProxyAutoConfigParent.cpp"
"ProxyAutoConfigChild.cpp"
"RemoteDecoderParent.cpp"
"RDDChild.cpp"

Which is way lesser files than what it should be. Do you think there's a way I could fix this? I appreciate your help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@smowton @mies47 @jketema