Decompression of blobs #4979
Comments
|
Sound too specific. There are dozens of different compression algorithms moreover it is not clear how particular data types should be compressed (e.g. numbers, timestamps) - in some binary format, plain text or somehow else. |
The most popular are: deflate, lzma, lzma2, brotli, zstd. Data stream type can be either determined by header, or can be provided (some compression libs allow to put raw data without any container headers).
Compression algorythms usually work on binary data. So the transformation proposed is |
|
I think this can be easily (relatively) implemented as a custom value transformer ( |
serge-rider
added
feature request
help wanted
good first issue
and removed
question
labels
|
I'd like to tackle this issue if it is still relevant. |
|
Is this issue still open? |
|
I cannot decompressing navision 2009R2, somebody can help? BLOB fields starts with magic number 0x01447C5A in nav2009R2 my fields in HEX: it is simple txt file - with text: compressed string look like: i try to decompress string with code (can see below) but unsuccefuly because error... (System.IO.InvalidDataException: "The archive entry was compressed using an unsupported compression method.") can somebody help to decompress NAV2009R2 blob? mycode: |
|
I hacked a working AttributeTransformer, patch attached. But looking at both the AttributeTransformer as well as as the BinaryFormatters, both are a little cumbersome for this task. What would help is de-coupling the decompression from the resulting formatting of the contained data, like a transparent layer (wireshark has this for buffers). A good alternative would be to provide decompression functions inside the experssion language and then the user can define a virtual column with nested decoding. like I wonder how the image decoding is done and how that would integrate. Index: plugins/org.jkiss.dbeaver.model/plugin.xml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/plugins/org.jkiss.dbeaver.model/plugin.xml b/plugins/org.jkiss.dbeaver.model/plugin.xml
--- a/plugins/org.jkiss.dbeaver.model/plugin.xml (revision 83031d74725599b49015b08877f76d10ab0e0625)
+++ b/plugins/org.jkiss.dbeaver.model/plugin.xml (date 1622723453702)
@@ -229,6 +229,19 @@
</propertyGroup>
</transformer>
+ <transformer
+ class="org.jkiss.dbeaver.model.impl.data.transformers.AutoDecompressAttributeTransformer"
+ id="org.jkiss.dbeaver.core.AutoDecompressAttributeTransformer"
+ name="AUTO_DECOMPRESS"
+ description="%AutoDecompressTransformer.general.description"
+ applyByDefault="false"
+ custom="true">
+ <type kind="BINARY"/>
+ <type kind="BINARY"/>
+ <propertyGroup label="Properties">
+ <property id="algorithm" label="%AutoDecompressTransformer.property.algorithm.label" type="string" description="%AutoDecompressTransformer.property.algorithm.description" defaultValue="auto" required="true" validValues="auto,snappy-raw,lz4-block,deflate64"/>
+ <property id="max_size" label="%AutoDecompressTransformer.property.max_size.label" type="string" description="%AutoDecompressTransformer.property.max_size.description" defaultValue="10485760" required="true" validValues="1024,102400,1048576,10485760,104857600"/>
+ </propertyGroup>
+ </transformer>
</extension>
-
</plugin>
Index: features/org.jkiss.dbeaver.runtime.feature/feature.xml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/features/org.jkiss.dbeaver.runtime.feature/feature.xml b/features/org.jkiss.dbeaver.runtime.feature/feature.xml
--- a/features/org.jkiss.dbeaver.runtime.feature/feature.xml (revision 83031d74725599b49015b08877f76d10ab0e0625)
+++ b/features/org.jkiss.dbeaver.runtime.feature/feature.xml (date 1622660780512)
@@ -45,5 +45,8 @@
<plugin id="org.apache.commons.jexl" download-size="0" install-size="0" version="0.0.0" unpack="false"/>
<plugin id="net.sf.opencsv" download-size="0" install-size="0" version="0.0.0" unpack="false"/>
<plugin id="com.google.gson" download-size="0" install-size="0" version="0.0.0"/>
-
+ <plugin id="org.apache.commons.compress" download-size="0" install-size="0" version="0.0.0"/>
+ <plugin id="org.tukaani.xz" download-size="0" install-size="0" version="0.0.0"/>
+ <!--<plugin id="org.brotli.dec" download-size="0" install-size="0" version="0.0.0"/>
+ <plugin id="com.github.luben.zstd" download-size="0" install-size="0" version="0.0.0"/>-->
</feature>
Index: features/org.jkiss.dbeaver.rcp.feature/feature.xml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/features/org.jkiss.dbeaver.rcp.feature/feature.xml b/features/org.jkiss.dbeaver.rcp.feature/feature.xml
--- a/features/org.jkiss.dbeaver.rcp.feature/feature.xml (revision 83031d74725599b49015b08877f76d10ab0e0625)
+++ b/features/org.jkiss.dbeaver.rcp.feature/feature.xml (date 1622660784204)
@@ -135,7 +135,10 @@
<plugin id="com.jcraft.jsch" download-size="0" install-size="0" version="0.0.0" unpack="false"/>
<plugin id="org.apache.commons.logging" download-size="0" install-size="0" version="0.0.0" unpack="false"/>
<plugin id="org.objectweb.asm" download-size="0" install-size="0" version="0.0.0" unpack="false"/>
-
+ <plugin id="org.apache.commons.compress" download-size="0" install-size="0" version="0.0.0"/>
+ <plugin id="org.tukaani.xz" download-size="0" install-size="0" version="0.0.0"/>
+ <!--<plugin id="org.brotli.dec" download-size="0" install-size="0" version="0.0.0"/>
+ <plugin id="com.github.luben.zstd" download-size="0" install-size="0" version="0.0.0"/>-->
<!-- OS dependent -->
<!--
Index: features/org.jkiss.dbeaver.standalone.feature/feature.xml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/features/org.jkiss.dbeaver.standalone.feature/feature.xml b/features/org.jkiss.dbeaver.standalone.feature/feature.xml
--- a/features/org.jkiss.dbeaver.standalone.feature/feature.xml (revision 83031d74725599b49015b08877f76d10ab0e0625)
+++ b/features/org.jkiss.dbeaver.standalone.feature/feature.xml (date 1622660780508)
@@ -26,6 +26,10 @@
<import plugin="org.apache.commons.logging" match="greaterOrEqual"/>
<import plugin="com.jcraft.jsch" match="greaterOrEqual"/>
<import plugin="org.objectweb.asm" match="greaterOrEqual"/>
+ <import plugin="org.apache.commons.compress" match="greaterOrEqual"/>
+ <!--<import plugin="org.brotli.dec" match="greaterOrEqual"/>
+ <import plugin="com.github.luben.zstd" match="greaterOrEqual"/>-->
+ <import plugin="org.tukaani.xz" match="greaterOrEqual"/>
</requires>
<includes id="org.jkiss.dbeaver.rcp.feature" version="0.0.0"/>
Index: features/org.jkiss.dbeaver.appstore.feature/feature.xml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/features/org.jkiss.dbeaver.appstore.feature/feature.xml b/features/org.jkiss.dbeaver.appstore.feature/feature.xml
--- a/features/org.jkiss.dbeaver.appstore.feature/feature.xml (revision 83031d74725599b49015b08877f76d10ab0e0625)
+++ b/features/org.jkiss.dbeaver.appstore.feature/feature.xml (date 1622660788988)
@@ -26,6 +26,10 @@
<import plugin="org.apache.commons.logging" match="greaterOrEqual"/>
<import plugin="com.jcraft.jsch" match="greaterOrEqual"/>
<import plugin="org.objectweb.asm" match="greaterOrEqual"/>
+ <import plugin="org.apache.commons.compress" match="greaterOrEqual"/>
+ <!--<import plugin="org.brotli.dec" match="greaterOrEqual"/>
+ <import plugin="com.github.luben.zstd" match="greaterOrEqual"/> -->
+ <import plugin="org.tukaani.xz" match="greaterOrEqual"/>
</requires>
<includes id="org.jkiss.dbeaver.rcp.feature" version="0.0.0"/>
Index: plugins/org.jkiss.dbeaver.model/src/org/jkiss/dbeaver/model/impl/data/transformers/AutoDecompressAttributeTransformer.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/plugins/org.jkiss.dbeaver.model/src/org/jkiss/dbeaver/model/impl/data/transformers/AutoDecompressAttributeTransformer.java b/plugins/org.jkiss.dbeaver.model/src/org/jkiss/dbeaver/model/impl/data/transformers/AutoDecompressAttributeTransformer.java
new file mode 100644
--- /dev/null (date 1622723520926)
+++ b/plugins/org.jkiss.dbeaver.model/src/org/jkiss/dbeaver/model/impl/data/transformers/AutoDecompressAttributeTransformer.java (date 1622723520926)
@@ -0,0 +1,92 @@
+package org.jkiss.dbeaver.model.impl.data.transformers;
+
+import org.apache.commons.compress.compressors.CompressorException;
+import org.apache.commons.compress.compressors.CompressorInputStream;
+import org.apache.commons.compress.compressors.CompressorStreamFactory;
+import org.jkiss.dbeaver.DBException;
+import org.jkiss.dbeaver.model.DBPDataKind;
+import org.jkiss.dbeaver.model.data.DBDAttributeBinding;
+import org.jkiss.dbeaver.model.data.DBDAttributeTransformer;
+import org.jkiss.dbeaver.model.data.DBDDisplayFormat;
+import org.jkiss.dbeaver.model.data.DBDValueHandler;
+import org.jkiss.dbeaver.model.exec.DBCSession;
+import org.jkiss.dbeaver.model.impl.data.ProxyValueHandler;
+import org.jkiss.dbeaver.model.impl.jdbc.data.JDBCContentBytes;
+import org.jkiss.dbeaver.model.struct.DBSTypedObject;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+public class AutoDecompressAttributeTransformer implements DBDAttributeTransformer {
+ public void transformAttribute(
+ DBCSession session,
+ DBDAttributeBinding attribute,
+ List<Object[]> rows,
+ Map<String, Object> options
+ ) throws DBException {
+
+ attribute.setPresentationAttribute(
+ new TransformerPresentationAttribute(attribute, "COMPRESSED", -1, DBPDataKind.BINARY));
+
+ String algorithm = options.getOrDefault("algorithm", "auto").toString().trim().toLowerCase();
+ int maxSize = Integer.parseInt(options.getOrDefault("max_size", "10485760").toString().trim());
+ attribute.setTransformHandler(new AutoDecompressAttributeTransformer.AutoDecompressHandler(
+ attribute.getValueHandler(),
+ algorithm,
+ maxSize
+ ));
+
+
+ }
+
+ private static class AutoDecompressHandler extends ProxyValueHandler {
+ private static final int MAX_SIZE = 10*1024*1024;
+ private final String algorithm;
+ private final int maxSize;
+
+ public AutoDecompressHandler(DBDValueHandler target, String algorithm, int maxSize) {
+ super(target);
+ this.algorithm = algorithm;
+ this.maxSize = maxSize;
+ }
+
+ @Override
+ public String getValueDisplayString(DBSTypedObject column, Object value, DBDDisplayFormat format) {
+ byte[] bytes = null;
+ if (value instanceof byte[]) {
+ bytes = (byte[]) value;
+ } else if (value instanceof JDBCContentBytes) {
+ bytes = ((JDBCContentBytes) value).getRawValue();
+ }
+ try(InputStream rawStream = new ByteArrayInputStream(bytes)) {
+ try (CompressorInputStream cin = makeStream(rawStream)) {
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ byte[] buf = new byte[32];
+ int read = 0;
+ do {
+ out.write(buf, 0, read);
+ read = cin.read(buf);
+ } while(read > -1 && out.size() < this.maxSize);
+ return new String(out.toByteArray(), StandardCharsets.UTF_8);
+ }
+ } catch (Exception e) {
+ return "UNABLE TO DECOMPRESS: "+e.getMessage()+":::"+new String(bytes);
+ }
+ }
+
+ private CompressorInputStream makeStream(InputStream plain) throws CompressorException {
+ CompressorStreamFactory fac = new CompressorStreamFactory();
+
+ if(fac.getCompressorInputStreamProviders().containsKey(this.algorithm)) {
+ return fac.createCompressorInputStream(this.algorithm, plain, true);
+ } else {
+ return fac.createCompressorInputStream(CompressorStreamFactory.detect(plain), plain, true);
+ }
+ }
+ }
+}
Index: plugins/org.jkiss.dbeaver.model/META-INF/MANIFEST.MF
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/plugins/org.jkiss.dbeaver.model/META-INF/MANIFEST.MF b/plugins/org.jkiss.dbeaver.model/META-INF/MANIFEST.MF
--- a/plugins/org.jkiss.dbeaver.model/META-INF/MANIFEST.MF (revision 83031d74725599b49015b08877f76d10ab0e0625)
+++ b/plugins/org.jkiss.dbeaver.model/META-INF/MANIFEST.MF (date 1622660800127)
@@ -94,6 +94,8 @@
org.jkiss.utils;visibility:=reexport,
com.google.gson,
org.bouncycastle.bcpkix;resolution:=optional,
- org.bouncycastle.bcprov;resolution:=optional
+ org.bouncycastle.bcprov;resolution:=optional,
+ org.apache.commons.compress,
+ org.tukaani.xz
Bundle-Localization: OSGI-INF/l10n/bundle
Automatic-Module-Name: org.jkiss.dbeaver.model
Index: plugins/org.jkiss.dbeaver.model/OSGI-INF/l10n/bundle.properties
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/plugins/org.jkiss.dbeaver.model/OSGI-INF/l10n/bundle.properties b/plugins/org.jkiss.dbeaver.model/OSGI-INF/l10n/bundle.properties
--- a/plugins/org.jkiss.dbeaver.model/OSGI-INF/l10n/bundle.properties (revision 83031d74725599b49015b08877f76d10ab0e0625)
+++ b/plugins/org.jkiss.dbeaver.model/OSGI-INF/l10n/bundle.properties (date 1622723403959)
@@ -172,4 +172,11 @@
BinaryTransformer.property.format.label = Binary format
BinaryTransformer.property.format.description = Binary value format.
BinaryTransformer.property.encoding.label = Character encoding
-BinaryTransformer.property.encoding.description = Character encoding used to decode string value into bytes.
\ No newline at end of file
+BinaryTransformer.property.encoding.description = Character encoding used to decode string value into bytes.
+
+
+AutoDecompressTransformer.general.description = Decompress binary data by detecting header
+AutoDecompressTransformer.property.algorithm.label = Algorithm
+AutoDecompressTransformer.property.algorithm.description = Compression algorithm
+AutoDecompressTransformer.property.max_size.label = Max Size
+AutoDecompressTransformer.property.max_size.description = De-Compress up to N bytes before stopping
\ No newline at end of file |
lambdaupb
added a commit
to lambdaupb/dbeaver
that referenced
this issue
Jun 3, 2021
|
Thank you, @lambdaupb |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
Sometimes it is needed to store compressed data in the DB. Unfortunately not all the DBs have built-in compression and FUSE compressed FSes are not available for every OS. So it may make sense to store compressed binary blobs in the DB.
Unfortunately when one sees them in DBeaver he sees them compressed, but often they are needed uncompressed. So it'd be nice to have a feature to decompress the contents of certain columns.
This may be done by injecting an external function into SQL doing decompression.
The text was updated successfully, but these errors were encountered: