Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Newlines in data exported from sql query #3222

Open
CaptainStabs opened this issue Apr 13, 2022 · 6 comments
Open

Newlines in data exported from sql query #3222

CaptainStabs opened this issue Apr 13, 2022 · 6 comments
Labels
bug good first issue

Comments

@CaptainStabs
Copy link

@CaptainStabs CaptainStabs commented Apr 13, 2022

Using dolt sql -q "<query>" -r csv > file.csv results in newlines in the output data:
image
@timsehn
Copy link
Sponsor Contributor

@timsehn timsehn commented Apr 13, 2022

It really helps if you provide the query and database you were running it on so we can repro.

@CaptainStabs
Copy link
Author

@CaptainStabs CaptainStabs commented Apr 14, 2022 •
edited

Sorry, hospital-price-transparency-v3, and

select * from prices where code like '%,%' and cms_certification_num in ( '050125', '060014', '060032', '060034', '060065', '060100', '060112', '110003', '050380', '110036', '050481', '110125', '020017', '110128', '141335', '050549', '110177', '063027', '150046', '100023', '100054', '100080', '100110', '100126', '100131', '100156', '100161', '100166', '100179', '100180', '100189', '100191', '100204', '100206', '100209', '100212', '100213', '100223', '100224', '100226', '100228', '100231', '100232', '100236', '100238', '100242', '100243', '100246', '100248', '100252', '100254', '100256', '100259', '100260', '100264', '100267', '100269', '100320', '050022', '260027', '130014', '130018', '170176', '170182', '180124', '180127', '190026', '190176', '260095', '260214', '261320', '290003', '290039', '290047', '300014', '300017', '300029', '450087', '450097', '450203', '450222', '450403', '450431', '450617', '450634', '450638', '450644', '450647', '450651', '450659', '450662', '450669', '450670', '450672', '450674', '450675', '450711', '450713', '450718', '450775', '450788', '450804', '450809', '450822', '450853', '420030', '420079', '420085', '440006', '440046', '440065', '440150', '440161', '440194', '440197', '440227', '441311', '454094', '460005', '460013', '460017', '460042', '460047', '460052', '460054', '460060', '490020', '490048', '490107', '490110', '490112', '490116', '490118', '490126', '490141', '490145', '494023', '530034', '670103', '670106');

with dolt sql -q "<above query>" -r csv > file.csv

@fulghum
Copy link
Contributor

@fulghum fulghum commented Apr 14, 2022

A much simpler repro is:

select * from prices where code like 'CPT/HCPC 27415,%' and cms_certification_num = '450670' and payer='Aetna' limit 1;

I see the newlines in both the CSV output and when directly querying through the SQL shell. This doesn't seem like an issue with Dolt, but with the hospital-price-transparency-v3 dataset. Was this causing an issue for you when processing or working with the data?

@alecstein – I'll let you see if we need to do anything for the data. At minimum, it seems like having a check that validates no newlines are in certain fields would be useful.

@CaptainStabs
Copy link
Author

@CaptainStabs CaptainStabs commented Apr 14, 2022

A much simpler repro is:

select * from prices where code like 'CPT/HCPC 27415,%' and cms_certification_num = '450670' and payer='Aetna' limit 1;

I see the newlines in both the CSV output and when directly querying through the SQL shell. This doesn't seem like an issue with Dolt, but with the hospital-price-transparency-v3 dataset. Was this causing an issue for you when processing or working with the data?

@alecstein – I'll let you see if we need to do anything for the data. At minimum, it seems like having a check that validates no newlines are in certain fields would be useful.

Yes, it was a huge pain to fix...

@timsehn
Copy link
Sponsor Contributor

@timsehn timsehn commented Apr 15, 2022

I think the solution is we have to quote the value if it contains a newline.

@zachmu zachmu added bug good first issue labels May 5, 2022
@zachmu
Copy link
Member

@zachmu zachmu commented May 5, 2022

Is this a valid CSV file? If so, I don't think this is a bug.

We should have a test that we can round-trip strings with newlines into dolt via dolt sql -r csv and dolt import.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug good first issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@fulghum @zachmu @timsehn @alecstein @CaptainStabs