Pause on this. It won't work on Windows (ref libuv/libuv#1236 (comment)). So it appears that len on Windows is an unsigned long. So internally it'll need to manually write large buffers in smaller chunks.

@trevnorris are you going to follow up on this?

@BridgeAR Forgot I had started this. It stalled b/c I wanted to see about changing uv_buf_init(char*, unsigned int) to uv_buf_init(char*, size_t), but that's not ABI compatible and won't land in node. Thus len will need to be changed manually.

I'll try to get this rebased and fixing again in the near future. Unless you are asking b/c you want to pick this up and run with it. :-)

@trevnorris as time is limited for all of us, I was not planning on following up on this currently. So it would be nice if you find the time to rebase this. In that case this could be properly reviewed.

@BridgeAR Will do.

@BridgeAR Has been rebased and running new CI

CI: https://ci.nodejs.org/job/node-test-pull-request/10084/

I think it may make sense to expand this at some point to support any ArrayBufferView (TypedArray or DataView). But for now, this LGTM

@jasnell This is about allowing ArrayBuffers, not ArrayBufferViews; the latter would actually be a lot easier to do.

I don’t think the added complexity here justifies the feature (esp. given that it’s perfectly reasonably implementable in userland and not a common use case), but then again it’s not a huge problem either.

Yes, I know.

@addaleax

I don’t think the added complexity here justifies the feature (esp. given that it’s perfectly reasonably implementable in userland and not a common use case), but then again it’s not a huge problem either.

I'd like to have full ArrayBufferView support, but doing so would have the same amount of complexity. Consider the fact that a Float64Array supports 8 GB, the additional read/write loops are necessary. This is meant to be a step in full ArrayBufferView support.

EDIT: I agree that the added complexity sucks, but it's not that much, and if you see a way to simplify it I'll happily update the PR.

@trevnorris what do we need for full ArrayBufferView support beyond looser typechecking?

@addaleax read/write is currently broken. Example writing:

const fs = require('fs');
const file_path = '/tmp/buf_file.blah';

const buf = Buffer.alloc(require('buffer').kMaxLength, 1);
const fd = fs.openSync(file_path, 'w+');
const written = fs.writeSync(fd, buf, 0, buf.byteLength, 0);

console.log(fs.statSync(file_path).size, buf.byteLength);
// output: 2147479552 2147483647

now for reading. first create the file:

$ fallocate -l 2147483647 /tmp/buf_file.blah

now reading the file:

const fs = require('fs');
const file_path = '/tmp/buf_file.blah';
const buf = Buffer.allocUnsafe(fs.statSync(file_path).size);

const fd = fs.openSync(file_path, 'r+');
const bytes = fs.readSync(fd, buf, 0, buf.byteLength, 0);
console.log(bytes, buf.byteLength);
// output: 2147479552 2147483647

We've been ignoring a documented kernel limitation:

On Linux, write() (and similar system calls) will transfer at most
0x7ffff000 (2,147,479,552) bytes, returning the number of bytes
actually transferred. (This is true on both 32-bit and 64-bit
systems.)

This PR fixes that and adds new functionality.

EDIT: @addaleax I was taking the opinion that it should always write all the data to the file unless there was an error because I'd expect the same of doing a TCP write, but guess they don't work that way. node is just so cozy and handles all that hard stuff for me so often that I became lazy.

@santigimeno mentioned the libuv bug libuv/libuv#1501 which we'll need to work around. if the libuv fix doesn't land soon i'll change the PR so the maximum data written at a single time is 2GB.

Closing this after having a discussion on IRC. To summarize: I had expected fs.read() and fs.write() to handle writing out all the data for me. Instead it's actually the users responsibility to make sure to check the number of bytes read/written and loop until the operation is complete. Reason I thought fs.read() and fs.write() would handle it for me is because other operations like fs.readFile(), fs.writeFile() or net.Socket#write() handle sending everything when too much data is sent.

So, instead we'll need a PR for updating the docs with language to clarify that it's the user's responsibility to check the return value and make sure all the data has been read/written.