Replies: 4 comments 10 replies
-
|
You already can do the following with simdjson::ondemand::parser parser;
simdjson::padded_string docdata = R"({"value":"12321323213213213213213213213211223"})"_padded;
simdjson::ondemand::document doc = parser.iterate(docdata);
simdjson::ondemand::object obj = doc.get_object();
string_view token = obj["value"].raw_json_token();
// token has value "12321323213213213213213213213211223", it points inside the input string |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for answering How can I avoid this allocation? |
Beta Was this translation helpful? Give feedback.
-
|
@kirillp What is your operating system ? |
Beta Was this translation helpful? Give feedback.
-
|
I am developing on Windows, but it must also run on Linux backend |
Beta Was this translation helpful? Give feedback.
-
|
@kirillp Yeah, so under Windows that's an issue for you. Your best option is to have lots of RAM and reuse the parser. This being said, documents that large than are not a stream of JSON document (see our iterate_many functionality) are generally an issue. It is not a great idea to have monolithic JSON documents that are that big. You should consider switching to streams of JSON documents. |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately, it is requirements from customer... |
Beta Was this translation helpful? Give feedback.
-
|
Note that there is an open issue aiming to optimize this aspect further... |
Beta Was this translation helpful? Give feedback.
-
|
Yes I see, this issue describes string copy. The issue is reserving memory for copy of the document that I never be using. |
Beta Was this translation helpful? Give feedback.
-
|
Under Linux or macOS, it is not an issue. You are only likely to have an issue under Windows. Is it avoidable? Sure, it might be, but I don't think we support this scenario at this time. It is rather specific. |
Beta Was this translation helpful? Give feedback.
-
|
Is it possible to make the parser to reserve memory for string copies on demand ? Thanks for help |
Beta Was this translation helpful? Give feedback.
-
|
Yeah so what you have is a comma separated list of documents. We do not currently support these in an infinite stream, but we have better support coming thanks to work by @cjauvin. |
Beta Was this translation helpful? Give feedback.
-
|
@kirillp What is your specific use case ? |
Beta Was this translation helpful? Give feedback.
-
|
Open view to a huge file, parse and reorganize it, make some data filtering and analysis, then write results to new json + binary |
Beta Was this translation helpful? Give feedback.
-
|
One does not generally encounter JSON files that big. So it must be a very specific application. Large files usually come as JSON streams. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello, thank you vm for grate library.
I was surprised that get_string(std::string_view &target) always copy file content to internal buffer (even if a string doesn
t have any escaping). I have to parse very huge files (2-3-4Gb) and I'm using mapped memory to open it. I need to provide most time and memory efficient file processing. During parsing I don't need to un-escape anything, Im fine with original strings however the parser still allocates memory to store all possible strings in internal buffer.Can I control this moment somehow ?
How can I save memory and promise to avoid using get_string(std::string_view &target),
and use std::strings for that rare moments where I need to make a copy or unescape content ?
Thanks
Beta Was this translation helpful? Give feedback.
All reactions