segmentation fault with HTTP Response with big JSON body

[ishansheth]

Hi,

I am using your library and i am getting a crash inside the library when the HTTPResponse has around 78 K big JSON payload.
The crash happens at IoReaderImpl.cpp line number 33 according to gdb.

There i think the buffer is filled with payload data iteratively. On the first iteration its ok and around 16 K data is filled in the buffer. In the second iteration its failing and crashing

Would you be able to tell what is the cause of it? Is there a place where i need to configure the buffer size?

Here are the traces :

[2021-04-21 15:54:04.929226] [0x00007fec8f94b700] [trace] Created timer StartReceiveFromServer for {Connection c4996c71-4b6a-4263-9589-167b77817459 {Socket socket# 9 10.161.70.168:38894 <--> 10.161.70.150:80}}
[2021-04-21 15:54:04.929344] [0x00007fec8f94b700] [trace] DataReaderStream: Chained to N9restc_cpp10DataReaderE
[2021-04-21 15:54:04.929493] [0x00007fec8f94b700] [trace] Created timer IoReaderImpl for {Connection c4996c71-4b6a-4263-9589-167b77817459 {Socket socket# 9 10.161.70.168:38894 <--> 10.161.70.150:80}}
[2021-04-21 15:54:04.939901] [0x00007fec8f94b700] [trace] Canceled timer IoReaderImpl
[2021-04-21 15:54:04.939979] [0x00007fec8f94b700] [trace] Read #14480 bytes from 0x7fec88001e20
[2021-04-21 15:54:04.940007] [0x00007fec8f94b700] [trace] DataReaderStream::Fetch: Fetched buffer with 14480 bytes.
[2021-04-21 15:54:04.940039] [0x00007fec8f94b700] [trace] ReadServerResponse: getc_bytes is 17
[2021-04-21 15:54:04.940062] [0x00007fec8f94b700] [trace] HTTP Response: HTTP/1.1 200 OK
[2021-04-21 15:54:04.940115] [0x00007fec8f94b700] [trace] Server: nginx/1.16.1
[2021-04-21 15:54:04.940153] [0x00007fec8f94b700] [trace] Date: Wed, 21 Apr 2021 13:54:17 GMT
[2021-04-21 15:54:04.940210] [0x00007fec8f94b700] [trace] Content-Type: application/json
[2021-04-21 15:54:04.940248] [0x00007fec8f94b700] [trace] Content-Length: 14339
[2021-04-21 15:54:04.940301] [0x00007fec8f94b700] [trace] Connection: keep-alive
[2021-04-21 15:54:04.940329] [0x00007fec8f94b700] [trace] ReadHeaderLines: getc_bytes is 157
[2021-04-21 15:54:04.940392] [0x00007fec8f94b700] [trace] Canceled timer StartReceiveFromServer
[2021-04-21 15:54:04.940496] [0x00007fec8f94b700] [trace] Done-handler is destroyed
[2021-04-21 15:54:04.940685] [0x00007fec919d1c00] [trace] DataReaderStream::ReadSome: Returning buffer with 14323 bytes.
[2021-04-21 15:54:04.940995] [0x00007fec919d1c00] [trace] Created timer IoReaderImpl for {Connection c4996c71-4b6a-4263-9589-167b77817459 {Socket socket# 9 10.161.70.168:38894 <--> 10.161.70.150:80}}
Segmentation fault

your input would be really helpful.

Regards,

[jgaa]

I can't do much without a backtrace from gdb.

However, can you try the k8deployer branch in stead of master? It has some fixes that is not merged to master yet. (I have not payed the ransom to Apple this year, so my macos compiler don't work, and that means that I can't compile and run the test on all the platforms - so I'm holding back the changes for now).

Also, can you run cmake with this flag: -DRESTC_CPP_LOG_JSON_SERIALIZATION=1. That will output lots of information from the json serialization that may be useful to pinpoint the problem.

[ishansheth]

Hi,

I am compiling on linux (Ubuntu 18.04). Here is the GDB backtrace

#1 0x0000555555afec1a in restc_cpp::IoReaderImpl::ReadSome (this=0x7ffff0016640)
at /home/ishan.sheth/workspace/ccalib_rest_api_test/restc-cpp/src/IoReaderImpl.cpp:32
#2 0x0000555555b02b74 in restc_cpp::DataReaderStream::Fetch (this=0x7ffff0002320)
at /home/ishan.sheth/workspace/ccalib_rest_api_test/restc-cpp/src/DataReaderStream.cpp:21
#3 0x0000555555b02ed0 in restc_cpp::DataReaderStream::ReadSome (this=0x7ffff0002320)
at /home/ishan.sheth/workspace/ccalib_rest_api_test/restc-cpp/src/DataReaderStream.cpp:38
#4 0x0000555555b00cf9 in restc_cpp::PlainReaderImpl::ReadSome (this=0x7ffff0015ca0)
at /home/ishan.sheth/workspace/ccalib_rest_api_test/restc-cpp/src/PlainReaderImpl.cpp:33
#5 0x0000555555ad28dc in restc_cpp::ReplyImpl::GetBodyAsString[abi:cxx11](unsigned long) (this=0x7ffff0016570, maxSize=16777216)
at /home/ishan.sheth/workspace/ccalib_rest_api_test/restc-cpp/src/ReplyImpl.cpp:192

Hope it helps
Regards

[jgaa]

So, it segfaults here:

const auto bytes = conn->GetSocket().AsyncReadSome(
                {buffer_.data(), buffer_.size()}, ctx_.GetYield());

The most reasonable explanation is that the rest client is being destroyed for example by the main thread exiting and the rest client is going out of scope while the worker-thread is trying to fetch data. I have seen that in the past, but it's supposed to be fixed in simple use-cases (it still occurs for example if you share the asio ioservice between rest clients, and then delete the asio context while some rest clients are still active).

It could also be a bug. Can you try to compile the library in debug mode with optimizations turned off? The crash may happen in some code that is invisible to gdb with optimizations turned on.

[ishansheth]

yes, its on that line. I already built library in Debug mode and this is the only information from GDB i got.

I have just one client inside a class as a variable and doing HTTP GET from the member function using the client. I am also calleng request.get() function to wait until request is processed. So its not the problem that client is destroyed unexpectedly

I think it has to do with buffer_ and the way its fetched/allocated memory for the paylod.
Hope it helps.