FFMPEG failure on certain videos

There seems to be an issue with FFMPEG and videos that zoom at some point - they end up producing incomplete files, usually <1 second long.
Here's an example file: https://drive.google.com/file/d/0BzcUsIzmvuTfamJiSnI5VHF4SU0/view?usp=sharing
Here's the complete log https://gist.github.com/StoryStar/c9f8d9f621f2adef753b for this file, and here are the relevent lines
/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/jusreign/2015-11-29-23-41-32-story-5078327813701632.mp4: Invalid data found when processing input
[h264 @ 0x7f92228b1800] AVC: nal size 2969
    Last message repeated 1 times
[h264 @ 0x7f92228b1800] no frame!
/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/jusreign/2015-11-29-23-41-32-story-5078327813701632.mp4: Invalid data found when processing input
    Last message repeated 1 times
Error while decoding stream #0:1: Invalid data found when processing input
frame=    2 fps=0.0 q=-1.0 Lsize=      18kB time=00:00:01.06 bitrate= 138.2kbits/s
video:12kB audio:5kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 8.942801%```

After reviewing this snap and others it appears to happen in any video that contains zoom

EDIT: not sure this is zoom related actually, or at least not all videos containing zoom have this issue

Are you trying to process it without logging in, or you managed to log in ?

@mzahlag I'm able to log in successfully.  The issue appears during the download of stories, using $snapchat->getFriendStories(true);

@StoryStar how did you manage to login and getting the attestation i believe snapchat has closed this method please share

@StoryStar he kept the auth file, like everyone should have done

@fis60 @mgp25 Yup, kept the auth file.
Can we keep discussion in here focused on the posted issue please? I'm still having issues with incomplete files being downloaded.  Maybe it could be related to the version of ffmpeg?

@mgp25 how long does the auth stays for and does it link to your username for example if a user logout but we saved the auth in our server and he tries to login again will it generate a new one or the old one can be pulled from the server?

@StoryStar check if ffmpeg is getting the file correctly, otherwise maybe the command we wrote isnt perfect and cause some error with some medias. Im not very experienced with ffmpeg

By Auth file you mean the session AuthToken?

@mgp25 I'm not sure what you mean by check if it's getting the file correctly - are the raw files downloaded somewhere first and then processed? How can I check that?
From what I can see, the png overlay and the mp4 are both downloaded to SC-API/src/stories/[mysnapchatusername]/[contentcreatorusername]/ and then ffmpeg is immediately called to perform the overlay with the mp4 and pngs as inputs, where the output overwrites the mp4.
I'm not very experience with ffmpeg myself, but I have been writing a script to concatenate all the videos downloaded from SC-API (including building mp4s from jpgs, and previously overlaying pngs on mp4s) - this was the command I was using for the overlay previously, with success:
ffmpeg -i "orig/$MP4NAME" -i "orig/$PNGNAME" -filter_complex \
	"[1:v]scale=$VID_WIDTH:$VID_HEIGHT[wat];[0:v][wat]overlay=0:main_h-overlay_h-10[outv]" \
	-map "[outv]" -map 0:a "$MP4NAME"

$VID_WIDTH and $VID_HEIGHT were taken from the output of:
eval $(ffprobe -v error -of flat=s=_ -select_streams v:0 -show_entries stream=height,width "$MP4NAME")

@StoryStar feel free to send a pull request

@mgp25 I'll look into it and see what I can do, at least check what command you're using right now and if a different one will work better.
In the meantime, can anyone confirm? I'm seeing broken/partial files (usually <1sec, frequently its a static image with sound but not always), stretched text/image overlays, files without file extensions...

Yes, I also frequently get video files exactly as you describe.

@mgfigueroa thanks for confirming
@mgp25 It looks like you were already using just about the same command:
ffprobe -v error -select_streams v:0 -show_entries stream=width,height \-of default=nokey=1:noprint_wrappers=1 $files[0]

ffmpeg -y -i $files[1] -vf scale=$videoSize[0]:$videoSize[1] $files[1]
ffmpeg -y -i $files[0] -i $files[1] -strict -2 -filter_complex overlay -c:a copy -flags global_header $files[0]

I've changed one line, the command with filter complex.  Here are the changes, unfortunately it looks like my auth has expired or something as I'm not able to log back in to snapchat now.  Maybe someone else can test it?
snapchat.php, line 2587:
shell_exec("ffmpeg -i $files[0] -i $files[1] -filter_complex '[1:v]scale=$videoSize[0]:$videoSize[1][wat];[0:v][wat]overlay=0:main_h-overlay_h-10[outv]' -map '[outv]' -map 0:a $files[1]");

I dont have a pc right now, but you can test it by using a sample video and png filter.

On 2/12/2015, at 20:21, StoryStar<EMAIL_ADDRESS>wrote:
@mgfigueroa thanks for confirming
@mgp25 It looks like you were already using just about the same command:
ffprobe -v error -select_streams v:0 -show_entries stream=width,height -of default=nokey=1:noprint_wrappers=1 $files[0]
ffmpeg -y -i $files[1] -vf scale=$videoSize[0]:$videoSize[1] $files[1]
ffmpeg -y -i $files[0] -i $files[1] -strict -2 -filter_complex overlay -c:a copy -flags global_header $files[0]
I've changed one line, the command with filter complex. Here are the changes, unfortunately it looks like my auth has expired or something as I'm not able to log back in to snapchat now. Maybe someone else can test it?
snapchat.php, line 2587:
shell_exec("ffmpeg -i $files[0] -i $files[1] -filter_complex '[1:v]scale=$videoSize[0]:$videoSize[1][wat];[0:v][wat]overlay=0:main_h-overlay_h-10[outv]' -map '[outv]' -map 0:a $files[1]");
—
Reply to this email directly or view it on GitHub.

Yes, I had files as described by you.

On 2/12/2015, at 19:35, StoryStar<EMAIL_ADDRESS>wrote:
@mgp25 I'll look into it and see what I can do, at least check what command you're using right now and if a different one will work better.
In the meantime, can anyone confirm? I'm seeing broken/partial files (usually <1sec, frequently its a static image with sound but not always), stretched text/image overlays, files without file extensions...
—
Reply to this email directly or view it on GitHub.

@LanchaBasualdo I've tested that command previously, when the api downloaded pngs and mp4s separately - it worked for me.  I'd like to test it in snaptchat.php itself, but I'm still unable to login

I was able to log in and test my command, it doesn't seem to be working :(
I'm downloading stories though, and after reviewing snapchat.php it looks like there is a different function for downloading snaps with png overlays - around line 1777.  The command that performs the overlay looks like it has a different output - could it be as simple as that the overlay command in the story function overwrites the existing file output (one of the inputs), while the one in the story function doesn't overwrite? I believe ffmpeg is not able to overwrite one of it's inputs...
Here are the two commands:
shell_exec("ffmpeg -loglevel panic -y -i $files[0] -i $files[1] -strict -2 -filter_complex overlay -c:a copy -flags global_header $out");
shell_exec("ffmpeg -y -i $files[0] -i $files[1] -strict -2 -filter_complex overlay -c:a copy -flags global_header $files[0]");
Notice the output.
@mgp25 Was the command for downloading snaps fixed, but the command for downloading stories not fixed?

Alright, I think that was it! I changed some lines in snapchat.php and it seems to be working properly now.  So far just tested on a couple videos, I'll confirm more later.
Here are my changes:
line 2581
					if ($returnvalue === 0)
					{
						//MYCHANGES
						// Ffmpeg can´t export over the same input file path, thus we create an output path, then we rename it
						$path_parts = pathinfo($files[0]);
						$original= $path_parts['filename'].'.' . $path_parts['extension'];
						$original=$path_parts['dirname']."/".$original;

						$name=$path_parts['filename'] . "-" . '.' . $path_parts['extension'];
						$out= $path_parts['dirname']."/".$name;
						//ENDMYCHANGES

						$videoSize = shell_exec("ffprobe -v error -select_streams v:0 -show_entries stream=width,height \-of default=nokey=1:noprint_wrappers=1 $files[0]");
						$videoSize = array_filter(explode("\n", $videoSize));

						shell_exec("ffmpeg -y -i $files[1] -vf scale=$videoSize[0]:$videoSize[1] $files[1]");
						//MYCHANGES -- changed $files[0] to $out
						shell_exec("ffmpeg -y -i $files[0] -i $files[1] -strict -2 -filter_complex overlay -c:a copy -flags global_header $out");
						//ENDMYCHANGES
						unlink($files[1]);

						//MYCHANGES
						rename($out,$original);
						//ENDMYCHANGES
					}

to line 2605

Changes have fixed the issue with overlaying pngs on mp4s.
However, there's another issue still, I think it's related to the order of the files in the $files array - I believe it assumes the first file ( $file[0] ) is the mp4, but that's not always the case, so sometimes it fails.  Here's an example of the error that ffmpeg gives, along with some other output.
ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
  built with Apple LLVM version 7.0.0 (clang-700.1.76)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, png_pipe, from '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072.png':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: png, rgba(pc), 640x1136 [SAR 5669:5669 DAR 40:71], 25 tbr, 25 tbn, 25 tbc
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf56.40.101
  Duration: 00:00:01.15, start: 0.036281, bitrate: 280 kb/s
    Stream #1:0(und): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc), 640x1136, 1615 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #1:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, mono, fltp, 129 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
[swscaler @ 0x7f87ab04ea00] deprecated pixel format used, make sure you did set range correctly
Output #0, image2, to '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072-.png':
  Metadata:
    encoder         : Lavf56.40.101
    Stream #0:0: Video: png, rgba, 640x1136 [SAR 1:1 DAR 40:71], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc56.60.100 png
Stream mapping:
  Stream #0:0 (png) -> overlay:main
  Stream #1:0 (h264) -> overlay:overlay
  overlay -> Stream #0:0 (png)
Press [q] to stop, [?] for help
[image2 @ 0x7f87aa809800] Could not get frame filename number 2 from pattern '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072-.png' (either set updatefirst or use a pattern like %03d within the filename pattern)
av_interleaved_write_frame(): Invalid argument
frame=    3 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.08 bitrate=N/A dup=0 drop=1
video:246kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Conversion failed!

Notice that it's trying to output to a png, which is obviously wrong.  There's also another error, though I'm not sure of the cause of that one - potentially partial/corrupt files. Heres the output
ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
  built with Apple LLVM version 7.0.0 (clang-700.1.76)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: mp41mp42isom
    creation_time   : 2015-12-13 06:33:40
  Duration: 00:00:09.94, start: 0.000000, bitrate: 1113 kb/s
    Stream #0:0(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, mono, fltp, 65 kb/s (default)
    Metadata:
      creation_time   : 2015-12-13 06:33:40
      handler_name    : Core Media Audio
    Stream #0:1(und): Video: h264 (Main) (avc1 / 0x31637661), yuvj420p(pc), 320x568, 1040 kb/s, 29.97 fps, 29.97 tbr, 600 tbn, 1200 tbc (default)
    Metadata:
      creation_time   : 2015-12-13 06:33:40
      handler_name    : Core Media Video
[swscaler @ 0x7f9ef1829200] deprecated pixel format used, make sure you did set range correctly
No pixel format specified, yuvj420p for H.264 encoding chosen.
Use -pix_fmt yuv420p for compatibility with outdated media players.
[libx264 @ 0x7f9ef1000c00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0x7f9ef1000c00] profile High, level 3.1
[libx264 @ 0x7f9ef1000c00] 264 - core 148 r2601 a0cd7d3 - H.264/MPEG-4 AVC codec - Copyleft 2003-2015 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: mp41mp42isom
    encoder         : Lavf56.40.101
    Stream #0:0(und): Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuvj420p(pc), 640x1136, q=-1--1, 29.97 fps, 30k tbn, 29.97 tbc (default)
    Metadata:
      creation_time   : 2015-12-13 06:33:40
      handler_name    : Core Media Video
      encoder         : Lavc56.60.100 libx264
    Stream #0:1(und): Audio: aac (libvo_aacenc) ([64][0][0][0] / 0x0040), 44100 Hz, mono, s16, 128 kb/s (default)
    Metadata:
      creation_time   : 2015-12-13 06:33:40
      handler_name    : Core Media Audio
      encoder         : Lavc56.60.100 libvo_aacenc
Stream mapping:
  Stream #0:1 -> #0:0 (h264 (native) -> h264 (libx264))
  Stream #0:0 -> #0:1 (aac (native) -> aac (libvo_aacenc))
Press [q] to stop, [?] for help
[h264 @ 0x7f9ef0855400] AVC: nal size 5869
    Last message repeated 1 times
[h264 @ 0x7f9ef0855400] no frame!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f9ef1000000] stream 1, offset 0x8488: partial file
/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072.mp4: Invalid data found when processing input
Error while decoding stream #0:0: Invalid argument
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f9ef1000000] stream 0, offset 0x2249: partial file
/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072.mp4: Invalid data found when processing input
Error while decoding stream #0:1: Invalid data found when processing input
frame=    3 fps=0.0 q=-1.0 Lsize=      39kB time=00:00:01.12 bitrate= 287.1kbits/s
video:20kB audio:18kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 4.057956%
[libx264 @ 0x7f9ef1000c00] frame I:1     Avg QP:20.14  size:  8146
[libx264 @ 0x7f9ef1000c00] frame P:2     Avg QP:21.62  size:  5689
[libx264 @ 0x7f9ef1000c00] mb I  I16..4: 46.7% 52.4%  1.0%
[libx264 @ 0x7f9ef1000c00] mb P  I16..4: 18.8% 29.3%  0.2%  P16..4: 17.4%  1.4%  0.4%  0.0%  0.0%    skip:32.5%
[libx264 @ 0x7f9ef1000c00] 8x8 transform intra:56.5% inter:93.6%
[libx264 @ 0x7f9ef1000c00] coded y,uvDC,uvAC intra: 13.4% 39.0% 7.6% inter: 4.8% 23.1% 1.2%
[libx264 @ 0x7f9ef1000c00] i16 v,h,dc,p: 37% 31%  9% 22%
[libx264 @ 0x7f9ef1000c00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 45% 17% 24%  2%  4%  3%  2%  2%  1%
[libx264 @ 0x7f9ef1000c00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 32% 22%  3%  7%  4%  5%  3%  2%
[libx264 @ 0x7f9ef1000c00] i8c dc,h,v,p: 54% 21% 18%  7%
[libx264 @ 0x7f9ef1000c00] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7f9ef1000c00] ref P L0: 89.5%  7.4%  3.0%
[libx264 @ 0x7f9ef1000c00] kb/s:1560.36

If that is the case you must order the array, its simple

On 13/12/2015, at 19:36, StoryStar<EMAIL_ADDRESS>wrote:
Changes have fixed the issue with overlaying pngs on mp4s.
However, there's another issue still, I think it's related to the order of the files in the $files array - I believe it assumes the first file ( $file[0] ) is the mp4, but that's not always the case, so sometimes it fails. Here's an example of the error that ffmpeg gives, along with some other output.
ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
built with Apple LLVM version 7.0.0 (clang-700.1.76)
configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
libavutil      54. 31.100 / 54. 31.100
libavcodec     56. 60.100 / 56. 60.100
libavformat    56. 40.101 / 56. 40.101
libavdevice    56.  4.100 / 56.  4.100
libavfilter     5. 40.101 /  5. 40.101
libavresample   2.  1.  0 /  2.  1.  0
libswscale      3.  1.101 /  3.  1.101
libswresample   1.  2.101 /  1.  2.101
libpostproc    53.  3.100 / 53.  3.100
Input #0, png_pipe, from '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072.png':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: png, rgba(pc), 640x1136 [SAR 5669:5669 DAR 40:71], 25 tbr, 25 tbn, 25 tbc
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072.mp4':
Metadata:
major_brand     : isom
minor_version   : 512
compatible_brands: isomiso2avc1mp41
encoder         : Lavf56.40.101
Duration: 00:00:01.15, start: 0.036281, bitrate: 280 kb/s
Stream #1:0(und): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc), 640x1136, 1615 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
Metadata:
handler_name    : VideoHandler
Stream #1:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, mono, fltp, 129 kb/s (default)
Metadata:
handler_name    : SoundHandler
[swscaler @ 0x7f87ab04ea00] deprecated pixel format used, make sure you did set range correctly
Output #0, image2, to '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072-.png':
Metadata:
encoder         : Lavf56.40.101
Stream #0:0: Video: png, rgba, 640x1136 [SAR 1:1 DAR 40:71], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
Metadata:
encoder         : Lavc56.60.100 png
Stream mapping:
Stream #0:0 (png) -> overlay:main
Stream #1:0 (h264) -> overlay:overlay
overlay -> Stream #0:0 (png)
Press [q] to stop, [?] for help
[image2 @ 0x7f87aa809800] Could not get frame filename number 2 from pattern '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072-.png' (either set updatefirst or use a pattern like %03d within the filename pattern)
av_interleaved_write_frame(): Invalid argument
frame=    3 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.08 bitrate=N/A dup=0 drop=1
video:246kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Conversion failed!
Notice that it's trying to output to a png, which is obviously wrong. There's also another error, though I'm not sure of the cause of that one - potentially partial/corrupt files. Heres the output
ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
built with Apple LLVM version 7.0.0 (clang-700.1.76)
configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
libavutil      54. 31.100 / 54. 31.100
libavcodec     56. 60.100 / 56. 60.100
libavformat    56. 40.101 / 56. 40.101
libavdevice    56.  4.100 / 56.  4.100
libavfilter     5. 40.101 /  5. 40.101
libavresample   2.  1.  0 /  2.  1.  0
libswscale      3.  1.101 /  3.  1.101
libswresample   1.  2.101 /  1.  2.101
libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072.mp4':
Metadata:
major_brand     : mp42
minor_version   : 1
compatible_brands: mp41mp42isom
creation_time   : 2015-12-13 06:33:40
Duration: 00:00:09.94, start: 0.000000, bitrate: 1113 kb/s
Stream #0:0(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, mono, fltp, 65 kb/s (default)
Metadata:
creation_time   : 2015-12-13 06:33:40
handler_name    : Core Media Audio
Stream #0:1(und): Video: h264 (Main) (avc1 / 0x31637661), yuvj420p(pc), 320x568, 1040 kb/s, 29.97 fps, 29.97 tbr, 600 tbn, 1200 tbc (default)
Metadata:
creation_time   : 2015-12-13 06:33:40
handler_name    : Core Media Video
[swscaler @ 0x7f9ef1829200] deprecated pixel format used, make sure you did set range correctly
No pixel format specified, yuvj420p for H.264 encoding chosen.
Use -pix_fmt yuv420p for compatibility with outdated media players.
[libx264 @ 0x7f9ef1000c00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0x7f9ef1000c00] profile High, level 3.1
[libx264 @ 0x7f9ef1000c00] 264 - core 148 r2601 a0cd7d3 - H.264/MPEG-4 AVC codec - Copyleft 2003-2015 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072.mp4':
Metadata:
major_brand     : mp42
minor_version   : 1
compatible_brands: mp41mp42isom
encoder         : Lavf56.40.101
Stream #0:0(und): Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuvj420p(pc), 640x1136, q=-1--1, 29.97 fps, 30k tbn, 29.97 tbc (default)
Metadata:
creation_time   : 2015-12-13 06:33:40
handler_name    : Core Media Video
encoder         : Lavc56.60.100 libx264
Stream #0:1(und): Audio: aac (libvo_aacenc) ([64][0][0][0] / 0x0040), 44100 Hz, mono, s16, 128 kb/s (default)
Metadata:
creation_time   : 2015-12-13 06:33:40
handler_name    : Core Media Audio
encoder         : Lavc56.60.100 libvo_aacenc
Stream mapping:
Stream #0:1 -> #0:0 (h264 (native) -> h264 (libx264))
Stream #0:0 -> #0:1 (aac (native) -> aac (libvo_aacenc))
Press [q] to stop, [?] for help
[h264 @ 0x7f9ef0855400] AVC: nal size 5869
Last message repeated 1 times
[h264 @ 0x7f9ef0855400] no frame!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f9ef1000000] stream 1, offset 0x8488: partial file
/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072.mp4: Invalid data found when processing input
Error while decoding stream #0:0: Invalid argument
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f9ef1000000] stream 0, offset 0x2249: partial file
/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/nastygal/2015-12-13-06-33-42-story-5997297463427072.mp4: Invalid data found when processing input
Error while decoding stream #0:1: Invalid data found when processing input
frame=    3 fps=0.0 q=-1.0 Lsize=      39kB time=00:00:01.12 bitrate= 287.1kbits/s
video:20kB audio:18kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 4.057956%
[libx264 @ 0x7f9ef1000c00] frame I:1     Avg QP:20.14  size:  8146
[libx264 @ 0x7f9ef1000c00] frame P:2     Avg QP:21.62  size:  5689
[libx264 @ 0x7f9ef1000c00] mb I  I16..4: 46.7% 52.4%  1.0%
[libx264 @ 0x7f9ef1000c00] mb P  I16..4: 18.8% 29.3%  0.2%  P16..4: 17.4%  1.4%  0.4%  0.0%  0.0%    skip:32.5%
[libx264 @ 0x7f9ef1000c00] 8x8 transform intra:56.5% inter:93.6%
[libx264 @ 0x7f9ef1000c00] coded y,uvDC,uvAC intra: 13.4% 39.0% 7.6% inter: 4.8% 23.1% 1.2%
[libx264 @ 0x7f9ef1000c00] i16 v,h,dc,p: 37% 31%  9% 22%
[libx264 @ 0x7f9ef1000c00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 45% 17% 24%  2%  4%  3%  2%  2%  1%
[libx264 @ 0x7f9ef1000c00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 32% 22%  3%  7%  4%  5%  3%  2%
[libx264 @ 0x7f9ef1000c00] i8c dc,h,v,p: 54% 21% 18%  7%
[libx264 @ 0x7f9ef1000c00] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7f9ef1000c00] ref P L0: 89.5%  7.4%  3.0%
[libx264 @ 0x7f9ef1000c00] kb/s:1560.36
—
Reply to this email directly or view it on GitHub.

@LanchaBasualdo Yup that fixed it :)
Can anyone shed light on the other error above? Additionally have a few other errors:
Unsure about this one, maybe there is no second file?
Notice: Undefined offset: 1 in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2604
PHP Notice:  Undefined offset: 1 in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2604

Notice: Undefined offset: 1 in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2604
ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
  built with Apple LLVM version 7.0.0 (clang-700.1.76)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
-vf: No such file or directory
PHP Notice:  Undefined offset: 1 in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2606

Notice: Undefined offset: 1 in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2606
ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
  built with Apple LLVM version 7.0.0 (clang-700.1.76)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Unrecognized option '2'.
Error splitting the argument list: Option not found
PHP Notice:  Undefined offset: 1 in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2608

Notice: Undefined offset: 1 in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2608
PHP Warning:  unlink(): No such file or directory in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2608

Warning: unlink(): No such file or directory in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2608
PHP Warning:  rename(/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/rachelteetyler/2015-12-14-02-59-42-story-4868086223831040-.mp4,/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/rachelteetyler/2015-12-14-02-59-42-story-4868086223831040.mp4): No such file or directory in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2611

Warning: rename(/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/rachelteetyler/2015-12-14-02-59-42-story-4868086223831040-.mp4,/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/rachelteetyler/2015-12-14-02-59-42-story-4868086223831040.mp4): No such file or directory in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2611

This one might be related to this ffmpeg issue ( https://trac.ffmpeg.org/ticket/4822 ) - I'm running on OSX also
ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
  built with Apple LLVM version 7.0.0 (clang-700.1.76)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
[png @ 0x7feb44808600] inflate returned error -3
Input #0, png_pipe, from '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/hongkong1515/2015-12-14-09-22-44-story-6100129922383872.png':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: png, rgba(pc), 1532x2560, 25 tbr, 25 tbn, 25 tbc
Output #0, image2, to '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/hongkong1515/2015-12-14-09-22-44-story-6100129922383872.png':
  Metadata:
    encoder         : Lavf56.40.101
    Stream #0:0: Video: png, rgba, 600x360, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc56.60.100 png
Stream mapping:
  Stream #0:0 -> #0:0 (png (native) -> png (native))
Press [q] to stop, [?] for help
[png @ 0x7feb44260c00] inflate returned error -3
Error while decoding stream #0:0: Generic error in an external library
frame=    0 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.00 bitrate=N/A
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)
Conversion failed!
ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
  built with Apple LLVM version 7.0.0 (clang-700.1.76)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/hongkong1515/2015-12-14-09-22-44-story-6100129922383872.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf56.25.101
  Duration: 00:00:08.58, start: 0.042667, bitrate: 1181 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 600x360, 1049 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
    Metadata:
      rotate          : 90
      handler_name    : VideoHandler
    Side data:
      displaymatrix: rotation of -90.00 degrees
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 139 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
[png @ 0x7fc553002c00] inflate returned error -3
Input #1, png_pipe, from '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/hongkong1515/2015-12-14-09-22-44-story-6100129922383872.png':
  Duration: N/A, bitrate: N/A
    Stream #1:0: Video: png, rgba(pc), 1532x2560, 25 tbr, 25 tbn, 25 tbc
[libx264 @ 0x7fc552812600] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0x7fc552812600] profile High, level 3.0
[libx264 @ 0x7fc552812600] 264 - core 148 r2601 a0cd7d3 - H.264/MPEG-4 AVC codec - Copyleft 2003-2015 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=1 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/hongkong1515/2015-12-14-09-22-44-story-6100129922383872-.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf56.40.101
    Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 360x600, q=-1--1, 30 fps, 15360 tbn, 30 tbc (default)
    Metadata:
      encoder         : Lavc56.60.100 libx264
    Stream #0:1(eng): Audio: aac ([64][0][0][0] / 0x0040), 48000 Hz, stereo, 139 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Stream mapping:
  Stream #0:0 (h264) -> overlay:main
  Stream #1:0 (png) -> overlay:overlay
  overlay -> Stream #0:0 (libx264)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[png @ 0x7fc553038000] inflate returned error -3
Error while decoding stream #1:0: Generic error in an external library
frame=   90 fps=0.0 q=29.0 size=     179kB time=00:00:03.13 bitrate= 467.4kbits/frame=  151 fps=148 q=29.0 size=     419kB time=00:00:05.18 bitrate= 661.5kbits/frame=  210 fps=138 q=29.0 size=     656kB time=00:00:07.14 bitrate= 752.2kbits/frame=  254 fps=112 q=-1.0 Lsize=    1093kB time=00:00:08.57 bitrate=1044.5kbits/s dup=1 drop=0
video:937kB audio:146kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.958468%
[libx264 @ 0x7fc552812600] frame I:2     Avg QP:22.32  size: 20196
[libx264 @ 0x7fc552812600] frame P:102   Avg QP:24.17  size:  6960
[libx264 @ 0x7fc552812600] frame B:150   Avg QP:28.24  size:  1389
[libx264 @ 0x7fc552812600] consecutive B-frames:  1.2% 40.9% 57.9%  0.0%
[libx264 @ 0x7fc552812600] mb I  I16..4: 10.3% 58.6% 31.1%
[libx264 @ 0x7fc552812600] mb P  I16..4:  0.9%  3.2%  0.6%  P16..4: 33.1% 25.1% 14.9%  0.0%  0.0%    skip:22.1%
[libx264 @ 0x7fc552812600] mb B  I16..4:  0.0%  0.0%  0.1%  B16..8: 34.8% 10.5%  2.6%  direct: 1.8%  skip:50.2%  L0:43.0% L1:52.0% BI: 5.0%
[libx264 @ 0x7fc552812600] 8x8 transform intra:64.2% inter:42.5%
[libx264 @ 0x7fc552812600] coded y,uvDC,uvAC intra: 36.0% 42.1% 6.0% inter: 14.2% 6.7% 0.0%
[libx264 @ 0x7fc552812600] i16 v,h,dc,p: 38% 23% 14% 25%
[libx264 @ 0x7fc552812600] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 13% 56%  3%  2%  2%  3%  3%  3%
[libx264 @ 0x7fc552812600] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 19% 20%  8%  6%  4%  9%  7%  8%
[libx264 @ 0x7fc552812600] i8c dc,h,v,p: 61% 23% 12%  4%
[libx264 @ 0x7fc552812600] Weighted P-Frames: Y:12.7% UV:0.0%
[libx264 @ 0x7fc552812600] ref P L0: 61.1% 18.3% 15.2%  5.1%  0.4%
[libx264 @ 0x7fc552812600] ref B L0: 86.0% 12.6%  1.4%
[libx264 @ 0x7fc552812600] ref B L1: 93.2%  6.8%
[libx264 @ 0x7fc552812600] kb/s:905.79

I think that it means there is no file in the 2 (offset 1) object on the array that why the unlink can not delete anything as there ir no actual file. Print array to see what is it made of.

On 14/12/2015, at 17:56, StoryStar<EMAIL_ADDRESS>wrote:
@LanchaBasualdo Yup that fixed it :)
Can anyone shed light on the other error above? Additionally have a few other errors:
Unsure about this one, maybe there is no second file?
Notice: Undefined offset: 1 in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2604
PHP Notice:  Undefined offset: 1 in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2604
Notice: Undefined offset: 1 in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2604
ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
built with Apple LLVM version 7.0.0 (clang-700.1.76)
configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
libavutil      54. 31.100 / 54. 31.100
libavcodec     56. 60.100 / 56. 60.100
libavformat    56. 40.101 / 56. 40.101
libavdevice    56.  4.100 / 56.  4.100
libavfilter     5. 40.101 /  5. 40.101
libavresample   2.  1.  0 /  2.  1.  0
libswscale      3.  1.101 /  3.  1.101
libswresample   1.  2.101 /  1.  2.101
libpostproc    53.  3.100 / 53.  3.100
-vf: No such file or directory
PHP Notice:  Undefined offset: 1 in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2606
Notice: Undefined offset: 1 in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2606
ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
built with Apple LLVM version 7.0.0 (clang-700.1.76)
configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
libavutil      54. 31.100 / 54. 31.100
libavcodec     56. 60.100 / 56. 60.100
libavformat    56. 40.101 / 56. 40.101
libavdevice    56.  4.100 / 56.  4.100
libavfilter     5. 40.101 /  5. 40.101
libavresample   2.  1.  0 /  2.  1.  0
libswscale      3.  1.101 /  3.  1.101
libswresample   1.  2.101 /  1.  2.101
libpostproc    53.  3.100 / 53.  3.100
Unrecognized option '2'.
Error splitting the argument list: Option not found
PHP Notice:  Undefined offset: 1 in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2608
Notice: Undefined offset: 1 in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2608
PHP Warning:  unlink(): No such file or directory in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2608
Warning: unlink(): No such file or directory in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2608
PHP Warning:  rename(/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/rachelteetyler/2015-12-14-02-59-42-story-4868086223831040-.mp4,/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/rachelteetyler/2015-12-14-02-59-42-story-4868086223831040.mp4): No such file or directory in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2611
Warning: rename(/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/rachelteetyler/2015-12-14-02-59-42-story-4868086223831040-.mp4,/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/rachelteetyler/2015-12-14-02-59-42-story-4868086223831040.mp4): No such file or directory in /Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/snapchat.php on line 2611
This one might be related to this ffmpeg issue ( https://trac.ffmpeg.org/ticket/4822 ) - I'm running on OSX also
ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
built with Apple LLVM version 7.0.0 (clang-700.1.76)
configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
libavutil      54. 31.100 / 54. 31.100
libavcodec     56. 60.100 / 56. 60.100
libavformat    56. 40.101 / 56. 40.101
libavdevice    56.  4.100 / 56.  4.100
libavfilter     5. 40.101 /  5. 40.101
libavresample   2.  1.  0 /  2.  1.  0
libswscale      3.  1.101 /  3.  1.101
libswresample   1.  2.101 /  1.  2.101
libpostproc    53.  3.100 / 53.  3.100
[png @ 0x7feb44808600] inflate returned error -3
Input #0, png_pipe, from '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/hongkong1515/2015-12-14-09-22-44-story-6100129922383872.png':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: png, rgba(pc), 1532x2560, 25 tbr, 25 tbn, 25 tbc
Output #0, image2, to '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/hongkong1515/2015-12-14-09-22-44-story-6100129922383872.png':
Metadata:
encoder         : Lavf56.40.101
Stream #0:0: Video: png, rgba, 600x360, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
Metadata:
encoder         : Lavc56.60.100 png
Stream mapping:
Stream #0:0 -> #0:0 (png (native) -> png (native))
Press [q] to stop, [?] for help
[png @ 0x7feb44260c00] inflate returned error -3
Error while decoding stream #0:0: Generic error in an external library
frame=    0 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.00 bitrate=N/A
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)
Conversion failed!
ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
built with Apple LLVM version 7.0.0 (clang-700.1.76)
configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
libavutil      54. 31.100 / 54. 31.100
libavcodec     56. 60.100 / 56. 60.100
libavformat    56. 40.101 / 56. 40.101
libavdevice    56.  4.100 / 56.  4.100
libavfilter     5. 40.101 /  5. 40.101
libavresample   2.  1.  0 /  2.  1.  0
libswscale      3.  1.101 /  3.  1.101
libswresample   1.  2.101 /  1.  2.101
libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/hongkong1515/2015-12-14-09-22-44-story-6100129922383872.mp4':
Metadata:
major_brand     : isom
minor_version   : 512
compatible_brands: isomiso2avc1mp41
encoder         : Lavf56.25.101
Duration: 00:00:08.58, start: 0.042667, bitrate: 1181 kb/s
Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 600x360, 1049 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
Metadata:
rotate          : 90
handler_name    : VideoHandler
Side data:
displaymatrix: rotation of -90.00 degrees
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 139 kb/s (default)
Metadata:
handler_name    : SoundHandler
[png @ 0x7fc553002c00] inflate returned error -3
Input #1, png_pipe, from '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/hongkong1515/2015-12-14-09-22-44-story-6100129922383872.png':
Duration: N/A, bitrate: N/A
Stream #1:0: Video: png, rgba(pc), 1532x2560, 25 tbr, 25 tbn, 25 tbc
[libx264 @ 0x7fc552812600] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0x7fc552812600] profile High, level 3.0
[libx264 @ 0x7fc552812600] 264 - core 148 r2601 a0cd7d3 - H.264/MPEG-4 AVC codec - Copyleft 2003-2015 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=1 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '/Users/matt/Documents/JACKTHEAUTORIPPER/api/SC-API/src/stories/storystarcc/hongkong1515/2015-12-14-09-22-44-story-6100129922383872-.mp4':
Metadata:
major_brand     : isom
minor_version   : 512
compatible_brands: isomiso2avc1mp41
encoder         : Lavf56.40.101
Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 360x600, q=-1--1, 30 fps, 15360 tbn, 30 tbc (default)
Metadata:
encoder         : Lavc56.60.100 libx264
Stream #0:1(eng): Audio: aac ([64][0][0][0] / 0x0040), 48000 Hz, stereo, 139 kb/s (default)
Metadata:
handler_name    : SoundHandler
Stream mapping:
Stream #0:0 (h264) -> overlay:main
Stream #1:0 (png) -> overlay:overlay
overlay -> Stream #0:0 (libx264)
Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[png @ 0x7fc553038000] inflate returned error -3
Error while decoding stream #1:0: Generic error in an external library
frame=   90 fps=0.0 q=29.0 size=     179kB time=00:00:03.13 bitrate= 467.4kbits/frame=  151 fps=148 q=29.0 size=     419kB time=00:00:05.18 bitrate= 661.5kbits/frame=  210 fps=138 q=29.0 size=     656kB time=00:00:07.14 bitrate= 752.2kbits/frame=  254 fps=112 q=-1.0 Lsize=    1093kB time=00:00:08.57 bitrate=1044.5kbits/s dup=1 drop=0
video:937kB audio:146kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.958468%
[libx264 @ 0x7fc552812600] frame I:2     Avg QP:22.32  size: 20196
[libx264 @ 0x7fc552812600] frame P:102   Avg QP:24.17  size:  6960
[libx264 @ 0x7fc552812600] frame B:150   Avg QP:28.24  size:  1389
[libx264 @ 0x7fc552812600] consecutive B-frames:  1.2% 40.9% 57.9%  0.0%
[libx264 @ 0x7fc552812600] mb I  I16..4: 10.3% 58.6% 31.1%
[libx264 @ 0x7fc552812600] mb P  I16..4:  0.9%  3.2%  0.6%  P16..4: 33.1% 25.1% 14.9%  0.0%  0.0%    skip:22.1%
[libx264 @ 0x7fc552812600] mb B  I16..4:  0.0%  0.0%  0.1%  B16..8: 34.8% 10.5%  2.6%  direct: 1.8%  skip:50.2%  L0:43.0% L1:52.0% BI: 5.0%
[libx264 @ 0x7fc552812600] 8x8 transform intra:64.2% inter:42.5%
[libx264 @ 0x7fc552812600] coded y,uvDC,uvAC intra: 36.0% 42.1% 6.0% inter: 14.2% 6.7% 0.0%
[libx264 @ 0x7fc552812600] i16 v,h,dc,p: 38% 23% 14% 25%
[libx264 @ 0x7fc552812600] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 13% 56%  3%  2%  2%  3%  3%  3%
[libx264 @ 0x7fc552812600] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 19% 20%  8%  6%  4%  9%  7%  8%
[libx264 @ 0x7fc552812600] i8c dc,h,v,p: 61% 23% 12%  4%
[libx264 @ 0x7fc552812600] Weighted P-Frames: Y:12.7% UV:0.0%
[libx264 @ 0x7fc552812600] ref P L0: 61.1% 18.3% 15.2%  5.1%  0.4%
[libx264 @ 0x7fc552812600] ref B L0: 86.0% 12.6%  1.4%
[libx264 @ 0x7fc552812600] ref B L1: 93.2%  6.8%
[libx264 @ 0x7fc552812600] kb/s:905.79
—
Reply to this email directly or view it on GitHub.

@LanchaBasualdo That's exactly what I thought as well. I'm running the api again hopefully I'll be able to reproduce with the print statements in, it's just strange since it checks explicitly if $result is an array (snapchat.php line 2566) before doing any of this.
I guess $result is an array containing only one object, we will see.  If that's the case we could just check if there are multiple items in the array - but I'm not sure why snapchat would be sending a single item in a array, maybe someone else more experienced has an idea why.

Confirmed only one item in array

Are you shure that the video has text??

On 14/12/2015, at 20:24, StoryStar<EMAIL_ADDRESS>wrote:
Confirmed only one item in array
—
Reply to this email directly or view it on GitHub.

@LanchaBasualdo Not at all, though SC-API seems to think there should be.  I have no real way of knowing actually -- apart from maybe checking in the official snapchat client, though if it's an issue on snapchat's side then I'm not sure we can find out at all.  If someone wants to look into it, it's a snap from rachelteetyler's story today (can't check myself at the moment)
I believe the mp4 is still downloaded normally so it should be in there, just without any overlay that is maybe supposed to be on top.

You do... Send a video snap with text! And previously write the code to check if the array has more than 1 file. If it doesn't overpass the merging process.

On 14/12/2015, at 22:21, StoryStar<EMAIL_ADDRESS>wrote:
@LanchaBasualdo Not at all, though SC-API seems to think there should be. I have no real way of knowing actually -- apart from maybe checking in the official snapchat client, though if it's an issue on snapchat's side then I'm not sure we can find out at all. If someone wants to look into it, it's a snap from rachelteetyler's story today (can't check myself at the moment)
I believe the mp4 is still downloaded normally so it should be in there, just without any overlay that is maybe supposed to be on top.
—
Reply to this email directly or view it on GitHub.

@LanchaBasualdo haha sure I could send one, but I already know it works for the vast majority of snaps - I'm just not sure what causes there to be only one file in the array, or how to force that to reproduce.
I'll definitely be adding a check to make sure the files array has two items though, and skip the merging if that's not the case

And can it be that that specific video has no layer to merg?

On 14/12/2015, at 23:44, StoryStar<EMAIL_ADDRESS>wrote:
@LanchaBasualdo haha sure I could send one, but I already know it works for the vast majority of snaps - I'm just not sure what causes there to be only one file in the array, or how to force that to reproduce.
I'll definitely be adding a check to make sure the files array has two items though, and skip the merging if that's not the case
—
Reply to this email directly or view it on GitHub.

@LanchaBasualdo It could be - but usually snaps without a layer to merge don't come in an array like this - usually only those with a png to overlay come in an array like this

I understand, did you say that you have printed the array and that the array contained exactly only 1 file?

On 15/12/2015, at 0:33, StoryStar<EMAIL_ADDRESS>wrote:
@LanchaBasualdo It could be - but usually snaps without a layer to merge don't come in an array like this - usually only those with a png to overlay come in an array like this
—
Reply to this email directly or view it on GitHub.

Yup, just an mp4

Full Snapchat API using a new and complex tweak.
You can use it to:
Create snapchat bots.
Features:

Send a media to a story
Send a snap
Add a friend
Delete a friend
Mark a snap video as viewed
Create a third app on the App Store or Play Store.
more dev informations will be sent in PDF.
Monthly subscription: $450! only. Unlimited requests to snapchat.
FREE 24h trial!
Skype: mzaboss


Fix enabling linenumbers by default

linenos has become a flag, meaning it cannot have a value. This produces line numbers when testing locally

Great! Thank you.


ofxGifEncoder reset

I added a reset similar to ofxGifDecoder so now we can save multiple independent gif during runtime.

sweet. thanks!


GRPC python client cpu 100% sometimes

What version of gRPC and what language are you using?
1.25.0  python client
install by pip  and package is  grpcio-1.25.0-cp27-cp27mu-manylinux1_x86_64.whl
What operating system (Linux, Windows,...) and version?
debian 7
What runtime / compiler are you using (e.g. python version or version of gcc)
python 2.7.3
emm. is a old python project  running in old linux , for some reason , we need transfer old communicator to new grpc
project start all  func thread by thread , include grpc client write in a while loop.
What did you do?

simple but stream both server and client .
I define proto file as this
service AgentNotify {
  rpc SendHeartBeat ( stream HeartBeatRequest) returns (stream HeartBeatReply) {}
}
message HeartBeatRequest {
  string request = 1;
  string msgId = 2;
}
message HeartBeatReply {
  string reply = 1;
  string TimeStamp = 2;
}

generate code cmd
python -m grpc_tools.protoc --proto_path=./ --python_out=./ --grpc_python_out=./ heartbeat.proto

and the grpc client code like this
class channel:
    def init(self):
            self.stub = heartbeat_pb2_grpc.AgentNotifyStub(self.channel)
            response_iterator = self.stub.SendHeartBeat(self.gen_heartbeat())
            for response in response_iterator:
                xxxx

    def gen_heartbeat(self):
        while self.channel_alive:
            try:
                hb = self.request_queue.get(timeout=10)
            except Empty:
                continue
            hb_string = json.dumps(hb)
            request = heartbeat_pb2.HeartBeatRequest(request=hb_string)
            yield request

What did you expect to see?
running with no problem
What did you see instead?
python client message send thread stop in stuck for some often, and cpu running in 100% mode.  but the tcp connection was still keep with server.
before add grpc . project runs well, and when turn grpc switch on,  the project was lay down to  the street(扑街),  short time may in few minitus, and longer may in some hours.
Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).
See TROUBLESHOOTING.md for how to diagnose problems better.
Anything else we should know about your project / environment?

@nathanielmanistaatgoogle  I see this issue  #14911  ,  was python grpc sdk got  some  err can cause cpu 100?

@charlesld Can you help us by creating a full reproduce case for this issue? By "thread stop in stuck", do you mean there is no traffic go through? Or simply because 100% CPU usage?

Hi, @lidizheng  when cpu 100% , no traffic , and whole project stop running，seems empty loop was running。
I will  continue watching this ,and feed back.

@charlesld Still, I'm not entirely sure what is the behavior here, please try to create a minimal reproduction case that can trigger the bug you are describing.

Hi, @lidizheng seems  I've found some reason why the problem.

I use result=os.popen(cmd)   with a very high freq to fetch some app status.
When starting an application, the application will be blocked with a very high probability
I add log everywhere , then I found some stuck point when os.open
I change os.open to this code

pipe = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE)
lines = pipe.stdout.readlines()
pipe.stdout.close()

Fine . it's much better , and programs rarely get blocked
Throughout the afternoon, I  only found one endpoint was blocked, so I printed the stack information and flame chart with py-spy. I found  grpc._channel and subprocess may  have some system-level signal contention or  dead lock or something??  (because when  CPU100  they cost the same CPU time)

stack info

flame info

    [flame file link here   ](https://pan.baidu.com/s/1No4VQ5TwAu8tlxlIPWOcrQ)
    and file link code  is : **rc1k**

So I guess , the bug was here.

last comment , I use debain 9 and python 2.7.13

Both os.popen and subprocss.Popen will trigger our fork handlers. The CPython implementation of those command is not ideal before 3.7 that it is using fork + exec pattern.
To make fork work properly, we registered a bunch of logic pre-fork and post-fork to manage FD / locks / threads. If you call a command with a very high frequency, it is likely to trigger some sort of data race.
So, can you try to set the environment variable GRPC_ENABLE_FORK_SUPPORT=0? If possible, please upgrade to Python 3.7+.

THx @lidizheng

Emm.   I've modified this variable ， and  I am watching the effect。

@lidizheng  was GRPC_ENABLE_FORK_SUPPORT default set to 1. I close it on my runtime , and it worked!

@charlesld By default, it is turned on. Glad to see the deadlock not appearing again.
If the solution is acceptable, please close this issue.

Fine, thx again~~ @lidizheng


chore(*): update build to support release build

update Travis configuration to use encrypted build parameters saved in the web UI over terminal based encryption
changed the deployment to only push to maven if it's master that is being built

@pahimar,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

@pahimar let's get this in


Unused information in the metadata.yml format

The metadata for electrode groups has both a location and targeted_location field. However only targeted_location is used in conversion (there isn't space in the pynwb electrode group object for both). Propose removing the location from the metadata to avoid confusion.

[ ] Change example metadata files
[ ] Change the metadata generator to remove this field
[ ] Change the metadata validator to not require this field

@samuelbray32 Please keep me updated on a final decision. A change in any of these require other things to change.


sensor.samba_backup shows as unavailable on every HA restart

I have Start on Boot, Watchdog and Auto Update set to ON
Samba Backup has a Green dot in the top right
But on my Lovelace, the section that should show backup status (Idle usually) instead shows as unavailable every time I restart HA.
If I restart Samba Backup, the sensor reappears in Lovelace.  And if I then click on it to show the 24 hour history, it does not show that it was ever unavailable.
Version 4.5.0
core-2021.6.4
Home Assistant OS 5.13
supervisor-2021.06.0
Log file though doesn't seem to show anything related to above:
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[09:50:24] INFO: Host: <IP_ADDRESS>
[09:50:24] INFO: Share: Pi4b
[09:50:24] INFO: Target Dir: Pi4b
[09:50:24] INFO: Keep local: 5
[09:50:24] INFO: Keep remote: 15
[09:50:24] INFO: Trigger time: 03:00
[09:50:24] INFO: Trigger days: Mon Tue Wed Thu Fri Sat Sun
[03:00:37] INFO: Backup running ...
[03:00:38] INFO: Creating snapshot "Samba Backup 2021-06-13 03:00"
[03:03:53] INFO: Copying snapshot f34a6ba5 (Samba_Backup_2021_06_13_03_00.tar) to share
[03:03:56] INFO: Deleting cc17f2e2 local
[03:03:57] INFO: Deleting 8ee6dc29 local
[03:03:57] INFO: Deleting 249bba7e local
[03:03:57] INFO: Deleting Samba_Backup_2021_05_28_03_00.tar on share
[03:04:08] INFO: Backup finished

Yes, that's a known issue. Please read the docs --> https://github.com/thomasmauerer/hassio-addons/blob/master/samba-backup/DOCS.md#home-assistant-sensor
You can use the given automation to restore the sensor on startup.

Thank you Thomas - sorry I missed that in the docs!
Awesome and very useful plugin for HA
Thanks again

Yes, that's a known issue. Please read the docs --> https://github.com/thomasmauerer/hassio-addons/blob/master/samba-backup/DOCS.md#home-assistant-sensor
You can use the given automation to restore the sensor on startup.

Thank you for posting this one, Thomasmauerer


Please add a way to stop/cancel the inference, and Suggestions

Hi, there!
Nice project! Thank you for developing and maintaining it!
Maybe I missed something in the help, but is there a way to cancel an inference? If not, I think this could be really useful, specially when trying models locally (specially the big ones that may run very slowly).
Some suggestions (please feel free to ask me to open a separate issue for the ones you may be implemented):

Option to delete chat
Add a secondary keybinding to send a message, maybe Shift + Enter or Ctrl + Enter. Personally, I would prefer that it worked like some messengers can be configured: Enter to send and Shift/Ctrl+Enter to add a line break (maybe add a preference for this?)
Other options to the app "theme"
Tune some variables per chat, like system prompt, temperature, etc
Export the chat (as a MarkDown file?)
Add image to the chat (for multimodal models)

Thanks again. Regards

Hey, sorry for the delay.

You can press a when highlighting a chat on the home screen to archive it (it will remain in your local database but is hidden from the UI).
I've added alt+enter in the latest version, but support is down to your terminal. Customisable keymaps will be available in the coming months.
I've added support for custom themes in the latest version.
Good idea! Right now you can change the system prompt in ctrl+o menu from the home page, and that will apply for the remainder of the session. It'll be set back to your config file prompt in the next session however.
Good idea!
This may come in the future but it's not a priority for me as I don't really use any multimodal stuff. Happy to accept contributions though!


High resolution

Can the network be trained so that the features of the style image (for example, large brush strokes) be applied to content images that are larger than the trained images?
For example, if I train the network on the 250px dataset using a style image of 750px, when I evaluate/style a content image that is 1500px all of the style image features are very small. I would ideally like the features to be of a similar size in relation to the style image.
Is this possible?

This method could be a solution: https://github.com/LouieYang/stroke-controllable-fast-style-transfer


error with field edit_history_tweet_ids

This appears to be an error with the Tweet.py file
Stream encountered an exception Traceback (most recent call last):   File "/home/ciptools/data-loaders/covid19-api-v2/venv/lib/python3.9/site-packages/tweepy/streaming.py", line 92, in _connect     self.on_data(line)   File "/home/ciptools/data-loaders/covid19-api-v2/venv/lib/python3.9/site-packages/tweepy/streaming.py", line 926, in on_data     includes = self._process_includes(data["includes"])   File "/home/ciptools/data-loaders/covid19-api-v2/venv/lib/python3.9/site-packages/tweepy/client.py", line 164, in _process_includes     includes["tweets"] = [Tweet(tweet) for tweet in includes["tweets"]]   File "/home/ciptools/data-loaders/covid19-api-v2/venv/lib/python3.9/site-packages/tweepy/client.py", line 164, in <listcomp>     includes["tweets"] = [Tweet(tweet) for tweet in includes["tweets"]]   File "/home/ciptools/data-loaders/covid19-api-v2/venv/lib/python3.9/site-packages/tweepy/tweet.py", line 121, in __init__     map(int, data["edit_history_tweet_ids"]) KeyError: 'edit_history_tweet_ids'
My code
`
# This is where we set our filter rule
streamer.add_rule()

expansions =  "author_id,referenced_tweets.id,in_reply_to_user_id,geo.place_id,entities.mentions.username,referenced_tweets.id.author_id",
user_fields =  "created_at,description,entities,id,location,name,protected,public_metrics,url,username,verified,withheld",
tweet_fields= "attachments,author_id,context_annotations,conversation_id,created_at,entities,geo,id,in_reply_to_user_id,lang,public_metrics,possibly_sensitive,referenced_tweets,source,text,withheld",
place_fields= "contained_within,country,country_code,full_name,geo,id,name,place_type",

#return hydrated tweet
cs.filter(user_fields=user_fields, tweet_fields=tweet_fields, place_fields=place_fields, expansions=expansions)

`

This seems like it might be a bug with the Twitter API, since edit_history_tweet_ids should be a default Tweet field.
For code block usage, see https://docs.github.com/en/github/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks.

This is also breaking my existing code when I initialize a tweepy.Tweet from a dict
import tweepy
tweepy.Tweet(data={"id":1, "text":"example"})
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/home/meiji163/miniconda3/lib/python3.8/site-packages/tweepy/tweet.py", line 128, in __init__
   map(int, data["edit_history_tweet_ids"])
KeyError: 'edit_history_tweet_ids'

While adding support for the edit metadata, it did occur to me that it might be a breaking change in terms of how Tweet is initialized. I supposed there are possible use cases for wanting to use old data to construct the model. For now, I've changed it to handle the KeyError and warn about it being missing instead, with https://github.com/tweepy/tweepy/commit/3dea0df2b8ee797264de67afc9f2d670e68aa634. This is a stopgap, but I don't really see another way to keep backwards-compatibility when the Twitter API adds new default fields to models.


Need a new name (again)

'Adflow' is triggering ad blockers!

found on this list: https://github.com/easylist/easylist/blob/master/easylist/easylist_general_block.txt#L1646

working on a new list of ideas, haven't yet put it to a vote


Fixed Code Quality Issues

Description
Summary:

Removed useless Object inheritance
Removed the usage of self
Add .deepsource.toml

I ran a DeepSource Analysis on my fork of this repository. You can see all the issues raised by DeepSource here.
DeepSource helps you to automatically find and fix issues in your code during code reviews. This tool looks for anti-patterns, bug risks, performance problems, and raises issues. There are plenty of other issues in relation to Bug Discovery and Anti-Patterns which you would be interested to take a look at.
If you do not want to use DeepSource to continuously analyze this repo, I'll remove the .deepsource.toml from this PR and you can merge the rest of the fixes. If you want to setup DeepSource for Continuous Analysis, I can help you set that up.

Hi @SharonBrizinov
Can you please have a look at this PR? The Travis Build has succeeded, so I guess the PR is ready to be reviewed.

Hi,
Thanks for the PR. These changes are breaking the functionality of the functions/classes (changed class functions to static) so I'm closing this PR.


Upgrade to micrometer-bom 1.10.7

Fixed in https://github.com/micrometer-metrics/tracing/commit/8ae7d6ebc083753740c83dab4b989205347101d2


Create an influxd subcommand to repair corrupt DBRP mappings

A bug in early 2.0.x versions of InfluxDB caused the influxd upgrade command to generate DBRP mappings without bucket or organization IDs. The bug was fixed in 2.0.2, but anyone who migrated their data before that version will start to see panics when using the V1 APIs after upgrading to the latest V2 binaries. A work-around exists to repair the mappings from the outside using the API, but it's fairly complex. It'd be nice if we could provide a CLI command that performs the same fixup logic.

@danxmoran we should be able to validate and fix these with a DB migration. We have infrastructure to handle this very case:

Create a new migration using the kvmigrate internal command:
go run internal/cmd/kvmigrate/main.go create <name of migration>

This will generate an empty Go migration file in the kv/migration/all folder

Implement an UpOnlyMigration such as:

https://github.com/influxdata/influxdb/blob/8ca2989717bb3db9e0bfcafe4e2d1e367826f285/kv/migration/all/0011_populate-dashboards-owner-id.go#L13


Update service-fabric-diagnostics-event-generation-perf.md

These perf counters are for Service Remoting only apparently.

@jpoon : Thanks for your contribution! The author, @dkkapur, has been notified to review your proposed change.

@jpoon thanks for going through and trying all this.
#sign-off


Update Let's Encrypt documentation about resolvers for certificateresolvers

What does this PR do?
Update documentation about certificatesresolvers to point out that domain names are not supported as resolvers.
Motivation
I had to find out the hard way that domain names are not supported. I migrated my domain from Cloudflare toDdigitalOcean and thus had to update my certificatesresolvers with the new config and new API auth token, etc etc. After a lot of time struggling and failing to get it to work. I came up with the idea to try an IP address instead of a domain name, and it actually worked.
Additional Notes
Feel free to change the text I put. I just wanted to point this out. Please excuse me if this is already documented elsewhere and I looked over it.

Hi @tomwiggers,
you're assertion is not true.
But you need to use a domain name that your system is able to resolve. (cf https://github.com/go-acme/lego/blob/fb72e1a292f256c85fade0f4ae146f5f810efbdf/challenge/dns01/nameserver.go#L24-L27)

I am sure my system is able to resolve ns1.digitalocean.com, I just verified this using nslookup.

I am sure my system is able to resolve ns1.digitalocean.com, I just verified this using nslookup.

It's possible that within the Traefik container it's unable to resolve domain names. Traefik does not come with network tools, or even a shell, so you may want to try and spin up busybox and check to make sure that your containers can do DNS lookups.

The traefik container is also able to resolve domain names. The nslookup command is available inside the container and it successfully resolves. Bash is not available, no, but sh is.
Anyway if the container did not come with networking tools, then why would the default be DNS-based nameservers? As @jbdoumenjou pointed out.

The traefik container is also able to resolve domain names. The nslookup command is available inside the container and it successfully resolves. Bash is not available, no, but sh is.

I just double checked and this is accurate. I was referring to command line networking tools, it appears the code @jbdoumenjou referenced leverages the go library https://github.com/miekg/dns
In any case, it seems peculiar that the default lookups use FQDN, while you're unable to.

https://github.com/containous/traefik/tree/v2.2.1/integration/fixtures/acme
I didn't see where custom DNS servers being used in these tests, and it's possible this case isn't covered - but I spent ~5 minutes looking, so this may be something worth investigating further.

test


Fix dimension labels for static Jobs.

The dimensions weren't being included in the returned data from
scrapeStaticJob, and therefore weren't being set as labels in the
metrics output.
Fixes #75

Thanks a lot. Will release on Friday.


Improve Russian translation

we should rebase this so it can get integrated in the new translations framework, after merging we can handle any future fixes on the translation ui


Duplicate Keys Item when use SXSSF

Hey guys,
I've using XSSF mode like for 6 months but now my Excel file increase a lot of its size so I switch to Streaming Mode but I've been seeing a lot of weird issues related to "Cannot save item with duplicate Key".
Error comes exactly from the Dictionary object (C# exception actually), but why this does not implement the same structure as XSSF?

I think this is a bug of SXSSF. How many rows and columns do you have in your current Excel file in average?

So far more than 10000 keys, I see the code I think it's a bug from the Dictionary structure. I'll clon the project and check If I can think something.
Thanks Tony

Looks your company is using NPOI. Can you contribue your use case to https://github.com/nissl-lab/npoi/issues/705 ?

Looks your company is using NPOI. Can you contribue your use case to #705 ?

I'm not working at PwC anymore :sob:

I solved this bug by directly changing the source code in version 2.5.1. add code " if (!numberFormats.ContainsKey(formatId))"

cause:


Could not open library: libduckdb.so

I'm tryna use your duckdb for my very simple Deno project, but getting stuck on these:
error: Uncaught (in promise) Error: Could not open library: Could not open library: libduckdb.so: cannot open shared object file: No such file or directory
  return Deno.dlopen(file, symbols);
              ^
    at new DynamicLibrary (ext:deno_ffi/00_ffi.js:440:46)
    at Object.dlopen (ext:deno_ffi/00_ffi.js:577:10)
    at prepare (https://deno.land/x/plug@0.5.2/plug.ts:117:15)
    at eventLoopTick (ext:core/01_core.js:181:11)
    at async https://deno.land/x/duckdb@0.1.1/lib.js:47:

I've been reloading the dependencies:
$ deno install -r<EMAIL_ADDRESS>✅ Successfully installed duckdb
/home/vi/.deno/bin/duckdb

but it still doesn't work.
ICYMI, I'm using Deno 1.33.4 on my Ubuntu 18.04 (WSL2):
$ deno --version
deno 1.33.4 (release, x86_64-unknown-linux-gnu)
v8 <IP_ADDRESS>
typescript 5.0.4

I had to download the C/C++ installer for DuckDB from here, https://duckdb.org/docs/installation/index
Then extract the .so file from within the package zip, and place that on the library path. This may be something like /usr/local/lib. I also had some success exporting LD_LIBRARY_PATH to a known directory and placing the library file there.
Hope this helps!

I had to download the C/C++ installer for DuckDB from here, https://duckdb.org/docs/installation/index
Then extract the .so file from within the package zip, and place that on the library path. This may be something like /usr/local/lib. I also had some success exporting LD_LIBRARY_PATH to a known directory and placing the library file there.
Hope this helps!

Thank you, I've switched to a real Linux environment (instead of using WSL2) and didn't face this issue then. Anyway, let me check your approach in my old WSL2. :D

On second look, this is still a problem - looks like the use of the plug library is failing. I'll have a further dig.

@gilesdring Any success?
I am facing the same issue. Running the example from the README:
$ deno run --allow-ffi --unstable-ffi -A duckdb-deno.ts
error: Uncaught (in promise) Error: Could not open library: Could not open library: libduckdb.so: cannot open shared object file: No such file or directory
  return Deno.dlopen<S>(await download(options), symbols as any);
              ^
    at new DynamicLibrary (ext:deno_ffi/00_ffi.js:454:42)
    at Object.dlopen (ext:deno_ffi/00_ffi.js:558:10)
    at dlopen (https://jsr.io/@divy/plug/1.0.3/mod.ts:155:15)
    at eventLoopTick (ext:core/01_core.js:168:7)
    at async https://jsr.io/@divy/duckdb/0.2.1/lib.js:41:27


Graphs : Dijkstra's Algorithm

Hi @amritansh22 please assign me this issue.

Yes sure. I have assigned it to you.


issue #218 - Add conditional event handling with inspector

Motivation
The author of the #218 issue noticed that on Android inspection does not block touches which is a great inconvenience in developing
Changes
As there's no native or JS hook on enabling "Inpsect", I decided to use 'toggleElementInspector' which seems to be fairly fine.
I suppose it's a matter for discussion, @kmagiera, @angly-cat if it is a good idea to block every touch event depending on inspector's state

done


Can't find ini file for NickelClock

Hi,
I wanted to install this in my Kobo, but the file is not on the specified folders. I have turned on the "Show hidden files" option, but I only see the "Uninstall" file. Hope you can please help me out. I'm using Mac and Windows, but not results so far.
Thanks and thank you for the hard work

I am having the same issue with the latest version as well.


More control over BufReader

Rendered.

I think you need to decide on one behavior.

do a single read that appends to the buffer
or read until EOF or buffer.len() == n

Right now the function suggests 2. but the guaranteed behavior is actually 1.
So either drop the min argument and rename to read_to_bufor sth, or change the function to guarantee EOF or filled to the minimum.

@oli-obk Ideally, fill_buf() would read into the buffer every time it's called. I can kind of see why that would be undesirable, but the method name sort of implies that's what it does.
Looping until EOF or the minimum is met would be the best bet, but it would break back-compat if it was added to the BufRead trait because it couldn't be implemented in terms of fill_buf(), at least with its current semantics. I do ask that question in the last section; I don't know if you read that far. But even then, the buffer length can't be guaranteed because the source could be at EOF, so the user would still have to check it.

You are adding a new function. That function could by default simply return an Err(Error::new(ErrorKind::Other, FillBufMinNotImplemented)). The standard library can then actually implement that function for all trait implementors. This would cause some repetition, but it would not be a breaking change.
alternatively as @seanmonstar suggested, the body could simply be unimplemented!().

Ideally, fill_buf() would read into the buffer every time it's called. I can kind of see why that would be undesirable, but the method name sort of implies that's what it does.

yea, this was discussed here: https://github.com/rust-lang/rfcs/pull/1015
I wonder if an RFC changing fill_buf() to read on every call (until the buffer is full?) would be accepted.

Returning an error or panicking by default is undesirable because it would violate users' expecations. I don't think any other part of the stdlib stubs APIs out like that. However, it might be good to discuss this so we can have an established strategy for adding methods to traits.
It doesn't look like the semantics change to fill_buf() that @seanmonstar proposed as an alternative was actually discussed at all. That could definitely be a viable approach, and I don't think it would actually break any existing usage of the method.
Another alternative would be to add just read_into_buf() and provide a default impl that returns try!(self.fill_buf()).len(). Or implement read_into_buf() and get_buf() as inherent methods on BufReader, since that's the struct that's primarily giving us trouble here.

Or implement read_into_buf() and get_buf() as inherent methods on BufReader, since that's the struct that's primarily giving us trouble here.

Why not combine both:
get_buf() simply forwards to fill_buf() but is overridden in all std-impls to not cause a read.
fill_buf() always reads if the buffer isn't full yet. (in case the backend is a fixed sized array e.g.)

get_buf() simply forwards to fill_buf() but is overridden in all std-impls to not cause a read.

How would that work?

http://is.gd/mqRi6W

@cybergeek94 I'm concerned about this RFC for a few reasons, and I'll try to explain them here:

Primarily, this RFC as written is a breaking change, which unfortunately is no longer acceptable. This must be written assuming that the change is backwards compatible, so one of the alternatives you've listed may want to be taken.
I'm a little uneasy about the motivation here. You don't have any guarantee about the internal size of the buffer for each buffered reader, so it's somewhat unclear to me in what case you can rely on it being able to give a larger buffer after this method is called.
If you want previous data to be preserved, it may force all existing data in the buffer to be shuffled down to the beginning of the vector its residing in.
The documentation on this function says that implementations are not required to meet this minimum, but this seems to me like it's saying you basically can't rely on this function acting its purpose? That seems somewhat odd to me.

@alexcrichton
I did submit a pre-RFC on Internals, yet after almost a week and over a hundred views and one "like", no one has responded to it. I am aware Rustcamp happened last weekend, but other similar discussions that were submitted later have loads of comments. I assumed that meant no one had a strong enough opinion on it to speak their piece.
The primary motivation is made very clear, in that without an API change like this, there's no way to explicitly tell the BufRead impl that you want more data in the buffer without emptying it. This is unergonomic for the reasons stated in the RFC. There's no promise about the minimum length of the buffer because it can't always be guaranteed, such as when the source is at EOF.
The API contract only guarantees a "best effort" to meet that minimum, which could be one bulk read, or as many as needed until the minimum is met or the source returns EOF.

Primarily, this RFC as written is a breaking change, which unfortunately is no longer acceptable. This must be written assuming that the change is backwards compatible, so one of the alternatives you've listed may want to be taken.

I'm very much aware it is unacceptable because it is breaking. That's precisely the reason why I provided the two alternatives, because I didn't expect the base proposal to be accepted at face-value. I structured it that way to give the base design first, and then I expand on the more practical approaches. If this isn't exactly in the spirit of the RFC process, I apologize. It's only my first go at it.
My preferred alternative is the latter of the two (implement as an inherent method on BufReader), because that is what I ended up implementing in my own custom solution. The semantics of BufReader is really the primary concern here, because most of the other concrete BufRead impls (e.g. &[u8], Cursor<Vec<u8>>) return all the data that will ever be available. The only real exception is StdinLock, which may benefit from a similar API.

If you want previous data to be preserved, it may force all existing data in the buffer to be shuffled down to the beginning of the vector its residing in.

This is something I hadn't considered until recently, but I wasn't sure what should be done about it (my implementation actually does nothing). If cache locality isn't a huge concern, then the data only needs to be shifted down once the end of the buffer allocation, or some threshold, has been reached. If all the data in the buffer ends up being consumed, the impl can just start back at the beginning (which is what mine does).

@oli-obk I don't think anyone would go for that because it would depend too much on method selection semantics, which could change later on and silently break it.

@cybergeek94 they can't change, that would break ridiculous amounts of code. And it would not ever break it truly, just be less efficient due to reading on every get_buf call (which can actually be unit tested against, so shouldn't be a problem).

I assumed that meant no one had a strong enough opinion on it to speak their piece.

Unfortunately we're all quite busy so we don't always have time to vet all RFCs, Pre-RFCs, and internals posts. Silence doesn't necessarily mean consensus!

There's no promise about the minimum length of the buffer because it can't always be guaranteed, such as when the source is at EOF, or when the requested size is larger than the internal buffer can hold. The API contract only guarantees a "best effort" to meet that minimum, which could be one bulk read, or as many as needed until the minimum is met or the source returns EOF.

The weird part about this API to me is that it basically provides no guarantees. It says that in some situations you may be able to read some more data, but you can never concretely say in which situations without otherwise knowing everything else in play.
As a result this sounds like a much more appropriate method for concrete instances rather than the trait. For example it doesn't necessarily make sense for anyone operating over a generic BufRead to make use of this method because there are so few guarantees about what happens after it's called.

If this isn't exactly in the spirit of the RFC process, I apologize. It's only my first go at it.

No worries! It's much easier to focus discussion on an RFC if the text is basically perpetually in a state of "ready to be merged".

@alexcrichton What might be the best idea here is to go back to the previous RFC and take the read_into_buf() and get_buf() methods it proposed for the BufRead trait, and implement those as inherent methods on BufReader. Then its BufRead impl can be implemented in terms of those, without changing the semantics. That would suit my use case without breaking anything.
Should I update the RFC in-place or withdraw, revise, and resubmit?

Should I update the RFC in-place or withdraw, revise, and resubmit?

Feel free to update in-place!

What should people do, then, if they feel like their post isn't getting enough attention?

I think you basically did the right thing opening an RFC afterwards, just don't be super surprised if you hear some new opinons :)

@alexcrichton Beginning my revised draft now. I do have one concern we haven't talked about: should we provide a method to allow the user to grow the internal buffer if they need more than 64k bytes? It seems like a desirable feature since we're already giving them control over when the buffer is read into.

@alexcrichton I have these in my "Unresolved Questions" section. Perhaps we should address them before I commit the revisions so I can leave that section empty:

Should read_into_buf() return the new length of the buffer instead of the number of bytes read?

This would be more ergonomic for the author's specific use-case but this might not always be the case.
It might be useful to do while try!(buf.read_into_buf()) > 0 {} to make sure the buffer is filled.

Should another method be added that allows the user to grow BufReader's internal buffer, in case they need more bytes in-memory than BufReader allocates by default? (64KiB (64 * 1024) at the time of this writing)

Should read_into_buf() shuffle existing bytes down if there is room at the beginning of the buffer?

Perhaps if there is more room at the beginning than there is at the end?

I think there should definitely be a method to grow the internal buffer.

 DroidLogician: perhaps, yeah, although there may be some form of composition where you get the buffer/cursor out and then put it back in later
 e.g. so you can modify the buffer or w/e
 DroidLogician: for now though I'd keep the RFC as slim as possible

So another RFC later on, perhaps.

I have pushed the new revisions.

@cybergeek94 do you have some example code which makes use of these methods? I'm still a little uneasy about the guarantees that functions like read_info_buf() will provide. For example if you're working with a generic BufReader then there's no guarantee that any data was actually read, so you can't rely on the method. Additionally the get_buf method seems so close to fill_buf that I'm not totally convinced its worth it.
That being said, I'm curious to see how this would be leveraged, as I may just be missing something?

@alexcrichton an implementation of this can be found in hyper: https://github.com/hyperium/hyper/blob/master/src/buffer.rs
And usage of those methods: https://github.com/hyperium/hyper/blob/master/src/http/h1.rs#L698-L718
Key difference between get_buf and fill_buf: get_buf will NOT perform a read internally. This is important if you want to get the buffer, and not have the chance of blocking.

@alexcrichton While this uses my implementation of fill_buf_min(), the principle is the same: https://github.com/cybergeek94/multipart/blob/master/src/server/boundary.rs#L36
Basically, if the buffer isn't long enough, more data needs to be read into it. The guarantee? That BufReader will actually try to read into the buffer if there's room. fill_buf() will only read if the buffer is completely empty, which is highly problematic if you're trying to work with data in the buffer and you realize you need more.
While this would ideally be fixed by fill_buf() performing a read every time it's called, I can understand why that would be undesirable. That was the problem that fill_buf_min() was supposed to solve.

I may publish the buffer used in hyper, if this isn't accepted. Or perhaps
that's the best course?
On Fri, Aug 14, 2015, 1:22 PM Austin Bonander<EMAIL_ADDRESS>wrote:

@alexcrichton https://github.com/alexcrichton While this uses my
implementation of fill_buf_min(), the principle is the same:
https://github.com/cybergeek94/multipart/blob/master/src/server/boundary.rs#L36
Basically, if the buffer isn't long enough, more data needs to be read
into it. The guarantee? That BufReader will actually try to read into
the buffer if there's room. fill_buf() will only read if the buffer is
completely empty, which is highly problematic if you're trying to work with
data in the buffer and you realize you need more.
While this would ideally be fixed by fill_buf() performing a read every
time it's called, I can understand why that would be undesirable. That was
the problem that fill_buf_min() was supposed to solve.
—
Reply to this email directly or view it on GitHub
https://github.com/rust-lang/rfcs/pull/1235#issuecomment-131230768.

My own custom impl works for me. I just figured it'd be nice to have it back in the stdlib type I started from so I didn't have to maintain my own copy.

@seanmonstar
Doesn't that example function return spurious errors sometimes though? If the buffered reader had a only a few bytes left in it but located at the very end, read_into_buf would always return Ok(0), and wouldn't that function return an error? E.g. it'd return Error::TooLarge for having only read 2 bytes so far but there may still be bytes on the stream which can be read.
--
@cybergeek94
I think what you've got makes sense, but what's the benefit over just calling fill_buf? It looks like it's basically the same logic as it has to handle the case that the buffer returned is quite small and doesn't have much data inside it. In that sense this may be a performance optimization, but I don't think it helps structure code any differently?

@alexcrichton
I dunno if there's some part of this that I'm not getting across very well. The problem: I have some data in the buffer. I realize that I need more, because there's not enough data in the buffer to make a decision on. If that data isn't what I'm looking for, I need to pass it on to the user of my API and not just discard it, because it's probably meaningful to them in some way.
With the current BufReader implementation of fill_buf(), I can't always work directly with the buffer because it might give me an incomplete picture. Sometimes, the one read it performs on the source doesn't give me enough data in the buffer to work with. So I need more, but not without discarding the existing data from memory.
I have two possible solutions: clear the buffer and push it to a temporary allocation, or copy BufReader and make it work for my purposes. Neither is satisfactory.
If I work with a temporary allocation, that's more code to shuffle data in and out of it, and to decide when I need to use it and when not. I need to store that buffer in my struct because I might need it later, that's another 12 or 24 bytes, and another field to initialize and drop later. It's also more strain on the CPU cache hierarchy. Some might call this premature optimization, but I would call it "not doing more work than necessary".
Copying the struct to my crate increases my crate's footprint, it's more to maintain, download and build. And technically, it violates the DRY principle because it's an almost-exact copy of an in-distribution struct that doesn't quite work for me as-is.
I still personally believe in fill_buf_min(), I just think I failed to clearly establish its motivations and utility. It still bugs me that "guarantees" are an issue, because in comparison, fill_buf() guarantees nothing. It's unintuitive that a method named fill_buf() would return a nearly empty buffer while more remains to be read just because it's not actually empty yet. I understand that performing a read every time it's called defeats the purpose of buffering, but there could at least be a saner threshold than empty.
read_into_buf() lets the user tell BufReader directly, "I need more data in your buffer, so you should perform a read now, not when it's empty." get_buf() is kind-of a gimme from my perspective; you shouldn't have to call read_into_buf() and then fill_buf() to get the buffer, because then you have to deal with two io::Result values and the second wouldn't really mean anything if you handled the return value of the first.
For read_into_buf() to work sanely, it should copy bytes down to the beginning of the buffer after some threshold is reached, like there's more room at the beginning of the buffer than at the end.

@cybergeek94 I think I understand your use case here, and I definitely agree that this sort of abstraction should be consolidated and it shouldn't need to be maintained separately. That being said, I'm still hung up on this point:

Sometimes, the one read it performs on the source doesn't give me enough data in the buffer to work with. So I need more, but without discarding the existing data from memory.

In your specific use case this may work out, but in general this isn't a problem the current BufReader type can solve. A BufReader does not dynamically grow its internal buffer, so it's possible that a request to read more data simply can't succeed. This means that the API proposed here, read_info_buf, doesn't actually provide the guarantee that it reads any more data, so you already have to be ready for this situation. As a result, you'll already need to have your own side buffer and handle this logic, so this extra method would purely be a performance optimization rather than a fundamental addition I think.

@alexcrichton But if I can guarantee that I'll only need a block up to a certain size, and I can create my BufReader with at least that capacity, then I know that I can keep calling read_into_buf until I either get enough or the socket errors, no need for an extra side buffer.
For example, if I want to wrap around the schannel API for SSL, I have to provide entire encrypted blocks to schannel, I can't give them piecemeal otherwise it'll error and tell me
tell me to read more first. However I do at least have the guarantee of SSL record size limits, so as long as my buffer capacity is greater than the limit, using read_into_buf will work perfectly.

@retep998 you aren't always guaranteed though that the data in your buffer is always at the beginning? The implementations of read_into_buf so far all don't shuffle data to the beginning of the buffer, so even if your buffer is large enough for the largest message you could be looking at the buffered data at the very end, unable to move it down and read in some more.

True, I should add a 'reset_buf' or something to reset after a request is
completed, so it's always starting at the beginning.
On Mon, Aug 17, 2015, 1:12 PM Alex Crichton<EMAIL_ADDRESS>wrote:

@retep998 https://github.com/retep998 you aren't always guaranteed
though that the data in your buffer is always at the beginning? The
implementations of read_into_buf so far all don't shuffle data to the
beginning of the buffer, so even if your buffer is large enough for the
largest message you could be looking at the buffered data at the very end,
unable to move it down and read in some more.
—
Reply to this email directly or view it on GitHub
https://github.com/rust-lang/rfcs/pull/1235#issuecomment-131946648.

@alexcrichton I've made it pretty clear that at some point, read_into_buf should move the existing data back to the beginning before reading in more. It's mostly about deciding when is the optimal time to do so. This calls for the kind of benchmarking and microoptimizing that I don't really specialize in. I'm not averse to adding a method that lets the user do this themselves, but I'm sure you can find some problem with this too.
I think there's a critical difference you're trivializing between the case when there's no more room in the buffer to read into and when the buffer is almost empty and won't be read into until it's completely empty. The user has control over the length of the buffer, at least when creating the BufReader. If they know ahead of time what they're looking for, they can make sure it's long enough. A method to grow the buffer would help, but you've already said it's out of scope for this discussion.

Yeah certainly this case can be handled, but what I've been pointing out is that your existing implementation of read_into_buf doesn't handle this case, nor does the RFC in its current form mention this.
To me this seems like a pretty serious drawback of the functionality as you're not always in a concrete context with complete control over a BufReader and in a generic context this function unfortunately doesn't have much meaning (b/c you don't know the size of the internal buffer)

@alexcrichton Actually it's in the "Unresolved Questions" section. That's why I wanted to discuss it.