Rewrite the Jpeg (De/En)coder

The current Jpeg tools within the library are too slow and buggy, see #307 (Also cannot process 1x1 image)
I need to rewrite them or the library will be unusable for large images. I've been basing my work on https://github.com/briandonahue/FluxJpeg.Core but I cannot get it to work. (See jpeg branch) This is not going well...
Perhaps it would be possible to translate https://golang.org/src/image/jpeg/ from Golang, that, however is beyond my expertise.
This is a showstopper as far as I'm concerned and I really need help.

@JimBobSquarePants Do you have a feel for whether it is possible to get FluxJpeg.Core working? i.e. how close is the jpeg branch to a working jpeg decoder/encoder? I cannot tell by looking at that branch - are there jpeg tests that fail?

@voidstar69   It's not got any tests yet.... (I know! shocking :scream:)  as FluxJpeg doesn't have any unit tests. I've been doing an input/output comparison using a hex editor so far. I'm gonna see if I can pilfer some examples from the golang repo or libjpeg itself.
I think the encoder is pretty much correct except for the actual compression part. All the marker headers before then seem ok. Working a bit blind really.

@voidstar69 I've just added a unit test for the Forward DCT method and that seems to check out correctly. It's either the Huffman encoder or my looping strategy when splitting up the image into 8x8 blocks that's the issue now.
https://github.com/JimBobSquarePants/ImageProcessor/commit/7933d5fcf4da5bf8eaf42e9e0d6709a70d68ce7d

@JimBobSquarePants Could you test FluxJpeg against the existing libjpeg library, writing tests to compare the Huffman encoder, Forward DCT method, and any other steps? Assuming that libjpeg produces the correct result, it might point out where the issue is in FluxJpeg.
I am afraid that I won't have time to work on this for a few days, maybe until the weekend.

@voidstar69   I've been doing a physical comparison for the most part. Testing the Huffman Encoder is difficult though since it's so tightly coupled. It'd be great if you could figure this out. I'm scratching my head! :confounded:

@JimBobSquarePants I have stepped through the new jpeg Encode method, calling writer.Flush() in the Watch window to force it to dump each chunk of the binary data to file, and comparing the binary file in a text editor to the same file (rbg.jpg) produced by the old jpeg encoder.

WriteApplicationHeader has only a few different bytes.
WriteDescreteQuantizationTables has many different bytes.
WriteStartOfFrame I think was identical.
WriteHuffmanTables was quite different, in content and maybe in size.
WriteStartOfScan hard to tell because of previous differences.

I also tried to use BeyondCompare's image comparison mode, but it complained with "Jpeg Marker expected", even after the entire file had been written and flushed by the new encoder.
Eyeballing the data in a text editor, I think I saw lots of character 255 (y with double dots above it, in my character set) in the old encoder output, but missing from the new encoder output. I think this character sometimes represents the start of a new chunk (but is not a marker on its own).
I suggest that you step through the new encoder in the same way as I have, and try modifying/hacking the code to force the new encoder to produce the same bytes as the old encoder. Once a jpeg viewer app accepts the jpeg header, you will know that you are making good progress.   :-)

@JimBobSquarePants BTW on my PC, Windows Explorer thumbnails and Windows Photo Viewer do appear to load the rgb.jpg produced by the new jpeg encoder, albeit with incorrect colours. So maybe the jpeg structure is correct, but the quantization tables / huffman tables / encoded pixel data is wrong?

@voidstar69 Thanks yeah, I think it's got something to do with the way I'm looping through and encoding the pixels. I can add the prior headers to an existing jpeg and it will continue to display correctly.

WriteApplicationHeader is different because for some reason the image resolution is not getting preserved. (I don't think that is a factor)
WriteDescreteQuantizationTables  is different because the original encoder writes the quantization tables separately with a marker between each one. I write the tables combined with only one marker. This is the same as photoshop and also the golang  implementation as far as I can see and shouldn't be an issue. (Values after each marker will be different due to quality settings I believe).
WriteHuffmanTables I thought this one was pretty much identical. I'll have to double check
WriteStartOfScan Yeah that starts ok (The marker) but then goes completely out of whack. FDCT output has been unit tested for an 8x8 block but I don't know If I'm loading the correct data in when I'm calling it. As far as the following huffman encoding goes though the original code is uncommented and does not have unit tests so I'm working blind.

I've been comparing my input/output using HxD

Here's a dump of the input RGB file as a byte array to help with unit testing.
rgb.cs.txt

@voidstar69 Ok So I'm been doing some comparison testing against FluxJpeg and it turns out that their DCT encoder is deviating what what I would expect from the information derived from Wikipedia
My unit tests in the solution match the described output but FluxJpeg doesn't
Expected this.fdct.QuantizeBlock(dct, quantizationTableNumber[componantIndex]);
-26, -3, -6,  2, 2, -1, 0, 0,
  0, -2, -4,  1, 1,  0, 0, 0,
 -3,  1,  5, -1, -1, 0, 0, 0,
 -3,  1,  2, -1, 0,  0, 0, 0,
  1,  0,  0,  0, 0,  0, 0, 0,
  0,  0,  0,  0, 0,  0, 0, 0,
  0,  0,  0,  0, 0,  0, 0, 0,
  0,  0,  0,  0, 0,  0, 0, 0

FluxJpeg _dct.QuantizeBlock(dctArray2, JpegEncoder.QtableNumber[comp]);
-165, -7, 1, -1, 0, 0, 0, 0,
  55, -6, 1, -1, 0, 0, 0, 0,
  23, -4, 1,  0, 0, 0, 0, 0,
  -6, -2, 1,  0, 0, 0, 0, 0,
 -12,  0, 0,  0, 0, 0, 0, 0,
  -2,  0, 0,  0, 0, 0, 0, 0,
   3,  0, 0,  0, 0, 0, 0, 0,
   2,  0, 0,  0, 0, 0, 0, 0

That will mean that their Huffman encoder will not be performing the expected algorithm against the array so I will need to find an example to replace it with or an explanation of the algorithm simple enough for me to derive a method from.

Sent the wrong quality to FluxJpeg above, number still wrong though.

@JimBobSquarePants When I run the same test, it seems to produce the correct output from FluxJpeg for the first component:
FDCT fdct = new FDCT(50);
float[] dctData = fdct.FastFDCT(input);
int[] actual = fdct.QuantizeBlock(dctData, 0);
Assert.Equal(expected, actual);

It is only the output of the second component, from FluxJpeg, that does not match the expected output:
actual = fdct.QuantizeBlock(dctData, 1);
Assert.Equal(expected, actual);

I cannot see how to test LibJpeg in the same way. Can you share your test code for LibJpeg? I want to help you create some more tests of this behaviour, and then we might narrow down on the bug.
Thanks.

@voidstar69 I created my unit test based purely on the info supplied in the Wikipedia page. I can't find any decent unit tests anywhere for libjpeg or language derivatives. The best are in the golang repo. The Wikipedia page lists each expected array for each function, though sadly they don't format the output of the huffman encoding in any way I can follow.

Here's the report from JpegSnoop on my output.
Looks like it's definitely my SOS portion.

JPEGsnoop 1.7.5 by Calvin Hass
  http://www.impulseadventure.com/photo/
  -------------------------------------

  Filename: [E:\github\ImageProcessor\tests\ImageProcessorCore.Tests\TestOutput\Encode\rgb.jpg]
  Filesize: [758] Bytes

Start Offset: 0x00000000
*** Marker: SOI (xFFD8) ***
  OFFSET: 0x00000000

*** Marker: APP0 (xFFE0) ***
  OFFSET: 0x00000002
  Length     = 16
  Identifier = [JFIF]
  version    = [1.1]
  density    = 0 x 0 DPI (dots per inch)
  thumbnail  = 0 x 0

*** Marker: DQT (xFFDB) ***
  Define a Quantization Table.
  OFFSET: 0x00000014
  Table length = 132
  ----
  Precision=8 bits
  Destination ID=0 (Luminance)
    DQT, Row #0:   1   1   1   1   1   1   1   1
    DQT, Row #1:   1   1   1   1   1   1   1   1
    DQT, Row #2:   1   1   1   1   1   1   1   1
    DQT, Row #3:   1   1   1   1   1   1   1   1
    DQT, Row #4:   1   1   1   1   1   1   1   1
    DQT, Row #5:   1   1   1   1   1   1   1   1
    DQT, Row #6:   1   1   1   1   1   1   1   1
    DQT, Row #7:   1   1   1   1   1   1   1   1
    Approx quality factor = 100.00 (scaling=2.99 variance=6.13)
  ----
  Precision=8 bits
  Destination ID=1 (Chrominance)
    DQT, Row #0:   1   1   1   1   1   1   1   1
    DQT, Row #1:   1   1   1   1   1   1   1   1
    DQT, Row #2:   1   1   1   1   1   1   1   1
    DQT, Row #3:   1   1   1   1   1   1   1   1
    DQT, Row #4:   1   1   1   1   1   1   1   1
    DQT, Row #5:   1   1   1   1   1   1   1   1
    DQT, Row #6:   1   1   1   1   1   1   1   1
    DQT, Row #7:   1   1   1   1   1   1   1   1
    Approx quality factor = 100.00 (scaling=1.54 variance=1.58)

*** Marker: SOF0 (Baseline DCT) (xFFC0) ***
  OFFSET: 0x0000009A
  Frame header length = 17
  Precision = 8
  Number of Lines = 3
  Samples per Line = 9
  Image Size = 9 x 3
  Raw Image Orientation = Landscape
  Number of Img components = 3
    Component[1]: ID=0x01, Samp Fac=0x22 (Subsamp 1 x 1), Quant Tbl Sel=0x00 (Lum: Y)
    Component[2]: ID=0x02, Samp Fac=0x11 (Subsamp 2 x 2), Quant Tbl Sel=0x01 (Chrom: Cb)
    Component[3]: ID=0x03, Samp Fac=0x11 (Subsamp 2 x 2), Quant Tbl Sel=0x01 (Chrom: Cr)

*** Marker: DHT (Define Huffman Table) (xFFC4) ***
  OFFSET: 0x000000AD
  Huffman table length = 418
  ----
  Destination ID = 0
  Class = 0 (DC / Lossless Table)
    Codes of length 01 bits (000 total):
    Codes of length 02 bits (001 total): 00
    Codes of length 03 bits (005 total): 01 02 03 04 05
    Codes of length 04 bits (001 total): 06
    Codes of length 05 bits (001 total): 07
    Codes of length 06 bits (001 total): 08
    Codes of length 07 bits (001 total): 09
    Codes of length 08 bits (001 total): 0A
    Codes of length 09 bits (001 total): 0B
    Codes of length 10 bits (000 total):
    Codes of length 11 bits (000 total):
    Codes of length 12 bits (000 total):
    Codes of length 13 bits (000 total):
    Codes of length 14 bits (000 total):
    Codes of length 15 bits (000 total):
    Codes of length 16 bits (000 total):
    Total number of codes: 012

  ----
  Destination ID = 0
  Class = 1 (AC Table)
    Codes of length 01 bits (000 total):
    Codes of length 02 bits (002 total): 01 02
    Codes of length 03 bits (001 total): 03
    Codes of length 04 bits (003 total): 00 04 11
    Codes of length 05 bits (003 total): 05 12 21
    Codes of length 06 bits (002 total): 31 41
    Codes of length 07 bits (004 total): 06 13 51 61
    Codes of length 08 bits (003 total): 07 22 71
    Codes of length 09 bits (005 total): 14 32 81 91 A1
    Codes of length 10 bits (005 total): 08 23 42 B1 C1
    Codes of length 11 bits (004 total): 15 52 D1 F0
    Codes of length 12 bits (004 total): 24 33 62 72
    Codes of length 13 bits (000 total):
    Codes of length 14 bits (000 total):
    Codes of length 15 bits (001 total): 82
    Codes of length 16 bits (125 total): 09 0A 16 17 18 19 1A 25 26 27 28 29 2A 34 35 36
                                         37 38 39 3A 43 44 45 46 47 48 49 4A 53 54 55 56
                                         57 58 59 5A 63 64 65 66 67 68 69 6A 73 74 75 76
                                         77 78 79 7A 83 84 85 86 87 88 89 8A 92 93 94 95
                                         96 97 98 99 9A A2 A3 A4 A5 A6 A7 A8 A9 AA B2 B3
                                         B4 B5 B6 B7 B8 B9 BA C2 C3 C4 C5 C6 C7 C8 C9 CA
                                         D2 D3 D4 D5 D6 D7 D8 D9 DA E1 E2 E3 E4 E5 E6 E7
                                         E8 E9 EA F1 F2 F3 F4 F5 F6 F7 F8 F9 FA
    Total number of codes: 162

  ----
  Destination ID = 1
  Class = 0 (DC / Lossless Table)
    Codes of length 01 bits (000 total):
    Codes of length 02 bits (003 total): 00 01 02
    Codes of length 03 bits (001 total): 03
    Codes of length 04 bits (001 total): 04
    Codes of length 05 bits (001 total): 05
    Codes of length 06 bits (001 total): 06
    Codes of length 07 bits (001 total): 07
    Codes of length 08 bits (001 total): 08
    Codes of length 09 bits (001 total): 09
    Codes of length 10 bits (001 total): 0A
    Codes of length 11 bits (001 total): 0B
    Codes of length 12 bits (000 total):
    Codes of length 13 bits (000 total):
    Codes of length 14 bits (000 total):
    Codes of length 15 bits (000 total):
    Codes of length 16 bits (000 total):
    Total number of codes: 012

  ----
  Destination ID = 1
  Class = 1 (AC Table)
    Codes of length 01 bits (000 total):
    Codes of length 02 bits (002 total): 00 01
    Codes of length 03 bits (001 total): 02
    Codes of length 04 bits (002 total): 03 11
    Codes of length 05 bits (004 total): 04 05 21 31
    Codes of length 06 bits (004 total): 06 12 41 51
    Codes of length 07 bits (003 total): 07 61 71
    Codes of length 08 bits (004 total): 13 22 32 81
    Codes of length 09 bits (007 total): 08 14 42 91 A1 B1 C1
    Codes of length 10 bits (005 total): 09 23 33 52 F0
    Codes of length 11 bits (004 total): 15 62 72 D1
    Codes of length 12 bits (004 total): 0A 16 24 34
    Codes of length 13 bits (000 total):
    Codes of length 14 bits (001 total): E1
    Codes of length 15 bits (002 total): 25 F1
    Codes of length 16 bits (119 total): 17 18 19 1A 26 27 28 29 2A 35 36 37 38 39 3A 43
                                         44 45 46 47 48 49 4A 53 54 55 56 57 58 59 5A 63
                                         64 65 66 67 68 69 6A 73 74 75 76 77 78 79 7A 82
                                         83 84 85 86 87 88 89 8A 92 93 94 95 96 97 98 99
                                         9A A2 A3 A4 A5 A6 A7 A8 A9 AA B2 B3 B4 B5 B6 B7
                                         B8 B9 BA C2 C3 C4 C5 C6 C7 C8 C9 CA D2 D3 D4 D5
                                         D6 D7 D8 D9 DA E2 E3 E4 E5 E6 E7 E8 E9 EA F2 F3
                                         F4 F5 F6 F7 F8 F9 FA
    Total number of codes: 162

*** Marker: SOS (Start of Scan) (xFFDA) ***
  OFFSET: 0x00000251
  Scan header length = 12
  Number of img components = 3
    Component[1]: selector=0x01, table=0(DC),0(AC)
    Component[2]: selector=0x02, table=1(DC),1(AC)
    Component[3]: selector=0x03, table=1(DC),1(AC)
  Spectral selection = 0 .. 63
  Successive approximation = 0x00

*** Decoding SCAN Data ***
  OFFSET: 0x0000025F
  Scan Decode Mode: No IDCT (DC only)
    NOTE: Low-resolution DC component shown. Can decode full-res with [Options->Scan Segment->Full IDCT]

*** ERROR: @ 0x00000287.6, nNumCoeffs>64 [65]
*** ERROR: Bad scan data in MCU(0,0): Lum CSS(1,0) @ Offset 0x00000289.0
           MCU located at pixel=(8,0)
*** ERROR: @ 0x0000029E.5, nNumCoeffs>64 [66]
*** ERROR: Bad scan data in MCU(0,0): Lum CSS(0,1) @ Offset 0x000002A0.3
           MCU located at pixel=(0,8)

  Compression stats:
    Compression Ratio:  1.05:1
    Bits per pixel:    22.81:1

  Huffman code histogram stats:
    Huffman Table: (Dest ID: 0, Class: DC)
      # codes of length 01 bits:        0 (  0%)
      # codes of length 02 bits:        1 ( 25%)
      # codes of length 03 bits:        2 ( 50%)
      # codes of length 04 bits:        0 (  0%)
      # codes of length 05 bits:        0 (  0%)
      # codes of length 06 bits:        0 (  0%)
      # codes of length 07 bits:        1 ( 25%)
      # codes of length 08 bits:        0 (  0%)
      # codes of length 09 bits:        0 (  0%)
      # codes of length 10 bits:        0 (  0%)
      # codes of length 11 bits:        0 (  0%)
      # codes of length 12 bits:        0 (  0%)
      # codes of length 13 bits:        0 (  0%)
      # codes of length 14 bits:        0 (  0%)
      # codes of length 15 bits:        0 (  0%)
      # codes of length 16 bits:        0 (  0%)

    Huffman Table: (Dest ID: 1, Class: DC)
      # codes of length 01 bits:        0 (  0%)
      # codes of length 02 bits:        1 ( 50%)
      # codes of length 03 bits:        0 (  0%)
      # codes of length 04 bits:        0 (  0%)
      # codes of length 05 bits:        0 (  0%)
      # codes of length 06 bits:        0 (  0%)
      # codes of length 07 bits:        0 (  0%)
      # codes of length 08 bits:        1 ( 50%)
      # codes of length 09 bits:        0 (  0%)
      # codes of length 10 bits:        0 (  0%)
      # codes of length 11 bits:        0 (  0%)
      # codes of length 12 bits:        0 (  0%)
      # codes of length 13 bits:        0 (  0%)
      # codes of length 14 bits:        0 (  0%)
      # codes of length 15 bits:        0 (  0%)
      # codes of length 16 bits:        0 (  0%)

    Huffman Table: (Dest ID: 0, Class: AC)
      # codes of length 01 bits:        0 (  0%)
      # codes of length 02 bits:       52 ( 60%)
      # codes of length 03 bits:        2 (  2%)
      # codes of length 04 bits:       10 ( 12%)
      # codes of length 05 bits:        2 (  2%)
      # codes of length 06 bits:        1 (  1%)
      # codes of length 07 bits:       10 ( 12%)
      # codes of length 08 bits:        0 (  0%)
      # codes of length 09 bits:        3 (  3%)
      # codes of length 10 bits:        2 (  2%)
      # codes of length 11 bits:        2 (  2%)
      # codes of length 12 bits:        0 (  0%)
      # codes of length 13 bits:        0 (  0%)
      # codes of length 14 bits:        0 (  0%)
      # codes of length 15 bits:        0 (  0%)
      # codes of length 16 bits:        2 (  2%)

    Huffman Table: (Dest ID: 1, Class: AC)
      # codes of length 01 bits:        0 (  0%)
      # codes of length 02 bits:        3 ( 50%)
      # codes of length 03 bits:        1 ( 17%)
      # codes of length 04 bits:        0 (  0%)
      # codes of length 05 bits:        0 (  0%)
      # codes of length 06 bits:        1 ( 17%)
      # codes of length 07 bits:        0 (  0%)
      # codes of length 08 bits:        0 (  0%)
      # codes of length 09 bits:        0 (  0%)
      # codes of length 10 bits:        0 (  0%)
      # codes of length 11 bits:        0 (  0%)
      # codes of length 12 bits:        0 (  0%)
      # codes of length 13 bits:        0 (  0%)
      # codes of length 14 bits:        1 ( 17%)
      # codes of length 15 bits:        0 (  0%)
      # codes of length 16 bits:        0 (  0%)

  YCC clipping in DC:
    Y  component: [<0=    0] [>255=    0]
    Cb component: [<0=    0] [>255=    0]
    Cr component: [<0=    0] [>255=    0]

  RGB clipping in DC:
    R  component: [<0=    0] [>255=    0]
    G  component: [<0=    0] [>255=    0]
    B  component: [<0=    0] [>255=    0]

  Average Pixel Luminance (Y):
    Y=[ 81] (range: 0..255)

  Brightest Pixel Search:
    YCC=[ -275, -242,    0] RGB=[ 93,103, 38] @ MCU[  0,  0]

  Finished Decoding SCAN Data
    Number of RESTART markers decoded: 0
    Next position in scan buffer: Offset 0x000002AC.4

*** Marker: EOI (End of Image) (xFFD9) ***
  OFFSET: 0x000002F4

*** Searching Compression Signatures ***

  Signature:           01BBB1709AC9C1F89220D955A31A8F34
  Signature (Rotated): 01BBB1709AC9C1F89220D955A31A8F34
  File Offset:         0 bytes
  Chroma subsampling:  2x2
  EXIF Make/Model:     NONE
  EXIF Makernotes:     NONE
  EXIF Software:       NONE

  Searching Compression Signatures: (3347 built-in, 0 user(*) )

          EXIF.Make / Software        EXIF.Model                            Quality           Subsamp Match?
          -------------------------   -----------------------------------   ----------------  --------------
     CAM:[CASIO COMPUTER CO.,LTD   ] [EX-Z750                            ] [                ] Yes
     CAM:[CASIO COMPUTER CO.,LTD.  ] [EX-Z1000                           ] [                ] Yes
     CAM:[PENTAX                   ] [PENTAX Optio S5i                   ] [                ] Yes
     CAM:[SIGMA                    ] [SIGMA SD9                          ] [                ] Yes
     SW :[ACDSee                   ]                                       [100             ]
     SW :[Apple ImageIO.framework  ]                                       [100 (Best)      ]
     SW :[Digital Photo Professiona]                                       [10              ]
     SW :[IJG Library              ]                                       [100             ]
     SW :[MS Office Pic Mgr        ]                                       [                ]
     SW :[Nikon Scan               ]                                       [Excellent Qualit]
     SW :[Picasa                   ]                                       [100 (Maximum)   ]
     SW :[ZoomBrowser EX           ]                                       [highest         ]
     SW :[EOS Viewer Utility       ]                                       [                ]

    The following IJG-based editors also match this signature:
     SW :[GIMP                     ]                                       [100             ]
     SW :[IrfanView                ]                                       [100             ]
     SW :[idImager                 ]                                       [100             ]
     SW :[FastStone Image Viewer   ]                                       [100             ]
     SW :[NeatImage                ]                                       [100             ]
     SW :[Paint.NET                ]                                       [100             ]
     SW :[Photomatix               ]                                       [100             ]
     SW :[XnView                   ]                                       [100             ]

  Based on the analysis of compression characteristics and EXIF metadata:

  ASSESSMENT: Class 1 - Image is processed/edited

  This may be a new software editor for the database.
  If this file is processed, and editor doesn't appear in list above,
  PLEASE ADD TO DATABASE with [Tools->Add Camera to DB]

Here's the input for comparison

JPEGsnoop 1.7.5 by Calvin Hass
  http://www.impulseadventure.com/photo/
  -------------------------------------

  Filename: [E:\github\ImageProcessor\tests\ImageProcessorCore.Tests\TestImages\Formats\Jpg\rgb.jpg]
  Filesize: [794] Bytes

Start Offset: 0x00000000
*** Marker: SOI (xFFD8) ***
  OFFSET: 0x00000000

*** Marker: APP0 (xFFE0) ***
  OFFSET: 0x00000002
  Length     = 16
  Identifier = [JFIF]
  version    = [1.1]
  density    = 96 x 96 DPI (dots per inch)
  thumbnail  = 0 x 0

*** Marker: APP1 (xFFE1) ***
  OFFSET: 0x00000014
  Length          = 102
  Identifier      = [Exif]
  Identifier TIFF = 0x[4D4D002A 00000008]
  Endian          = Motorola (big)
  TAG Mark x002A  = 0x002A

  EXIF IFD0 @ Absolute 0x00000026
    Dir Length = 0x0004
    [XResolution                         ] = 96/1
    [YResolution                         ] = 96/1
    [ResolutionUnit                      ] = Inch
    [Software                            ] = "paint.net 4.0.9"
    Offset to Next IFD = 0x00000000

*** Marker: DQT (xFFDB) ***
  Define a Quantization Table.

  OFFSET: 0x000000C1
  Table length = 67
  ----
  Precision=8 bits
  Destination ID=1 (Chrominance)
    DQT, Row #0:   1   1   1   1   1   1   1   1
    DQT, Row #1:   1   1   1   1   1   1   1   1
    DQT, Row #2:   1   1   1   1   1   1   1   1
    DQT, Row #3:   1   1   1   1   1   1   1   1
    DQT, Row #4:   1   1   1   1   1   1   1   1
    DQT, Row #5:   1   1   1   1   1   1   1   1
    DQT, Row #6:   1   1   1   1   1   1   1   1
    DQT, Row #7:   1   1   1   1   1   1   1   1
    Approx quality factor = 100.00 (scaling=1.54 variance=1.58)

*** Marker: SOF0 (Baseline DCT) (xFFC0) ***
  OFFSET: 0x00000106
  Frame header length = 17
  Precision = 8
  Number of Lines = 3
  Samples per Line = 9
  Image Size = 9 x 3
  Raw Image Orientation = Landscape
  Number of Img components = 3
    Component[1]: ID=0x01, Samp Fac=0x22 (Subsamp 1 x 1), Quant Tbl Sel=0x00 (Lum: Y)
    Component[2]: ID=0x02, Samp Fac=0x11 (Subsamp 2 x 2), Quant Tbl Sel=0x01 (Chrom: Cb)
    Component[3]: ID=0x03, Samp Fac=0x11 (Subsamp 2 x 2), Quant Tbl Sel=0x01 (Chrom: Cr)

*** Marker: DHT (Define Huffman Table) (xFFC4) ***
  OFFSET: 0x00000119
  Huffman table length = 31
  ----
  Destination ID = 0
  Class = 0 (DC / Lossless Table)
    Codes of length 01 bits (000 total):
    Codes of length 02 bits (001 total): 00
    Codes of length 03 bits (005 total): 01 02 03 04 05
    Codes of length 04 bits (001 total): 06
    Codes of length 05 bits (001 total): 07
    Codes of length 06 bits (001 total): 08
    Codes of length 07 bits (001 total): 09
    Codes of length 08 bits (001 total): 0A
    Codes of length 09 bits (001 total): 0B
    Codes of length 10 bits (000 total):
    Codes of length 11 bits (000 total):
    Codes of length 12 bits (000 total):
    Codes of length 13 bits (000 total):
    Codes of length 14 bits (000 total):
    Codes of length 15 bits (000 total):
    Codes of length 16 bits (000 total):
    Total number of codes: 012

*** Marker: DHT (Define Huffman Table) (xFFC4) ***
  OFFSET: 0x0000013A
  Huffman table length = 181
  ----
  Destination ID = 0
  Class = 1 (AC Table)
    Codes of length 01 bits (000 total):
    Codes of length 02 bits (002 total): 01 02
    Codes of length 03 bits (001 total): 03
    Codes of length 04 bits (003 total): 00 04 11
    Codes of length 05 bits (003 total): 05 12 21
    Codes of length 06 bits (002 total): 31 41
    Codes of length 07 bits (004 total): 06 13 51 61
    Codes of length 08 bits (003 total): 07 22 71
    Codes of length 09 bits (005 total): 14 32 81 91 A1
    Codes of length 10 bits (005 total): 08 23 42 B1 C1
    Codes of length 11 bits (004 total): 15 52 D1 F0
    Codes of length 12 bits (004 total): 24 33 62 72
    Codes of length 13 bits (000 total):
    Codes of length 14 bits (000 total):
    Codes of length 15 bits (001 total): 82
    Codes of length 16 bits (125 total): 09 0A 16 17 18 19 1A 25 26 27 28 29 2A 34 35 36
                                         37 38 39 3A 43 44 45 46 47 48 49 4A 53 54 55 56
                                         57 58 59 5A 63 64 65 66 67 68 69 6A 73 74 75 76
                                         77 78 79 7A 83 84 85 86 87 88 89 8A 92 93 94 95
                                         96 97 98 99 9A A2 A3 A4 A5 A6 A7 A8 A9 AA B2 B3
                                         B4 B5 B6 B7 B8 B9 BA C2 C3 C4 C5 C6 C7 C8 C9 CA
                                         D2 D3 D4 D5 D6 D7 D8 D9 DA E1 E2 E3 E4 E5 E6 E7
                                         E8 E9 EA F1 F2 F3 F4 F5 F6 F7 F8 F9 FA
    Total number of codes: 162

*** Marker: DHT (Define Huffman Table) (xFFC4) ***
  OFFSET: 0x000001F1
  Huffman table length = 31
  ----
  Destination ID = 1
  Class = 0 (DC / Lossless Table)
    Codes of length 01 bits (000 total):
    Codes of length 02 bits (003 total): 00 01 02
    Codes of length 03 bits (001 total): 03
    Codes of length 04 bits (001 total): 04
    Codes of length 05 bits (001 total): 05
    Codes of length 06 bits (001 total): 06
    Codes of length 07 bits (001 total): 07
    Codes of length 08 bits (001 total): 08
    Codes of length 09 bits (001 total): 09
    Codes of length 10 bits (001 total): 0A
    Codes of length 11 bits (001 total): 0B
    Codes of length 12 bits (000 total):
    Codes of length 13 bits (000 total):
    Codes of length 14 bits (000 total):
    Codes of length 15 bits (000 total):
    Codes of length 16 bits (000 total):
    Total number of codes: 012

*** Marker: DHT (Define Huffman Table) (xFFC4) ***
  OFFSET: 0x00000212
  Huffman table length = 181
  ----
  Destination ID = 1
  Class = 1 (AC Table)
    Codes of length 01 bits (000 total):
    Codes of length 02 bits (002 total): 00 01
    Codes of length 03 bits (001 total): 02
    Codes of length 04 bits (002 total): 03 11
    Codes of length 05 bits (004 total): 04 05 21 31
    Codes of length 06 bits (004 total): 06 12 41 51
    Codes of length 07 bits (003 total): 07 61 71
    Codes of length 08 bits (004 total): 13 22 32 81
    Codes of length 09 bits (007 total): 08 14 42 91 A1 B1 C1
    Codes of length 10 bits (005 total): 09 23 33 52 F0
    Codes of length 11 bits (004 total): 15 62 72 D1
    Codes of length 12 bits (004 total): 0A 16 24 34
    Codes of length 13 bits (000 total):
    Codes of length 14 bits (001 total): E1
    Codes of length 15 bits (002 total): 25 F1
    Codes of length 16 bits (119 total): 17 18 19 1A 26 27 28 29 2A 35 36 37 38 39 3A 43
                                         44 45 46 47 48 49 4A 53 54 55 56 57 58 59 5A 63
                                         64 65 66 67 68 69 6A 73 74 75 76 77 78 79 7A 82
                                         83 84 85 86 87 88 89 8A 92 93 94 95 96 97 98 99
                                         9A A2 A3 A4 A5 A6 A7 A8 A9 AA B2 B3 B4 B5 B6 B7
                                         B8 B9 BA C2 C3 C4 C5 C6 C7 C8 C9 CA D2 D3 D4 D5
                                         D6 D7 D8 D9 DA E2 E3 E4 E5 E6 E7 E8 E9 EA F2 F3
                                         F4 F5 F6 F7 F8 F9 FA
    Total number of codes: 162

*** Marker: SOS (Start of Scan) (xFFDA) ***
  OFFSET: 0x000002C9
  Scan header length = 12
  Number of img components = 3
    Component[1]: selector=0x01, table=0(DC),0(AC)
    Component[2]: selector=0x02, table=1(DC),1(AC)
    Component[3]: selector=0x03, table=1(DC),1(AC)
  Spectral selection = 0 .. 63
  Successive approximation = 0x00

*** Decoding SCAN Data ***
  OFFSET: 0x000002D7
  Scan Decode Mode: No IDCT (DC only)
    NOTE: Low-resolution DC component shown. Can decode full-res with [Options->Scan Segment->Full IDCT]

  Scan Data encountered marker   0xFFD9 @ 0x00000318.0

  Compression stats:
    Compression Ratio:  1.27:1
    Bits per pixel:    18.96:1

  Huffman code histogram stats:
    Huffman Table: (Dest ID: 0, Class: DC)
      # codes of length 01 bits:        0 (  0%)
      # codes of length 02 bits:        2 ( 50%)
      # codes of length 03 bits:        0 (  0%)
      # codes of length 04 bits:        0 (  0%)
      # codes of length 05 bits:        0 (  0%)
      # codes of length 06 bits:        0 (  0%)
      # codes of length 07 bits:        2 ( 50%)
      # codes of length 08 bits:        0 (  0%)
      # codes of length 09 bits:        0 (  0%)
      # codes of length 10 bits:        0 (  0%)
      # codes of length 11 bits:        0 (  0%)
      # codes of length 12 bits:        0 (  0%)
      # codes of length 13 bits:        0 (  0%)
      # codes of length 14 bits:        0 (  0%)
      # codes of length 15 bits:        0 (  0%)
      # codes of length 16 bits:        0 (  0%)

    Huffman Table: (Dest ID: 1, Class: DC)
      # codes of length 01 bits:        0 (  0%)
      # codes of length 02 bits:        0 (  0%)
      # codes of length 03 bits:        0 (  0%)
      # codes of length 04 bits:        0 (  0%)
      # codes of length 05 bits:        0 (  0%)
      # codes of length 06 bits:        0 (  0%)
      # codes of length 07 bits:        1 ( 50%)
      # codes of length 08 bits:        0 (  0%)
      # codes of length 09 bits:        1 ( 50%)
      # codes of length 10 bits:        0 (  0%)
      # codes of length 11 bits:        0 (  0%)
      # codes of length 12 bits:        0 (  0%)
      # codes of length 13 bits:        0 (  0%)
      # codes of length 14 bits:        0 (  0%)
      # codes of length 15 bits:        0 (  0%)
      # codes of length 16 bits:        0 (  0%)

    Huffman Table: (Dest ID: 0, Class: AC)
      # codes of length 01 bits:        0 (  0%)
      # codes of length 02 bits:        0 (  0%)
      # codes of length 03 bits:        0 (  0%)
      # codes of length 04 bits:        4 ( 36%)
      # codes of length 05 bits:        0 (  0%)
      # codes of length 06 bits:        0 (  0%)
      # codes of length 07 bits:        2 ( 18%)
      # codes of length 08 bits:        1 (  9%)
      # codes of length 09 bits:        0 (  0%)
      # codes of length 10 bits:        1 (  9%)
      # codes of length 11 bits:        0 (  0%)
      # codes of length 12 bits:        0 (  0%)
      # codes of length 13 bits:        0 (  0%)
      # codes of length 14 bits:        0 (  0%)
      # codes of length 15 bits:        0 (  0%)
      # codes of length 16 bits:        3 ( 27%)

    Huffman Table: (Dest ID: 1, Class: AC)
      # codes of length 01 bits:        0 (  0%)
      # codes of length 02 bits:        2 ( 13%)
      # codes of length 03 bits:        0 (  0%)
      # codes of length 04 bits:        0 (  0%)
      # codes of length 05 bits:        0 (  0%)
      # codes of length 06 bits:        1 (  6%)
      # codes of length 07 bits:        2 ( 13%)
      # codes of length 08 bits:        0 (  0%)
      # codes of length 09 bits:        3 ( 19%)
      # codes of length 10 bits:        1 (  6%)
      # codes of length 11 bits:        0 (  0%)
      # codes of length 12 bits:        1 (  6%)
      # codes of length 13 bits:        0 (  0%)
      # codes of length 14 bits:        0 (  0%)
      # codes of length 15 bits:        0 (  0%)
      # codes of length 16 bits:        6 ( 38%)

  YCC clipping in DC:
    Y  component: [<0=    0] [>255=    0]
    Cb component: [<0=    0] [>255=    0]
    Cr component: [<0=    0] [>255=    0]

  RGB clipping in DC:
    R  component: [<0=    0] [>255=    0]
    G  component: [<0=    0] [>255=    0]
    B  component: [<0=    0] [>255=    0]

  Average Pixel Luminance (Y):
    Y=[ 39] (range: 0..255)

  Brightest Pixel Search:
    YCC=[ -288,  445,  -75] RGB=[ 77, 80,189] @ MCU[  0,  0]

  Finished Decoding SCAN Data
    Number of RESTART markers decoded: 0
    Next position in scan buffer: Offset 0x00000317.4

*** Marker: EOI (End of Image) (xFFD9) ***
  OFFSET: 0x00000318

*** Searching Compression Signatures ***

  Signature:           01BBB1709AC9C1F89220D955A31A8F34
  Signature (Rotated): 01BBB1709AC9C1F89220D955A31A8F34
  File Offset:         0 bytes
  Chroma subsampling:  2x2
  EXIF Make/Model:     NONE
  EXIF Makernotes:     NONE
  EXIF Software:       OK   [paint.net 4.0.9]

  Searching Compression Signatures: (3347 built-in, 0 user(*) )

          EXIF.Make / Software        EXIF.Model                            Quality           Subsamp Match?
          -------------------------   -----------------------------------   ----------------  --------------
     CAM:[CASIO COMPUTER CO.,LTD   ] [EX-Z750                            ] [                ] Yes
     CAM:[CASIO COMPUTER CO.,LTD.  ] [EX-Z1000                           ] [                ] Yes
     CAM:[PENTAX                   ] [PENTAX Optio S5i                   ] [                ] Yes
     CAM:[SIGMA                    ] [SIGMA SD9                          ] [                ] Yes
     SW :[ACDSee                   ]                                       [100             ]
     SW :[Apple ImageIO.framework  ]                                       [100 (Best)      ]
     SW :[Digital Photo Professiona]                                       [10              ]
     SW :[IJG Library              ]                                       [100             ]
     SW :[MS Office Pic Mgr        ]                                       [                ]
     SW :[Nikon Scan               ]                                       [Excellent Qualit]
     SW :[Picasa                   ]                                       [100 (Maximum)   ]
     SW :[ZoomBrowser EX           ]                                       [highest         ]
     SW :[EOS Viewer Utility       ]                                       [                ]

    The following IJG-based editors also match this signature:
     SW :[GIMP                     ]                                       [100             ]
     SW :[IrfanView                ]                                       [100             ]
     SW :[idImager                 ]                                       [100             ]
     SW :[FastStone Image Viewer   ]                                       [100             ]
     SW :[NeatImage                ]                                       [100             ]
     SW :[Paint.NET                ]                                       [100             ]
     SW :[Photomatix               ]                                       [100             ]
     SW :[XnView                   ]                                       [100             ]

  Based on the analysis of compression characteristics and EXIF metadata:

  ASSESSMENT: Class 1 - Image is processed/edited

  This may be a new software editor for the database.
  If this file is processed, and editor doesn't appear in list above,
  PLEASE ADD TO DATABASE with [Tools->Add Camera to DB]

Maybe this will be useful?
http://www.opennet.ru/docs/formats/jpeg.txt

what's about getting source of imagemagick or something and do a "1-on-1"-copy in c#?

Copying from ImageMagick is not really an option because it uses libjpeg(-turbo). You could translate libjpeg to C# but I assume that it is going to be a lot of work. I noticed you starred https://github.com/BitMiracle/libjpeg.net, maybe that project can help you figure out what to do?

I'm using BitMiracle already, it's incredibly slow for larger images and buggy for tiny ones plus the code is a minefield to read. I've spent weeks pouring through the source and got nowhere.

I've been trying to do a byte by byte comparison between my code and FJCore (difficult due to differences in rounding) and I think it's the endianness of my output from the Huffman Table.
I'm using a specialist EndianBinaryWriter with Big Endian encoding to write to the stream and I think that the HuffmanTable class is already doing some sort of conversion already.
I suppose I have two choices. Use a normal stream end big endian encode everything outside the pixel data or convert the Huffman Table so that it does not switch the endianess.
The latter is certainly beyond me but maybe not someone else reading.
Here's hoping....

Well using the underlying stream works.... I'd much, much rather use the encoder throughout though.

Has anyone looked at libjpeg.net -- it appears to be a port of the libjpeg native library for C#?
https://github.com/BitMiracle/libjpeg.net

I will take a crack at converting https://golang.org/src/image/jpeg/ if you are still interested in that method.  Let me know if you still want to pursue that route.

Hi there,
That would be by far and away my preferred route so if you can get that working that would be amazing! :+1:
Let me know if you need anything from me.
Cheers
James

Ok -- the initial conversion of the library is done.  Right now it only supports decoding and the encapsulation is a little messy and needs to be cleaned up, but it works. I have not run any speed tests, but with some profiling we should be able to get it running quick (it's not really that complicated).
The sample program I used is basically
Image img1 = new Image(new FileStream(args[0], FileMode.Open));
img1.Save(new FileStream(args[1], FileMode.Create), new PngFormat());

I am building on a mac/mono, so I don't know what you have to do to get it to compile, but I'm assuming it will just work.
https://github.com/mweber26/ImageProcessor

Encoding is up in the latest commit from that same repository.

AMAZING!
I'm absolutely blown away by the speed you have done this! Simply getting a working encoder with less functionality has taken me months and I was working from a C# source. This is an absolute game changer as far as the project is concerned. I'm so happy!
I've not really played with Mono, I got the project building using Visual Studio Code on the Mac (Which I think is the recommended approach just now?) though there appears to be DNX support for Xamarin Studio via a plugin. http://lastexitcode.com/blog/2016/01/01/DNXSupportInXamarinStudio/ I'd like to keep a single codebase without having to add code for System.Numerics.Vectors just now if possible.
Getting it neatly encapsulated is certainly important now as it will allow others to more easily add functionality like progressive support and the ability to choose the chroma subsampling approach. (Eventually EXIF support hopefully also).
There are a few things in my   jpeg branch that you may find useful like constants and plenty of comments, I also have some endian readers/writers in my core branch that could be useful.
Let me know if you need anything from me

Thanks,  it wasn't too bad.  There are a couple weird things in go that needed to be fixed after the line-for-line conversion.   But adding printfs to both and comparing the files allowed me to track down the conversion issues pretty quick.
I do everything from the command line, so no visual studio code for me, but am only targeting 4.5 now (it appears that numerics were added natively in 4.6).  I have to try to upgrade xamarin and see if it gets a new target.  I wasn't including Numerics as a permanent solution, I just needed something to get it up and running.  I also have no problem with you merging the Jpeg decoder stuff without my numerics additions since I have very strange release requirements for my application of this library.
I will work on the encapsulation and clean the code up a little bit.  What are you looking for for calling this issue "complete"?

I also figured out how to build via command line:
curl -sSL https://raw.githubusercontent.com/aspnet/Home/dev/dnvminstall.sh | DNX_BRANCH=dev sh && source ~/.dnx/dnvm/dnvm.sh
dnvm upgrade
cd ~/dev/ImageProcessor/src/ImageProcessorCore
dnu restore
dnu build

Nice! The new CLI they've been putting together is great for that kind of stuff. Way better than MSBuild.
I think if you can make it match the existing coding format a bit more and clean up the encapsulation we can mark it as complete. Any perf improvements or code transition to use existing functionality (e.g. implicit colorspace casting) can be figured out after.
If you fancy giving the subsampling option for the encoder a try please have a go though. :wink:

I will look at the subsampling, but it's not super high on my priority list since the project I'm working on only decodes jpegs at this time.  But I might take a crack at it -- it shouldn't be too hard.  We should probably create another issue for it after the gojpeg is merged.

Ace... I'll have a look myself also now this is converted. Just merged your commit, it's a real relief to get this closed! :smile:

2,661 additions and 7,979 deletions
That is always nice to see

Yup!

From looking around.  The only two encoding subsamples that are really used at 420 and 444.  Is there a reason to support the other, way less common, ones?  444 is very easy to do, I already have a pull that does that.  The other, more exotic, subsamples are more complicated.

Yeah that works for me. If someone specifically wants the more rare variants they can add them. From what I see from your PR that should be a case of adding another switch and a method so shouldn't be too daunting for anyone with the requirement.

Yeah -- that's the concept.  In practically, I couldn't get it to work that easily.  I attempted 422 (which is 2 per width, 1 per height), and had all sorts of problems.  You have to implement the 422 code to output the blocks and another 16x16_8x8_scale to combine the two width/two height and the 4 versions to complete.
After a few failed attempts and some research, basically nobody uses anything but 444 and 420, so I skipped it.

Cool. We'll stick to that then definitely for now.


'%' is redundant

Unit of measure

@tomscholz you don't have the auth to merge this kind of obvious improvements without side effects?

This is honestly a very nitpicky pull request. This is more of a style convention than anything and really does not need to be changed.


MapFromAttributes not working with Enum fields

The MapFromAttributes is not working when the class includes an enum.  Here is a simple example:
    public void InitializeES()
    {
        //drop the index for debugging purposes, remove this later
        var deleteResponse = _client.DeleteIndex(f => f.Index(_client.Infer.DefaultIndex));
        VerifyResponse(deleteResponse);
        if (!_client.IndexExists(i => i.Index(_client.Infer.DefaultIndex)).Exists)
        {
            var response = _client.CreateIndex(_client.Infer.DefaultIndex);
            VerifyResponse(response);
        }
        VerifyResponse(_client.Map<TestClass>(m => m.MapFromAttributes()));
    }

    private static void VerifyResponse(IResponse response)
    {
        if (!response.IsValid)
        {
            throw new ElasticsearchServerException(response.ServerError);
        }
    }

    [ElasticType(Name = "testclass")]
    public class TestClass
    {
        public int Id { get; set; }

        [ElasticProperty(Index = FieldIndexOption.NotAnalyzed)]
        [JsonConverter(typeof(StringEnumConverter))]
        public EnumTester EnumTest { get; set; }
    }

    public enum EnumTester
    {
        Value1,
        Value2
    }

This is the resulting structure from calling GET testindex/_mapping:
{
"testindex": {
"mappings": {
"testclass": {
"properties": {
"id": {
"type": "integer"
}
}
}
}
}
}
The enum is missing from the mapped data structure.
I'm on ES 1.3.0, NEST 1.0.2
Am I doing something wrong or is this a bug?
Thanks in advance guys, I love ES and I love NEST!

Related: http://stackoverflow.com/questions/29194750/attribute-based-index-hints-change-my-results
I noticed this question pop up over at StackOverflow. It sounds more like a bug to silently skip it just because the type is unset, especially because other attributes are being set.
The type should be mandatory, it should be defaulted, or it should at least log a very high level warning to the user.

We can either default to integer, string, or throw an exception.  I'm torn- defaulting to integer seems technically correct, since that's how enums are treated naturally.  On the other hand, I think most people are expecting/want a string.  Thoughts @Mpdreamz ?

@gmarz I think that I like the exception because it covers cases more generally. Enums can perhaps be supported automatically later, but a good start would be failing fast here.

Weighing in here, as the guy who was blindsided by this issue and asked the question.
We are talking about documents, not POCO objects.  How the runtime stores and works with enums is irrelevant from a document standpoint.  What's important is what those numbers convey, and that meaning is expressed in words.  I would argue for storing the enum as a string, the way that ElasticSearch 1.4.x does now.
I would go a step further, and when you automap based on types, all enums would be treated as strings that are not analyzed.  The number of values an enum can hold is finite, and they should never be tokenized and split apart (i.e. with underscores, etc.).  Feel free to prove me wrong, but I would argue enum types would be most often used with terms filters.

In the very near term, I agree with @pickypg.    At least then I'm not spending a week trying to figure out why a query is suddenly broken.

@bloritsch first of all sorry this was such a nuisance to you, I have to admit that I totally misread this issue as enums defaulting to ints instead of strings. Not that they were skipped altogether, this is a real bug.
However in our 1.x releases we have to default to int since thats what Json.NET defaults to as well.
If you write a query/filter right now, without setting the StringEnumConverter, it will write out integers as well. See this unit test and its expected output. The same when you index an enum without that converter in place it will also default to writing enums as ints.
I just submitted a PR so that MapFromAttributes() maps enums as int's here: https://github.com/elastic/elasticsearch-net/pull/1290
Also to clarify:
MapFromAttributes() is documented to not be an exhaustive and fully inclusive mapping, it takes the attributes you set explicitly into account and automatically maps object types and known value types (long, int, double, datetime, etcetera) for you out of the box while still allowing you to override them afterwards.
With that in mind it should throw an exception only when an attribute is present on an unknown value type and no FieldType is explicitly set.
Also the linked xml comment's english is dreadful and totally unclear, I will push an update for that as well.
Both of these are now also part of the PR and should go in the next release.
Thanks @bloritsch and @pickypg for chasing our asses on this!

@Mpdreamz, is it a bug then that when my record was serialized to ElasticSearch it was indeed a string?  This is how it automapped without the explicit field type (using the _mapping?pretty URI):
"releasableTo" : {
        "type" : "string"
}

The records had the enum names in them.  I just thought that was how it was supposed to be, and it seemed to be expected behavior to me.  I was only surprised that the NotAnalyzed wasn't being applied.
When you have a new version pushed to NuGet I'll update my project.  There's more attributes I have to add, but I haven't quite gotten to that part of the functionality yet.

Do you have a StringEnumConverter set on your enum ?
The way to have enums always serialized as strings would be by registering it as a contract converter for enums:
https://github.com/elastic/elasticsearch-net/blob/develop/src/Tests/Nest.Tests.Unit/Internals/Serialize/CustomConvertersTests.cs#L40
Otherwise they will get indexed as ints.
Json.NET also has a notion of global settings too that could set the StringEnumConverter maybe one of your other dependencies does this?

I never explicitly set one, but I'm also using Avro on the same set of
records.  I wonder if there is some interplay between the two libraries.
On Fri, Mar 27, 2015 at 5:38 AM, Martijn Laarman<EMAIL_ADDRESS>wrote:

Do you have a StringEnumConverter set on your enum ?
The way to have enums always serialized as strings would be by registering
it as a contract converter for enums:
https://github.com/elastic/elasticsearch-net/blob/develop/src/Tests/Nest.Tests.Unit/Internals/Serialize/CustomConvertersTests.cs#L40
Otherwise they will get indexed as ints.
Json.NET also has a notion of global settings too that could set the
StringEnumConverter maybe one of your other dependencies does this?
—
Reply to this email directly or view it on GitHub
https://github.com/elastic/elasticsearch-net/issues/901#issuecomment-86881325
.


Enable memory and disk space monitoring

Load the script embedded into the AMI.

PR : https://github.com/AirVantage/av-sched/pull/20

Validated on QA


allow custom message for android on show

One more thing :+1:

Ideally we'd need a cross-platform way to do this.

The api's are different in this regard. Since with iOS, you configure the instance after creation. With Android it can be passed in. To make the JS api the same we could configure the instance inside js like this:
indicator.show = function(msg) {
  var hud = MBProgressHUD.showHUDAddedToAnimated(this._getRootWindow(), true);
  hud.labelText = msg || 'Loading';
  return hud;
};

Thoughts?

:+1: from me.
I think we should have the argument to the show as an object so we can extend it with further options in the future without breaking the API.
var loader = loader.show({
  message: "Hello"
});

I have something really nice in the works here and will submit a different PR for it when it's ready this week :+1:


Write a R program to find smallest element in a matrix

Description
Write a R program to find smallest element in a matrix
Input :
4 7 3
9 4 2
0 6 4

Output : 0

Contributed by - @ShubhraSuman

How to contribute

Save the solution in find_smallest_element_in_a_matrix.r file
Add find_smallest_element_in_a_matrix.r file in program/find-smallest-element-in-a-matrix folder

!assign


Deployment via Railway

We want to deploy the backend and frontend via Railway. Note: for OAuth via GitHub, you have to change the params

Done!!!!


Use core instead of std for net types.

Helpful for using .into() with these types on no_std platforms.

To fix MSRV should this be gated behind an ip_in_core feature?

IMO we can just increase the MSRV, unless someone has some objection.

Have brought the msrv up. If it's okay, I will fix the clippy lints in another PR to not pollute this one with unrelated changes.

Bumping the MSRV brings with it the changes to Clippy's lints. In my opinion, it's fine to have the linting changes in this PR. If they're done in a separate commit, it's still easy to follow.

Ready for review.

there's more places where MSRV should be updated, like the readme. please grep for "1.65" and update them all.


releases.json Breaking Change

releases.json Breaking Change
releases.json is a family of files (and schema) that the .NET Team publishes to enable programatic access to the catalog of .NET releases.
We are making some targeted changes to these files, primarily to reduce user confusion, in response to consistent and broad feedback.
We are making three primary changes, ordered from most to least breaking:

Replace the lts and current value with active, in the support-phase property.
Replace the rc value with go-live in the in the support-phase property.
Document the allowable values for support-phase.
Add a new release-type property, with immutable values of lts or sts, as appropriate. standard stands for Standard Support and replaces the Current terminology to describe a release.

Timing
The change will be made in two phases:

The release-type property will be added via https://github.com/dotnet/core/pull/7499.
The support-phase property for .NET 7 will transition from rc to active on November 8th. Previously, this would have been current.

Context
"Current" releases will now be referred to as "Standard Term Support" releases, in obvious contrast to "Long Term Support (LTS)" releases. The duration of the support periods and related policies are not changing.
The "Current" term will no longer be used.
See .NET Release Labels for more information.

I like it. I also really like the new release label naming.
Quick question regarding process and nomenclature: The second and third bullet aren't really breaking changes in the schema though, right? For APIs we generally don't consider adding new members a breaking change; returning different values for existing data would be, which is what I think the first bullet means.

The support-phases property for .NET 7 will transition from go-live to sts on November 8th.

This should be support-phase, singular, right?

support-phases

Oops. Fixed. Good catch!

breaking change

100% agree. The first two changes are breaking (to different degrees). The last one is benign.

go-live means rc? If yes, why don't we just use rc instead

Should we pin this issue? Thanks for providing the timing of the changes. It helps us to plan the website changes.

Updated issue with the latest guidance

Hey folks,
These changes have been published. You should see them live on dot.net and public endpoints for releases.json files shortly.

Should we close this one @richlander?

This happened a long time ago.


Installation instructions using Vundle

😄

@arelangi Nice, would you mind squashing the commits and then I'll get it merged?

@jbranchaud Done

Thanks :heart:


Listening to events with the normal way.

Make listening to events working with the normal behavior instead of passing events as props.
I wanted to do the following to prevent passing invalid methods (events) to the Cropper but I think it doesn't worth and it is not a good idea, so I'll leave it for you to decide.
for (var event in this.$listeners) {
  // if (['ready', 'cropstart', 'cropmove', 'cropend', 'crop', 'zoom'].indexOf(event) > -1) {
    props[event] = this.$listeners[event]
  // } else {
    // console.warn(`The ${event} is not an event`)
  // }
}

Of course, we can now do so.
<vue-cropper
  @ready="cropperReay"
  @zoom="zooming"
  @crop="cropping">
</vue-cropper>

@Agontuk  Please merge!!!

Sorry for the late reply, this change will break the current behaviour. I'd like to keep the existing behaviour (as function props) and add event support if possible.

Event support added in v4.1.0 with existing functionality intact.


Git Sync fails with error: unable to write file .git/objects/c5/eb4274e2f1f9d7f567c0d01412d3905a498604: Invalid argument

Type: Bug
When trying to commit changes to repo using VS Code Source Control Git, it gives error: unable to write file .git/objects/c5/eb4274e2f1f9d7f567c0d01412d3905a498604: Invalid argument. I have had no isssues pushing before only in the last few days have I experienced this issue. Currently on newest updated version of VS Code. I have staged a commit and whenever I try to sync/pull/fetch it fails with this error.
VS Code version: Code 1.79.1 (4cb974a7aed77a74c7813bdccd99ee0d04901215, 2023-06-12T16:14:05.102Z)
OS version: Windows_NT x64 10.0.19044
Modes:

System Info

Item
Value

CPUs
12th Gen Intel(R) Core(TM) i5-1250P (16 x 2112)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: enabled

Load (avg)
undefined

Memory (System)
15.24GB (2.54GB free)

Process Argv
--crash-reporter-id 3456828a-f755-4e94-a467-40ad368f5750

Screen Reader
no

VM
0%

Extensions (5)

Extension
Author (truncated)
Version

aadb2c
Azu
1.3.5

xml
Dot
2.5.1

vscode-azureresourcegroups
ms-
0.7.5

vscode-azurestorage
ms-
0.15.3

azure-account
ms-
0.11.5

A/B Experiments
vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30410667
vsclangdf:30486550
c4g48928:30535728
dsvsc012:30540252
pynewext54:30695312
azure-dev_surveyone:30548225
vscccc:30610679
2e4cg342:30602488
pyind779:30671433
89544117:30613380
vscrp:30673768
pythonsymbol12:30671437
showlangstatbar:30737416
vsctsb:30748421
pythonms35:30701012
a2ce3375:30757347
pythonfmttext:30731395
pythoncmv:30756943
fixshowwlkth:30730052
pythongtdpath:30739704
dh2dc718:30763024
pythonnosmt12:30765603

@jmalloci, does the same error occurs if you try to run these operation from the terminal?

@jmalloci, that seems to indicate that this is probably an issue with the git repository store and not VS Code. I would like to suggest that you post your question on stackoverflow as that would be a more appropriate forum. Thank you!


drivers: eth: gmac: Remove extra variable

The msg_start variable was declared twice.
Signed-off-by: Jukka Rissanen<EMAIL_ADDRESS>

Codecov Report

Merging #12103 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #12103   +/-   ##
=======================================
  Coverage   48.07%   48.07%
=======================================
  Files         284      284
  Lines       43328    43328
  Branches    10389    10389
=======================================
  Hits        20828    20828
  Misses      18363    18363
  Partials     4137     4137

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 32ce219...2bfbe6f. Read the comment docs.


property $parsedViews is not used, so del it

Related: https://github.com/scil/framework/commit/c9edabc05ac61c3a1dfe7fe4f564f5c8592049e8.


Enforce lower-level declaration for overrides

You shouldn't be able to assign arbitrary values to a class that isn't "self". That is, a.b.c = 1 should fail if c isn't already declared for a.b.

After upgrading from v0.2.57 to v0.2.58 the following ato code:
from "ams1117-33/elec/src/ams1117-33.ato" import AMS111733  # ato install ams1117-33

module MyModule:
    ⋮
    # 3V3 regulator
    ldo_3v3 = new AMS111733
    ldo_3v3.power_in ~ power_connector.power
    ldo_3v3.power_out ~ power_3v3

Generates the following warnings:
[14:33:52] WARNING  Implicit Declaration Future Deprecation Warning                     errors.py:80
    .../.ato/modules/ams1117-33/elec/src/ams1117-33.ato:32:4
    Field 'value' not declared for .../elec/src/entry.ato:MyModule::ldo_3v3.input_cap.
    Declaring implicitly for now, but in the future this may become an error...
WARNING  Implicit Declaration Future Deprecation Warning                                errors.py:80
    .../.ato/modules/ams1117-33/elec/src/ams1117-33.ato:34:4
    Field 'package' not declared for .../elec/src/entry.ato:MyModule::ldo_3v3.input_cap.
    Declaring implicitly for now, but in the future this may become an error...
WARNING  Implicit Declaration Future Deprecation Warning                                errors.py:80
    .../.ato/modules/ams1117-33/elec/src/ams1117-33.ato:37:4
    Field 'value' not declared for .../elec/src/entry.ato:MyModule::ldo_3v3.output_cap.
    Declaring implicitly for now, but in the future this may become an error...
WARNING  ... forgoing more "Implicit Declaration Future Deprecation Warning" errors of  errors.py:193

$ ato --version
ato, version 0.2.58

The following specifies the main power input voltage and also the allowed input voltage of a sub module. This ato code did not generate any warnings or errors in version 0.2.57.
from "generics/interfaces.ato" import Power

module SubModule:
    power = new Power
    # specify allowable voltage for this sub module
    power.voltage = 3.1V to 3.3V

module Main:
    power = new Power
    # specify main power input voltage
    power.voltage = 3.1V to 3.3V

    sub_module = new SubModule
    sub_module.power ~ power

Version 0.2.58 generates the following warnings and error:
$ ato build
[15:35:40] INFO     Using project .../atopile-433/test              common.py:71
    INFO     Building default                                       build.py:38
    WARNING  Implicit Declaration Future Deprecation Warning        errors.py:80
    .../atopile-433/test/elec/src/test.ato:13:4
    Field 'voltage' not declared for .../atopile-433/test/elec/src/test.ato:Main::power.
    Declaring implicitly for now, but in the future this may become an error...
    WARNING  Implicit Declaration Future Deprecation Warning        errors.py:80
    .../atopile-433/test/elec/src/test.ato:7:4
    Field 'voltage' not declared for .../atopile-433/test/elec/src/test.ato:Main::sub_module.power.
    Declaring implicitly for now, but in the future this may become an  error...
    ERROR Error                                                     errors.py:80
    .../atopile-433/test/elec/src/test.ato:16:4
    The source and target separately defined values for the attribute "voltage"

Setting do_not_populate on a component in a third-party package generates a deprecation warning
It may be necessary to set the do_not_populate (or exclude_from_bom) on a component provided by a third-party package. These two attributes were introduced in #419.
Version 0.2.58 indicates that this feature will be deprecated. It is likely not desired nor reasonable to expect third-party packages to explicitly define do_not_populate and exclude_from_bom.
Environment
$ ato --version
ato, version 0.2.58

How to reproduce
from "xt-connectors/xt-connectors.ato" import XT30_Male_Right_Angle

module Main:
    connector = new XT30_Male_Right_Angle
    connector.do_not_populate = True

Run ato build
When the project is built the following warning is generated:
$ ato build
[11:02:03] INFO     Using project .../atopile-433/test                                                                   common.py:71
           INFO     Building default                                                                                build.py:38
           WARNING  Implicit Declaration Future Deprecation Warning                                                      errors.py:80
                    .../atopile-433/test/elec/src/test.ato:5:4
                        Field 'do_not_populate' not declared for .../atopile-433/test/elec/src/test.ato:Main::connector.
                        Declaring implicitly for now, but in the future this may become an error...

Thanks for the reports @sethfischer . Don't stress too much. There's a bunch of places we should upgrade to avoid this warning, and it'll gradually disappear. Nothing to be concerned of for the minute


feature issue: saving Find/Replace options

VSCode should save the Find / Replace options like: Match case and Whole word.
This would allow to using them after restart.
It could be an user option below search settings.

also does the multiple selections need to use the find settings or can it have it's own, set in settings.json or somewhere?

We'll start with the persistence with three search options  Match Case, Match Word and Regex first. The goal is aligning with current Search Viewlet.
As we discussed in https://github.com/Microsoft/vscode/issues/11574#issuecomment-245623366 , we are going to persist user preference in following way

anytime a find widget toggle is changed, I will persist it (last one wins)
when the workbench starts up, I will read the persisted values and start with those by default.

In addition to that, the setting persistence is workspace-based. Workspaces won't share the data. Let's start with it first and see if it's necessary to add a new setting which defines the default toggle setting for Find Widget.

Hi @rebornix. I was wondering which version of VSCode persists it?
Any way to use the Nightly version or I need to wait for 1.12.1 be updated?
Thanks

Is it intended that vscode only remembers the search settings for directory where these have been set?
If I start vscode e.g. with: code example-dir
and I check case sensitive search I will remember these settings, after I closed and reopened the editor.
But if start vscode in an other directory the settings are gone. Is this how it should work?
As far as I know there is no global setting to define the default one or am I missing something?


Compile error in VS2015 (msvc 19.0.3) due to issues with noexcept() operator in function overloading

From https://lists.boost.org/Archives/boost/2019/12/247907.php

This might actually be a bug in msvc 14.0!

Then I guess VS 2015 is not supported.  STLInterfaces works "with any conforming C++14 compiler."


fix: modify peer dependencies

close  https://github.com/vercel/webpack-asset-relocator-loader/issues/188

This does not solve the problem


unrecognized option `--install-extension'

I cannot install any extensions via the command line. I constantly get:
$ code --install-extension ms-vscode.cpptools
open: unrecognized option `--install-extension'
Usage: open [-e] [-t] [-f] [-W] [-R] [-n] [-g] [-h] [-b <bundle identifier>] [-a <application>] [filenames] [--args arguments]
Help: Open opens files from a shell.
      By default, opens each file using the default application for that file.
      If the file is in the form of a URL, the file will be opened as a URL.
Options:
      -a                Opens with the specified application.
      -b                Opens with the specified application bundle identifier.
      -e                Opens with TextEdit.
      -t                Opens with default text editor.
      -f                Reads input from standard input and opens with TextEdit.
      -F  --fresh       Launches the app fresh, that is, without restoring windows. Saved persistent state is lost, excluding Untitled documents.
      -R, --reveal      Selects in the Finder instead of opening.
      -W, --wait-apps   Blocks until the used applications are closed (even if they were already running).
          --args        All remaining arguments are passed in argv to the application's main() function instead of opened.
      -n, --new         Open a new instance of the application even if one is already running.
      -j, --hide        Launches the app hidden.
      -g, --background  Does not bring the application to the foreground.
      -h, --header      Searches header file locations for headers matching the given filenames, and opens them.

Attempting to install an extension keeps failing for me even via the marketplace within the app. Every install attempt just stalls.

VSCode Version: 1.21.1
OS Version: OSX 10.10.3

Steps to Reproduce:

Open command line
Run command code --install-extension <extension-name>

@JayyWalker did you follow the setup step to install Code on the command line, pls see https://code.visualstudio.com/docs/setup/mac#_command-line

@egamma Yes, I have just did it again just to make sure but the issue still persists.

@JayyWalker If you run which code, bash will tell you code is an alias, which is wrong.
Open your ~/.bashrc file, remove that alias code found in there.


fix: Correct state machine definition for non-variable values

Description

Correct state machine definition for non-variable values

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

:tada: This PR is included in version 1.1.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


OpenCE 1.11 P9 and P10

[x] Builder PR for UBI9 support for cpu images.
[x] GCC 12 changes for all feedstocks for P10
[x] Apache Beam Build Failure PPC CPU (Not checked on cuda)
[x] scikit-learn failure at test stage on PPC P9 and P10
[x] Onnx runtime failure on P9
[x] TF Failure on P10
[x] TF-io-gcs failure on both P9 and P10
[x] jaxlib failure on both
[ ] TF Build failure on PPC cuda - https://github.com/tensorflow/tensorflow/issues/63055

Failure seen for Pytorch using gcc-toolset-12, which was solved after setting '-Werror' flag as 'nonnull' (previously fixed in open-ce-1.9.4 release):
    inlined from ‘_OI std::copy(_II, _II, _OI) [with _II = __gnu_cxx::__normal_iterator<const long unsigned int*, vector<long unsigned int> >; _OI = __gnu_cxx::__normal_iterator<long unsigned int*, vector<long unsigned int> >]’ at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/stl_algobase.h:620:7,
    inlined from ‘std::vector<_Tp, _Alloc>& std::vector<_Tp, _Alloc>::operator=(const std::vector<_Tp, _Alloc>&) [with _Tp = long unsigned int; _Alloc = std::allocator<long unsigned int>]’ at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/vector.tcc:244:21:
/opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/stl_algobase.h:431:30: error: argument 1 null where non-null expected [-Werror=nonnull]
  431 |             __builtin_memmove(__result, __first, sizeof(_Tp) * _Num);
      |             ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Fixed: https://github.com/open-ce/pytorch-feedstock/pull/126

Failure of TF-IO on PPC
/home/builder/.cache/bazel/_bazel_builder/fe550e2cf85dc9ea502a597dab3d7f2d/external/python/BUILD.bazel:10:6: no such package '@python_ppc64le-unknown-linux-gnu//': The repository '@python_ppc64le-unknown-linux-gnu' could not be resolved: Repository '@python_ppc64le-unknown-linux-gnu' is not defined and referenced by '@python//:python_headers'
ERROR: Analysis of target '//tensorflow_io/core/grpc:endpoint_py_pb2_grpc' failed; build aborted:

Fixed :
https://github.com/open-ce/tensorflow-io-gcs-filesystem-feedstock/pull/28


adding support for intra-group dependency validation

Issue #, if available:
#178
Description of changes:
Adding validation for intra-group dependencies, exiting the deploy if found
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

amen


test: add tests for ruby

This test makes sure that ruby importer works!

Closed due to https://github.com/nexB/vulnerablecode/pull/185


ZincProblemPosition: fix infinite loops

JavaOptionConverter.convertInt overflowed the stack, probably because there is no Optional#isEmpty

Should fix https://github.com/lihaoyi/mill/issues/814

Thanks! Does this mean it's now available as a development version?

Thanks! Does this mean it's now available as a development version?

Yes. You may wait until the Travis CI job finished successfully (sometime it doesn't) and give Maven Central some minutes to sync though.


Loading examples tests data from cache before from the web

SUMMARY
There is no need to fetch the tests data examples from the web over and over, for each test execution.
Furthermore if there is an issue with accessing the web, the test can not be executed.
More cleaver to load the data from the cache
After first downloading, the data is saved to regular files.
On second data loading, it will be loaded from the files.
TEST PLAN
There is no need, this is helps the tests, the only thing is to run tests twice and notice if there are problems

please delete it, I'll open a new one


Feat/native bigint

Creating a separate PR from https://github.com/blockstack/stacks.js/pull/1029 because this one introduces breaking changes, and adds a bunch more unit test update clutter.
Int and UInt Clarity values now store a native bigint, and the cvToValue(...) function returns a bigint rather than number or string. The cvToJSON(...) function preserves valid JSON output, and returns integers as string quoted. These are both breaking changes.
Generally, most of the function input types were made permissive so that numeric arguments like fee, nonce, amount, etc accept the type type IntegerType = number | string | bigint | Uint8Array | BN. This is not a breaking change and should make these functions a bit friendlier/easy to use. E.g. constructing a BN.js instance is no longer required.
Generally, most of the function output types that were previously a BN.js instance are now a bigint instance. This one is a breaking change.

@kyranjamie could use a review and approval when you get a chance


add option to pin "epoch curves" in confmat cells in the detail view

we should add the option to pin certain confmat cells to the detail view --> either as a list of plots or (what probably would be better) simple adding them to the detail view plot --> you can add and remove the curves by clicking on the corresponding confmat cells.
clicking on the aggregated cells automatically selects all the cells in the row/column. By doing this we can quickly let the user build "contrasts" for comparing classes
(if the have this contrast feature it would be even better to have the option to create multiple contrasts with separate curves)

@thinkh
didn't we say we would add this for the paper


[BUG]oss gg test

Issue Summary
test
Steps to Reproduce

(for example) Went to ...
Clicked on...
...

Expected behavior
tset
Other information
No response
Screenshots
No response
Desktop (please complete the following information)
- OS:tes
- Node:
- npm:

/assign

/assign

/assign

/assign


fix: Encoding of List offsets was incorrect when slice offsets begin with zero

When encoding offsets the code had an optimization to reuse the offsets if the first offset was zero assuming the slice already pointed to first element. But the offset can also be zero if all previous lists were empty. When this occured it mold make all lists in the slice as empty, even if they shouldn't be.
Which issue does this PR close?
Closes #6803 .
Rationale for this change
Fixing a bug that can lead to loss of information when encoding record batches.
What changes are included in this PR?

Change the encoding and a test to demonstrate the fix.

Are there any user-facing changes?
No, though some clients will experience smaller encoded messages since only the relevant slice will be sent.

CI seems to be legitimately failing

CI seems to be legitimately failing

missed that.  fixed

Looks good to me, thank you


opts is not defined, addon broken with select2 3.5.3

oddly the new point release seem to have broken my code that looked like this, reverting back to 3.5.2 worked
{{select-2
        placeholder="Select a user"
        value=chosenUser
        typeaheadSearchingText="Searching users"
        typeaheadNoMatchesText="No users found for '%@'"
        typeaheadErrorText="Loading failed: %@"
        optionLabelPath="userName"
        query="queryUsers"
      }}

https://github.com/select2/select2/releases/tag/3.5.3

This is a known bug in 3.5.3, a new 3.5.4 release is being cooked up to fix the issue.
https://github.com/select2/select2/issues/3694


Add implementations for Rand trait

The rand crate provides Rand implementations for several of its generators, which serve as a convenient way of obtaining a randomly-seeded instance of a generator (see e.g. here).  For example, this function is able to seed any such Rng randomly from a seed obtained from OsRng:
fn from_osrng<T: Rng + Rand>() -> io::Result<T> {
    let mut osrng = try!(OsRng::new());
    Ok(osrng.gen())
}

This pull request adds such impls to this library's generators.

I tried to add a couple of unit test cases to this and ran into the problem that the test suite requires non-stable features.  I'm a relative newcomer to Rust, so I don't know how to get around this just yet, but I did test the changes by trying them from another program I've been writing.

Thanks! I'm fine merging this without a test, but would you mind doing two things first? I've been intending to relicense under dual MIT/Apache-2.0, so I did it today. Please rebase on 2fe2808 or later and include in the commit message language like the following:
Unless explicitly stated otherwise, I license this and future contributions under the dual MIT/Apache-2.0 license, allowing licensees to chose either at their option.

I have rebased the pull request and reworded the commit message as requested.


Changing the default escape character to " to be compatible with RFC 4180

RFC 4180 says:

If double-quotes are used to enclose fields, then a double-quote
appearing inside a field must be escaped by preceding it with
another double quote.  For example:
"aaa","b""bb","ccc"

So, I think default escape parameter should be \".

@sakama can I ask you to review this change?

@frsyuki

So, I think default escape parameter should be ".

I think so too.
I used \\ to be the same as the default option of CSV parser , but almost application or CSV writer use \" as default escape char.

:+1:
I merged.


Update ThingsMacHelper to 3.9

After making all changes to the cask:

[x] brew cask audit --download {{cask_file}} is error-free.
[x] brew cask style --fix {{cask_file}} reports no offenses.
[x] The commit message includes the cask’s name and version.

The app name is different in finder, normally we would do something like this but I'm not sure if it will break anything in this case.
https://github.com/caskroom/homebrew-cask/blob/7bed7fb0041eec4988680ca9188155c51496c7e0/Casks/teamviewer-quickjoin.rb#L10-L11

Thanks for the PR!

@commitay No problem 👍 It's my first time updating a cask, so any tips like that are really useful! Thank you 😄


An environmental issue

I followed the Git workflow to set up the entire environment and executed this command:
python eureka.py env=shadow_hand sample=16 iteration=2 model=gpt-3.5-turbo  (In order to increase the success rate, a sample size of 16 was set.)

When I check each failed log, they all show the same error. Have you encountered this issue before?
Traceback (most recent call last):
File "/homeb/yulong/Eureka/Eureka-main/eureka/../isaacgymenvs/isaacgymenvs/train.py", line 214, in
launch_rlg_hydra()
File "/home/yulong/miniconda3/envs/eureka_v2/lib/python3.8/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/home/yulong/miniconda3/envs/eureka_v2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/home/yulong/miniconda3/envs/eureka_v2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/home/yulong/miniconda3/envs/eureka_v2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/home/yulong/miniconda3/envs/eureka_v2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
File "/home/yulong/miniconda3/envs/eureka_v2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 458, in
lambda: hydra.run(
File "/home/yulong/miniconda3/envs/eureka_v2/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/home/yulong/miniconda3/envs/eureka_v2/lib/python3.8/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/home/yulong/miniconda3/envs/eureka_v2/lib/python3.8/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "/homeb/yulong/Eureka/Eureka-main/eureka/../isaacgymenvs/isaacgymenvs/train.py", line 203, in launch_rlg_hydra
statistics = runner.run({
File "/homeb/yulong/Eureka/Eureka-main/rl_games/rl_games/torch_runner.py", line 124, in run
self.run_train(args)
File "/homeb/yulong/Eureka/Eureka-main/rl_games/rl_games/torch_runner.py", line 101, in run_train
self.agent.train()
File "/homeb/yulong/Eureka/Eureka-main/rl_games/rl_games/common/a2c_common.py", line 1251, in train
step_time, play_time, update_time, sum_time, a_losses, c_losses, b_losses, entropies, kls, last_lr, lr_mul = self.train_epoch()
File "/homeb/yulong/Eureka/Eureka-main/rl_games/rl_games/common/a2c_common.py", line 1115, in train_epoch
batch_dict = self.play_steps()
File "/homeb/yulong/Eureka/Eureka-main/rl_games/rl_games/common/a2c_common.py", line 686, in play_steps
self.obs, rewards, self.dones, infos = self.env_step(res_dict['actions'])
File "/homeb/yulong/Eureka/Eureka-main/rl_games/rl_games/common/a2c_common.py", line 504, in env_step
obs, rewards, dones, infos = self.vec_env.step(actions)
File "/homeb/yulong/Eureka/Eureka-main/isaacgymenvs/isaacgymenvs/utils/rlgames_utils.py", line 256, in step
return  self.env.step(actions)
File "/homeb/yulong/Eureka/Eureka-main/isaacgymenvs/isaacgymenvs/tasks/base/vec_task.py", line 355, in step
self.post_physics_step()
File "/homeb/yulong/Eureka/Eureka-main/isaacgymenvs/isaacgymenvs/tasks/shadow_handgpt.py", line 653, in post_physics_step
File "/homeb/yulong/Eureka/Eureka-main/isaacgymenvs/isaacgymenvs/tasks/shadow_handgpt.py", line 370, in compute_reward
AttributeError: 'ShadowHandGPT' object has no attribute 'target_orientation'

change to gpt4.0,Can greatly improve the success rate，

thank you for you reply! But I don't think it has anything to do with whether or not you use gpt3 or gpt4. Please check the output in file eg.eureka/outputs/eureka/2023-11-14_14-41-48/env_iter4_response4.txt to see if any other problems occured

Yes, I still had a GPT3 success rate of 0 after excluding log errors, and then successfully adopted 4.0
------------------ 原始邮件 ------------------
发件人:                                                                                "eureka-research/Eureka"                                                                                @.>;
发送时间: 2023年11月24日(星期五) 下午5:13
@.>;
<EMAIL_ADDRESS>主题: Re: [eureka-research/Eureka] An environmental issue (Issue #17)
thank you for you reply! But I don't think it has anything to do with whether or not you use gpt3 or gpt4. Please check the output in file eg.eureka/outputs/eureka/2023-11-14_14-41-48/env_iter4_response4.txt to see if any other problems occured
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.***>

there are so many error similar to "AttributeError: 'ShadowHandGPT' object has no attribute 'target_orientation'", any solution to solve this problem? I don't know how to solve it ? every error is the "... has no attribute ..."


Setup Alarmo

Requirements

[ ] Install Alarmo integration.
[ ] Configurearmed_vacation.
[ ] Add condition to air purifier automation to not run if sensor.armed_vacation = true.
[ ] Add condition to vacuum schedule automation to not run if sensor.armed_vacation = true.
[ ] Add condition to climate automation to not run if sensor.armed_vacation = true.

Resources
Alarmo integration

The goal of this ticket was to create a vacation mode for Home Assistant in order to turn off some automations when away from home for extended periods of time.  Unfortunately, I don't think this is the best way to achieve this goal.


too few features during training

Hi,
Thank you for open-sourcing your training code.
I want to change the backbone and train a model using your training scheme. I use the default configuration. However, the model cannot extract any features during training (~300 steps) and it causes errors when calculating the "distance" and "affinity" matrix. Could you give me some advice?
Looking forward to your reply :)

Thanks for your kindly help and useful suggestions. I have increased the steps of warm up and it works.


Error: ENOTDIR

This might have been better as a Stackoverflow, but I think I would get a faster answer here.
This will not work
{from: '^/rest/users', to: '/json/users/user-list.json'},
{from: '^/rest/users/banned/123', to: '/json/users/banned/123.json'}
Because /rest/users/banned/123 will map to '/json/users/user-list.json/banned/123'
With or without $ it will produce the same outcome.

Not sure I understand the issue/question, but can you try this?
{from: '^/rest/users/banned/123$', to: '/json/users/banned/123.json'},
{from: '^/rest/users$', to: '/json/users/user-list.json'}

@eduardoinnorway is your issue solved?


Confusing behaviour for the navigation bottom bar.

In fact if we follow the same sequence of actions with any activity (offlineEdit ,BarcodeSearch...) navigable from the mainActivity the bottomNavigation bar clings on to the lists icon (in this case).

Something to do with the onResume() in MainActivity I guess !!

We should only switch fragments, not activities

But some of the buttons in bottom nav lead to different activities not fragments ,we'll have to change that !!

Or we could uncheck every option whenever the MainActivity resumes .

But some of the buttons in bottom nav lead to different activities not fragments ,we'll have to change that !!

That's what i want to change. Is there any reason for not using fragments in all the 5 activities covered by the bottom navigation.

Ok, wrong idea. I looked at the code and we use Intents everywhere so this is not a good solution.

or Instead of doing this tedious refactoring we could make an invisible item in the bottom nav and select it whenever we are not selecting any of the 5 bottom nav fragments.
temp fix but will make things consistent !!

fixed


Is it possible to implement all traits on SubscriberClient?

If RESP3 is used (see HELLO), a client can issue any commands while in the subscribed state (from https://redis.io/docs/manual/pubsub/).
So maybe SubscriberClient could implement the same traits as RedisClient and we wouldn't need 2 clients when working with SubscriberClient. What do you think?

Thats a good idea. I'll put this in 6.1.0 (via https://github.com/aembke/fred.rs/pull/122)

Added in https://github.com/aembke/fred.rs/pull/117


[nrf fromlist] drivers: serial: nrfx: disable before setting pin config

Per the nrf9160 datasheet, UARTE must be disabled prior to configuring
PSEL.TXD, PSEL.RXD, PSEL.CTS, and/or PSEL.RTS.  This also ensures that
any prior running code, such as a bootloader, which leaves the UART
enabled will not prevent the Zephyr application from changing the
configuration.
Fixes 27080
Signed-off-by: Pete Skeggs<EMAIL_ADDRESS>

This looks stale; I think I just got a ping about it because master rebased.
@plskeggs I'm going to close for now -- could you please rebase and reopen if still relevant?


Release a new version with support for labels

TL;DR
Support for labels was added in #250, but the latest release is v6.2.0 from Dec 7 2021. Please release a new version with the latest changes included. Thanks!
Terraform Resources
No response
Detailed design
No response
Additional information
No response

@willroberts We will need to add an upgrade guide. Tracking in https://github.com/terraform-google-modules/terraform-google-lb-http/issues/254 but happy to review a PR if you have bandwidth

I see that v6.3.0 has been released. Thanks!


Latest Images Not Present on DockerHub

The latest helm chart referencing image: lightbend/kafka-lag-exporter:0.5.6-SNAPSHOT, doesn't seem to be able to pull the image.
I checked on DockerHub, and only images upto tag 0.5.5 seem to be available: https://hub.docker.com/r/lightbend/kafka-lag-exporter/tags
Is the CI/CD to push docker images working as expected @seglo ?

@abhishekjiitr Are you referring to the unpackaged helm chart in master?  This is expected as part of the release.  If you want to deploy the 0.5.5 chart from unpackaged source, you can browse to the v0.5.5 tag https://github.com/lightbend/kafka-lag-exporter/tree/v0.5.5 .
Usually, if you're deploying the chart this way you would override the location of the docker image to a local repo, or your own docker registry.

Got it @seglo , thanks!


2.2.0rc1 state of click isues (tested on Windows 11)

In Kivy==2.2.0rc1, a bug has appeared related to detecting the state of click due to the new version of SDL in the following scenario: a user presses a widget triggering on_touch_move => the user moves the mouse outside of the application and the application loses focus (via alt+tab, or a drag n drop of a file from the kivy application in Windows, or a program) , then user up left mouse button. After this, when hovering over the kivy application, it constantly uses on_touch_move as though the mouse button is still being pressed. This continues until the application regains focus. Also, when using os.environ["SDL_MOUSE_FOCUS_CLICKTHROUGH"] = '1' to allow clicks on the application without focus, in this scenario the first click of the left mouse button does not register, but the right button does.

Hi @Buratini !
Seems I can't reproduce the issue on macOS (my everyday platform), can you please post a reproducible example that can be run on Windows? (from someone else, or hopefully on the weekend)

Video:
https://youtu.be/VtfpPLuee2Y
Code:
from kivy.app import App
from kivy.uix.widget import Widget
class TouchWidget(Widget):
def on_touch_down(self, touch):
    if self.collide_point(*touch.pos):
        print("Touch down")
        return True
    return super(TouchWidget, self).on_touch_down(touch)

def on_touch_move(self, touch):
    if self.collide_point(*touch.pos):
        print("Touch move")
        return True
    return super(TouchWidget, self).on_touch_move(touch)

def on_touch_up(self, touch):
    if self.collide_point(*touch.pos):
        print("Touch up")
        return True
    return super(TouchWidget, self).on_touch_up(touch)

class TouchApp(App):
def build(self):
return TouchWidget()
if name == 'main':
TouchApp().run()

Well, I can confirm is an SDL2-related issue, and we're likely be able to fix it (if feasible, still have to dig into SDL2 codebase) in future, by releasing a new version of kivy-deps.sdl2, without the need of a new Kivy version.
It's not meant to be used in production, but, as a workaround:

With Python 3.7, 3.8, 3.9 or 3.10 install kivy 2.2.0rc1 (the old kivy-deps.sdl2==0.4.5 is unsupported on Python 3.11)
In the same Python environment, install kivy-deps.sdl2==0.4.5
The issue disappears.


Broken demo data

pbix file and excel file are broke
"To follow along, open the Customer Feedback PBIX file in Power BI Desktop. You also can download the Customer Feedback Excel file for Power BI service or Power BI Desktop"

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 4200dd45-15ab-ee4a-9818-10cc81475f64
Version Independent ID: 0b96ac28-e2a2-41c9-eff0-efd864fc0529
Content: Key influencers visualizations tutorial - Power BI
Content Source: powerbi-docs/visuals/power-bi-visualization-influencers.md
Service: powerbi
Sub-service: powerbi-service
GitHub Login: @mihart
Microsoft Alias: mihart

Thanks for your comment, @[github person who posted]. I've assigned this issue to the doc author for review.

@chizhova - the files are there, click Download on the GitHub page. @mihart - can you add a quick explanation to the doc?


HK50

symbol="HK50",
exchange="GLOBALPRIME",
screener="cfd"

Hi @rararrr,
Currently, all indices are not available via the scanner API. HK50 is a part of indices, not CFD as it is not available in https://scanner.tradingview.com/cfd/scan. In order for a symbol to be added to the database, it must have a 'Technicals' tab, see this example.
I'm going to close this issue as it is unfixable from TradingView's side. If you have any questions don't hesitate to reply. Have a nice day.


Restore python3-tk as an apt requirement

I needed to install this to get matplotlib working

Any idea why this changed from your last update? https://github.com/quantumlib/Cirq/issues/1728

I'm not sure. Perhaps uninstalling python3-tk didn't really remove it, or perhaps I had installed some other dependency that was filling in the blank. In the future we should only remove this after testing that it works on a fresh virtual machine.


Can I pass a parameter without creating new config.py?

For example, I have two types of gpu servers, one is 4gpus and another is 8. So I need to set two kinds of learning rate to fit one certain GPU, which doubles the number of config files.
Can I just pass the parameter using existing API? just like the way using --set in detectron?

This feature was considered before but not adopted. We will have new discussions about the config system.

the config system is good now,  you can create two config file

Already supported by setting --options


CLI suport for parser

Fixes #10

Thanks for this! I'll give it a try in the next day or so, but it looks great. Really appreciate it.

:+1:

@rmurphey Is this fine or do you want me to change anything?


Full yarn upgrade, including moving to new major versions

There's an issue with the esbuild major version upgrade that I'll need to fix later. This is no longer ready for merge!


Little misbeahvior in code example for SpinWait

Code example for SpinWait: https://learn.microsoft.com/en-us/dotnet/standard/threading/how-to-use-spinwait-to-implement-a-two-phase-wait-operation#example
In the method Latch.Wait(int timeout) Stopwatch uses to measure amount of time elapsed during the first phase of wait operation. Then in the second phase  timeout passed to this method decreases by this measured timelapse.
The problem: stopwatch is reinitialized every iteration of outer while cycle. Thus by the start of second phase we  didn't  actually measured elapsed time at all.

This issue was moved to dotnet/docs#36678


Lost param in first access

Every first time when access the 'home' and navigate to 'pagetwo' the param from URL is losted
Error in console: "Uncaught TypeError: Cannot set property 'textContent' of null" this happening because onHashChanged was not fired, but how fix this?
You can see this issue in http://phonon.quarkdev.com/App/public/phonon/examples/pizza-app/ too. First access, no have hash, so need back to home and navigate again to work.

Thank you for reporting this issue.
I just fixed it.
You can now download the latest release v1.1.0

lol thank you


Update defender-for-cloud-planning-and-operations-guide.md

@LiatCo23
Thank you for your contribution. Would you take a moment to sign the Contributor License Agreement (CLA)?
After the CLA is signed, someone can review your pull request.
Thanks!

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team


Acme Client fails to retry fetching the certificate when the CA returns processing.

See:
https://github.com/Ephenodrom/Dart-Acme-Client/issues/10
for details.

I have pushed a PR with a fix for this.

resolved via:
https://github.com/Ephenodrom/Dart-Acme-Client/issues/10


DROOLS-1073 Option to bypass start task operation in REST/JMS

this PR depends on https://github.com/droolsjbpm/jbpm/pull/416

@ibek could you please take a look at the failure? looks like a compilation error...

@mswiderski hurrayy, it passed - the error was strange - there was no Map on that line so I rerun the tests
I will create 6.4.x PRs, then