increase timeout for beforeEach to make node4 tests pass

Increasing timeout for beforeEach to make node4 tests pass.

@alexander-fenster Is this PR still active, or should it be closed?


Make Enhancements to the option to mark the tasks Started, stopped or Completed.

Good afternoon everyone. I get confused when I start a task on the maproulette site and I want to stop and start again later, because when I click on finished task and I click on several tasks I do not know which option to click finish after completing several tasks. the button below does not load for 100% is and a doubt!

Hi, you can save / copy the URL for the task and re-load it later. Unless someone else fixed the task in the mean time the screen should look the same.


Update picodfplayer.py

Added function to play numbered files in the MP3 Directory which is available on some versions of the DFPlayer modules

I'm not sure if this functionality is available on all modules because all my older ones have stopped working or are AWOL among all my junk but I needed it for a project and it worked as documented on the modules I have available at the moment (the ones with the blue playback LED).   YMMV.


Inconsistent treatment of self-referencing union-typed bound.

Example:
import "dart:async" show FutureOr;
class C<T extends FutureOr<T>> {}
class D<T extends T?> {}

These two classes both have self-referential type parameter bounds, both bounds are union types which are proper supertypes of the type variable itself.

The first class is accepted, the second causes an error in both analyzer and CFE.
Analyzer (in DartPad, master channel):

error
line 3 • 'T' can't be a supertype of its upper bound.[ (view docs)] (https://dart.dev/diagnostics/type_parameter_supertype_of_its_bound)
Try using a type that is the same as or a subclass of 'dynamic'.

CFE (dart2js in DartPad, master channel):
Error compiling to JavaScript:
lib/main.dart:3:9:
Error: Type 'T' can't use itself as a bound.
class D<T extends T?> {}
        ^

I believe that either both classes should be allowed, or both classes should be disallowed.
(And if I'm wrong, I'd like to question the missing symmetry of the rules in question. :wink:)
Most likely the former, since the reported error conditions aren't actually true.
The analyzer says that T is a supertype of its bound, and it isn't.
CFE says T is itself its bound, and it isn't.
(Probably the usual issue of forgetting to check the nullability stored on the type.)

Duplicate of https://github.com/dart-lang/language/issues/433?


Is TUF a library or a collection of tools or both?

To help with #129... I need to ask...
How do you envision TUF to appear on distributions?
Does TUF provide files that are supposed to be imported, i.e. a python library?
Would you prefer having it installed in /usr/bin/...?
Obviously /usr/bin/sig.py wouldn't be a good name. Something like /usr/bin/tuf-sig?
Or would it suffice to take the whole tuf folder (https://github.com/theupdateframework/tuf/tree/develop/tuf) and to install it's contents to /usr/lib/tuf? I.e. so you'd end up with a file /usr/lib/tuf/sig.py and so forth?
Applications based on TUF would then call external utilities such as /usr/lib/tuf/sig.py?

"How do you envision TUF to appear on distributions?  Does TUF provide files that are supposed to be imported, i.e. a python library?"
Developers of software update systems can download and install the tuf-0.9.9-py2.py3-none-any.whl or tuf-0.9.9.tar.gz Python distributions so that they can integrate the framework with their software update system.  TUF is a library that is imported on the client side of the software update system.  The Python distribution also includes tools to generate and sign the TUF metadata that is added to the update system's software repository, but the framework does not require code to run on the server side.
The current pip installation steps should help illustrate how the framework is packaged:
$ pip install tuf

This step above installs TUF's pure-python modules.  The software update system's client application may then import the library to perform software updates (import tuf.client.updater)
Importing the framework and requesting updates is outlined here
$ pip install tuf[tools]

This step fetches TUF's C dependencies (PyCrypto & PyNaCl) so that repository maintainers can generate & sign TUF metadata.  Example use of the repository tools:
from tuf.repository_tool import *
repository = load_repository("/path/to/repository")
generate_and_write_ed25519_keypair('/path/to/ed25519_key')
...
repository.write()

"Would you prefer having it installed in /usr/bin/...? "
If you would rather avoid pip, then putting all the TUF python modules in /usr/lib/tuf makes sense.  However, you would need the C dependencies for the repository tools to work.  Note: The tuf.repository_tool module uses many of the modules imported by tuf.client.updater, so we just package them together.  tuf-0.9.9-py2.py3-none-any.whl is 192 KB.

Good news:
PyCrypto is in Debian, apparently. (https://packages.debian.org/search?keywords=python-pycrypto)
Bad news:
PyNaCl is not, apparently? So someone would have to package it for Debian first.
Quite harder than I anticipated.

Developers of software update systems can download and install the tuf-0.9.9-py2.py3-none-any.whl or tuf-0.9.9.tar.gz Python distributions so that they can integrate the framework with their software update system.

That's something as far I understand, Debian doesn't like. They don't like some application based on TUF (such as in future hopefully: pip), to include third party code:
https://wiki.debian.org/UpstreamGuide#No_inclusion_of_third_party_code
(From perspective of pip: TUF is considered third party code.)
Debian wants to avoid code duplication:
https://wiki.debian.org/dedup.debian.net
They would require [that part of] TUF to be packages and installable comparable to python-yaml (https://packages.debian.org/de/jessie/python-yaml).

If you would rather avoid pip

Well, as per "you", it's not about me personally. Rather Debian policy (and most likely others as well) are very strict. Too bureaucratic and strict for my taste sometimes.
While python-pip is installable from Debian (https://packages.debian.org/search?keywords=python-pip) and while you as a user of Debian, developing our own stuff, are welcome to install stuff using pip, it's different when it comes to packaging. As far I know, in these cases you are forbidden to depend on python-pip to download stuff.
Ironically what could help getting TUF packaged for Debian, would be getting TUF being integrated into pip first. Because when pip cannot be used without anymore, the maintainers of python-pip in Debian would have to package PyNaCl and TUF for Debian. Or if they can't do that for whatever reasons (time, interest, difficulty), at some point python-pip would be removed from Debian. But the latter is rather unlikely, I hope, since pip is quite popular.
Sorry to say, but due to the unexpected difficulty, I might not be able to help as much with packaging as I wanted.

The repository tools do work without PyNaCl, you just lose the option of signing metadata with the Ed25519 signature scheme.  Generating RSA-PSS signatures with PyCrypto is still an option.
Unfortunately, the link you provided for PyCrypto is actually PyCryptopp (Python wrappers for the Crypto++ library).  What we currently support as an option for repository maintainers is PyCrypto), however, we can also add PyCryptopp to our "configurable crypto library" interface. (PyCryptopp appears to have bindings for Ed25519, AES, RSA signatures, etc., which will work nicely for us.)
For our repository tools it would be nice to eventually support the following libraries (we currently support the first three):

PyCA's Ed225519 (pure python, and only used to verify signatures on the client side): https://github.com/pyca/Ed25519
PyNaCl (libsodium backend to generate Ed25519 signatures): https://github.com/pyca/pynacl
PyCrypto (AES and RSA-PSS signatures): https://github.com/dlitz/pycrypto
PyCA's Cryptography (openssl backend)
PyCryptopp (Crypto++ backend)

Getting TUF packaged for Debian (today) shouldn't be a problem (the core client stuff is all pure-python and supports Ed25519), it's just that repository maintainers using our tools would have limited options if the PyNaCl, PyCrypto, etc., bindings are unavailable.

I got someone to kickstart packaging PyNaCl for debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=776452
We'll see where it goes.

:+1:
Thanks for making this happen.

I have just uploaded a python-nacl package, but it will need to go through NEW processing before it will be available in Debian unstable. Once jessie-backports is open, I can also look at providing a backport for Debian 8.0 ("jessie").

Thank you, @mithrandi.
I will begin testing python-nacl and Terry's upcoming pyca/Cryptography pull request soon after they are available.

Made good progress on packaging tuf as a lib, so you can import it using import tuf.client.updater (#259 and #262).
To get back to the original question... Do you also want users to be able to run tuf-something in a terminal? I mean, do you want users to be able to use some of tuf's stuff as tools?
Or do you want users/programs to be able to run something like /usr/lib/tuf/some-tuf-tool.py in a terminal?

We can treat tuf/client/basic_client.py as a command-line tool.
We used to have a few other command-line scripts, but the repository management tools are now imported or run in the Python interpreter (in interactive mode).  Adding a command-line interface for the repository management tools as been discussed, but this may be done later.  Let's just add tuf/client/basic_client.py for now.

By the way, if the following suffices, then the current packaging (#259 and #262) suffices.
/usr/share/pyshared/tuf/client/basic_client.py

Usage: basic_client.py [options]

basic_client.py: error: "--repo" must be set on the command-line.

We learn more about what Debian developers think about this as we progress with #259, #262 and #263 and start a discussion with them.
Okay, my original question in this ticket has been answered. Closeable.

The basic_client.py usage output looks correct.  I'm closing this ticket and we'll continue with Debian packaging in #259 and #262.


Getting XZ file size without the whole file

Parking this here until I can spend some more time on it and thinking about potential issues. Working my way through seeing why the initial creating of a Bag from a blocked xz object in s3 is so slow. The first bottleneck is in reading the entire file AFAICT from s3 in getsize to get the uncompressed size and make the offsets. You can get this information from the XZ footer in many cases.
I'm not sure this will work on files with multiple streams or the other failure modes in which to fail gracefully yet. I think this is rare, but I'm not sure.
from io import BytesIO

import boto3
from lzmaffi import (SEEK_SET,
                     SEEK_CUR,
                     SEEK_END,
                     STREAM_HEADER_SIZE,
                     _peek,
                     decode_stream_footer,
                     decode_index
                     )

def get_xz_size(fp):
    # mostly cribbed from lzmaffi.LZMAFile
    fp.seek(0, SEEK_END)

    fp.seek(-4, SEEK_CUR)
    padding = 0
    while _peek(fp, 4) == b'\x00\x00\x00\x00':
        fp.seek(-4, SEEK_CUR)
        padding += 4

    fp.seek(-STREAM_HEADER_SIZE + 4, SEEK_CUR)
    stream_flags = decode_stream_footer(_peek(fp, STREAM_HEADER_SIZE))
    fp.seek(-stream_flags.backward_size, SEEK_CUR)

    new_index = decode_index(_peek(fp, stream_flags.backward_size), padding)

    return new_index.uncompressed_size

def getsize(path, compression, s3):
    # no real way to know about the padding AFAICT
    # could fail more gracefully and do increasing range sizes
    client = s3.session.client('s3')
    bucket, key = path.split('/', 1)
    obj = client.get_object(Bucket=bucket, Key=key, Range='bytes=-64000')
    return get_xz_size(BytesIO(obj['Body'].read()))

Incidentally, and this is probably known, but if you set a small blocksize on a big file, you can spend a lot of time in helper functions. Some profiling I did showed ~30 seconds being spent calling isinstance, for example.
Edit Adding some profiling info.
File information. xz zipped file of the Reddit data. Made with blocksizes of 2 ** 28 bytes.
|12 $ xz -lv RC_2012-01.xz
RC_2012-01.xz (1/1)
  Streams:            1
  Blocks:             36
  Compressed size:    1,564.6 MiB (1,640,610,088 B)
  Uncompressed size:  8,985.2 MiB (9,421,643,851 B)
  Ratio:              0.174
  Check:              CRC32
  Stream padding:     0 B
  Streams:
    Stream    Blocks      CompOffset    UncompOffset        CompSize      UncompSize  Ratio  Check      Padding
         1        36               0               0   1,640,610,088   9,421,643,851  0.174  CRC32            0
  Blocks:
    Stream     Block      CompOffset    UncompOffset       TotalSize      UncompSize  Ratio  Check
         1         1              12               0      44,515,064     268,435,456  0.166  CRC32
         1         2      44,515,076     268,435,456      46,363,016     268,435,456  0.173  CRC32
         1         3      90,878,092     536,870,912      47,664,172     268,435,456  0.178  CRC32
         1         4     138,542,264     805,306,368      47,556,716     268,435,456  0.177  CRC32
         1         5     186,098,980   1,073,741,824      47,077,776     268,435,456  0.175  CRC32
         1         6     233,176,756   1,342,177,280      47,012,772     268,435,456  0.175  CRC32
         1         7     280,189,528   1,610,612,736      46,765,040     268,435,456  0.174  CRC32
         1         8     326,954,568   1,879,048,192      45,930,444     268,435,456  0.171  CRC32
         1         9     372,885,012   2,147,483,648      46,271,224     268,435,456  0.172  CRC32
         1        10     419,156,236   2,415,919,104      47,666,328     268,435,456  0.178  CRC32
         1        11     466,822,564   2,684,354,560      46,713,536     268,435,456  0.174  CRC32
         1        12     513,536,100   2,952,790,016      47,080,768     268,435,456  0.175  CRC32
         1        13     560,616,868   3,221,225,472      46,696,412     268,435,456  0.174  CRC32
         1        14     607,313,280   3,489,660,928      46,860,648     268,435,456  0.175  CRC32
         1        15     654,173,928   3,758,096,384      47,020,680     268,435,456  0.175  CRC32
         1        16     701,194,608   4,026,531,840      46,243,948     268,435,456  0.172  CRC32
         1        17     747,438,556   4,294,967,296      46,529,068     268,435,456  0.173  CRC32
         1        18     793,967,624   4,563,402,752      46,770,552     268,435,456  0.174  CRC32
         1        19     840,738,176   4,831,838,208      46,890,440     268,435,456  0.175  CRC32
         1        20     887,628,616   5,100,273,664      46,445,100     268,435,456  0.173  CRC32
         1        21     934,073,716   5,368,709,120      46,544,852     268,435,456  0.173  CRC32
         1        22     980,618,568   5,637,144,576      47,012,008     268,435,456  0.175  CRC32
         1        23   1,027,630,576   5,905,580,032      46,247,284     268,435,456  0.172  CRC32
         1        24   1,073,877,860   6,174,015,488      45,990,216     268,435,456  0.171  CRC32
         1        25   1,119,868,076   6,442,450,944      46,573,748     268,435,456  0.174  CRC32
         1        26   1,166,441,824   6,710,886,400      47,157,424     268,435,456  0.176  CRC32
         1        27   1,213,599,248   6,979,321,856      47,028,536     268,435,456  0.175  CRC32
         1        28   1,260,627,784   7,247,757,312      46,936,560     268,435,456  0.175  CRC32
         1        29   1,307,564,344   7,516,192,768      47,137,180     268,435,456  0.176  CRC32
         1        30   1,354,701,524   7,784,628,224      46,858,996     268,435,456  0.175  CRC32
         1        31   1,401,560,520   8,053,063,680      46,922,832     268,435,456  0.175  CRC32
         1        32   1,448,483,352   8,321,499,136      46,239,540     268,435,456  0.172  CRC32
         1        33   1,494,722,892   8,589,934,592      46,672,132     268,435,456  0.174  CRC32
         1        34   1,541,395,024   8,858,370,048      47,904,024     268,435,456  0.178  CRC32
         1        35   1,589,299,048   9,126,805,504      46,677,644     268,435,456  0.174  CRC32
         1        36   1,635,976,692   9,395,240,960       4,633,052      26,402,891  0.175  CRC32

Using the following snippet.
%prun bag = db.read_text(S3_URL, blocksize=2 ** 28)

With original getsize
         4256026 function calls (4254493 primitive calls) in 68.786 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   379350   58.845    0.000   58.845    0.000 {method 'read' of '_ssl._SSLSocket' objects}
   379350    1.199    0.000   61.972    0.000 ssl.py:913(recv_into)
   379350    1.196    0.000   65.091    0.000 socket.py:561(readinto)
        7    1.133    0.162   68.512    9.787 core.py:747(_fetch)
     3129    1.107    0.000   65.922    0.021 {method 'read' of '_io.BufferedReader' objects}
    52/48    1.055    0.020    1.055    0.022 {method 'join' of 'bytes' objects}
   379350    0.902    0.000   60.437    0.000 ssl.py:778(read)
   379350    0.873    0.000    0.873    0.000 {method '_checkClosed' of '_io._IOBase' objects}
   379350    0.553    0.000   59.398    0.000 ssl.py:563(read)
   379355    0.542    0.000    0.542    0.000 socket.py:600(readable)
   379350    0.508    0.000    1.050    0.000 {method '_checkReadable' of '_io._IOBase' objects}
   758711    0.303    0.000    0.303    0.000 ssl.py:766(_checkClosed)
        3    0.217    0.072   67.229   22.410 client.py:422(read)
384048/383773    0.173    0.000    0.173    0.000 {built-in method builtins.len}
...

with getsize defined above
         31610 function calls (30556 primitive calls) in 1.582 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       15    1.496    0.100    1.496    0.100 {method 'read' of '_ssl._SSLSocket' objects}
        2    0.013    0.006    0.013    0.006 {method 'load_verify_locations' of '_ssl._SSLContext' objects}
        2    0.011    0.006    0.011    0.006 {method 'do_handshake' of '_ssl._SSLSocket' objects}
        3    0.007    0.002    0.007    0.002 decoder.py:345(raw_decode)
      185    0.003    0.000    0.003    0.000 {built-in method posix.stat}
        2    0.002    0.001    0.002    0.001 {method 'connect' of '_socket.socket' objects}
       83    0.002    0.000    0.002    0.000 {built-in method posix.listdir}
        2    0.002    0.001    0.002    0.001 {built-in method _socket.getaddrinfo}
     5370    0.002    0.000    0.003    0.000 {built-in method builtins.isinstance}
        1    0.001    0.001    1.577    1.577 s3.py:16(read_bytes)
...

Small block-size and the original getsize.
I suppose I should have expected this. Not sure there's much to do here. I expected this to be marginally more efficient, coupled with a smaller file block-size. It wasn't.

For reference. There's probably a lot of smart, custom logic that could be written around the blocked compress file formats and re-used. It's not for free, but it's fairly straightforward.

I guess I'm a little confused in general on how these blocked formats stored on something like s3 could work with dask efficiently without some more work.
It looks like even doing something like take(1) is going to download the entire file for each partition, wrap the entire file with LZMAFile, then use read_block to seek to an offset? Then it will pull all those together and grab a slice from it? Is that right?
I expected to be able to set something like 48 MB blocksize in the file, point dask to it and tell it the same blocksize, and that take(1) would get the first (or whichever) 48MB chunk using Range the range header to s3 and return in a few seconds. Am I totally misunderstanding?

I guess it's hitting the cache eventually? Is there a thundering herd problem?

Ah, digging in a bit more here, I was able to get the optimized dask graph, which is what's being computed not the default output from visualize. Is there an easy way to get this? The visualization tends to hide everything interesting that happens in a list(...). I had to copy the logic in visualize
{('take-5a7336fb928041207b9a5bbb2a83acf5', 0): (list,
  (<function toolz.itertoolz.take>,
   1,
   (map,
    <function ujson.loads>,
    (<function dask.bag.text.decode>,
     (<function dask.bytes.s3.read_block_from_s3>,
      'S3_URL',
      0,
      268435456,
      (dict, [['anon', False]]),
      b'\n',
      'xz'),
     'utf-8',
     'strict'))))}

So I guess it doesn't download the entire file for this. I guess the only potential optimization that's missing is using ranges to download the blocks from s3.

@jseabold you might appreciate https://github.com/dask/dask/pull/1224
There are two mechanisms to grab data from S3, one is reading in blocks using range queries, the other is to provide an s3fs.S3File object that streams things through.  If you provide blocksize= you'll get the first, if you don't provide that then you'll get the second.
Adding LZMA on top of this complicates things because it has its own blocksize which makes things a bit funky.

Thanks, yeah, I was reading the PR that included the bytes refactor. Nice bit of work.
I saw that it should do streaming with blocksize=None, but I didn't notice any appreciable difference doing the take in speed or in my profiling. The instantiation of the bag is certainly faster.
I also didn't see the logic to do Range queries in s3fs. It looked like it would read from the beginning of the file and seek to the offset you requested rather than explicitly passing a Range header as I did above. Am I missing it?

I'll defer to @martindurant for the Range on S3 question.
Regarding streaming and performance, LZMA will have to download the entire block before it can decompress the first byte.  By default their blocksizes can be fairly large (or even the entire file is common) and so there will be a large upfront cost regardless.  You may want to download the file and then check the blocksizes using the xz command line tool.   There is, to my knowledge, no way to introspect within a block.

I'll look into what you've written above regarding timings tomorrow. Certainly, S3 is not meant for many short reads, as there is a significant cost to establishing connections.
The logic to pass Range= is in the s3fs.core._fetch_range() function: you can seek within a S3File before the first read and only get some specific piece (the block_size parameter defines the read buffer size), or call the S3FileSystem.read_block. If this isn't occurring as it should, let us know!

I created this file and was trying out different blocksizes, so I have control over that. I posted the blocksizes for this run above, but I'll need to go back and post profiling results for the smaller ones. I expected as you say, that I could pay a larger upfront cost, spending minutes in things like isinstance as noted above with the smaller blocksizes and get a faster take, but I'll need to go back and try with a smaller xz blocksize and a blocksize of None to dask to get closer to streaming behavior. I think I tried it but didn't notice substantially better behavior.
That said, and it's entirely possible I have a network bottleneck, but 44 MB (compressed) blocksizes should be a few seconds not the 300s spent in posix.read that I saw.

OK, I should take a closer look at your numbers before I comment further.

I didn't post the profiling results for take that I saw yet. That's why I was trying to read the code more and get the optimized graph, so I could say something coherent about that.
This isn't anything urgent I'm just trying to fix my expectations and understanding.

@martindurant I'll have to look closer here to see if I can avoid my boto3 call above and use your range. I didn't see it, but I didn't look very hard. Would be nice to read the footer and save a minute here, but it's going to take a bit more work than my ideal case above. Some of the xz spec notes that things are optional and I just guessed at how big the tail of the file should be.
Full disclosure: I didn't know anything about xz until I read the seekable compression code in bytes.compression and then read the xz spec and the biopython posts about this, so this isn't something mission critical for me. Just seems like it could be an easy simple win for text files. Dev time would probably be better spent on something like parquet support.

If you feel like mucking around with this you can use
bunzip2 -d -k -c $INFILE | pixz -5 -f 32 -o $OUTFILE

to recreate the reddit data as an xz file. man xz tells you the dictionary size for the compression levels and -f is a multiple of that size. The above seemed to be a decent tradeoff between blocksize and compression.

The graph you show does seem to explicitly download the whole file in one go
{('take-5a7336fb928041207b9a5bbb2a83acf5', 0): (list,
  (<function toolz.itertoolz.take>,
   1,
   (map,
    <function ujson.loads>,
    (<function dask.bag.text.decode>,
     (<function dask.bytes.s3.read_block_from_s3>,
      'S3_URL',
      0,
      268435456,
      (dict, [['anon', False]]),
      b'\n',
      'xz'),
     'utf-8',
     'strict'))))}```
(the two numerical arguments to read_block_from_s3 are offset and length)

You can pass an S3File directly to your get_xz_size and it works totally fine - only reads the footer. Actually, the code could be cleaner: `f._peek(x)` does `f.read(x); f.seek(-x, 1)`. The cache is optimised for read-ahead rather than backwards as here, but the few required reads should be fast (boto reuses it's connection at least a few times).

It feels like there might be a few issues here.
Getsize
Our current implementation is as follows:
def getsize(path, compression, s3):
    if compression is None:
        return s3.info(path)['Size']
    else:
        with s3.open(path, 'rb') as f:
            g = seekable_files[compression](f)
            g.seek(0, 2)
            result = g.tell()
            g.close()
        return result

So I guess our questions boils down to whether the combination of s3fs.S3File and the lzma.File objects play sufficiently nicely to get the size without reading in the entire file.  My intuition here is that yes they can, but the 68s cost similarity is pretty strong in @jseabold 's profiles.
We could special case this per-compression format but I'd really like to think that that logic is already in the lzma library.  Does anyone know?
Overhead cost
We should reduce the overhead to bag creation (for example calls to isinstance and tokenize) however there is only so much good we can do here.  Generally you should budget around 1ms per task/partition.  I suspect that, as you point out, your bag is very finely partitioned.

On a quick test (148-byte file!), lzma seems to read the whole file when seeking to the end:
f = s3.open('MDtemp/lzfile', 'rb')
lf = lzma.open(f)
lf.seek(0, 2)
len(f.cache) == 148

Whereas passing f to get_xy_size above only required the end
f = s3.open('MDtemp/lzfile', 'rb')
get_xz_size(f)
len(f.cache) == 24

From the docstring for lzma's seek:
        Note that seeking is emulated, so depending on the parameters,
        this operation may be extremely slow.

You have to use lzmaffi I believe.
On Wed, Jun 8, 2016, 8:28 AM Martin Durant<EMAIL_ADDRESS>wrote:

From the docstring for lzma's seek:
    Note that seeking is emulated, so depending on the parameters,
    this operation may be extremely slow.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1229#issuecomment-224588448, or mute
the thread
https://github.com/notifications/unsubscribe/AASE5JRjmWh9c00gA904rbL-OV9NGbxDks5qJsNygaJpZM4Iuj1T
.

lzmaffi.open(f) causes a read even before any seek. Which such a small file, I don't know how much of a larger file it would have read.

OK, that solves the mystery of why things are slow.  Assuming that lzmaffi handles things well it seems like we have two options:

Register getsize functions per compression (doable)
Just depend on lzmaffi (also doable)

Thoughts on the maintenance/support tradeoff?

lzmaffi should be tested to see how it behaves within getsize and bag first. I suspect one-to-one replacement might not be enough, and option 1) will have to be the one.

My guess is it's reading the index of the stream like I'm doing above, so
it knows how to seek, but we can always check.
On Wed, Jun 8, 2016, 8:53 AM Martin Durant<EMAIL_ADDRESS>wrote:

lzmaffi.open(f) causes a read even before any seek. Which such a small
file, I don't know how much of a larger file it would have read.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1229#issuecomment-224595628, or mute
the thread
https://github.com/notifications/unsubscribe/AASE5PpbKG-WeEHs90XuP_l27ZPXHdvnks5qJslQgaJpZM4Iuj1T
.

Actually, I'm fairly certain I cribbed that from LZMAFile.open in lzmaffi.

open seems to read the whole of my little file, while the get-size function above correctly only gets the footer. That doesn't matter of course for this little file. I'm interested to know how much is read for much bigger files like the ones you are dealing with.

This is the relevant code. Depending on how you created the file, it would
need to read from parts of the whole file to hit the blocks that it needs
to get all of the blocksize information, assuming that isn't all in the
index. I need to read the spec linked above a bit more carefully. That's
why I cribbed the part just to get the stream size since dask doesn't care
about anything else since it makes its own blocks.
I'm having trouble thinking about this in terms of the S3File because I
haven't spent enough time with that code. I'm a bit wary about seeking
through files by making separate s3 API calls through a file-like
abstraction though.
https://github.com/r3m0t/backports.lzma/blob/master/lzmaffi/__init__.py#L300
On Wed, Jun 8, 2016 at 12:29 PM Martin Durant<EMAIL_ADDRESS>wrote:

open seems to read the whole of my little file, while the get-size
function above correctly only gets the footer. That doesn't matter of
course for this little file. I'm interested to know how much is read for
much bigger files like the ones you are dealing with.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1229#issuecomment-224666896, or mute
the thread
https://github.com/notifications/unsubscribe/AASE5DGIbtX1150W4xFN8EyborHqhw3Fks5qJvv1gaJpZM4Iuj1T
.

So it looks like opening using lzmaffi creates the whole block index, which involves stepping through the file block-by-block in reverse order.
This will indeed read the whole file into cache in S3File, with one S3 read per block.
lzma's File only did a read on the first seek, I haven't looked yet to see if that too steps through the blocks at that point.

I addition, at lzmaffi always checks the magic value at the head of the file.
S3File was designed to get some contiguous block out, so if you read at the head and then then foot, you will get the intermediate data too... Repeatedly calling S3FileSystem.read_block would avoid this.
The blog post here http://blastedbio.blogspot.ca/2013/04/random-access-to-blocked-xz-format-bxzf.html suggests that the index should be at the end, and there should be no need to page through all the blocks.

As a reference, lzmaffi already is loaded with #1096 if installed. But perhaps one should consolidate the documentation and/or give a warning when lzma is used.

Note: I think we may have lost the usage of lzmaffi. I don't see usages of dask.compatability.lzma_compress anywhere inside dask/dataframe.

fsspec support xz files: https://github.com/intake/filesystem_spec/blob/master/fsspec/compression.py#L31
However, the "blocks" mode of xz is not standard and seems rare, so pseudo-random-access is not implemented.
lzma_compress can probably be removed from dask.

(I recommend closing this issue)

FWIW my experience is similar to Martin's.  I'm not sure that supporting xz
is worth the time.
On Thu, Aug 8, 2019 at 8:21 AM Martin Durant<EMAIL_ADDRESS>wrote:

fsspec support xz files:
https://github.com/intake/filesystem_spec/blob/master/fsspec/compression.py#L31
However, the "blocks" mode of xz is not standard and seems rare, so
pseudo-random-access is not implemented.
lzma_compress can probably be removed from dask.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1229?email_source=notifications&email_token=AACKZTDOTKCPK2Z67IB2RL3QDQ2ZBA5CNFSM4CF2HVJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD33652I#issuecomment-519565033,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACKZTCZP2SZ5AXK33BL25TQDQ2ZBANCNFSM4CF2HVJQ
.

Oh yeah, given the age of the issue I'm also in favor of just closing with a "this has gone stale, please reopen if this is still a live issue"

Recognizing that conversation has some non-trivial cost.


[cinder-csi-plugin] Fixing ca-file reference in README + adding link

The binaries affected:

[ ] openstack-cloud-controller-manager
[*] cinder-csi-plugin
[ ] k8s-keystone-auth
[ ] client-keystone-auth
[ ] octavia-ingress-controller
[ ] manila-csi-plugin
[ ] manila-provisioner
[ ] magnum-auto-healer
[ ] barbican-kms-plugin

What this PR does / why we need it:
Updates the ca-file reference in the README + a link to parameters doc for more info
Which issue this PR fixes:
N/A
Special notes for reviewers:
Just a simple doc update/change
Release note:

NONE

I signed it

@oybed Thanks for the PR. Looks like CLA still not being recognized as signed, please follow the steps again mentioned here , https://github.com/kubernetes/cloud-provider-openstack/pull/948#issuecomment-592987292 to make sure CLA is signed.

/ok-to-test

@ramineni thx - looks like I had missed a step. Should be good now. Please let me know if not.

/lgtm

/lgtm
/approve
Thanks


Doesn't appear to work with VUE 3

vue-router.esm-bundler.js:3284 TypeError: createElement is not a function
    at Proxy.render (webpack-internal:///./node_modules/vue-dndrop/dist/vue-dndrop.esm.js:2881)
    at renderComponentRoot (webpack-internal:///./node_modules/@vue/runtime-core/dist/runtime-core.esm-bundler.js:891)
    at ReactiveEffect.componentUpdateFn [as fn] (webpack-internal:///./node_modules/@vue/runtime-core/dist/runtime-core.esm-bundler.js:4636)
    at ReactiveEffect.run (webpack-internal:///./node_modules/@vue/reactivity/dist/reactivity.esm-bundler.js:197)
    at setupRenderEffect (webpack-internal:///./node_modules/@vue/runtime-core/dist/runtime-core.esm-bundler.js:4762)
    at mountComponent (webpack-internal:///./node_modules/@vue/runtime-core/dist/runtime-core.esm-bundler.js:4545)
    at processComponent (webpack-internal:///./node_modules/@vue/runtime-core/dist/runtime-core.esm-bundler.js:4503)
    at patch (webpack-internal:///./node_modules/@vue/runtime-core/dist/runtime-core.esm-bundler.js:4107)
    at mountChildren (webpack-internal:///./node_modules/@vue/runtime-core/dist/runtime-core.esm-bundler.js:4294)
    at mountElement (webpack-internal:///./node_modules/@vue/runtime-core/dist/runtime-core.esm-bundler.js:4203)

i will take a look at it right away. thanks for the feedback!

I run into this issue as well, btw I'm using script-setup syntax, not sure if it'll be an issue.

Pretty easy to fix:
in Container.js and Draggable.js, please:

Edit $slots.default to $slots.default()
Add import { h } from 'vue' in the top of the files
Change the notation of render: function (createElement) to render: function ()
Change createElement to h in both render functions

Let me know if you would like a pull request.

Hey, @codingcronus, thank you very much for your time and this awesome solution! 💡
I have published a stable version of this repo supporting both Vue2 and Vue3 projects:

For vue2:

npm i vue-dndrop

For vue3:

npm i vue-dndrop@next

Feel free to try it out 🎉


Add_units_table also adds unit_name

Motivation
This is the second PR to break down all the code changes in https://github.com/catalystneuro/nwb-conversion-tools/pull/450 as requested by @CodyCBakerPhD .
Here functionality for dealing with non int units_ids in sorting extractor objects from spikeinterface is added. Plus, as in the case for the electrode table a new column called "units_name" is added with the goal of serving as a cross-writing identifier for row matching in the units table.
Minor modifications of naming and style preparing for the last and larger refactor are also push forward in this PR.
How to test the behavior?
Simple tests were added to test_si for three cases:

Writing int units_ids
Writing string units_ids
Not overwriting the property of units_name if this is already in the sorting object.

@h-mayorquin Thanks again for taking the time to break this up. I'll take a look at this in detail tomorrow!

@h-mayorquin Overall looks very good, I'll give it another pass later tonight to be sure but will probably approve/merge prior to our meeting tomorrow.


Fix aura re resort and tighten form reuse logic

The bugfix is only in the aura deep clone, everything else is an extra fix to what inluki did that fixes the root of the problem (but what he did is still important to avoid re-prep spam).


Windows: jadx-gui-0.8.0.exe can't be placed in the Open With context menu

On Windows 10 I tried to add the jadx-gui-0.8.0.exe in the Open With list of APK files.
However it does not matter what I do Jad-X never appears in this context menu even if it you do open an APK using jadx-gui-0.8.0.exe once. Other apps then appear in the open with context menu, but not JadX.
Really strange. There must be something different/wrong with the jadx-gui-0.8.0.exe that prevents Windows from displaying it in the open with list.

I searched the net for this problem and found many similar issues in other applications.
It seems to be a problem with Windows 10 and the included app meta data in the EXE file. WIndows seems to assume that every EXE file has at least CompanyName and a FileDescription entry that is not empty.
For jad-x this means that the launch4j configuration options should include a companyName = "....."  entry. It seems like the FileDescription is automatically added by launch4j. In my personal build it worked after adding the comanyName line.
I would provide a pull request, but I have to admit that I don't have a clue what we should set as companyName for an open source software like jad-x.
In my tests I used companyName = "Open Source Software (Apache License 2.0)"
Any comments or other suggestions?

@jpstotz I think just jadx as CompanyName is fine. Thanks for solving this!

OK. Change is include din pull request #379


Add support for multiple custom_structured_configuration prefixes

Change Summary

Allow multi layered custom_structured_configuration and more flexible naming.
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes)
[X] Documentation content changes
[ ] Other (please describe):

Related Issue(s)

Component(s) name
eos_l3ls_evpn
Proposed changes

custom_structured_configuration_prefix allows the user to customize the prefix for Custom Structured Configuration variables.
Default value is custom_structured_configuration_.	Default value is custom_structured_configuration_. Remember to include any delimiter like the last _ in this case.
It is possible to specify a list of prefixes, which will all be merged one by one. The order of merge will start from beginning of the list, which means that keys defined in the later prefixes will be able to override keys defined in previous ones.
custom_structured_configuration_prefix: < variable_prefix, default -> "custom_structured_configuration_" >	custom_structured_configuration_prefix: < variable_prefix, default -> "custom_structured_configuration_" >
#or
custom_structured_configuration_prefix: [ < variable_prefix_1 > , < variable_prefix_2 > , < variable_prefix_3 > ]

Example using multiple prefixes:
custom_structured_configuration_prefix: [ my_dci_ , my_special_dci_ ]

my_special_dci_ethernet_interfaces:
  Ethernet4000:
    ip_address: <IP_ADDRESS>/21

my_dci_ethernet_interfaces:
  Ethernet4000:
    description: My test
    ip_address: <IP_ADDRESS>/12
    shutdown: false
    type: routed
    mtu: 1500
    peer: MY-own-peer
    peer_interface: Ethernet123
    peer_type: my_precious

In this example  Ethernet4000 will be added to the ethernet_interfaces dictionary in the Structured Configuration and the ip_address will be <IP_ADDRESS>/21 since ip_adddress was overridden on the later custom_scructured_configuration_prefix
How to test

Tested with my test repo: https://github.com/ClausHolbechArista/claus-l3ls-dev/commits/superspine-alternative
Following commits show each test scenario.

[x] Test without custom config
[x] Test with default prefix and custom config
[x] Test with 1 custom prefix but no matching config
[x] Test with 1 custom prefix and both matching and non-matching custom config (non-matching is just ignored)
[x] Test with multiple custom prefixes. One using the default name and others using custom names. Also including overrides.

Checklist:

[X] My code has been rebased from devel before I start
[X] I have read the CONTRIBUTING document.
[X] My change requires a change to the documentation.
[X] I have updated the documentation accordingly.
[x] I have updated molecule CI testing accordingly
[ ] All new and existing tests passed (pre-commit, make linting and make sanity-lint).

Rebased and refreshed CI artifacts

Rebased and refreshed CI artifacts


vk-sys/vulkano{-win}: correct type of native window

This commit changes the definitions for window in XlibSurfaceCreateInfoKHR
and XcbSurfaceCreateInfoKHR with to u64 (xlib) and u32 (xcb) (see
definitions below).
vk-sys defined XlibSurfaceCreateInfoKHR and XcbSurfaceCreateInfoKHR with
pub window: *const c_void. This propagates through to the API in
vulkano::swapchain::Surface::from_{xlib,xcb}(). As the native window types
in xcb/xlib are defined as unsigned ints, this may lead to the belief that a
pointer to such an integer is required. vulkano will create a Surface but
this surface will not be supported by any queue family.
xlib: https://cgit.freedesktop.org/xorg/proto/x11proto/tree/X.h#n96 and
https://cgit.freedesktop.org/xorg/proto/x11proto/tree/X.h#n66
  (search `Windows`)

xcb: https://xcb.freedesktop.org/manual/group__XCB____API.html
(search `xcb_window_t`)

(winit defines the field in XWindow correcty but casts into a *mut c_void in get_xlib_window().)

Oh, good point. Thanks for the fix.
While xcb_window_t is defined as u32, XID is defined as unsigned long, which is 32bits on 32bits Linux and 64bits on 64bits Linux.
Please use std::os::raw::c_ulong instead of u64 for xlib.

Thanks!


Disable write/read Parquet when Parquet field IDs are used

Temporarily disable write/read parquet when schema has a specified Parquet field ID
Solution is: Fallback to CPU when writing Parquet with field id column. Fallback to CPU if
"spark.sql.parquet.fieldId.read.enabled" is true when reading Parquet.
Contributes #4846
Signed-off-by: Chong Gao<EMAIL_ADDRESS>

build

build


Added Best Practices

@ConnorLand

I'll also add links later to other best practices docs: https://docs.mendix.com/refguide/dev-best-practices/


Update PetDB graph on IEDA page

The graph at this page needs to be updated: https://www.iedadata.org/community/usage/
Currently it is showing all time lows because the numbers do not include new PetDB statistics. We need to update this graph to include new PetDB statistics.

When old PetDB is shut down, the graph should be populated only by new PetDB statistics
The old statistics should remain for everything before that date
Any months (December, January, February) where both old and new PetDB were operating should have statistics that represent usage by both systems.

Lulin says that if new PetDB web service has the same API it is easy to combine the old and new statistics into one plot.

@sparkji Can you implement a web service which will return the value as below for the ECDB. You can left out unique_email field because we are using that now.
http://isotope.ldeo.columbia.edu:7001/petdbWeb/search/download_stat.jsp?start=2018-12-01&end=2019-02-01

Will do.

The API for download statistics has been added. Please check https://ecapi.earthchem.org/statistics/downloadstatistics?start=2018-12-01&end=2019-02-01
<STATISTICS type="DOWNLOAD">
  <RECORD>
    <YEAR>2018</YEAR>
    <MONTH>12</MONTH>
    <UNIQUE_IP>3</UNIQUE_IP>
    <UNIQUE_EMAIL>2</UNIQUE_EMAIL>
    <MONTHLY_DOWNLOAD>4</MONTHLY_DOWNLOAD>
  </RECORD>
  <RECORD>
    <YEAR>2019</YEAR>
    <MONTH>1</MONTH>
    <UNIQUE_IP>106</UNIQUE_IP>
    <UNIQUE_EMAIL>70</UNIQUE_EMAIL>
    <MONTHLY_DOWNLOAD>332</MONTHLY_DOWNLOAD>
  </RECORD>
</STATISTICS>

@sparkji When was the NEW ECDB Search released? LS

In the middle of Nov.
Try this
https://ecapi.earthchem.org/statistics/downloadstatistics?start=2018-11-01&end=2019-03-01

In the middle of Nov.
Try this
https://ecapi.earthchem.org/statistics/downloadstatistics?start=2018-11-01&end=2019-03-01

@sparkji Is there any Nov stats? The above does not return November stats.

@lulin-song actually we advertise new search to users until Nov. 28, so there is no stats for Nov and fewer numbers for Dec.

@sparkji  Can you check the web service to return XML instead of 'String'? Maybe '' missing in the returned string?
Thanks, LS

@kelseymarkey The plot is updated. I created a test plot URL here. https://test-ecl.earthchem.org/plot/PetDBStatsPlotView.php Please check.

@lulin-song this looks great! Please proceed.

Done.


The default size of "Delete Files" dialog is too small

Storage Explorer Version: 1.8.1_master_20190610.4
Platform/OS: Linux Ubuntu/macOS High Sierra/Windows 10
Architecture: ia32/x64
Regression From: Previous release(1.8.1)
Steps to reproduce:

Expand a storage account -> File Shares.
Create a new file share and upload a file to it.
Select the uploaded file and try to delete it.
Check the prompt dialog.

Expect Experience:
Pop up the "Delete Files" dialog that show messages completely.
Actual Experience:
The default size of "Delete Files" dialog is too small to show complete messages.

More Info:
This issue also reproduces for one blob.

Fix merged into master.

This issue reproduces on feature\main-host branch 20190614.4. So we re-open it.

We just need to merge master into the feature branch.

Verified this issue on build feature/main-host 20190624.2. It has been fixed. So we close it.


Fire change events on mutually exclusive elements when their value changes

Will resolve #187

 Fire change events on mutually exclusive options


Support M1 & Intel Mac

We have experience with this. Let us know if you need help.

I'll create a gist and share it, don't know if it entirely solves your issue. But we do have leap running on M1

Yeah we could run on mac with other configurations, not when starting from genesis tho

@poplexity found that if you disable OC mode it works, in tevmc.json this can be done at:
config['nodeos']['ini']['vm_oc_enable'] = false


Redirected searches have an extra period appended to them

When I launch a Cortana voice search the redirected query has an extra period at the end. It doesn't affect usage, but it would be a nice quality of life fix :smile:

I'm very much struggling to reproduce this. Still an issue?

@TheoBr definitely not, iirc I ended up disabling Cortana on my laptop (due to unrelated reasons). I'll close this out. Thanks for the diligent follow up tho! Not a lot of devs will flup on an almost-two year old issue.


data process

For data of training process,
If I have image size 100x160(h x w) and I want to design a 224 x 224 network.
How to deal with the images is better?

dirctly resize the image size 100x160 to 224x224
resize the image size 100x160 to 224x224 with padding 0

2 is better

Hi zhanghang
How about your data process for image resize?
Here is your process, right?
resize the image size 100x160 to 224x224 with padding 0
https://github.com/zhanghang1989/ResNeSt/blob/master/scripts/torch/verify.py#L140

Yea, that's the crop method for evaluation


Can't get GUI properly

Can't get GUI after running vncserver, just displaying square curser with some options on top left corner

Run vncstart to start the vncserver and vncstop to stop the vncserver

same problem here, vncstart or vncserver-start doesn't do anything

same problem here, vncstart or vncserver-start doesn't do anything

show the output when you use vncstart. If you dont give us proper info then we could not fix the issue because most of the users are getting the GUI properly.

same problem here, vncstart or vncserver-start doesn't do anything

show the output when you use vncstart. If you dont give us proper info then we could not fix the issue because most of the users are getting the GUI properly.

well then good luck finding an alternative sir 🫡. Linux=errors/problems so you have to try to fix it not just uninstalling it.


Put the Cookbook contributors guide into the main site

As discussed at recent Pythia meetings, we want to elevate the Cookbook Contributors guide by styling it with the Pythia theme and making it easier to find.

Plan:

[ ] Move the markdown to this repo
[ ] Modify the prominent button on the gallery page to say "How can I make a new Cookbook?", and have it point to the guide
[ ] Put the "Submit a new Cookbook" button at the end of the Cookbook contributors guide instead of on the gallery page
[ ] Update all links to the Cookbook guide for internal consistency

All done!


✅ Use one instance of envtest across tests

There has been issues with running multiple instances of envtest in
parallel. This changes the tests to use a single instance of envtest by
using a testify test suite and its SetupSuite and TearDownSuite to
setup and teardown envtest.

fixes #226


Support incompatible schema changes

Describe your changes
In this diff we build list of deltas of incompatible schema changes, from the history of the collections schema changes.
On read and update we apply all the schema deltas, which has newer version then the version of the row.
Incompatible schema changes supported:

field deletions
field type changes
reducing maxLength of strings and bytes

For the type changed we are doing best effort convertion. If the type is not convertible we are returning empty value for it. For the convertion rules please refer to TestTypeConversionRules.
Future work:

Currently the feature doesn't work if search is enabled. (turned out TypeSense doesn't allow field drop/add if data present in the collection)
Support max_length truncation for the strings
We can also support adding default value to the field. Populating empty fields in the existing rows with new default value. This can work for static defaults and updated_at(), created_at().

Feature can be configured by schema.allow_incompatible = true setting
How best to test these changes
Unit and integration tests added
Issue ticket number and link
TIG-749

Codecov Report
Base: 32.83% // Head: 33.77% // Increases project coverage by +0.94% :tada:

Coverage data is based on head (0d77e52) compared to base (a42403e).
Patch coverage: 71.77% of modified lines in pull request are covered.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #748      +/-   ##
==========================================
+ Coverage   32.83%   33.77%   +0.94%
==========================================
  Files         133      133
  Lines       14963    15251     +288
==========================================
+ Hits         4913     5151     +238
- Misses       9574     9620      +46
- Partials      476      480       +4

Impacted Files
Coverage Δ

schema/search.go
31.20% <0.00%> (-5.77%)
:arrow_down:

server/metadata/tenant.go
38.52% <0.00%> (-0.10%)
:arrow_down:

server/services/v1/database/query_runner.go
4.64% <0.00%> (-0.03%)
:arrow_down:

server/services/v1/database/search_indexer.go
43.75% <0.00%> (-0.40%)
:arrow_down:

server/services/v1/database/sessions.go
5.55% <0.00%> (-0.03%)
:arrow_down:

schema/collection.go
71.91% <66.66%> (+0.64%)
:arrow_up:

schema/migration.go
82.59% <87.74%> (+72.59%)
:arrow_up:

schema/fields.go
78.68% <100.00%> (+1.31%)
:arrow_up:

schema/rules.go
90.00% <100.00%> (+0.34%)
:arrow_up:

schema/schema.go
50.51% <100.00%> (ø)

... and 8 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Looks good overall. One thing I realized is that we probably need better documentation on what all changes are allowed in the case of Arrays and Objects?


Typo in documentation comments, 'initialise' for 'initialize'

Typo in documentation comments, 'initialise' for 'initialize'

Good observation! :p


refactor: Replace <IP_ADDRESS> with HOST_IP

Parse the host IP from route
Set env

Fixes 54

Fixes #54


Last Week in Pony - December 31, 2023

@SeanTAllen Did you intend for this to be a modification of the 2023-12-24 file?

I modified the file that is included here in case that is the intention. I suspect the file last-week-in-pony-123123.md was meant to be added though rather than modifying last-week-in-pony-122423.md.

I did not

The contents need to be added to a new file and the change to 24 reverted.

I'm fixing the files


Fix compilation under -Werror

With strict warnings and warnings as error, openssl currently does not compile
due to a missing include.
Signed-off-by: Juergen Christ<EMAIL_ADDRESS>
Checklist

Just for reference, @juergenchrist s commit fixes the following warning:

02:12:19: crypto/ec/ecx_meth.c:918:9: error: implicit declaration of function 'RAND_priv_bytes_ex' [-Werror=implicit-function-declaration]
02:12:19:   918 |     if (RAND_priv_bytes_ex(ctx->libctx, privkey, X25519_KEYLEN) <= 0)
02:12:19:       |         ^~~~~~~~~~~~~~~~~~
02:12:20: cc1: all warnings being treated as errors
02:12:20: make[1]: *** [Makefile:15093: crypto/ec/libcrypto-lib-ecx_meth.o] Error 1

I suppose this manifests only on some platforms as on linux x86_64 this clearly does not. Perhaps the reason is the openssl/rand.h is being included on linux x86_64 from some of the other includes in the file. Anyway this does not matter.

24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set.  Please review the updates and set the label manually.

Merged to master, thanks for the fix.


optimize compilation speed

What problem does this feature solve?
in scripts/build.js
I found that @microsoft/api-extractor was used for the build

If you invoke API Extractor multiple times for a single tsconfig.json environment, this approach also allows you to reuse the same CompilerState object across multiple invocations. This can be a significant performance optimization, since the TypeScript compiler analysis is relatively expensive.

reference

[1] @microsoft/api-extractor

What does the proposed API look like?
optimize compilation speed

Thanks for the PR but please don't duplicate issues and PRs


chore(ci): Upgrade actions/upload-artifact from v2 to v3 to address deprecation

Description
Upgrade actions/upload-artifact from v2 to v3.
This update is necessary to address the deprecation of v2. The action is failing with the following error:
Error: This request has been automatically failed because it uses a deprecated version of `actions/upload-artifact: v2`. Learn more: https://github.blog/changelog/2024-02-13-deprecation-notice-v1-and-v2-of-the-artifact-actions/

The repository's README contains the following warning:
actions/upload-artifact@v3 is scheduled for deprecation on November 30, 2024. [Learn more.](https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/) Similarly, v1/v2 are scheduled for deprecation on June 30, 2024. Please update your workflow to use v4 of the artifact actions. This deprecation will not impact any existing versions of GitHub Enterprise Server being used by customers.

Therefore, we need to upgrade the action to address this issue.
Note: I submitted this PR directly because this PR addresses a straightforward update. If an issue discussion is preferred, please let me know and I'll create one promptly.
Related Issue(s)

Breaking Change?
No.
Checklist

[x] I have made corresponding changes to the documentation (if required)
[x] I have added tests that prove my fix is effective or that my feature works

Hi @kesompochy, thank you very much for contributing to this and improving our CI.
Just yesterday, we merged #1208, which upgrades to upgrade-artifact to v4.
We're still facing the issue:
Run coverage combine ./coverage/coverage*/.coverage
Couldn't combine from non-existent path './coverage/coverage*/.coverage'

By adding include-hidden-files: true to our CI, similar to what was done in https://github.com/astronomer/astro-provider-ray/pull/60/files, will overcome the issue

I just noticed this last fix to add include-hidden-files: true was done in #1210, so I'm closing this PR. But thank you very much, again, for helping out on this.!

@tatiana , thanks for the update. I appreciate you taking the time to review my PR and provide information about the recent changes. I'm glad the issue has been resolved.
Thank you again for your time and for maintaining such an active project.


Review domain model and architecture

Description
The proposed domain model and architecture need to be inspected and reviewed by at least two developers
Acceptance Criteria
dev1

[ ] domain model and architecture have been understood
[ ] there are no more open questions for now

dev2

[ ] domain model and architecture have been understood
[ ] there are no more open questions for now

Why is Media connected with 2 composition lines?
How are we going to solve Authentication? First Idea: If a user is part of the right course they have permission. When splitting the graphql request in the API gateway, check once for authentication and send the parts of the request to the different microservices, telling all of them that the user has permission to see the information. IMPORTANT: This only works if you have a strict tree structure for graphql requests. If a request references e.g. another course (so horizontal connections in the tree), it won't work!


fix Escape string when insert stats

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run feut


Debugger won't stop at breakpoints

Hi, I can't seem to get the debugger to pause at breakpoints; I see the console logs though.
This is the error in the console also: Debugger: error: adapter failed hover evaluation Stack frame not found
But, if I use debugger via Chrome and inspect the Node.js process and leave a breakpoint in Chrome, I see the breakpoint in Sublime Text Debugger.
Any clues as to whats going on here pls? Heres the config I am using:
{
    "folders": [
        {
            "path": "/Users/name/Projects/Web/w-project"
        }
    ],
    "debugger_configurations": [
        {
            "name": "Attach to Node.js",
            "type": "node",
            "request": "attach",
            "port": 9229,
            "address": "localhost",
            "restart": false,
            "localRoot": "${workspaceFolder}",
            "remoteRoot": "/Users/name/Projects/Web/w-project",
            "skipFiles": ["<node_internals>/**"],
            "outFiles": ["dist/**/*.js"],
            "resolveSourceMapLocations": [
				"dist/**/*.js.map",
                "dist/**/*.js"
			]
        }
    ]
}

Sublime Text build: 4180
macOS 14.6.1

Debugger: error: adapter failed hover evaluation Stack frame not found is from hovering the mouse over something that looked like an expression its not an issue.
If the breakpoints are gray when you start the debugger then the adapter was unable to verify them. You can hover over the breakpoint and it should have a message from the adapter which may or may not be useful.
My guess would be there is some configuration issue which is not allowing the adapter to determine where to place the breakpoints in the generated source files. I believe resolveSourceMapLocations, outFiles, localRoot, remoteRoot all impact breakpoints but thats a bit beyond my knowledge of how js debug adapter works (https://github.com/microsoft/vscode-js-debug)
Microsoft has a bunch of docs which might help such as https://code.visualstudio.com/docs/nodejs/nodejs-debugging#_source-maps


Expose route and app models to RunnerCall for extensions (alternative 2)

Another alternative to #878 (and #879)
This allows us to merge the Annotations from app and route (which are the useful information which is missing in Call) into the models.Call. The field is ephemeral like Config so it should not be persisted and should not cause issues.

I'm happy with this approach since it works for extensions (e.g. Call listeners) and is not persisted to the database. I also think that route annotations overriding app annotations is an acceptable solution and consistent with our approach to configuration. I'm closing #878 and #879 in favor of this one.


Cannot Handshake Client

I confirm (Loll)
Thanks!
Henlo,
So, I don't need help making the phishlet. (well, I guess technically I do, or I wouldn't be here)
What I do need help with is, is this:
[08:02:07] [+++] successfully set up SSL/TLS certificates for domains: [mydomain.com]
[08:02:16] [imp] [1] [company] new visitor has arrived: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0 (<IP_ADDRESS>)
[08:02:16] [inf] [1] [company] landing URL: https://mydomain.com/LureIDHere
[08:02:16] [+++] [1] detected authorization URL - tokens intercepted: /path/to/login
: 2019/04/27 08:02:37 [102] WARN: Cannot handshake client mydomain.com EOF
Whenever I or another person attempt to login in, it does nothing. just says "sorry, something went wrong, click back and resend the info".
If I try using a different domain (we have tons), it'll go to the login page, it'll capture the data
such as email/pass/tokens, but then still doesn't actually sign in? just keeps redirecting to the
page that says something went wrong, go back.
I thought maybe because it was the domains fault. So I set up A/CNAME records for every subdomain and main domain being phished. I gave it time to propagate
I thought it was my landing path, my auth url, even the phish domains, I tried changing all of
that, which just gives me no results. I'm nearly convinced it's maybe the way GO handles proxies? I looked into the documentation for both this tool and GO, and then looked up the handshake error elsewhere, and I did see other people using GO, have similar issues. So then I thought I'll write my own proxy script, or just change yours, which I did. Still does nothing.
One the first domain, where it's capturing everything, but doing nothing after hitting login, I manually added the site as an exception. The second one, I did no such thing, and it just doesn't capture or anything. Just redirects to the sorry something went wrong page.
I looked at the phishlets using 2FA bypasses, and tried to imitate those to no success.
Could you possibly point me in the right direction, of fixing the issue of "Cannot Handshake Client - EOF".
I'm currently running version 2.3.1 and my vps is running Debian
I've gone through the other some odd 200+ issues, and saw a few others had the issue, but no one actually posted a fix (that was even remotely relevant) the close I could find to my issue is this: /issues/248 - And there was no solution that helped me there.
:{}

Well, after messing around with my domain and mapping everything, pointing it to diff locations, etc etc.
I've solved the handshake issue.
For anyone else who comes across this, if it says "cannot handshake client domain.com EOF
Make an A record that points to your server, I'm also using a domain that has a real SSL cert as well, so that may help.

this is happening to me ive created an  A record that points to my server but no avil


WIP: Test infra improvements

I'll say this is "ready" once I get 5 error-free passes.

A few improvements:

Add test timing reporting information for simple perf comparisons and debugging.
Unpin pytest/pytest-cases since new ones are compatible
Update waiting strategy for flakiest tests
Move cache cleanup to weakref.finalizer rather than __del__. As far as I can tell from some threads (here, the PEP, here) seems to be the case that .finalize is more reliable because it is called when all references are gone rather than the arbitrary timing of garbage collection.

@jayqi Ready, but not urgent. Thinking this can be just post-release since it is not really user-facing. I hope it improves test reliability, but in my experience failures seem to be related in some way to resource load on shared GH runners (failures seem to cluster in time). It just needs a history change added, but didn't want to collide with #463


bug: Default answer based on previous questions does not work properly

Describe the problem
The default value of a question based on previous question does not render properly.
copier.yaml
### Questions for the user
project_name:
  type: str

project_client:
  type: str

project_id:
  type: str
  default: "{{ project_name }}-{{ project_client }}"

Template
To Reproduce

Run copier copy template project
Answer to 1 -> project
Answet to 2 -> pepito
Default answer of 3 question --> None-None

Expected behavior
Default answer to 3 question should be project-pepito
Screenshots/screencasts/logs

Environment

OS: Pop-OS 20.04
Copier version: 6.0.0a6

Can you confirm it happens in latest git master?

@Yajo I can confirm that the issue is already fixed in version  v6.0.0a9!
Thanks men for maintaining this geat repo!!


Bug fixes

Makefile use spaces instead of tab in all section
Execute mpi-run.sh explicitly with bash to avoid the ENOEXEC error
replace the efa installer to be the latest one

Signed-off-by: Shi Jin<EMAIL_ADDRESS>Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

The changes on installer is not correct, will resend the PR.


Document that Array operates correctly only with POD types

Update to documentation to specify that Array requires T to be a POD type.

PR
Author
Editor
Reviewers
Assignment
Approval
Merge

#1615
@rw-anderson
@tzanio
@stefanhenneking + @cjvogl
07/14/20
⌛due 07/28/20
⌛due 08/04/20

Merged in next for testing...


Expression tree parsing fails after solution migration from MVC4 (framework 4.6.1)

From @wurdum on August 8, 2017 14:53
After migration to asp net core previously working expression tree parsing fails now.
I use expressions to generate qualified names for inputs. This code @Model.GetElementName(m => m.Rows.Get(row).Cells.Get(cell).Value) should generate something like Rows[1].Cells[1].Value
What wonders me, this code works as expected:
@foreach (var r in Model.Rows) {
    foreach (var c in r.Cells) {
        var row = r;
        var cell = c;

Row: @row.Id, Cell: @cell.Id

Input name: @Model.GetElementName(m => m.Rows.Get(row).Cells.Get(cell).Value)

    }
}

But this fails:
@foreach (var row in Model.Rows) {
    foreach (var cell in row.Cells) {

Row: @row.Id, Cell: @cell.Id

Input name: @Model.GetElementName(m => m.Rows.Get(row).Cells.Get(cell).Value)

    }
}

I've created repo with project that reproduces error: https://github.com/wurdum/ExpressionTreeIssue
Copied from original issue: aspnet/Razor#1603

I don't believe there's a framework bug here. Certainly not a new MVC issue. Looks more like a minor change in the generated code for some complex Linq expressions.
The root problem is an ArgumentException thrown in the repro project's Parser class while it is visiting nodes in a Linq expression. The exception occurs after "Value" and "Cells[1]" are already in steps i.e. when visiting the m.Rows.Get(rows)` portion.
Put another way, the Parser should evaluate sub-Expressions more carefully. See for example the way ExpressionHelper isolates indexer expressions before evaluating them. Of course ExpressionHelper does not handle arbitrary method calls.
System.ArgumentException: Field 'row' defined on type 'AspNetCore._Views_Home_Index_cshtml+<>c__DisplayClass21_0' is not a field on the target object which is of type 'AspNetCore._Views_Home_Index_cshtml+<>c__DisplayClass21_1'.
   at System.Reflection.RtFieldInfo.CheckConsistency(Object target)
   at System.Reflection.RtFieldInfo.InternalGetValue(Object obj, StackCrawlMark& stackMark)
   at System.Reflection.RtFieldInfo.GetValue(Object obj)
   at ExpressionTreeIssue.Controllers.Parser`1.GetArgumentId(MemberExpression arg) in C:\dd\Projects\ExpressionTreeIssue\Controllers\Parser.cs:line 63
   at ExpressionTreeIssue.Controllers.Parser`1.VisitMethod(MemberExpression caller, MemberExpression arg, List`1 segments) in C:\dd\Projects\ExpressionTreeIssue\Controllers\Parser.cs:line 58
   at ExpressionTreeIssue.Controllers.Parser`1.VisitNode(Expression body, List`1 steps) in C:\dd\Projects\ExpressionTreeIssue\Controllers\Parser.cs:line 44
   at ExpressionTreeIssue.Controllers.Parser`1.VisitNode(Expression body, List`1 steps) in C:\dd\Projects\ExpressionTreeIssue\Controllers\Parser.cs:line 47
   at ExpressionTreeIssue.Controllers.Parser`1.VisitNode(Expression body, List`1 steps) in C:\dd\Projects\ExpressionTreeIssue\Controllers\Parser.cs:line 36
   at ExpressionTreeIssue.Controllers.Parser`1.GetCallSteps() in C:\dd\Projects\ExpressionTreeIssue\Controllers\Parser.cs:line 19
   at ExpressionTreeIssue.Controllers.Extensions.GetElementName[T](T model, Expression`1 expression) in C:\dd\Projects\ExpressionTreeIssue\Controllers\Extensions.cs:line 10
   at AspNetCore._Views_Home_Index_cshtml.<ExecuteAsync>d__21.MoveNext() in /Views/Home/Index.cshtml:line 8
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.<RenderPageAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.<RenderAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.<ExecuteAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.ViewResult.<ExecuteResultAsync>d__26.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeResultAsync>d__30.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextResultFilterAsync>d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame`1.<RequestProcessingAsync>d__2.MoveNext()
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]

(I removed stack frames matching System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() and System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task).)

FYI seemingly-similar code using HTML helpers such as the following works as expected:
@foreach (var row in Model.Rows) {
    foreach (var cell in row.Cells) {

Row: @row.Id, Cell: @cell.Id

Input name: @Html.IdFor(m => cell.Value)

    }
}

or
@for (var row = 0; row < Model.Rows.Count; row++) {
    for (var cell = 0; cell < Model.Rows[row].Cells.Count; cell++) {

Row: @Model.Rows[row].Id, Cell: @Model.Rows[row].Cells[cell].Id

Input name: @Html.IdFor(m => m.Rows[row].Cells[cell].Value)

    }
}

Thank you for reference to good example.
I've fixed Parser error (e661137) using ExpressionHelper's way to parse indexers.

Closing this since all questions seem to have been answered. @wurdum please reopen or otherwise let us know if I'm missing something.


[Laravel API] Missing protected(?) methods from Laravel API.

Laravel Version: 5.3.22

Description:
Why are protected(?) methods missing from API documentation?
Example:
https://laravel.com/api/5.3/Illuminate/Foundation/Auth/AuthenticatesUsers.html
https://github.com/laravel/framework/blob/5.3/src/Illuminate/Foundation/Auth/AuthenticatesUsers.php#L114
authenticated() method is clearly empty for one reason only: to let user know that one can influence login flow (after credentials are correct). Why is it not listed in API Documentation?
For example I check if user is banned or verified - AFTER credentials are checked so no other person with knowledge of username() can get that information.
#16385 might fix this

I believe it's the default setting of the api documentation generator (Sami) to only generate docs for public methods.

Yes I figured out that fact, and I find it annoying (kind of?).
I use "offline" API Documentation quiet a lot, why is username() method visibility set as public but not one I am talking about authenticated().
Please continue on PR #16385.

Closing since no a bug in framework code.

@GrahamCampbell Could you tell us the correct place to report this? The https://github.com/laravel/docs repository seems to be for the manually written docs, and I am unable to find the repository that contains the code for generating the api docs.


[1/2/2022]: Installing Windows Build Tools with NVM-Windows.

Now that windows build tools is supposed to be installed via the Node JS installer, is there/could there be a way for us to do this via nvm-windows?
The installer on NPM has some issues, such as the latest version freezing powershell when installing and older versions having security vulnerabilities.

Please use the reactions to cast a vote in favor of or against this feature suggestion »

The MSI installer is responsible for any additional build tool installation, but the MSI installer does not allow multiple versions to be installed simultaneously. This is why version managers use the zip archives instead.
In order to do what's requested, NVM4W would have to recreate the build tools installation process. That's not a trivial effort, and it's not something I have time to do in this project. However; it is something I'll look into for rt.


Add a default fetcher using Fetch API

Not sure if you want this, but I found most of the time a simple fetcher like url => fetch(url).then(res => res.json()) is more than enough to use useSWR until you have reasons to build your own fetcher (like handling errors or parsing the response in a specific way).
I believe adding a default one could make using SWR a little bit faster since now you will only need to install and use.
@shuding if you prefer to don't provide a default one feel free to close this PR.

I agree on this one, but would love to hear more discussions!

Love it. Would request a note about best practices for error responses; that particular default leads to a lot of vague SyntaxErrors

I think when you want to handle error errors is when you will start to use a custom build fetcher, this fetcher should allow you to start using SWR without any config or boilerplate until you need a better fetcher

But yeah we may need to better document how to handle errors, specially because Fetch does not throw in all the HTTP errors

I really like this idea, since it makes it easier to hit the ground running with SWR.

This is great, I always need to make that simple fetcher on small demos. Also, I think this would help new people using SWR easily

So, I tried Paco's idea for SWR as a local cache but this PR fills the console with localhost:3000/whatever 404 errors. 😱

@brandonpittman you will need to define a custom fetcher then, maybe you can try passing undefined as your fetcher 🤔

@sergiodxa Yeah, passing null in the fetcher spot fixed it. Thanks!


Unable to render Tuple(Float64) if holding NaN or Inf

What happened:
When querying dataset containing column of Float64/32 type nested in a Tuple, the query fails in case the value is NaN/Inf with the error:
json: unsupported value: NaN: Could not process SQL results
json: unsupported value: +Inf: Could not process SQL results

How to reproduce it:
Reproducible query within the datasource:
SELECT tuple(NaN)
SELECT tuple(Inf)
SELECT tuple(-Inf)

It works as expected when not in a tuple:
SELECT NaN, Inf, -Inf

Thanks for submitting this issue. It's possible the tuple is being interpreted as JSON and failing to convert in the Go backend


not found variable - PS

Hello
I would like to convert veritcal coordinate from hybrid sigma level to pressure level. But in zarr coordinate variables, I didn't found the variable PS. Could you help me? Please
Thank you so much.

I can make sure that those are added! Thanks for letting us know.

Thank you so much.

BTW, I saw you upload the new variables include PS, thank you so much. But, could you upload daily output PS to AWS, thanks.

The daily PS data have been added as well! The catalog has been updated accordingly.

Thank you so much. BTW, could you upload V variable for daily output?

This variable is now included on AWS/the catalog

GOOD! Thanks.


Add meta descriptions

This commit adds meta description tags to all email alert pages, the contents of which are set to the summary associated with the relevant alert. This adds descriptions to external search engine results when the pages are re-indexed.
Trello: https://trello.com/c/bbAwFAQ1/108-add-meta-descriptions-to-pages-missing-them

do you happen to have a screenshot ?

I think this would cause the email alert pages to have the same description as their topic page. Could we change it somehow like we did for that other description?

Yeah, good point. I've changed the meta description for the email signup page using some of the text already on the page.


Flipping directly from league to competitions and back gives an unsightly blip in the submenus

This is a lot better than it was after recent changes but definitely not perfect.

Whatever I was worried about here doesn't seem like a big deal now. Closing.


Update scalafmt-core to 3.0.5

Updates org.scalameta:scalafmt-core from 2.7.5 to 3.0.5.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scalameta", artifactId = "scalafmt-core" } ]

labels: library-update, semver-major

Can be closed if #197 is merged.

Closing since solved by #197.


build(client): Upgrade build-tools deps to latest prerelease

Upgrades build-tools in client to the latest prerelease version.
The primary reason to pick up this prerelease is to get the updated changeset and release note tools, as well as fixes for flub release history.

Looks fine What is the logic for deciding prerelease build-tools versus just a regular release?

The main reason is that there's no need to create and push a release branch or bump to the next version first, so it's just faster. I wasn't going to bump client to this release originally - I was just going to test with it and use the releaseNotes tool for the 2.2 release. But it will be nice for the changesets to have the new metadata, and I had a little extra time to do this bump today.


Make Configuration.OutputDirectory public

Azure plugin needs to override InputLibrary, which needs Configuration.OutputDirectory as input.
The actual usage is in https://github.com/Azure/azure-sdk-for-net/pull/47374

API change check
APIView has identified API level changes in this PR and created following API reviews.
Microsoft.Generator.CSharp

Offline synced with @JoshLove-msft, we will not introduce the inheritance of InputLibrary until we need to add new functionality of InputLibrary in Azure plugin.


incorrect namespace in privacy class

https://github.com/antoni-oliver/graderanking/blob/main/classes/privacy/provider.php#L24
should be block_graderanking - not calendar block.

Fixed in 076251bc37a6276b82465c1135ecad898feab840.


Update releases.csv

Added 2.1.0 release according to the JSON-release file

Thanks for the submission. Looks like @preetikr updated the file already.


Changes on the Input Bar

Description
I would love to contribute by making changes on the  input bar which has few minor bugs
Please give me the issue
Screenshots

Additional information
No response

I have a talk about that just yesterday with @origranot .
Could you explain what are you going to do?
The main task is getting rid of the outboxed line

I will just make sure it won't overflow like, I'll make the UI/UX clean
The border kinda looks ugly as of now

Make sure to follow the DaisyUI themes guidelines please! Thanks😊

Thanks, I'll edit the code soon and make the PR by tomorrow

I'm closing this issue
Its daisy UI feature, I dont think I can disable the input box border

Just override the class that generates that border


DateTimeZone only accepts a subset of the IDs from TimeZone

so build fails periodically, https://travis-ci.org/dropwizard/dropwizard/jobs/57990585

Thanks. I ran this test so many times locally, but didn't catch any errors (seems to be environment-specific).

no problem, easy for that to happen.


--word_timestamps = True not working

Hi.
I think this might be user error (hopefully!), but when I run whisper-ctranslate2 inside Google Colab with the following command
!whisper-ctranslate2 "{full_path}" --vad_filter True --model large-v2 --verbose True --language fi --word_timestamps True --output_dir "{transcripts_dir}"
I get perfectly nice sentence-level output in JSON and every other format, but no word-level tagging. I've tried removing some of the parameters, but haven't gotten it to work yet. Removing vad_filter doesn't seem to do anything.
Is this a language-dependant thing (i.e. Finnish & large-v2) or is there something else I'm missing?
Both {full_path} and {transcripts_dir} contain meaningful values.

Hello.
It's not language or model dependant. It works for me.
Some ideas:

Make sure that you use the whisper-ctranslate2 version 0.4.1 or higher
Look in the JSON for  '"words": [' it should be there, even if is not added you should see a   "words": null
Try it outside  Google Colab


Code doesn't run

When i run the following :
python data_preprocess.py SMD

I get the following error
ImportError: DLL load failed: Le module spécifié est introuvable.
Failed to load the native TensorFlow runtime.

I see that other people have trouble running the code.
Does anyone know how to fix it or know a version of the code that works ?

No sorry. Keep the feed updated if you find a solution


Added Galaxy 101 workflow

Summary of changes

Added workflow (https://github.com/galaxyproject/training-material/issues/1459).
Added workflow test.
Added link to Zenodo object with input data.

Both planemo test and planemo workflow_lint pass.

Wonderful, thanks a lot @malloryfreeberg!


WeakMap is not defined

Even after the changes in #1094, as best as I can tell, the webpacking process introduces a dependency on WeakMap, which is not available in older browser.
I think there's a way of polyfilling WeakMap, but I'm not sure how to best proceed to do that.

What browsers are we going after here?

An older version of Chromium... I'll send you a MSG

Nominally, I think we're shooting for similar requirements to JLab: Firefox 49+, Chrome 39+, Safari 10+, but also IE 11 for widgetsnbextension.

Ah, ok, closing this then. Thanks for the details, @jasongrout


Fix compiler warnings in sample app

[Who] As a Developer
[What] I want third-party code to be free of warnings
[Value] So that I can be confident in its stability and so that my own warnings do not get lost in the noise

Closing for age, DCO URL changed since PR issued.


No such file or directory - tpu civis (Errno:ENOENT)

C:/Ruby23/lib/ruby/gems/2.3.0/gems/inquirer-0.2.1/lib/inquirer/utils/iohelper.rb:109:in ``': No such file or directory - tput civis (Errno::ENOENT)
from C:/Ruby23/lib/ruby/gems/2.3.0/gems/inquirer-0.2.1/lib/inquirer/utils/iohelper.rb:109:in without_cursor'         from C:/Ruby23/lib/ruby/gems/2.3.0/gems/inquirer-0.2.1/lib/inquirer/prompts/list.rb:95:in run'
from C:/Ruby23/lib/ruby/gems/2.3.0/gems/inquirer-0.2.1/lib/inquirer/prompts/list.rb:120:in ask'         from C:/Ruby23/lib/ruby/gems/2.3.0/gems/inquirer-0.2.1/lib/inquirer.rb:12:in list'

This happened when using Ask.list

After a bit of digging: I was on Windows, and tput is a Unix command. Any way to add Windows support?


Refactor first_day indices into generics

Pull Request Checklist:

[x] This PR addresses an already opened issue (for bug fixes / features)

This PR fixes #1175

[x] Tests for the changes have been added (for bug fixes / features)

[ ] (If applicable) Documentation has been added / updated (for bug fixes / features)

[ ] HISTORY.rst has been updated (with summary of main changes)

[ ] Link to issue (:issue:number) and pull request (:pull:number) has been added

[x] The relevant author information has been added to AUTHORS.rst and .zenodo.json

What kind of change does this PR introduce?

Uses DayOfYearStr from same file source
Refactors indices into generics that compose temperature-based indices and indicators unique to variables
Adds deprecation notices where needed

Does this PR introduce a breaking change?
Yes. first_day_threshold_reached is a new generic indice, and users must transition to using first_day_temperature_below and first_day_temperature_above for previous behaviour. For previous indices/indicators (first_day_above|below), they remain available, but will show DeprecationWarnings when called by user and will be removed in xclim>=0.40.0
Other information:
~The first_day_XX_above|below indices all perform local imports of generic first_day_above and first_day_below. This was needed to ensure that the deprecated versionsof these indices remain available without overwriting the newer generics. This will no longer be needed in xclim>=0.40.0~ - no longer relevant

Coverage increased (+0.01%) to 92.212% when pulling 94a65fd0cd5a85ff5438547bb9b4e3bb851ad234 on generic_first_days into 2386b2bfeb1bbda113e01d2fa4d362e3acfd7440 on master.

This is now ready for final review


-Clean Parameter does not work

Expected Behavior
All packages are removed from packagepath when the -clean parameter is used.
Current Behavior
No packages are removed.
Failure Information (for bugs)
When an older update is present in the packagepath e.g an earlier cumulative update in the month (due to there being twice a month sometimes) it will error with the following
VERBOSE: Destination is DS004:\Packages\Windows_10_1803_x64
VERBOSE: Performing the operation "Remove package" on target "Package_for_KB4485449 neutral amd64 17134.580.1.0".
Get-ChildItem : This row has been removed from a table and does not have any data.  BeginEdit() will allow creation of new data in this row.
At C:\Program Files\WindowsPowerShell\Modules\LatestUpdate\<IP_ADDRESS>\Public\Import-LatestUpdate.ps1:100 char:13
+             Get-ChildItem | Where-Object { $_.Name -like "Package*" } ...
+             ~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Get-ChildItem], RowNotInTableException
    + FullyQualifiedErrorId : ExceptionCaught,Microsoft.PowerShell.Commands.GetChildItemCommand

Steps to Reproduce
Please provide detailed steps for reproducing the issue.

Ensure that an older update is present (Imported) in the package path. This must be one that is not currently offered by Get-LatestUpdate.
Run 'Import-LatestUpdate -UpdatePath "C:\Temp\Updates" -DeployRoot "\SERVER\DeploymentShare$" -PackagePath "Windows_10_1803_x64" -Clean -Verbose'
It will then error upon trying to clean up the first package found in the package path. No further cleaning is performed. Meaning over time packages will build up in MDT.

Context
Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions.

LatestUpdate Version: <IP_ADDRESS>
PowerShell Version: 5.1 Build 14409 Revision 1018
Operating System: Windows Server 2012 R2

Failure Logs
Please include any relevant log snippets or files here.

Fixed in commit 3ae65830d55273082484328592affc4860a126e7

Fixed in release v<IP_ADDRESS>.


Minor improvements for Node.js init template

Description of changes:
I've only renamed the handler (because camel case!).
ping @sanathkr
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Assuming camel case is the preferred style for Node?

Yes, exactly 💯


Feature request:

Sometimes the Collection model makes it unnecessary complicated to get linked items.
E.g. a constraint limits that an User can have only 10 linked active devices. In this case, it would be simpler to retrieve all devices with devices instead of devicesCollection.

I can see how that might be clunky in constrained cases but we think its important for generated APIs to be consistent across the graph and as flexible as possible to reduce the probability that the output graphql schema has to change in a backward incompatible way. For that reason, this isn't a feature that we're currently interesting in adding to the project


Update component reference for v1.29

Taking informal LGTM from https://github.com/kubernetes/website/pull/44346#pullrequestreview-1781017587 as grounds to:
/lgtm
/approve


[Pull Request] Implement local SSL proxy for running the app with a .test domain instead of ngrok

This pull request introduces the implementation of a local SSL proxy to facilitate running the application using a .test domain. Currently, the app relies on Ngrok for tunneling and exposing the local server to the internet. However, this approach has limitations and can be cumbersome for testing and development purposes.
The local SSL proxy provides a more streamlined solution by allowing the app to be accessed locally through a .test domain. This simplifies the setup process and ensures a secure and reliable connection for local testing.
Key Changes:
Added the local-ssl-proxy library to the project dependencies.
Added an npm script to generate the certificate and configure the proxy.
Updated the app's configuration to utilize the local SSL proxy for serving requests.
Documented the necessary steps for setting up and using the local SSL proxy in the README file.
Benefits:
Improved development workflow: With the local SSL proxy, developers can quickly and easily access the app using the .test domain, eliminating the need for external services like Ngrok.
Enhanced security: The local SSL proxy ensures secure connections between the app and the local machine, protecting sensitive data during testing and development.
Simplified setup: By utilizing the .test domain, the setup process becomes more straightforward and reduces the dependencies on third-party services.

Hey! While this would be a great addition, I do not accept PRs to the repo. Please feel free to fork the repo and build it there!

Also, while understandable why you potentially see running a local proxy server instead of using ngrok, it's easier to debug, easily replaced with other tunneling services and more importantly, more approachable. Explaining everyone how to find your hosts file and sudo it for permissions is going to drive beginners / mid level devs insane, is not easily portable to other machines (should they decide to change it) and requires device specific setup.
The repo should be npm i, npm run ngrok, npm run dev, which it delivers on. Anything that is added in, like npm run update:url is to improve dev experience and should work across all devices. The moment there's more complication to this process it's going to worsen the developer experience.
What I'd recommend is to fork the repo and keep it synced to this repo so it's up to date while maintaining your preferred dx.

Hello @kinngh, I hope this message finds you well. Firstly, I am grateful for reviewing my pull request. Your feedback is greatly appreciated, and I value your insights on improving the developer experience.
I understand your concern about the approach I took with running a local proxy server instead of using ngrok. However, I fully agree that ngrok provides a convenient and beginner-friendly solution for tunneling services. I initially implemented it to cater to the needs of advanced users who prefer a more robust experience.
However, my intention was also to provide an additional feature that could benefit advanced users without affecting the experience of beginners. By introducing the option to use a local proxy server, developers who are more experienced and comfortable with custom setups could enjoy a super fast loading speed and enhanced debugging capabilities.
It is important to note that ngrok has some limitations, especially in custom subdomains. Custom subdomains are only available for premium users, meaning non-premium users would have to change the application URLs each time they use ngrok. I guess this is why you build npm run update:url; this can be inconvenient and time-consuming, particularly for beginners who are just starting their development journey.
To strike a balance, it would be beneficial to provide a clear guide in the documentation that outlines both the ngrok option for beginners and the local proxy server option for advanced users. This way, developers can choose the method that aligns with their skill level and requirements. We can also use localhost to eliminate the hosts file step.
Once again, thank you for your valuable feedback. I am open to any other suggestions or modifications you may have about this pull request. Again, your guidance and expertise are precious.

Thanks, @kinngh. I completely understand your viewpoint on simplicity.