Inconsistent and confusing regex anchoring behavior

Regex anchoring got introduced by @juliusv and @brian-brazil in #996 in config files as well as for the label_replace() function in promql.
It was not changed anywhere else in promql, so label regex matchers are still not anchored. See this example: node_boot_time{instance=~"robustperception"}.
As I'm not aware of a single linux tool, programming langue or other regex interface which does anchoring (please give examples otherwise), I'm questioning the benefit of anchoring. It's a non-standard behavior, requires additional knowledge and therefore, makes it harder for people to get started with Prometheus. I'm able to write a regex in almost any tool immediately without the need to look up the documentation, but with prometheus it took me several tries to understand why my label_replace function did not work as expected.
I heard the argument that anchored regexs are shorter, they're not in all cases (^4 vs. 4.*). In general I don't think this should be an argument over consistency with the majority of other regex implementations.
I also heard that anchoring makes it harder for people to write wrong regular expressions. I checked tens of expressions using regular matchers at soundcloud and couldn't find a single one using ambiguous regexps. I'd argue that the majority of developers knows how to use regexps and disagree that it's Prometheus' role to prevent the minority of people from doing something stupid.
I see 4 options to solve this issue:

Leave the current state (anchoring behavior: config, label_replace; standard behavior: rest of promql)
Revert the label_replace change (anchoring: config; standard: promql)
Revert anchoring everywhere.
Update all places without exception to use anchoring.

While I personally don't see the benefit of anchoring, my realistic vote is 2. It does not require another disruptive change to configs and provides a clear rule for user: anchoring in config, standard behavior in promql.
@fabxc @beorn7

Option 2. ist roughly what my discussion about this with @beorn7 would have lead me. I don't have an issue with doing anchoring myself and find the wrapping odd. But people form different backgrounds may feel the opposite.
In configs there may be the risk of people not anchoring properly out of laziness, which could lead to conflicts later on, though unlikely. As queries are more volatile, I don't feel the same reasoning applies there – certainly not partially for the relabel functions only.

FWIW, as part of https://github.com/prometheus/prometheus/issues/996 I also meant to make regex label matchers anchored back then (it's even stated as a goal in the issue), but apparently forgot that part. Up until today, I thought that I had made them anchored too. The intent was to make everything consistent, although that part was forgotten.
As for usages within SC, I've seen some wrong ones in the past (assuming anchored matches when they were not), but they are probably fixed due to user education by now.
As for 2), wouldn't anchoring some regexes and not others (config) be even more confusing to users than having anchoring everywhere?
What I like about anchoring is that if I want to match multiple full label values (common use case), I just have to write foo|bar|baz instead of ^(foo|bar|baz)$. I agree that unanchored usage is what's common in Unix tools though.
Anyways, I trust y'all to do the right thing. Just remember, if you do decide for 2), it'll be a second breaking change to label_replace(), reverting to the old behavior after breaking it once to be anchored. A bit of a weird back and forth.
Personally I prefer 4), but that might just be what I like myself.
/cc @brian-brazil

As I'm not aware of a single linux tool, programming langue or other regex interface which does anchoring (please give examples otherwise),

There's many that do beginning anchors, the problem is they aren't usually at all documented in this regard. The most obvious would be match vs search in python.

I'm able to write a regex in almost any tool immediately without the need to look up the documentation

I always end up having to explicitly anchor, as otherwise I'd end up having to read the code to determine if this particular interface anchors.
Due to the wide inconsistencies in this area I'm for 4) as that's probably what the user wants and had thought selectors already worked this way.

I'd argue that the majority of developers knows how to use regexps and disagree that it's Prometheus' role to guard the minority of people from doing something stupid.

This is different from my experience. Sysadmins tend to know it reasonably well, and developer knowledge tends not to be too great.

Due to the wide inconsistencies in this area

Please give examples for this claim. You presented one function in Python where a similar function is also available with partial match behavior. To my knowledge at least, the standard libraries in Java, Go or Ruby, the cmdline tools grep, sed, awk, test or the databases influxdb, graphite, opentsdb do all partial matching. Just to name a few in the related fields.

I'm for 4) as that's probably what the user wants and had thought selectors already worked this way.

Maybe time for a survey. I checked a few of the expressions written at SoundCloud and found examples for half anchored expressions (code=~"^5") as well as explicit partial matches (job=~"cassandra" with values "--"). So I'm at least not alone to expect partial matching behavior.
As I can not see your described "wide inconsistencies", my vote is to not get into a guessing game what a user might want here, but to follow the most common implementation (which seems to be partial matching by far) unless we have evidence that  the absolute majority of Prometheus users indeed prefers anchored matching.

Please give examples for this claim. You presented one function in Python where a similar function is also available with partial match behavior.

For clarity please don't use the term "partial match", that has a completely different meaning when we're talking about text search. All the regexes we're talking about here are fully matched. The correct term is anchoring.
In my experience it's a coin toss as whether a given Python program or interface will use unanchored or beginning anchored regexes, so I gave up tracking which was which many years ago.
Go does not document if it is anchored (reading between the lines it's unanchored). Java is documented as anchored. Ruby is documented as unanchored. Given that OpenTSDB is written in Java and is backed by Cassandra which is also in Java I'd presume that it's also anchored (the docs give no indication, there's a hint that it's fully anchored).
There is no standard here. It boils down to which language(s) you're used to using.

Maybe time for a survey.

We've had several users get confused by the previously unanchored behaviour of Prometheus configs, particularly with relabelling. Looking at the end result of a code base doesn't tell you about what user expectations are, it just tells you what the user ended up doing after an unknown amount of trial and error.
I agree with @juliusv that most users are looking for anchored matches. In the case of properly used labels you're going to almost always want fully anchored matches, and if you think you don't then you probably haven't thought things all the way through.
Taking your second example, "{namespace}-{db}-{suffix}" is not a proper use of labels as that should be three separate labels. Due to this suboptimal label choice, you've also got an incorrect regex of job=~"cassandra". This needs to be anchored to be safe, and you also need to allow for collisions with other naming schemes that happen to contain 'cassandra' in their name. The correct regex is going to end up something like ^[^-]+-cassandra-.*$.
If that was three separate labels then job=~"cassandra" would still be wrong, as you'd pick up other jobs that happened to contain that string which I'm pretty sure isn't intended (if it was beginning anchored that'd be different as then there's fairly obvious namespacing). A more realistic example would be job=~"cassandra|someotherjob" which I'd expect to be equivalent to job!="cassandra",job!="someotherjob"
Any completely unanchored use of regexes with labels is likely incorrect. You need anchors to be safe.

First of all: +1 for consistency. (Good that we clarified that the current inconsistency was an oversight and not a deliberate decision.)
WRT "standard" behavior: grep, sed, awk all work line-based, where non-anchored matching makes sense (because you want to find part of a line). Matching label values is a different thing because you are usually interested in the whole label. As a Unix command line counter-example look at expr, which does anchoring at the beginning.
Library review:
Python
Has search, fullmatch, and match (which is non-anchored, fully anchored, anchored at the beginning, respectively). The most "naturally" named function is thus implementing a behavior that is not present at all in Prometheus.
C++
The popular RE2 library has FullMatch and PartialMatch (for fully anchored and non-anchored, respectively), so no "standard" at all.
Boost has regex_match for fully anchored and regex_search for non-anchored.
Java
java.util.regex.Matcher has matches, lookingAt, find for fully anchored, non-anchored, anchored at the beginning, respectively.
Conclusion
It doesn't make sense to talk about "standard" behavior. Libraries either do not acknowledge a standard or call anchored matching just "match" and non-anchored matching something like "find" or "search", implying anchored matching is the more "natural" matching. Command line tools usually do non-anchored matching.
As said initially, I first and foremost want consistency. I don't feel too strongly about the choice, but my preference in the context of Prometheus would be full anchoring.

Going with a completely unified approach is easiest in the long run.
Most things that are not lists do indeed indicate that one is dealing with a label that should have been split into multiple ones.
I do think this will be breaking rules for a few people – as one is usually aware of the possible label values, skipping any leading and trailing .*-variants was just very convenient so far.

Alright. Consistency inside of Prometheus is probably the most important thing, so the solution taken makes sense.
p.s. @brian-brazil The example was for a job label which has to be unique. As we currently don't support label definition for specific sd_config groups, each sd address is its own job with non-optimal naming.


apply cookiecutter

use cookiecutter to update the doc
Also fixes #29

Auto-footnotes do not like to be referenced in the first line.
It also does not like other strings with square brackets, e.g. [-5, -10] dB.

merges into #26

c30621e03fbd57ebdaa6648a75a2813f08b54c83

done


API specification doesn't seem correct

Greetings, I'm attempting to use the data from HealthSites for a COVID-19 related project.  I'm finding the API specification here doesn't seem to be correct.  For instance, try the "Interact" button for the GET /api/v2/countries/autocomplete method there.  It results in an HTTP 400 Insufficient characters error from the server.  I observe the same thing when attempting to interact from Python.
from coreapi import Client
client = Client()
schema = client.get("https://healthsites.io/api/docs/")
result = client.action(schema, ["api", "v2", "facilities", "autocomplete", "list"])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/coreapi/client.py", line 178, in action
    return transport.transition(link, self.decoders, params=params, link_ancestors=link_ancestors)
  File "/usr/local/lib/python3.7/site-packages/coreapi/transports/http.py", line 386, in transition
    raise exceptions.ErrorMessage(result)
coreapi.exceptions.ErrorMessage: <Error: 400 BAD REQUEST>
    message: "Insufficient characters."

There doesn't seem to be any way to specify the search term here.  I tried a variety of options (by including params={'q': 'Hospital'} as the last argument to client.action, but couldn't find a parameter name that was accepted.
When checking the requests being sent from the Map site, I can see that the correct URI appears to be: https://healthsites.io/api/v2/countries/autocomplete/e?q=SearchTerm
And, in fact, that works fine in Python as well (using the built-in requests library).
import requests
res = requests.get('https://healthsites.io/api/v2/countries/autocomplete/e?q=Unit')
res.json()
[{'label': 'United Arab Emirates', 'id': 348}, {'label': 'United Kingdom', 'id': 349}, {'label': 'United States', 'id': 350}]

Can anyone explain the discrepancy I'm seeing here?  Thanks in advance.

@jeff303 Sorry for the delay in getting back to you. I am going to ask our resident guru (@meomancer) to take a quick look.

the fix is here : https://github.com/healthsites/healthsites/pull/1429

When will this change be reflected in the production API?  I am still seeing this behavior:
>>> result = client.action(schema, ["api", "v2", "facilities", "autocomplete", "list"], params={'q': 'Hospital'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jeff/.local/share/virtualenvs/hospitalaid-Yme3N8YY/lib/python3.7/site-packages/coreapi/client.py", line 165, in action
    _validate_parameters(link, params)
  File "/Users/jeff/.local/share/virtualenvs/hospitalaid-Yme3N8YY/lib/python3.7/site-packages/coreapi/client.py", line 80, in _validate_parameters
    raise exceptions.ParameterError(errors)
coreapi.exceptions.ParameterError: {'q': 'Unknown parameter.'}

I assume there is some delay associated with this making its way from development into production.  Thanks for your help so far!


Inject (cross-)compiling flags

This is an implementation of the cross-compiling design at https://github.com/aantron/luv/issues/129 . It ended up being more complicated because I forgot that the include directories (-I/usr/include) in addition to the libraries (-L/usr/lib -luv) needed modifications.
Testing

Pending testing on Windows; only macOS has been tested

Native compiling on macOS
brew install libuv # 1.43.0
dune clean
unset LUV_USE_SYSTEM_LIBUV LUV_INCLUDE_LIBUV_DEFAULT LUV_LIB_LIBUV_DEFAULT LUV_INCLUDE_LIBUV_DEFAULT_DARWIN_X86_64 LUV_LIB_LIBUV_DEFAULT_DARWIN_X86_64
export LUV_USE_SYSTEM_LIBUV=yes
export LUV_INCLUDE_LIBUV_DEFAULT='-I;/opt/homebrew/include'
export LUV_LIB_LIBUV_DEFAULT='-L/opt/homebrew/lib;-luv'

LUV_USE_SYSTEM_LIBUV=yes LUV_INCLUDE_LIBUV_DEFAULT='-I;/opt/homebrew/include' LUV_LIB_LIBUV_DEFAULT='-L/opt/homebrew/lib;-luv' make test

Failing tests after PR are:
  [FAIL]        version              1   minor.
  [FAIL]        version              5   hex.
  [FAIL]        version              6   version.
  [FAIL]        version              7   string.

  [FAIL]        tty                  0   tty.
  [FAIL]        tty                  1   handle functions.

  # This stalls on macOS (Apple Silicon) so disabled
  ...           udp                  7   multicast.

  # This does SIGABRT so disabled
                thread                   mutex.

Failing tests without LUV_USE_SYSTEM_LIBUV=yes:
  [FAIL]        version              1   minor.
  [FAIL]        version              5   hex.
  [FAIL]        version              6   version.
  [FAIL]        version              7   string.

  [FAIL]        tty                  0   tty.
  [FAIL]        tty                  1   handle functions.

  # These stall on macOS so disabled
  ...           udp                  7   multicast.
  ...           fs_event             2   create.

Cross-compiling on macos

Host=darwin_arm64 Target=darwin_x86_64

Using Duniverse Opam repository at https://github.com/dune-universe/opam-overlays
with one extra patch for ctypes.0.17.1+dune at https://github.com/avsm/ocaml-ctypes/pull/4

dune clean
unset LUV_USE_SYSTEM_LIBUV LUV_INCLUDE_LIBUV_DEFAULT LUV_LIB_LIBUV_DEFAULT LUV_INCLUDE_LIBUV_DEFAULT_DARWIN_X86_64 LUV_LIB_LIBUV_DEFAULT_DARWIN_X86_64
export LUV_USE_SYSTEM_LIBUV=yes
export LUV_INCLUDE_LIBUV_DEFAULT='-I;/opt/homebrew/include'
export LUV_LIB_LIBUV_DEFAULT='-L/opt/homebrew/lib;-luv'
export LUV_INCLUDE_LIBUV_DEFAULT_DARWIN_X86_64="-I;/tmp/dckbuild/darwin_x86_64/vcpkg_installed/x64-osx/include"
export LUV_LIB_LIBUV_DEFAULT_DARWIN_X86_64="-L/tmp/dckbuild/darwin_x86_64/vcpkg_installed/x64-osx/lib;-luv"
dune build @install
dune build -x darwin_x86_64 @install

$ tail -n20 _build/*/src/c/config/compile_c.sh _build/*/src/c/config/uv_lib.sexp
==> _build/default.darwin_x86_64/src/c/config/compile_c.sh <==
#!/bin/sh
# context=default.darwin_x86_64
set -euf
src=$1; shift
ctypes=$1; shift
target=$1; shift
"$@" "$src" -I "$ctypes" -I "/tmp/dckbuild/darwin_x86_64/Debug/dksdk/ocaml/opt/mlcross/darwin_x86_64/lib/ocaml" "-I" "/tmp/dckbuild/darwin_x86_64/vcpkg_installed/x64-osx/include" -o "$target"

==> _build/default/src/c/config/compile_c.sh <==
#!/bin/sh
# context=default
set -euf
src=$1; shift
ctypes=$1; shift
target=$1; shift
"$@" "$src" -I "$ctypes" -I "/tmp/dckbuild/darwin_x86_64/Debug/dksdk/ocaml/lib/ocaml" "-I" "/opt/homebrew/include" -o "$target"

==> _build/default.darwin_x86_64/src/c/config/uv_lib.sexp <==
("-L/tmp/dckbuild/darwin_x86_64/vcpkg_installed/x64-osx/lib" "-luv")
==> _build/default/src/c/config/uv_lib.sexp <==
("-L/opt/homebrew/lib" "-luv")

$ file _build/install/*/lib/stublibs/dllluv_unix_stubs.so
_build/install/default.darwin_x86_64/lib/stublibs/dllluv_unix_stubs.so: Mach-O 64-bit dynamically linked shared library x86_64
_build/install/default/lib/stublibs/dllluv_unix_stubs.so:               Mach-O 64-bit dynamically linked shared library arm64

Windows has been tested as well.


[FEA] MultiGPU TensorFlow Dataloader

Similar to https://github.com/NVIDIA/NVTabular/issues/57 ,  we should add the ability to the TF dataloader to train on multiple GPU's

I'll need to do more research into this, but my intuition is that this is probably nontrivial and might not be realistic to make it into 0.2

Moved to 0.3 - though it'd be great to verify if we can get this going with a similar mechanism to what we're doing with PT before then.

@benfred Does this count as complete now that we have horovod?


/usr/local/bin not writeable by non-root (default)

Homebrew sets /usr/local/bin to be writable by the user, however this is not the default permission; in that case, installing will fail. (Homebrew changes the permissions of /usr/local/bin—seems bad but ok)
Instead, write binary to a tmp file, and move it into place after escalating privs.
r? @thieman

Seems legit. FWIW I think Homebrew stopped requiring weird permissions on /usr/local/bin a few versions ago.
CI seems sad, will have to see what's up with it

FWIW I think Homebrew stopped requiring weird permissions on /usr/local/bin a few versions ago.

Oh that's great news, thanks for the heads up 😃. And thanks for the review.

The install script isn't directly tested by CI so whatever's up with Jenkins shouldn't block this. Mergin'.


optimise dict_from_bytes

The current implementation of dict_from_bytes loops over all the items in an array, both of each array in the structure for i in range(size) and of structure if an array of structures for structure in range(0, array_size):

When reading large arrays this is slow, can this be optimised better use of struct.unpack?
Instead of unpacking one item of the array at a time, can this function be refactored to struct.unpack the whole array at once.

remove the for i in range(size) and change

from:
n_bytes = sizeof(plc_datatype)
struct.unpack(
    DATATYPE_MAP[plc_datatype],
    bytearray(byte_list[index: (index + n_bytes)]),
)[0]

to:
n_bytes = sizeof(plc_datatype) * size
datatype = DATATYPE_MAP[plc_datatype]
datatype = datatype[0] + str(size) + datatype
struct.unpack(
    datatype,
    bytearray(byte_list[index: (index + n_bytes)]),
)[0]

I'm also unsure if it is the unpack which is slow too so we could build up an unpack string, then unpack in one go, this would also benefit if for structure in range(0, array_size)?
I've been meaning to tackle this for a while and often when reading large amounts of data I use multiple read_by_names instead as this is actually faster, but you then have to workaround the fact that the data is being read over multiple ADS cycles.
I raise this now as could be thought about when refactoring for #288.
Happy to look at other alternatives too to decrease the time taken to go from bytes to the dictionary, from my testing of large amounts of data, the acatual ADS read of all the bytes is super quick but can take seconds to unpack it into the right format

from my testing of large amounts of data, the acatual ADS read of all the bytes is super quick but can take seconds to unpack it into the right format

That is interesting. I agree, looking at the code there could be some room for improvement in regards of speed. @chrisbeardy you're welcome to tackle on some improvements here 👍
Also we now have WSTRING support in pyads and I just see that it is not supported for structures, yet. I'll open an issue for that.

TLDR
On a read_structure_by_name with larger amounts of data being read:

Approx 45% of time is spent doing the bytes_to_dict conversion
Approx 45% of time is doing the [i for i in read_data] on line 274 of pyads_ex.py

For speedup, I suggest we target these two areas. Improving [i for i in read_data] on line 274 of pyads_ex.py will also have more benefits across the whole library.
Tests
I've just conducted some quick tests. Based on a real world example from a machine I have done where we have a 60 second rolling log for data capture.
I conducted these tests on both python 3.6 pyads 3.3.4 and python 3.8 and pyads 3.3.9, both show the same results. I have a relatively quick Intel(R) Xeon(R) CPU E3-1535M v6 @ 3.10GHz, 3101 Mhz, 4 Core(s), 8 Logical Processor(s) and 32gb RAM machine.
It is easy to take a quick look the [i for i in read_data] on line 274 of pyads_ex.py. It seems Using list(read_data) would show approx up to 20% improvement, for very small amounts of data this actually makes it slighty less performant but then the overall time difference is then negligible.
Interestingly using numpy frombuffer gives a lovely 80% increase in performance, again for very small amounts of data this actually makes it slighty less performant likely due to the overhead of numpy using c.
I'm not suggesting we use numpy here as that would seem overkill to add that dependency for just one line. But I will do a PR for using list(read_data) instead.
Given that it is already possible to return the ctypes I think trying to optimise this to list case futher using C would likely just make pyads more complex than it needs to be but for use cases of large amounts of data we could add to the documentation a suggestion to use numpy:
    data_ctype_raw = plc.read_by_name("", c_ubyte * size_of_data, handle=datalog_handle, return_ctypes=True)
    data = pyads.dict_from_bytes(np.frombuffer(data_ctype_raw, dtype=np.ubyte).tolist(), DATALOG_STRUCTURE)

Now to tackle to the bytes_to_dict conversion ...
Code
Python Code (if you set a breakpoint on the return statement, you can use a debugger to inspect the variables):
from ctypes import c_ubyte

import pyads
import time

import numpy as np

def main():
    size_of_data = pyads.size_of_structure(DATALOG_STRUCTURE)
    size_of_big_data = pyads.size_of_structure(DATALOG_STRUCTURE_BIG)
    plc = pyads.Connection(ADS_ADDRESS, ADS_PORT)
    plc.open()
    datalog_handle = plc.get_handle("MAIN.data")
    datalog_big_handle = plc.get_handle("MAIN.dataBig")

    data_read_start = time.perf_counter()
    data = plc.read_structure_by_name("", DATALOG_STRUCTURE, structure_size=size_of_data, handle=datalog_handle)
    data_read_time = time.perf_counter() - data_read_start
    plc.write_by_name("MAIN.lastRead", data["currentPointer"], pyads.PLCTYPE_UDINT)

    data_read_start = time.perf_counter()
    data2 = plc.read_structure_by_name("", DATALOG_STRUCTURE, structure_size=size_of_data, handle=datalog_handle)
    data_read_2_time = time.perf_counter() - data_read_start

    data_read_start = time.perf_counter()
    data_big = plc.read_structure_by_name("", DATALOG_STRUCTURE_BIG, structure_size=size_of_big_data, handle=datalog_big_handle)
    data_read_big_time = time.perf_counter() - data_read_start

    data_read_start = time.perf_counter()
    data_big_2 = plc.read_structure_by_name("", DATALOG_STRUCTURE_BIG, structure_size=size_of_big_data, handle=datalog_big_handle)
    data_read_big_time_2 = time.perf_counter() - data_read_start

    # test with just a read by name and return ctype
    data_read_start = time.perf_counter()
    data_big_no_to_dict_raw = plc.read_by_name("", c_ubyte * size_of_big_data, handle=datalog_big_handle)
    data_read_big_no_to_dict_time = time.perf_counter() - data_read_start
    data_read_start = time.perf_counter()
    data_big_no_to_dict = pyads.dict_from_bytes(data_big_no_to_dict_raw, DATALOG_STRUCTURE_BIG)
    data_read_big_to_dict_time = time.perf_counter() - data_read_start

    data_read_start = time.perf_counter()
    data_big_ctype_raw = plc.read_by_name("", c_ubyte * size_of_big_data, handle=datalog_big_handle, return_ctypes=True)
    data_read_big_ctype_time = time.perf_counter() - data_read_start
    data_read_start = time.perf_counter()
    data_big_ctype_to_array = [i for i in data_big_ctype_raw]
    data_read_big_to_list_time = time.perf_counter() - data_read_start
    data_read_start = time.perf_counter()
    data_big_ctype_to_array = list(data_big_ctype_raw)
    data_read_big_to_list_time_2 = time.perf_counter() - data_read_start
    data_read_start = time.perf_counter()
    data_big_ctype_to_array = np.frombuffer(data_big_ctype_raw, dtype=np.ubyte).tolist()
    data_read_big_to_list_time_3 = time.perf_counter() - data_read_start
    data_read_start = time.perf_counter()
    data_big_ctype_dict = pyads.dict_from_bytes(data_big_ctype_to_array, DATALOG_STRUCTURE_BIG)
    data_read_big_from_list_to_dict_time = time.perf_counter() - data_read_start

    data_read_start = time.perf_counter()
    data_ctype_raw = plc.read_by_name("", c_ubyte * size_of_data, handle=datalog_handle, return_ctypes=True)
    data_read_ctype_time = time.perf_counter() - data_read_start
    data_read_start = time.perf_counter()
    data_ctype_to_array = [i for i in data_ctype_raw]
    data_read_to_list_time = time.perf_counter() - data_read_start
    data_read_start = time.perf_counter()
    data_ctype_to_array = list(data_ctype_raw)
    data_read_to_list_time_2 = time.perf_counter() - data_read_start
    data_read_start = time.perf_counter()
    data_ctype_to_array = np.frombuffer(data_ctype_raw, dtype=np.ubyte).tolist()
    data_read_to_list_time_3 = time.perf_counter() - data_read_start
    data_read_start = time.perf_counter()
    data_ctype_dict = pyads.dict_from_bytes(data_ctype_to_array, DATALOG_STRUCTURE)
    data_read_from_list_to_dict_time = time.perf_counter() - data_read_start

    plc.release_handle(datalog_handle)
    plc.release_handle(datalog_big_handle)
    plc.close()

    return

ADS_ADDRESS = "<IP_ADDRESS>.1.1"
ADS_PORT = 851
DATALOG_LENGTH = 15001
DATALOG_STRUCTURE: pyads.StructureDef = (
    ("currentPointer", pyads.PLCTYPE_UDINT, 1),
    ("data1", pyads.PLCTYPE_LREAL, DATALOG_LENGTH),
    ("data2", pyads.PLCTYPE_LREAL, DATALOG_LENGTH),
    ("data3", pyads.PLCTYPE_LREAL, DATALOG_LENGTH),
    ("data4", pyads.PLCTYPE_LREAL, DATALOG_LENGTH),
    ("data5", pyads.PLCTYPE_LREAL, DATALOG_LENGTH),
    ("data6", pyads.PLCTYPE_LREAL, DATALOG_LENGTH),
    ("data7", pyads.PLCTYPE_LREAL, DATALOG_LENGTH),
    ("data8", pyads.PLCTYPE_LREAL, DATALOG_LENGTH),
    ("data9", pyads.PLCTYPE_LREAL, DATALOG_LENGTH),
    ("data10", pyads.PLCTYPE_LREAL, DATALOG_LENGTH),
    ("data11", pyads.PLCTYPE_LREAL, DATALOG_LENGTH),
    ("data12", pyads.PLCTYPE_LREAL, DATALOG_LENGTH),
    ("cycleCount", pyads.PLCTYPE_UDINT, DATALOG_LENGTH),
    ("dcTaskTime", pyads.PLCTYPE_STRING, DATALOG_LENGTH),
    ("lastExecTime", pyads.PLCTYPE_UDINT, DATALOG_LENGTH),
    ("cycleTimeExceeded", pyads.PLCTYPE_BOOL, DATALOG_LENGTH),
    ("RTViolation", pyads.PLCTYPE_BOOL, DATALOG_LENGTH),
)
DATALOG_LENGTH_BIG = 60001
DATALOG_STRUCTURE_BIG: pyads.StructureDef = (
    ("currentPointer", pyads.PLCTYPE_UDINT, 1),
    ("data1", pyads.PLCTYPE_LREAL, DATALOG_LENGTH_BIG),
    ("data2", pyads.PLCTYPE_LREAL, DATALOG_LENGTH_BIG),
    ("data3", pyads.PLCTYPE_LREAL, DATALOG_LENGTH_BIG),
    ("data4", pyads.PLCTYPE_LREAL, DATALOG_LENGTH_BIG),
    ("data5", pyads.PLCTYPE_LREAL, DATALOG_LENGTH_BIG),
    ("data6", pyads.PLCTYPE_LREAL, DATALOG_LENGTH_BIG),
    ("data7", pyads.PLCTYPE_LREAL, DATALOG_LENGTH_BIG),
    ("data8", pyads.PLCTYPE_LREAL, DATALOG_LENGTH_BIG),
    ("data9", pyads.PLCTYPE_LREAL, DATALOG_LENGTH_BIG),
    ("data10", pyads.PLCTYPE_LREAL, DATALOG_LENGTH_BIG),
    ("data11", pyads.PLCTYPE_LREAL, DATALOG_LENGTH_BIG),
    ("data12", pyads.PLCTYPE_LREAL, DATALOG_LENGTH_BIG),
    ("cycleCount", pyads.PLCTYPE_UDINT, DATALOG_LENGTH_BIG),
    ("dcTaskTime", pyads.PLCTYPE_STRING, DATALOG_LENGTH_BIG),
    ("lastExecTime", pyads.PLCTYPE_UDINT, DATALOG_LENGTH_BIG),
    ("cycleTimeExceeded", pyads.PLCTYPE_BOOL, DATALOG_LENGTH_BIG),
    ("RTViolation", pyads.PLCTYPE_BOOL, DATALOG_LENGTH_BIG),
)

if __name__ == "__main__":
    main()

PLC Code:
{attribute 'pack_mode' := '1'}
TYPE stData :
STRUCT
    currentPointer : UDINT;
    data1 : ARRAY [0..GVL.dataLogLength] OF LREAL;
    data2 : ARRAY [0..GVL.dataLogLength] OF LREAL;
    data3 : ARRAY [0..GVL.dataLogLength] OF LREAL;
    data4 : ARRAY [0..GVL.dataLogLength] OF LREAL;
    data5 : ARRAY [0..GVL.dataLogLength] OF LREAL;
    data6 : ARRAY [0..GVL.dataLogLength] OF LREAL;
    data7 : ARRAY [0..GVL.dataLogLength] OF LREAL;
    data8 : ARRAY [0..GVL.dataLogLength] OF LREAL;
    data9 : ARRAY [0..GVL.dataLogLength] OF LREAL;
    data10 : ARRAY [0..GVL.dataLogLength] OF LREAL;
    data11 : ARRAY [0..GVL.dataLogLength] OF LREAL;
    data12 : ARRAY [0..GVL.dataLogLength] OF LREAL;
    cycleCount : ARRAY [0..GVL.dataLogLength] OF UDINT;
    dcTaskTime : ARRAY [0..GVL.dataLogLength] OF STRING;
    lastExecTime : ARRAY [0..GVL.dataLogLength] OF UDINT;
    cycleTimeExceeded : ARRAY [0..GVL.dataLogLength] OF BOOL;
    RTViolation : ARRAY [0..GVL.dataLogLength] OF BOOL;
END_STRUCT
END_TYPE

{attribute 'pack_mode' := '1'}
TYPE stDataBig :
STRUCT
    currentPointer : UDINT;
    data1 : ARRAY [0..GVL.dataLogLengthBig] OF LREAL;
    data2 : ARRAY [0..GVL.dataLogLengthBig] OF LREAL;
    data3 : ARRAY [0..GVL.dataLogLengthBig] OF LREAL;
    data4 : ARRAY [0..GVL.dataLogLengthBig] OF LREAL;
    data5 : ARRAY [0..GVL.dataLogLengthBig] OF LREAL;
    data6 : ARRAY [0..GVL.dataLogLengthBig] OF LREAL;
    data7 : ARRAY [0..GVL.dataLogLengthBig] OF LREAL;
    data8 : ARRAY [0..GVL.dataLogLengthBig] OF LREAL;
    data9 : ARRAY [0..GVL.dataLogLengthBig] OF LREAL;
    data10 : ARRAY [0..GVL.dataLogLengthBig] OF LREAL;
    data11 : ARRAY [0..GVL.dataLogLengthBig] OF LREAL;
    data12 : ARRAY [0..GVL.dataLogLengthBig] OF LREAL;
    cycleCount : ARRAY [0..GVL.dataLogLengthBig] OF UDINT;
    dcTaskTime : ARRAY [0..GVL.dataLogLengthBig] OF STRING;
    lastExecTime : ARRAY [0..GVL.dataLogLengthBig] OF UDINT;
    cycleTimeExceeded : ARRAY [0..GVL.dataLogLengthBig] OF BOOL;
    RTViolation : ARRAY [0..GVL.dataLogLengthBig] OF BOOL;
END_STRUCT
END_TYPE

{attribute 'qualified_only'}
VAR_GLOBAL CONSTANT
    dataLogLength : UDINT := 15000;
    dataLogLengthBig : UDINT := 60000;
END_VAR

PROGRAM MAIN
VAR
    data : stData;
    dataBig : stDataBig;
    i : UDINT;
    iBig : UDINT;
    j : UDINT;
    taskIndex : GETCURTASKINDEX;
    taskInfo : PlcTaskSystemInfo;
    lastRead : UDINT;
END_VAR
taskInfo := _TaskInfo[taskIndex.index];

IF i > GVL.dataLogLength THEN
    i := 0;
END_IF

IF iBig > GVL.dataLogLengthBig THEN
    iBig := 0;
END_IF

IF j > 1_000_000 THEN
    j := 0;
END_IF

data.currentPointer := i;
data.data1[i] := UDINT_TO_LREAL(j);
data.data2[i] := UDINT_TO_LREAL(j);
data.data3[i] := UDINT_TO_LREAL(j);
data.data4[i] := UDINT_TO_LREAL(j);
data.data5[i] := UDINT_TO_LREAL(j);
data.data6[i] := UDINT_TO_LREAL(j);
data.data7[i] := UDINT_TO_LREAL(j);
data.data8[i] := UDINT_TO_LREAL(j);
data.data9[i] := UDINT_TO_LREAL(j);
data.data10[i] := UDINT_TO_LREAL(j);
data.data11[i] := UDINT_TO_LREAL(j);
data.data12[i] := UDINT_TO_LREAL(j);
data.cycleCount[i] := taskInfo.CycleCount;
data.dcTaskTime[i] := LINT_TO_STRING(taskInfo.DcTaskTime);
data.lastExecTime[i] := taskInfo.LastExecTime;
data.cycleTimeExceeded[i] := taskInfo.CycleTimeExceeded;
data.RTViolation[i] := taskInfo.RTViolation;

dataBig.currentPointer := iBig;
dataBig.data1[iBig] := UDINT_TO_LREAL(j);
dataBig.data2[iBig] := UDINT_TO_LREAL(j);
dataBig.data3[iBig] := UDINT_TO_LREAL(j);
dataBig.data4[iBig] := UDINT_TO_LREAL(j);
dataBig.data5[iBig] := UDINT_TO_LREAL(j);
dataBig.data6[iBig] := UDINT_TO_LREAL(j);
dataBig.data7[iBig] := UDINT_TO_LREAL(j);
dataBig.data8[iBig] := UDINT_TO_LREAL(j);
dataBig.data9[iBig] := UDINT_TO_LREAL(j);
dataBig.data10[iBig] := UDINT_TO_LREAL(j);
dataBig.data11[iBig] := UDINT_TO_LREAL(j);
dataBig.data12[iBig] := UDINT_TO_LREAL(j);
dataBig.cycleCount[iBig] := taskInfo.CycleCount;
dataBig.dcTaskTime[iBig] := LINT_TO_STRING(taskInfo.DcTaskTime);
dataBig.lastExecTime[iBig] := taskInfo.LastExecTime;
databig.cycleTimeExceeded[iBig] := taskInfo.CycleTimeExceeded;
dataBig.RTViolation[iBig] := taskInfo.RTViolation;

i := i + 1;
iBig := iBig + 1;
j := j + 1;

Code for only looking at [i for i in read_data] on line 274 of pyads_ex.py
import timeit
import ctypes
import numpy as np

number = 1_000_000 # _000
pylist = list(range(number))

def list_comp():
    t = timeit.Timer("data = [i for i in c_array]", globals=globals())
    print("[i for i in c_array] : " + str(t.repeat(repeat=5, number=10)))

def to_list():
    t = timeit.Timer("data = list(c_array)", globals=globals())
    print("list(c_array) : " + str(t.repeat(repeat=5, number=10)))

def using_numpy(dt):
    code = f"""\
from_buffer = np.frombuffer(c_array, dtype={dt})
buffer_list = from_buffer.tolist()
"""
    t = timeit.Timer(code, globals=globals())
    print("np.frombuffer : " + str(t.repeat(repeat=5, number=10)))

print("Int Array")
c_array_type = ctypes.c_int * len(pylist)
c_array = c_array_type(*pylist)
# c_array = (ctypes.c_int * len(pylist))(*pylist)

if type(c_array_type).__name__ == "PyCArrayType":
    list_comp()
    to_list()
    using_numpy("np.int32")
    from_buffer = np.frombuffer(c_array, dtype=np.int32)
    buffer_list = from_buffer.tolist()
    print([i for i in c_array] == list(c_array) == buffer_list)

print("\nBool Array")
c_array = (ctypes.c_bool * len(pylist))(*pylist)
list_comp()
to_list()
print([i for i in c_array] == list(c_array))

print("\nFloat Array")
c_array = (ctypes.c_float * len(pylist))(*pylist)
list_comp()
to_list()
using_numpy("np.float32")
from_buffer = np.frombuffer(c_array, dtype=np.float32)
buffer_list = from_buffer.tolist()
print([i for i in c_array] == list(c_array) == buffer_list)

print("\nChar Array")
pylist = [1] * number
c_array = (ctypes.c_char * len(pylist))(*pylist)
list_comp()
to_list()
print([i for i in c_array] == list(c_array))

print("\nu_byte array - What actually comes into pyads")
c_array = (ctypes.c_ubyte * len(pylist))(*pylist)
list_comp()
to_list()
using_numpy("np.ubyte")
from_buffer = np.frombuffer(c_array, dtype=np.ubyte)
buffer_list = from_buffer.tolist()
print([i for i in c_array] == list(c_array) == buffer_list)

Very interesting that the list() function performs so much better then the list-comprehension.

20% improvement is definetly worth a PR.

@chrisbeardy You already did some optimization with #292. Do you see room for more speed improvement or shall we close the issue?

There is definitely room for improvement here. I have a few ideas but limited time right now.
Using struct.unpack better is one. Instead of struck.upacking only one at a time we could do multiple if it's an array etc. That's just one.
Also I think there is potential to reorganise to allow unpacking more at a time too.
Leave it open for now.


Unable to use relative path for dockerfile

Before proceeding

[X] I didn't find a similar issue

Version
1.2.47
How to reproduce
Using the following image configuration fails
image: my-app
context: app
dockerfile: ../build/app.Dockerfile

Result
Error: unable to load werf config: 'dockerfile: PATH' required and should be relative to context!
Expected result
Successful build
Additional information
No response

Unfortunately, this is the current limitation of Docker. There is no way to use a Dockerfile not from context when building an image for a compressed context that reads from STDIN (default werf behavior).
There are no such limitations when using the Buildah backend, and we can add such a possibility. I'll make sure to keep you informed.


Missing external images causes crash

When an external image results in a WebException (404 or 500) the parser will crash.

Solved with version 1.2.69


converted to .net standard

I converted to .net standard and did som other fixes to get the demo project to work and build.

@dhindrik Thank you for the PR. It looks good. Just one question, is there any issue to update it to .NET Standard 2.0?

No, but everything needed is supported in .net Standard 1.0. A rule is too keep libraries as so low version as possible.

Thanks again. I made the update with other minor changes at my end.


python38Packages.inotify-simple: 1.1.8 -> 1.2.1

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/python3.8-inotify-simple/versions.
meta.description for python38Packages.inotify-simple is: '"A simple Python wrapper around inotify"'.
meta.homepage for python38Packages.inotify-simple is: '"https://github.com/chrisjbillington/inotify_simple"

Checks done (click to expand)

built on NixOS

Warning: a test defined in passthru.tests did not pass

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 1.2.1 with grep in /nix/store/sa9bvshxcq04l5dp7hh59qdi5j36i3ns-python3.8-inotify-simple-1.2.1

directory tree listing: https://gist.github.com/1b7486ec813ebe505e575f646693af48

du listing: https://gist.github.com/70119c02a82a42526038ffa5b10ec4be

Rebuild report (if merged into master) (click to expand)

13 total rebuild path(s)
4 package rebuild(s)
4 x86_64-linux rebuild(s)
3 i686-linux rebuild(s)
3 x86_64-darwin rebuild(s)
3 aarch64-linux rebuild(s)
First fifty rebuilds by attrpath
paperless
python27Packages.inotify-simple
python37Packages.inotify-simple
python38Packages.inotify-simple

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/sa9bvshxcq04l5dp7hh59qdi5j36i3ns-python3.8-inotify-simple-1.2.1 \
  --option binary-caches 'https://cache.nixos.org/ https://r-ryantm.cachix.org/' \
  --option trusted-public-keys '
  r-ryantm.cachix.org-1:gkUbLkouDAyvBdpBX0JOdIiD2/DP1ldF3Z3Y6Gqcc4c=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(r-ryantm's Cachix cache is only trusted for this store-path realization.)
Or, build yourself:
nix-build -A python38Packages.inotify-simple https://github.com/r-ryantm/nixpkgs/archive/ec719bc01df023619973d439b93d97280a17cbbe.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/sa9bvshxcq04l5dp7hh59qdi5j36i3ns-python3.8-inotify-simple-1.2.1
ls -la /nix/store/sa9bvshxcq04l5dp7hh59qdi5j36i3ns-python3.8-inotify-simple-1.2.1/bin

cc @erikarvstedt for testing.

Tested ACK ec719bc01df023619973d439b93d97280a17cbbe


How to convert a Amazon Sagemaker model into a Tensorflow model

I have trained and saved a model using Amazon SageMaker and I need to convert it into a Tensorflow model. Is there a way that this can be achieved?

Not familiar with Amazon SageMaker. Could you provide some for investigation? Thanks.

You can access the model from here https://drive.google.com/drive/folders/1rZiN25hxCtzr4vdyaPK9LFf8B63tQd51?usp=sharing

Hi @vasantivmahajan , your model is MxNet model, you can follow the mxnet -> tensorflow examples to do it, with model file (mx-mod-symbol.json and mx-mod-0000.params). Please ping us if there is any error. Thanks.


GoDaddy plugin works no more

Plugin Manual generated source gridnet.org with 2 identifiers
Plugin Single created 1 order
Renewing [Manual] gridnet.org
Cached order has status invalid, discarding
[.gridnet.org] Authorizing...
[.gridnet.org] Authorizing using dns-01 validation (Godaddy)
[.gridnet.org] Error preparing for challenge answer
[.gridnet.org] Deactivating pending authorization
[gridnet.org] Deactivating pending authorization
Renewal for [Manual] gridnet.org failed, will retry on next run
Error preparing for challenge answer
No certificate generated

We would need to see the detailed log (e.g. from disk or when run with the --verbose parameter on the command line) to diagnose this.

From the given error message Could not authenticate API key/secret it looks like your API key is no longer valid. Could that be the case? That would be my prime suspect at least.

I have triple-checked (trust me, I want this to work) the secret - by copy&pasting it from godaddy (with notepad as a middle-man to make sure that are no strange ASCII characters). And still, that is what I get.
 [VERB] [HTTP] Response content: {"code":"UNABLE_TO_AUTHENTICATE","message":"Unauthorized : Could not authenticate API key/secret"}
 [DBUG] [gridnet.org] Failed to create record under _acme-challenge.gridnet.org
 [EROR] [gridnet.org] Error preparing for challenge answer
System.Exception: [gridnet.org] Unable to prepare for challenge answer
   at PKISharp.WACS.Plugins.ValidationPlugins.DnsValidation`1.PrepareChallenge(ValidationContext context, Dns01ChallengeValidationDetails challenge)
   at PKISharp.WACS.Plugins.ValidationPlugins.Validation`1.PrepareChallenge(ValidationContext context)
   at PKISharp.WACS.RenewalValidator.Prepare(ValidationContext context, RunLevel runLevel)
 [VERB] Starting post-validation cleanup

Friends,so I had a single success when providing both the secret and API-key explicitly through command line parameters, in autonomous mode.
But, when choosing those same keys from the 'valut' - I am always faced with an error, the one depicted above.
I tried many many times, making sure that I am not confusing a key with a secret.
Then I run into a limit of failed attempts.

so I've switched to the 2nd acme DNS server to get around the limit...
now I get
[EROR] [gridnet.org] Authorization result: invalid
executing
wacs.exe --verbose --target manual --host gridnet.org,*.gridnet.org --validation godaddy --validationmode dns-02 --apikey Y --apisecret X --csr rsa --installation iis --installationsiteid 1 --store certificatestore
[EROR] [gridnet.org] {"type":"urn:ietf:params:acme:error:unauthorized","detail":"No TXT record found at _acme-challenge.gridnet.org","status":403,"instance":null}

Okay so it looks like two things may be happening here:

GoDaddys authentication using only an API key has been deprecated (may have been for a while already), so the secret should become mandatory. Support for API secrets was a later addition to the plugin, so I guess there was some sort of transition process.
Something may be wrong with the secret vault or the way you're trying to invoke it? If you manage the vault from the menu, do you see the right secrets? And what do you exactly type on the command line (or in the TUI) when you try to use the vault?

Fix to make the API secret mandatory is included in 2.2.5, did you get any further with the secret store?


Is there plan to implement some hack print like gruns/icecream

icecream#inspect-variables
expected:
func genInt() int {
  return 1
}
s := "hello"

dev.Print(s, genInt()) // s: hello, genInt(): 1

hi @potoo0 ,
I didn't know about icecream, thanks for bringing it up -- it seems a very useful tool.
But it seems to work well (mostly) in gonb, check out:

(Notice the bug with Ic() with no parameters, but I suspect that doesn't matter much ?)
Were you looking for something else ?
cheers

Sorry, I didn't do enough digging.
My first idea was to use ast to rewrite dev.Print(...) as fmt.Printf(<pattern>, ...), after the cell code is written to the file.
I also implemented a simple version, but it seems unnecessary after this.
selX.Name = "fmt"
sel.Sel.Name = "Printf"
label := identLabel(fset, expr)
expr.Args = append([]ast.Expr{&ast.BasicLit{Kind: token.STRING, Value: label}}, expr.Args...)
// ...
// format.Node overwrite main.go

// identLabel ...
for idx, arg := range expr.Args {
  builder.WriteString(AnsiCyan)
  err := printer.Fprint(&builder, fset, arg)
  if err != nil {
    builder.WriteString("-")
  }
  builder.WriteString(AnsiReset)
  builder.WriteString(": %+v")
  if idx+1 != argLen {
    builder.WriteString(", ")
  }
}


Error performing backfill with Firebase extensions

New to Algolia and trying to get the Firebase extension up-and-running, but having some issues with the backfill script.
This is the script I am running:
{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}
{"algoliaAPIKey":"********","severity":"INFO","message":"Initializing extension with configuration"}

WARNING: The back fill process will index your entire collection which will impact your Search Operation Quota.  Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details.  Do you want to continue? (y/N): y

{"severity":"ERROR","message":"Error when performing Algolia index TypeError: Cannot read property 'split' of undefined\n    at retrieveDataFromFirestore (/Users/simon/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:72:65)\n    at /Users/simon/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:80:9\n    at Interface._onLine (readline.js:340:5)\n    at Interface._line (readline.js:671:8)\n    at Interface._ttyWrite (readline.js:1015:14)\n    at ReadStream.onkeypress (readline.js:213:10)\n    at ReadStream.emit (events.js:376:20)\n    at emitKeys (internal/readline/utils.js:345:14)\n    at emitKeys.next (<anonymous>)\n    at ReadStream.onData (readline.js:1149:36)"}

I'm using the following command:
LOCATION=europe-west2\
PROJECT_ID=my-project-name\
ALGOLIA_APP_ID=0000000000\
ALGOLIA_API_KEY=0000000000\
ALGOLIA_INDEX_NAME=my_index\
COLLECTION_PATH=my-collection\
FIELDS=field1,field2,field3\
GOOGLE_APPLICATION_CREDENTIALS=/...json\
npx firestore-algolia-search

I tried reinstalling the extension and still having zero joy.
Any idea where I might be going wrong?

Found the answer here - needed to create and run a Bash Shell script.


[Request] image_from_spectrogram function

As discused in the discord, this could be useful for Audio2Audio

I will do this over the weekend!

See #16


feat(arrow): Provide conversions for more types

Expands the test coverage of our type conversions
Tightens up the code a bit so its less mind-numbing to read through
Removed the SQLite Flight SQL test, because there's something flakey in there. Unsure why that's the case, but I removed it for now so it's not causing us grief. I opened https://github.com/influxdata/grafana-flightsql-datasource/issues/78 to track that work.

DenseUnion (as returned from SQL info) isn't tested fully yet. Working on that next.

I'm realizing now that I think the flakiness in that integration query test is due to the concurrent readers. I'll play with that today before merging this. We might just have to vendor that whole example so we can tweak the SQLite setup options.

Here's the source of the failures: https://github.com/apache/arrow/pull/33983
I'll wait to see what happens here, but if this doesn't get merged, I'll vendor and patch it to fit our needs for integration tests.
The work I did on the Python library didn't involve concurrent requests so I never ran into this problem.


Suppressed error for long input data in new RNG

The following report was sent to the openssl-security mailing list by Dr. Falko Strenzke. The team members decided that it the security implications are minimal, so it will be treated as a normal bugfix. A pull request is in preparation.
The report was minimally edited and converted to markdown by me, but its content was not modified.
Hi OpenSSL team,
This is a security-related bug report for OpenSSL 1.1.1. It concerns the API functions
	void RAND_add(const void *buf, int num, double randomness);

and
	void RAND_seed(const void *buf, int num);

The default RNG implementation behind these functions is the new NIST DRBG generator. In this implementation, both functions fail to add any entropy contained within the the buf argument if num > 4096, i.e. if the size of the buffer is larger than 4096 bytes.
The reason are multiple checks:
https://github.com/openssl/openssl/blob/1708e3e85b4a86bae26860aa5d2913fc8eff6086/crypto/rand/drbg_lib.c#L963-L970
https://github.com/openssl/openssl/blob/1708e3e85b4a86bae26860aa5d2913fc8eff6086/crypto/rand/drbg_lib.c#L486-L490
https://github.com/openssl/openssl/blob/1708e3e85b4a86bae26860aa5d2913fc8eff6086/crypto/rand/drbg_lib.c#L504-L508
with drbg->max_entropylen and drbg->max_adinlen having the value 4096.
In case that more than 4096 bytes are input using either of the two above mentioned API functions, these checks let the respective function return with an error and without adding any entropy to the generator. Since both API functions have no return value, the caller never learns about the error conditions. I am not aware of any documentation that states this behaviour, and even if, this behaviour is a mere artefact and completely unnecessary, since the input data can simply be hashed in this case.
Furtunately, RAND_load_file() is not affected, since it slices the file into smaller portions of 1024 bytes for the purpose of seeding.
The following small program shows the problem during debugging:
void main()
{

  unsigned char seed_1 [4097] = { 1,3,4 };
  unsigned char output [10];
  RAND_add(seed_1, sizeof(seed_1), 0);
  RAND_seed(seed_1, sizeof(seed_1));
  //RAND_load_file("seedfile", 4097);
  int err_bytes = RAND_bytes(output, sizeof(output));
  printf("RAND_bytes returned %i\n", err_bytes);
}

Mit freundlichen Grüßen / With kind regards
Falko Strenzke

See Pull Request #7382.


Timetable ignoring a few courses due to minor error in slot timings

Small errors in time were not accounted for when generating the Time Table.
A difference of 1 min in ending time did not generate time table for that course.
5:30 slot not getting generated

Closed with #43 . Thanks, @Sigma1084 !


AttributeError: module 'papermill' has no attribute 'record'

Description
Baseline Deep Dive is using papermill's attribute: record
Unfortunately, record attribute was removed from papermill as per:
https://github.com/nteract/papermill/issues/507
Please update the code accordingly!
Thanks.

this is a duplicate of #641 currently the papermill version is pinned here: https://github.com/microsoft/recommenders/blob/837b8081a4421e144f2bc05ba949c5ac6c52320f/tools/generate_conda_file.py#L60
but we are certainly open to updating the notebooks to use scrapbook and then upgrading papermill version. if you have some time to help out PRs are welcome.
in the meantime you can downgrade the version of papermill you are using to match.
thanks, closing this for now


Rename to glacier-3dviz?

I think I'd like to keep the repo in the OGGM organisation namespace (since we seeked funding for it) but we can rename it to something more general

Yes, I just wanted to start with the renaming today. I thought about glacier3dviz with out the minus

Maybe we can but the tutorials also on a website like glacier3dviz.oggm.org?

Maybe we can but the tutorials also on a website like glacier3dviz.oggm.org?

Yes getting a subdomain is easy (we just have to ask Ben). It's all about getting a github workflows that works, or building locally and hosting on ghpages. Do you have a jupyterbook setup?

No not now. I wanted to copy the setup from the tutorials and adapt it. And building locally should be enough for starting.


Uncaught TypeError: fs.readFileSync is not a function in mapbox index.js

var fs = require('fs');
var path = require('path');
// readFileSync calls must be written out long-form for brfs.
module.exports = {
debug: {
fragmentSource: fs.readFileSync(path.join(__dirname, 'src/debug.fragment.glsl'), 'utf8'),   //Uncaught TypeError: fs.readFileSync is not a function
vertexSource: fs.readFileSync(path.join(__dirname, 'src/debug.vertex.glsl'), 'utf8')
},

var fs = require('fs');
var path = require('path');
// readFileSync calls must be written out long-form for brfs.
module.exports = {
debug: {
fragmentSource: fs.readFileSync(path.join(__dirname, 'src/debug.fragment.glsl'), 'utf8'), //Uncaught TypeError: fs.readFileSync is not a function
vertexSource: fs.readFileSync(path.join(__dirname, 'src/debug.vertex.glsl'), 'utf8')
},

Can you post your platform, Node version etc?
Node 4,5,6 and Linux, Windows, OSX are all working in continous integration.

node  v4.6.0
Windows 10 X64

node v4.6.0
Windows 10 X64

I can only test on Windows Server 2012 R2 (x64) via continous integration.
Do the following commands work?
md public
npm install
npm run build

I guess it's a build issue of Mapbox GL. But supporting everything on Windows is a bit difficult - that's why I will provide it as downloadable exe in the end.

I meet the same problem.

node v4.2.3
mac

I meet the same problem.

node v4.2.3
mac

it seems that node.js code has been bundled in the brower's javascript file

With the newest Mapbox GL package this should be solved now.


Mention that buffer-builder can be used if you want really-really-really fast encoding

See http://hackage.haskell.org/package/buffer-builder-<IP_ADDRESS>/docs/Data-BufferBuilder-Json.html.

About 2.5× slower according to one of the benchmarks.

(Or better yet, try to use buffer-builder for strict ByteString encoding.)


ThousandSeperator doesn't work after SaveAs a new excel

Description
original excel:
1.origin state:

2.click general

3.and then click thousand seperator

4.The number turn back to thousandSeperator.It's right.

after saveAs a new excel:
1.origin state:

2.click general

3.and then click thousand seperator

4.It's not work,that's the problem. Thank you.
Still at 1800.

Steps to reproduce the issue:
func ThousandSeperator() {
	f, _ := excelize.OpenFile("ThousandSeperator.xlsx")
	f.SaveAs("ThousandSeperator_new.xlsx")
}

ThousandSeperator.xlsx
Describe the results you received:
Not same as the original excel
Describe the results you expected:
Same as the original excel
Output of go version:
go1.13.7 windows/amd64

Excelize version or commit ID:
2020.05.05 master

Environment details (OS, Microsoft Excel™ version, physical, etc.):
win10,office2019

Thank you!

Thanks for your donation. I have added you're into the donator page.

😁😁

@xuri Hi,here has a follow-up questions:
click thousand seperator cause the style disappear after SaveAs function.
click:

then:


Changing trackers in announce-list corrupts save

Hi,
First thanks for taking the time to write this.
#
I’m trying to strip out existing (dead) trackers from torrent files and replace them with known good ones.  But, changing trackers in the announce-list results in a corrupted (non-working) announce-list.  [You can change the "announce" tracker successfully.]
For testing I’ve been using the MX-19.2_x64.iso torrent file.  It seems to be no longer available, but similar files are at https://mxlinux.org/torrent-files/ .
Output testing is by loading into qBittorrent v4.2.3 and checking the “Trackers” tab.  All tests fail by not loading the changed "announce-list" tracker to the Tracker List in qBittorrent.
See toradd.py code at bottom.  I’ve used “pytp” to visualize the outputed torrent files.
Request:
Have the code,
tracker_03a        = 'http://tracker.opentrackr.org:1337/announce'
data['announce-list'][0] = tracker_03a

correctly replace the tracker so that it loads successfully into qBittorrent.
Control pass:
Changes:
Straight read, no changes, and a write to separate file.
Results:
All trackers show up in  qBittorrent
michael@local [~/trash]# toradd.py arg1 arg2
michael@local [~/trash]# ./pytp '/home/michael/data/Downloads/isos/torrentfiles/MX-19.2_x64.iso.torrent' -d -i 2 > /home/michael/trash/torout.txt
michael@local [~/trash]# head torout.txt
{
  "announce": "http://l2.mxrepo.com:6969/announce",
  "announce-list": [
    [
      "http://l2.mxrepo.com:6969/announce"
    ],
    [
      "http://it.mxrepo.com:6969/announce"
    ]
  ],

Test 01-a:
Changes:
data['announce-list'][0] = tracker_03a
#data['announce-list'][1] = tracker_01

Results:
Loads only tracker  http://it.mxrepo.com:6969/announce
Output
michael@local [~/trash]# toradd.py arg1 arg2
michael@local [~/trash]# ./pytp '/home/michael/trash/test.torrent' -d -i 2 > /home/michael/trash/torout2.txt
michael@local [~/trash]# head torout2.txt
{
  "announce": "http://tracker.opentrackr.org:1337/announce",
  "announce-list": [
    "http://tracker.opentrackr.org:1337/announce",
    [
      "http://it.mxrepo.com:6969/announce"
    ]
  ],
  "created by": "mktorrent 1.0",
  "creation date":<PHONE_NUMBER>,

Test 01-b:
Changes:
data['announce-list'][0] = tracker_03a
data['announce-list'][1] = tracker_01

Results:
Loads only tracker  http://tracker.opentrackr.org:1337/announce
Output
michael@local [~/trash]# toradd.py arg1 arg2
michael@local [~/trash]# ./pytp '/home/michael/trash/test.torrent' -d -i 2 > /home/michael/trash/torout2.txt
michael@local [~/trash]# head torout2.txt
{
  "announce": "http://tracker.opentrackr.org:1337/announce",
  "announce-list": [
    "http://tracker.opentrackr.org:1337/announce",
    "http://tracker.dler.org:6969/announce"
  ],
  "created by": "mktorrent 1.0",
  "creation date":<PHONE_NUMBER>,
  "info": {
    "length":<PHONE_NUMBER>,

Test 02:
Changes:
data['announce-list'][0] = tracker_03b
data['announce-list'][1] = tracker_01

Results:
Loads only tracker  http://tracker.opentrackr.org:1337/announce
Output
michael@local [~/trash]# toradd.py arg1 arg2
michael@local [~/trash]# ./pytp '/home/michael/trash/test.torrent' -d -i 2 > /home/michael/trash/torout2.txt
michael@local [~/trash]# head torout2.txt
{
  "announce": "http://tracker.opentrackr.org:1337/announce",
  "announce-list": [
    "[\"http://tracker.opentrackr.org:1337/announce\"]",
    "http://tracker.dler.org:6969/announce"
  ],
  "created by": "mktorrent 1.0",
  "creation date":<PHONE_NUMBER>,
  "info": {
    "length":<PHONE_NUMBER>,

Test 03:
Changes:
data['announce-list'][0] = tracker_03c
data['announce-list'][1] = tracker_01

Results:
Loads only tracker  http://tracker.opentrackr.org:1337/announce
Output
michael@local [~/trash]# toradd.py arg1 arg2
michael@local [~/trash]# ./pytp '/home/michael/trash/test.torrent' -d -i 2 > /home/michael/trash/torout2.txt
michael@local [~/trash]# head torout2.txt
{
  "announce": "http://tracker.opentrackr.org:1337/announce",
  "announce-list": [
    "[http://tracker.opentrackr.org:1337/announce]",
    "http://tracker.dler.org:6969/announce"
  ],
  "created by": "mktorrent 1.0",
  "creation date":<PHONE_NUMBER>,
  "info": {
    "length":<PHONE_NUMBER>,

Test 04:
Changes:
data['announce-list'][0] = tracker_03d
data['announce-list'][1] = tracker_01

Results:
Loads only tracker  http://tracker.opentrackr.org:1337/announce
Output
michael@local [~/trash]# toradd.py arg1 arg2
michael@local [~/trash]# ./pytp '/home/michael/trash/test.torrent' -d -i 2 > /home/michael/trash/torout2.txt
michael@local [~/trash]# head torout2.txt
{
  "announce": "http://tracker.opentrackr.org:1337/announce",
  "announce-list": [
    "['http://tracker.opentrackr.org:1337/announce']",
    "http://tracker.dler.org:6969/announce"
  ],
  "created by": "mktorrent 1.0",
  "creation date":<PHONE_NUMBER>,
  "info": {
    "length":<PHONE_NUMBER>,

Program used
I was in the middle of passing trackers into the program, so ignore the other testing junk...
michael@local [~/common/bin]# cat toradd.py
#!/usr/bin/python3
# -*- coding: utf-8 -*-
import sys
import torrent_parser as tp

if  len(sys.argv) > 2:  # Need 2 arguments
  print(True)
else:
  print(False)
  sys.exit(43)

print ('Number of arguments:', len(sys.argv), 'arguments.')
print ('Argument List:', str(sys.argv))

tor_file_in       = '/home/michael/data/Downloads/isos/torrentfiles/MX-19.2_x64.iso.torrent'
tor_file_out      = '/home/michael/trash/test.torrent'
tracker_00        = "http://tracker.opentrackr.org:1337/announce"
tracker_01        = "http://tracker.dler.org:6969/announce"
tracker_02        = ''
tracker_03a        = 'http://tracker.opentrackr.org:1337/announce'
tracker_03b        = '["http://tracker.opentrackr.org:1337/announce"]'
tracker_03c        = '[http://tracker.opentrackr.org:1337/announce]'
tracker_03d        = "['http://tracker.opentrackr.org:1337/announce']"

data = tp.parse_torrent_file(tor_file_in)
print(data['announce'])
print(data['info']['name'])
print(data['announce-list'][0])
print(data['announce-list'][1])

data['announce'] = tracker_00
data['announce-list'][0] = tracker_03c
data['announce-list'][1] = tracker_01

print(data['announce'])
tp.create_torrent_file(tor_file_out, data)

sys.exit(0)

General output of toradd.py
It’s repetitive so I stripped it out of the above to reduce clutter...
michael@local [~/trash]# toradd.py arg1 arg2
True
Number of arguments: 3 arguments.
Argument List: ['/home/michael/common/bin/toradd.py', 'arg1', 'arg2']
http://l2.mxrepo.com:6969/announce
MX-19.2_x64.iso
['http://l2.mxrepo.com:6969/announce']
['http://it.mxrepo.com:6969/announce']
http://tracker.opentrackr.org:1337/announce

Edit:
Header bits for the input and output .torrent files
MX-19.2_x64.iso.torrent
d8:announce34:http://l2.mxrepo.com:6969/announce13:announce-listll34:http://l2.mxrepo.com:6969/announceel34:http://it.mxrepo.com:6969/announceee10:created by13:mktorrent 1.013:creation

test.torrent
d8:announce43:http://tracker.opentrackr.org:1337/announce13:announce-listl47:['http://tracker.opentrackr.org:1337/announce']37:http://tracker.dler.org:6969/announcee10:created by13:mktorrent 1.013:creation

Sorry for the late reply, it's Chinese Lunar New Year these few days, so I don't have time to check this issue.
I'll take a look tonight.

Due to Torrent file spec, type of announce-list field should be List<List<Tracker>>.
So, please try
tracker_00 = 'http://tracker.opentrackr.org:1337/announce'
tracker_01 = ['http://tracker.dler.org:6969/announce']
tracker_03 = ['http://tracker.opentrackr.org:1337/announce']

with your logic and see if it works.

Close due to not active.
Feel free to continue this issue if you want.

Hi 7sDream,
AFAICT my ticket include all of your try requests.  All break following saves.  That is why I didn’t reply, I figured you’d eventually read my op and chuckle to yourself that the information you wanted was already there.
Since you've closed this, indicating there is no bug, what is the proper syntax variable assignment to change trackers in the announce-list?
Please also add that (and how to change every other torrent value, as most people who would like to use this don’t know python or its syntax) to both your Wiki github provides and the README.  Doing so would stop wasting both of ours time.
Best Regards,
Michael

This is your code
tracker_00        = "http://tracker.opentrackr.org:1337/announce"
tracker_01        = "http://tracker.dler.org:6969/announce"
tracker_02        = ''
tracker_03a        = 'http://tracker.opentrackr.org:1337/announce'
tracker_03b        = '["http://tracker.opentrackr.org:1337/announce"]'
tracker_03c        = '[http://tracker.opentrackr.org:1337/announce]'
tracker_03d        = "['http://tracker.opentrackr.org:1337/announce']"

This is my suggestion:
tracker_00 = 'http://tracker.opentrackr.org:1337/announce'
tracker_01 = ['http://tracker.dler.org:6969/announce']
tracker_03 = ['http://tracker.opentrackr.org:1337/announce']

Please read it CAREFULLY.

Due to Torrent file spec, type of announce-list field should be List<List<Tracker>>.
So, please try
tracker_00 = 'http://tracker.opentrackr.org:1337/announce'
tracker_01 = ['http://tracker.dler.org:6969/announce']
tracker_03 = ['http://tracker.opentrackr.org:1337/announce']

with your logic and see if it works.

7sDream,
You obviously never bothered to test your own code, since instead of replying with the proper syntax, you said, “try this.”
As a suggestion, never, ever list this github account on any resume you submit to a potential employer.
You can close this for all I care, I’ll use something else.

I'm pretty sure the annonce-list can be modified(increased) normally using the code I've given.
This is not the "syntax problem" you mentioned, but you have absolutely no understanding of the format of the torrent file, nor a basic understanding of the Python language.
I have no responsibility or obligation to guide you on how to use my project, and I don't care what project you end up using to accomplish your goals.  Good lucks, and please do not use any of my code anymore.


Clarify Wavefront in vendor list

#1786 was merged before I could make an additional change, so that "Aria by VMware" is instead "Aria by VMware (Wavefront)". We intend this to be a stop-gap until a rebranding at VMware is finished.

@cartermp and @chalin: I made a small tweak to #1786, as discussed in that PR.


Contribution: Text toolbar

Is your feature request related to a problem? Please describe.
The usage of an old widget in a new UI to write runbooks or messages directly.

Describe the solution you'd like
I have designed a new component with the Carbon styleguide to type guided tasks easily (runbook automation) while improving the collaboration among teams.
All files here: https://ibm.box.com/s/4rv1jw7xwuv8c8rvqbxp9057qzlydkva
Screenshot details:

Some redlines:

Different color background:

cc @IBM/carbon-designers is this a pattern we want to consider for carbon-components?

Something like this will come along at one point so I would say yes but leave that to @aagonzales @jeanservaas and @chrisconnors-ibm

verse for example has a similar tool bar with similar functionality/features

I think this would be a good component to have! However, If we want to build an exact text toolbar, I think some of the smaller interactions will need to be fleshed out and spec-ed. Like how big does the type dropdown go before its starts scrolling. In the text color selector can you pick unique colors and if so how? Or is it only a pre-defined set.
Or if you want to contribute this more as a general file toolbar that can be configured in different ways then we'll need to define and spec what the general widget types are. I'm seeing at least 4 right now (single selection dropdown, Icon button - icon directly triggers an action, Icon menu, Search). Could there be any others?
I think there are a few style discrepancies as well, normally hover goes full bleed instead of inset we'll need to work through as well.
Thanks for reaching out and contributing!

Let’s resolve them and push further.

Thank you for feedback @aagonzales

The space before the type and with “check” in the dropdowns seems large.

@mjabbink Thanks for your feedback!

@aagonzales for context their team has a text toolbar compound component already - @m-battelli has been working to apply the new design language to that existing thing. I'd like to have them submit that component to our main set, so expect it to be 1:1 functionality from what they have.

we get the styling approved through this, wake, playbacks.
they develop the component based on their existing using the new IBM Design Language, our assets, etc.
2a) we help them develop a sketch symbol for eventual inclusion in the kit

@mjabbink @chrisconnors-ibm
I have improved several details, thanks for feedback and the recording link :-)
I have discussed this new stuff with @aagonzales
Please give me suggestions again:

40px height is nice with bleeding hovers. I don’t think the drop down needs hover does it? @aagonzales. I like 32px and assume this is an option?
For color version I like 2 or 3

in the 40px section I like the vertical lines spanning the 40px versus the vertical line floating. What do you think @aagonzales and @jeanservaas?

I think I take back what I said about color swatches. Not sure about the hover/select state when there is the white margin around the colors. When there is no margin the grey hover/select works better. What do you think @aagonzales @chrisconnors-ibm @jeanservaas

also wonder if adjusting the font dropdown so the rules align? There will be some headspace on the right side but that might be fine to hold the integrity of alignments in this case.

.
.
.
.
.

@mjabbink @aagonzales @chrisconnors-ibm
I've included all the feedback received until now. I'm also exploring the dark version.
Let me know what you think, thanks.

@m-battelli What determines the width of the attachment buttons?

@mjabbink I have used the file uploader component in this case:

This is looking really good! You've polished it up nicely.

dropdowns look good
hover and selected states look good
alignment looks good.
dark theme looks correct

Just a few things to add:

focus states are missing: which would probably just be a 2px outline around the buttons (takes up the same space as the hovers). It would be blue-60 for the light and white for the dark theme (color token is just $focus).

when two icons beside each other are selected, I think there should be a rule between them still. Maybe go darker?

responsiveness: I would maybe show what happens at some of the other break points. Does the tool bar just gain white space on the end if its on larger monitor? Do the icons slowly start to stack on a second row or does that change all at once? Is there a max and min width for the text bar?

@aagonzales

Focus state is ok for all themes.

For the two selected icon buttons beside each other, I think to add a separator simply. Also we need to consider the hover on a selected icon. This visual seems ok. What do you think?

I have designed the responsive behavior for the maximum and minimum width of the toolbar. I have adopted the ellipses menu with different version (list or panel). Let me know which of these is fine. Thanks!

For hover on selected, in the system we've been going an additional shade darker (Gray20) on the light and a shade lighter (Gray 60) on the dark .
The responsive strategy looks good. I think i would just add divides into the overflow menu to maintain the sections.

@aagonzales ok, thanks for feedback!I will add these details.

@aagonzales I have updated file version based on your feedback. Also I have checked all the colors applied in these two themes. That's all for now. All the stuff is here:
https://ibm.box.com/s/4rv1jw7xwuv8c8rvqbxp9057qzlydkva
Let me know how we can proceed, thanks a lot!

This looks great! I think the next steps would be to show it one more time at the IDL playback on Friday then dev can start on it. Is your dev team still building it?

A quick suggestion:

Icon needs to be redrawn, but spacing and size may need to be adjusted, but something like this could help.

@wonilsuh Thanks!

Hi Myriam, Great work!
During the playback, I mentioned the Adobe Illustrator toolbar has a good hierarchy set up to help users get the visual cue around there are more options within an individual tool. The screenshot below is the toolbar.
The mockup above from Wonil's reflects what I am saying and will be a good idea to try.

@JennySanchez Thanks! Sure, I will try to refine this.

@wonilsuh and @m-battelli
I think the mini chevron could be a nice solution. Agree it needs to be redrawn. It could also be a touch larger still. @conradennis could have a stab at creating the mini chevron option and get into the Library.

@m-battelli I believe that in your dropdown where you use the word "indend" you mean to have "indent"

I have refined all visual details after our last playback (March 29). Thanks for all your feedback!
The latest version with the 3 themes available is here:

Main improvement is related to mini-chevron #491 for color text and text alignment:

Let me know what you think on this final step.

@m-battelli this looks great thanks for fleshing out those details. It alls looks good to me, the only one concern I have is with the search results. In context, if the toolbar is flush to the top the of page or container where would the tag go? Maybe inside the field and could potentially be just a number?

I'm also wondering if maybe the text height shouldn't be a tag style? It looks like its will add some padding around the word that might make the content after a highlighted word move, which would not be ideal. Something more like a traditional highlight could work better.

@aagonzales agree with you. I will refine it!

@aagonzales
Option 1 (with interactive tag)

Option 2 (two separeted visuals: number and close icon)

What do you think?

@m-battelli I can't decide ... I know in our filterable and multi-select dropdowns we use the tag with the x but just looking around at what other things do with a find search they seem to separate them. I think either way would be acceptable.

In the separated option couldn’t the number still be in a tag rounded container without “x” rather than square container? I like the separated version.
These look great!

@mjabbink @aagonzales
I have fixed the search details as you suggested, thank you for the feedback!
<img width="603" alt="Screen Shot 2019-04-24 at 15 40 48" src="https://user-images.githubusercontent.com/47557215/56663925-6f09de00-66a7-11e9-9539-93f5c716410d.png”>
The updated files are here: https://ibm.box.com/s/4rv1jw7xwuv8c8rvqbxp9057qzlydkva
Please let me know if there is anything else to do or I can mark this design as completed :)

I would say its complete now! Looks great!

Great work and dedication. Shows.

@aagonzales @mjabbink @chrisconnors-ibm
Thank you for supporting me with great advice during this task. I have just aligned the shetch / pdf files with the 4 themes now available. I don't like to leave things halfway :-)
I remain available for further updates or questions.
Let me know when this new component will be implemented.
Ciao!

@m-battelli "Let me know when this new component will be implemented."
SInce your team, who built the v9 component, will be doing this development work, we're waiting for you to deliver them a spec, and your team to size, develop the refactor their existing component. from there you need to build a sketch Symbol (we'll help you) and document how to use the component (minimum as we have documented existing components and compound components).


Fixes MYRIAD-114

Fixed gulp build to not include local paths and gradle-ized build.
Removed public/ from git repo and added it to .gitignore.

Tested it on my dev setup, but the RM can't come up. It throws me the following null pointer:
Caused by: java.lang.NullPointerException
        at org.apache.myriad.webapp.MyriadWebServer.start(MyriadWebServer.java:64)
        at org.apache.myriad.Main.initWebApp(Main.java:119)
        at org.apache.myriad.Main.run(Main.java:91)
        at org.apache.myriad.Main.initialize(Main.java:76)
        at org.apache.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:51)
        ... 16 more

The corresponding code in the webserver is:
    String staticDir = this.getClass().getClassLoader().getResource("webapp/public").toExternalForm();

The myriad-scheduler-0.1.jar doesn't have "webapp/public". The build doesn't seem to copy "webapp/public" under the build directory.

I saw that once but couldn't reproduce it. I figured it was just from me
trying to get everything working. A few questions: did you gradle or
gradlew? Were the files generated in src/main/resources/webapp/public and
just didn't make it into the jar? I'll try to reproduce again this morning.
On Thursday, October 29, 2015, Santosh Marella<EMAIL_ADDRESS>wrote:

Tested it on my dev setup, but the RM can't come up. It throws me the
following null pointer:
Caused by: java.lang.NullPointerException
at org.apache.myriad.webapp.MyriadWebServer.start(MyriadWebServer.java:64)
at org.apache.myriad.Main.initWebApp(Main.java:119)
at org.apache.myriad.Main.run(Main.java:91)
at org.apache.myriad.Main.initialize(Main.java:76)
at org.apache.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:51)
... 16 more
The corresponding code in the webserver is:
String staticDir = this.getClass().getClassLoader().getResource("webapp/public").toExternalForm();

The myriad-scheduler-0.1.jar doesn't have "webapp/public". The build
doesn't seem to copy "webapp/public" under the build directory.
—
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-myriad/pull/28#issuecomment-152357160
.

I did "gradlew".
Yes. The files were generated under src/main/resources/webapp/public, but
didn't get copied over to myriad-scheduler/build/resources/main/webapp and
subsequently didn't make it into the myriad-scheduler-0.1.jar.
Santosh
On Fri, Oct 30, 2015 at 5:04 AM, Jim Klucar<EMAIL_ADDRESS>wrote:

I saw that once but couldn't reproduce it. I figured it was just from me
trying to get everything working. A few questions: did you gradle or
gradlew? Were the files generated in src/main/resources/webapp/public and
just didn't make it into the jar? I'll try to reproduce again this morning.
On Thursday, October 29, 2015, Santosh Marella<EMAIL_ADDRESS>wrote:

Tested it on my dev setup, but the RM can't come up. It throws me the
following null pointer:
Caused by: java.lang.NullPointerException
at
org.apache.myriad.webapp.MyriadWebServer.start(MyriadWebServer.java:64)
at org.apache.myriad.Main.initWebApp(Main.java:119)
at org.apache.myriad.Main.run(Main.java:91)
at org.apache.myriad.Main.initialize(Main.java:76)
at
org.apache.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:51)
... 16 more
The corresponding code in the webserver is:
String staticDir =
this.getClass().getClassLoader().getResource("webapp/public").toExternalForm();
The myriad-scheduler-0.1.jar doesn't have "webapp/public". The build
doesn't seem to copy "webapp/public" under the build directory.
—
Reply to this email directly or view it on GitHub
<
https://github.com/apache/incubator-myriad/pull/28#issuecomment-152357160>
.

—
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-myriad/pull/28#issuecomment-152506635
.

I updated the PR. I moved the web app build to before process resources.
Brandon and I both tried it and saw the files in the jar now.
On Friday, October 30, 2015, Santosh Marella<EMAIL_ADDRESS>wrote:

I did "gradlew".
Yes. The files were generated under src/main/resources/webapp/public, but
didn't get copied over to myriad-scheduler/build/resources/main/webapp and
subsequently didn't make it into the myriad-scheduler-0.1.jar.
Santosh
On Fri, Oct 30, 2015 at 5:04 AM, Jim Klucar<EMAIL_ADDRESS><javascript:_e(%7B%7D,'cvml','notifications@github.com');>>
wrote:

I saw that once but couldn't reproduce it. I figured it was just from me
trying to get everything working. A few questions: did you gradle or
gradlew? Were the files generated in src/main/resources/webapp/public and
just didn't make it into the jar? I'll try to reproduce again this
morning.
On Thursday, October 29, 2015, Santosh Marella<EMAIL_ADDRESS><javascript:_e(%7B%7D,'cvml','notifications@github.com');>>
wrote:

Tested it on my dev setup, but the RM can't come up. It throws me the
following null pointer:
Caused by: java.lang.NullPointerException
at
org.apache.myriad.webapp.MyriadWebServer.start(MyriadWebServer.java:64)
at org.apache.myriad.Main.initWebApp(Main.java:119)
at org.apache.myriad.Main.run(Main.java:91)
at org.apache.myriad.Main.initialize(Main.java:76)
at

org.apache.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:51)

... 16 more
The corresponding code in the webserver is:
String staticDir =

this.getClass().getClassLoader().getResource("webapp/public").toExternalForm();

The myriad-scheduler-0.1.jar doesn't have "webapp/public". The build
doesn't seem to copy "webapp/public" under the build directory.
—
Reply to this email directly or view it on GitHub
<

https://github.com/apache/incubator-myriad/pull/28#issuecomment-152357160>

.

—
Reply to this email directly or view it on GitHub
<
https://github.com/apache/incubator-myriad/pull/28#issuecomment-152506635>
.

—
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-myriad/pull/28#issuecomment-152567043
.

awesome. I'll test it out again before merging.
Santosh
On Fri, Oct 30, 2015 at 12:53 PM, Jim Klucar<EMAIL_ADDRESS>wrote:

I updated the PR. I moved the web app build to before process resources.
Brandon and I both tried it and saw the files in the jar now.
On Friday, October 30, 2015, Santosh Marella<EMAIL_ADDRESS>wrote:

I did "gradlew".
Yes. The files were generated under src/main/resources/webapp/public, but
didn't get copied over to myriad-scheduler/build/resources/main/webapp
and
subsequently didn't make it into the myriad-scheduler-0.1.jar.
Santosh
On Fri, Oct 30, 2015 at 5:04 AM, Jim Klucar<EMAIL_ADDRESS><javascript:_e(%7B%7D,'cvml','notifications@github.com');>>
wrote:

I saw that once but couldn't reproduce it. I figured it was just from
me
trying to get everything working. A few questions: did you gradle or
gradlew? Were the files generated in src/main/resources/webapp/public
and
just didn't make it into the jar? I'll try to reproduce again this
morning.
On Thursday, October 29, 2015, Santosh Marella <
<EMAIL_ADDRESS><javascript:_e(%7B%7D,'cvml','notifications@github.com');>>
wrote:

Tested it on my dev setup, but the RM can't come up. It throws me the
following null pointer:
Caused by: java.lang.NullPointerException
at
org.apache.myriad.webapp.MyriadWebServer.start(MyriadWebServer.java:64)
at org.apache.myriad.Main.initWebApp(Main.java:119)
at org.apache.myriad.Main.run(Main.java:91)
at org.apache.myriad.Main.initialize(Main.java:76)
at

org.apache.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:51)

... 16 more
The corresponding code in the webserver is:
String staticDir =

this.getClass().getClassLoader().getResource("webapp/public").toExternalForm();

The myriad-scheduler-0.1.jar doesn't have "webapp/public". The build
doesn't seem to copy "webapp/public" under the build directory.
—
Reply to this email directly or view it on GitHub
<

https://github.com/apache/incubator-myriad/pull/28#issuecomment-152357160>

.

—
Reply to this email directly or view it on GitHub
<

https://github.com/apache/incubator-myriad/pull/28#issuecomment-152506635>

.

—
Reply to this email directly or view it on GitHub
<
https://github.com/apache/incubator-myriad/pull/28#issuecomment-152567043>
.

—
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-myriad/pull/28#issuecomment-152636073
.

Tested it out. The build is good, RM comes up alright. But the UI doesn't seem to render. Only the favicon and the page title seem to appear.

I found an error in the shutdown component that kept it from rendering. I actually think I want to re-do the interface to make it more like the website design, but that can wait a bit.

@klucar - any chance this could be fixed today? This is a blocker for 0.1.0.

I thought I fixed it, maybe I didn't push. It was a simple error in the
JavaScript. Check the PR. I'm on mobile now but can look at it later.
On Tuesday, November 3, 2015, Santosh Marella<EMAIL_ADDRESS>wrote:

@klucar https://github.com/klucar - any chance this could be fixed
today? This is a blocker for 0.1.0.
—
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-myriad/pull/28#issuecomment-153410420
.

@klucar - you are right. I see a commit added to the PR. Not sure why I don't receive the git notifications, though. I'll test this out locally and merge in if the UI renders correctly. Thanks!


Allow connections to OVS over a unix domain socket.

@mapuri @jainvipin
This is the patch I submitted to libovsdb: https://github.com/socketplane/libovsdb/pull/35
Perhaps we can make use of it separately.

yes LGTM
We will need to update the netplugin's godeps to take this change.


Write during insert mode can corrupt undofile

Steps to reproduce

$ VIMRUNTIME=$PWD/runtime src/vim -u NORC /tmp/testfile.txt
:set nocompatible undofile
:imap . <cmd>write<cr>
:write
:call feedkeys("Otest.\<cr>boo!!!\<esc>")
:call feedkeys("u")

Expected behaviour
buffer should be empty instead of
boo!!!

Version of Vim
VIM - Vi IMproved 9.0 (2022 Jun 28, compiled Dec  8 2022 00:16:20) Included patches: 1-1027
Environment
debian testing
gnome terminal
xterm-256color
zsh 5.9
Logs and stack traces
No response

hmm, I guess we need:
diff --git a/src/edit.c b/src/edit.c
index 43f7d9abb..dc27fc31e 100644
--- a/src/edit.c
+++ b/src/edit.c
@@ -1049,6 +1049,7 @@ doESCkey:

        case K_COMMAND:             // <Cmd>command<CR>
        case K_SCRIPT_COMMAND:      // <ScriptCmd>command<CR>
+           start_arrow(&curwin->w_cursor);
            do_cmdkey_command(c, 0);
 #ifdef FEAT_TERMINAL
            if (term_use_loop())

This will break the undo sequence before the <cmd> so you then need 2 undos to have an empty buffer. Would that make sense?

Calling start_arrow() for every  mapping doesn't appear needed and may be unexpected.
The problem is that write() syncs undo, the next character typed should save for undo again.
The subject of this issue mentions "corrupt undofile", but I assume that's not actually the problem.


Game fails to run from editor

Godot version
3.4.3 stable
System information
Windows 11
Issue description
Sometimes (and I'm not sure what causes it), all the run buttons stop working. The output only shows

--- Debugging process started ---
--- Debugging process stopped ---

Steps to reproduce
I have no clue how to reproduce this. It just happens sometimes.
Minimal reproduction project
No response

I'm getting the error again. Here's my project.
ppc.zip
The only way it seems to fix this is to restart my PC.

I've had this issue on 4.0.alpha8 but don't know why it's occurring (or how to fix it). It only seems to occur in certain projects though.
There is one case where it will always happen: when the Godot binary you're running was replaced while the editor is running (which can only happen on macOS or Linux).


add support to 2 boards

Hi please add support to my 2 new boards: circuitart_esp32s2core & circuitart_stm32f4core
thx

look good, there is only a couple of requests (check out comment). Also as part of VID/PID requirement, could you tell how you obtain the VID/PID for your boards.

from the repo of pid.codes

board ID has been corrected in both board.h files, also website has this comment now (// TODO update link)
build.yml abc order has been corrected
VID/PID are from pid.codes but the pull requested has not been made yet

Can you give link to the PR, we will merge once that PR is merged.

board ID has been corrected in both board.h files, also website has this comment now (// TODO update link)
build.yml abc order has been corrected
VID/PID are from pid.codes but the pull requested has not been made yet

Can you give link to the PR, we will merge once that PR is merged.

ok thanks I will update the pull request with PID.CODES PR link

board ID has been corrected in both board.h files, also website has this comment now (// TODO update link)
build.yml abc order has been corrected
VID/PID are from pid.codes but the pull requested has not been made yet

Can you give link to the PR, we will merge once that PR is merged.

ok thanks I will update the pull request with PID.CODES PR link

I mean can you give the link to the PID.CODES PR link here in the comment section, adding it to the code is optional and up to you.


KP Entity subclass semantic operations -- July 15

From the Architecture Principles:
9. KPs that implement the Translator Reasoner API must perform the following kinds of reasoning in answering queries:
      i. Making identifiers more specific, e.g. responding to a query involving an entity with information related
         to a subclass of that entity.

We discussed this at the Spring Relay and described one solution in the prototype notebook linked here:
https://github.com/NCATSTranslator/TranslatorArchitecture/blob/master/Notebooks/Using OntologyKP to get subclasses.ipynb
This uses the ontology KP to find out what the subclasses of an item are.


How to add Attendance bot App to Group chat programmatically?

I have tried with Microsoft Graph Explorer for checking is it working or not but It gives me response with Specific permission mismatch but I have given proper RSC Application permission to App manifest and published it my dev portal. Please guide.

@ayon-maji - We are looking into this we will get back to you soon.

@ayon-maji -To install the Microsoft Teams bot in a team or group chat, add the teams or groupchat scope to your bot.
here is sample for reference-
https://github.com/microsoft/BotBuilder-Samples/tree/main/samples/csharp_dotnetcore/57.teams-conversation-bot

@ayon-maji -Did you get chance to check?

@sayali-MSFT Not yet. Will check it tonight.

@sayali-MSFT I have checked it not working with MS Graph API. I have created the manifest for Teams Attendance BOT and published it in my dev Tenant and after that I have tried to add that app using MS Graph API explorer but failed. Here is my graph result screenshots.

and this is my manifest-
{
"$schema": "https://developer.microsoft.com/en-us/json-schemas/teams/v1.11/MicrosoftTeams.schema.json",
"manifestVersion": "1.11",
"version": "1.0.0",
"id": "8d06f32f-0eac-4886-bb38-3998311ced93",
"packageName": "com.microsoft.teams.meetingattendancereport",
"developer": {
"name": "Microsoft",
"websiteUrl": "https://www.microsoft.com",
"privacyUrl": "https://www.microsoft.com/privacy",
"termsOfUseUrl": "https://www.microsoft.com/termsofuse"
},
"icons": {
"color": "color.png",
"outline": "outline.png"
},
"name": {
"short": "Meeting Attendance Bot",
"full": "Meeting Attendance Bot"
},
"description": {
"short": "Bot sent meeting attendance report in chat.",
"full": "This is a sample application which demonstrates how to get attendance reports using Graph API and send it in meeting chat."
},
"accentColor": "#235EA5",
"bots": [
{
"botId": "c8fb9760-c7eb-4d46-b615-249326266e10",
"scopes": [ "groupchat" ],
"isNotificationOnly": true
}
],
"validDomains": [
"*.ngrok.io",
"https://1d85-103-135-228-14.ngrok.io"
],
"permissions": [ "messageTeamMembers" ],
"webApplicationInfo": {
"id": "7e23a7cc-94e0-46cb-acaa-f8f8d4132e25",
"resource": "https://RscPermission",
"applicationPermissions": [
"OnlineMeeting.ReadBasic.Chat"
]
}
}

@ayon-maji -Currently, this operation does not support installation of apps that require resource-specific consent permissions. For details, see Known issues.
Reference Document-https://learn.microsoft.com/en-us/graph/api/chat-post-installedapps?view=graph-rest-1.0&tabs=http

@sayali-MSFT is there any solution to get meeting duration / attendees without OnlineMeetings.Read.All Application permission?

@ayon-maji -Without permission it is not possible to get meeting duration/attendees.
Here is one sample to get the meeting event like meeting start date time,duration, but you need to add the following permission in manifest json.
"applicationPermissions": [
"OnlineMeeting.ReadBasic.Chat"
]
Reference sample- https://github.com/OfficeDev/Microsoft-Teams-Samples/tree/main/samples/meetings-events/csharp/MeetingEvents/AppPackage

@sayali-MSFT I already have tried with that sample but this operation does not support installation of apps that require resource-specific consent permissions

@ayon-maji -Currently, Without permission it is not possible to get meeting duration/attendees.
Could you please help by suggesting this feature on - Microsoft Teams · Community

@sayali-MSFT ok

We are closing this issue for now, Please feel free to reopen it, if required.


chore(deps): update terraform terraform-ibm-modules/key-protect/ibm to v2.6.3

This PR contains the following updates:

Package
Type
Update
Change

terraform-ibm-modules/key-protect/ibm (source)
module
patch
2.6.2 -> 2.6.3

Release Notes

terraform-ibm-modules/terraform-ibm-key-protect (terraform-ibm-modules/key-protect/ibm)
v2.6.3
Compare Source
Bug Fixes

deps: require minimum 1.64.0 ibm provider to pull in fix for https://github.com/IBM-Cloud/terraform-provider-ibm/issues/5154 (#554) (59ba5ce)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

/run pipeline

/run pipeline

/run pipeline

changes made by @shemau in https://github.com/terraform-ibm-modules/terraform-ibm-kms-key-ring/pull/564 resolve this by un-pinning the provider version in this example, should be good now

Edited/Blocked Notification
Renovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR.
You can manually request rebase by checking the rebase/retry box above.
⚠ Warning: custom changes will be lost.

Failure is due to upgrade test.
TestRunCompleteExampleUpgrade 2024-04-17T15:11:59Z command.go:185: Terraform will perform the following actions:
TestRunCompleteExampleUpgrade 2024-04-17T15:11:59Z command.go:185:
TestRunCompleteExampleUpgrade 2024-04-17T15:11:59Z command.go:185:   # module.key_protect.ibm_kms_instance_policies.key_protect_instance_policies will be updated in-place
TestRunCompleteExampleUpgrade 2024-04-17T15:11:59Z command.go:185:   ~ resource "ibm_kms_instance_policies" "key_protect_instance_policies" {
TestRunCompleteExampleUpgrade 2024-04-17T15:11:59Z command.go:185:       + endpoint_type = "public"
TestRunCompleteExampleUpgrade 2024-04-17T15:11:59Z command.go:185:         id            = "crn:v1:bluemix:public:kms:ca-tor:a/abac0df06b644a9cabc6e44f55b3880e:51616daf-80f4-4dca-9a9f-8a7d4db33763::"
TestRunCompleteExampleUpgrade 2024-04-17T15:11:59Z command.go:185:         # (1 unchanged attribute hidden)
TestRunCompleteExampleUpgrade 2024-04-17T15:11:59Z command.go:185:
TestRunCompleteExampleUpgrade 2024-04-17T15:11:59Z command.go:185:         # (4 unchanged blocks hidden)
TestRunCompleteExampleUpgrade 2024-04-17T15:11:59Z command.go:185:     }
TestRunCompleteExampleUpgrade 2024-04-17T15:11:59Z command.go:185:
TestRunCompleteExampleUpgrade 2024-04-17T15:11:59Z command.go:185: Plan: 0 to add, 1 to change, 0 to destroy.

This change is in a module used in the example, not in the root module.
Skipping upgrade test.

/run pipeline


IMKTextInput inline composition buffer is not linebreaking when necessary.

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.72.2 d045a5eda657f4d7b676dedbfa7aab8207f8a075
OS Version: 22A379

Steps to Reproduce:

Use any macOS buit-in Chinese Input Method (or Japanese Input Method).
Enable auto-wrap.
Type something to force the inline composition buffer (i.e. "preedit area" in Windows TSF) breaks at the line end.

Update: Disabling all extensions will make this issue behaves differently, but still abnormal:

BTW, please find a native Traditional-Chinese bopomofo user to use the Zhuyin IME to test whether this issue is fixed.
My English ability is not well enough for describing other symptoms of this issue precisely.
(Simplified Chinese input methods shipped in macOS doesn't use inline composition buffer pretty much.)

Let's track supporting wrapping and IME at the same time in https://github.com/microsoft/vscode/issues/28291

Looks like this issue is already existed for years.
Considering that it looks like it won't be dealt in the following years, I'll add both VSCode and VSCodium to the IMKTextInput blacklist of vChewing IME (a Chinese Input Method project of mine) which enables popup-composition-buffer instead of the inline one.


do no use context as first document choice

fix #386

Warnings

:warning:
This pull request may need a CHANGELOG entry.

:warning:

Source/dom/models/Document.js changed, but not:

📚 its docs
🧪 its tests

That's OK as long as you're refactoring.

  Generated by :no_entry_sign: dangerJS


Modify the reindexer to support passing a FilterExpression

e.g. for the catalogue/Miro
Follows #3038 and #3102.

Here's an example of how this works in the Python SDK:
import boto3

d = boto3.client("dynamodb")

d.scan(
    TableName="vhs-sourcedata-miro",
    FilterExpression="isClearedForCatalogueAPI = :true",
    ExpressionAttributeValues={":true": {"BOOL": True}}
)

Closing as WONTFIX, in practice this wouldn't super useful – better off to supply a list of explicit IDs to the "start reindex" script.


Add GaugeIcon

Overview

PR Checklist

[ ] Related to designs:
[ ] Related to JIRA ticket: [ABC-123]
[ ] I have run this code to verify it works
[ ] This PR includes unit tests for the code change
[ ] This PR includes testing instructions tests for the code change
[ ] The alpha package of this PR is passing end-to-end tests in all relevant Codecademy repositories

Testing Instructions

Don't make me tap the sign.
PR Links and Envs

Repository
PR Link
PR Env

Monolith
Monolith PR
Monolith Env

Portal
Portal Link
Portal Env

Another Repo
Another Link
Another Env

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

I think this icon was just added the other day? See GaugeDashboardIcon https://gamut.codecademy.com/?path=/docs/atoms-icons-regular--editable-icon