Cannot concatenate dask.arrays that have a string index and float vectors.

This is based on example code from here: http://dask.pydata.org/en/latest/array-creation.html#concatenation-and-stacking
h5_files = glob.glob('*.h5')
datasets = [ h5py.File(fn)['/data/table'] for fn in h5_files ]
arrays = [ da.from_array(data, chunks=1000) for data in datasets ]
X = da.concatenate(arrays, axis=0)

The individual datasets are loaded just fine, but when trying to concatenate them I get the following error:
TypeError: invalid type promotion
I think this is due to dask.array not being able to concatenate a string-based index with float vectors (which is the case for my HDF5 files), as also hinted at here: https://github.com/pydata/xarray/issues/864
This is an excerpt when printing the arrays:
dask.array<array-c..., shape=(8000,), dtype=[('index', 'S50'), ('values_block_0', '<f4', (2048,))], chunksize=(1000,)>
dask.array<array-7..., shape=(839,), dtype=[('index', 'S50'), ('values_block_0', '<f4', (2048,))], chunksize=(839,)>
dask.array<array-5..., shape=(8000,), dtype=[('index', 'S50'), ('values_block_0', '<f4', (2048,))], chunksize=(1000,)>
dask.array<array-b..., shape=(1288,), dtype=[('index', 'S50'), ('values_block_0', '<f4', (2048,))], chunksize=(1000,)>
dask.array<array-3..., shape=(8000,), dtype=[('index', 'S50'), ('values_block_0', '<f4', (2048,))], chunksize=(1000,)>
dask.array<array-4..., shape=(6317,), dtype=[('index', 'S50'), ('values_block_0', '<f4', (2048,))], chunksize=(1000,)>

Surely, this must be easy to fix?

I'm unable to replicate. Can you create a reproducible example to help us debug? The following passes fine:
import h5py
import dask.array as da
import numpy as np

shapes = [8000, 839, 8000]
h5_files = []

for i, shape in enumerate(shapes):
    fn = 'test_{0}.hdf5'.format(i)

    x = np.ones(shape, dtype=[('index', 'S50'), ('values_block_0', '<f4', (2048,))])
    x['index'] = 'foobar'

    with h5py.File(fn) as f:
        f['/data/table'] = x

    h5_files.append(fn)

datasets = [h5py.File(fn)['/data/table'] for fn in h5_files]
arrays = [da.from_array(data, chunks=1000) for data in datasets]
X = da.concatenate(arrays, axis=0)

@jcrist You are correct, this works. My problem was that one HDF5 file had a different length of the string index, S52, whereas the others were S50. I had to coerce the h5py.Dataset.dtype in order to concatenate the arrays. I am leaving my solution here, in case someone else encounters the same problem. I am not closing the issue now, to hear if you have a better suggestion.
import glob

import dask.array as da
import h5py

class my_dataset(h5py.Dataset):
    def __init__(self, dataset, dtype):
        super(my_dataset, self).__init__(dataset.id)
        self._dtype = dtype

    @property
    def dtype(self):
        return self._dtype

h5_files = glob.glob('*.h5')  # HDF5 files with different length of string index

datasets = [ h5py.File(fn)['/data/table'] for fn in h5_files ]
datasets = [ my_dataset(data, [('index', 'S52'), ('values_block_0', '<f4', (2048,))]) for data in datasets ]

arrays = [ da.from_array(data, chunks=1000) for data in datasets ]
X = da.concatenate(arrays, axis=0)

print X

Hmmm, does the subclass result in h5py reading into a numpy array of the passed in dtype? If so, this seems fine. A simpler solution would be to cast the arrays to the proper type after loading:
import glob

import dask.array as da
import h5py

h5_files = glob.glob('*.h5')  # HDF5 files with different length of string index
datasets = [h5py.File(fn)['/data/table'] for fn in h5_files]
dtype = [('index', 'S52'), ('values_block_0', '<f4', (2048,))]
arrays = [da.from_array(data, chunks=1000).astype(dtype) for data in datasets]
X = da.concatenate(arrays, axis=0)

Glad you figured it out, closing for now.

Not only is your solution much more elegant @jcrist, it also works. My solution had some more bugs that I stopped solving once I tested your proposal. The preferred way to solve this problem would be your way.


Files not always being put to s3

using put from an ec2 to s3, I think we are seeing a hidden error that is not showing up, trying to upload a large number of files, some uploaded but only about half, files are about 5M, no errors being reported from
#!/bin/bash
s3-copy.sh dest src ...
here=${0%/*}
source $here/job.conf
$scl
dir=$1
shift
for file in "$@"
do
if [ -e "$file" ];
then
base=basename $file
s4cmd put --API-ContentEncoding=gzip --API-ContentType=text/tab-separated-values  $file s3://$hiveBucket$hiveBucketPrefix/$1/$base
fi
done

Hey,
I am unable to reproduce this issue. Can you let me know the following:

How many files you have
What is the content in the files? Are they all TSVs? Are they compressed?
What version of python you are running s4cmd on
What OS are you running on
How was s4cmd installed? pip/apt-get/built from code

We can take a look at this with more information :+1:


Decode CSV as Dict String (List String) and/or List (Dict String String)

Use case: I'dl ike providing my users a way to "explore" a CSV with "headers". So from a CSV which looks like:
name,first name,age
sheep,seb,18
hicks,brian,21

I'd like having either data by column:
Dict.fromList  [("name", ["sheep", "hicks"]), ("first name", ["seb", "brian"]), ("age", ["18", "21"])]

or by row:
[ Dict.fromList  [("name", "sheep"), ("first name", "seb"), ("age", "18")]
, Dict.fromList  [("name", "hicks"), ("first name", "brian"), ("age", "21")]
]

This falls squarely in the "use Csv.Parser" bucket of things you can do with this library. The functions in there say to open an issue if you need them—I guess you already saw that! If this turns out to be a common need, we can add it, so thank you for saying something!

Yes, I saw this sentence saying to open an issue, that's why I did it :)

I would need this exact thing. We have an application which lets users type in arbitrary combinations of fields
field1, field2, ... fieldn
value1, value2, ... valuen
value1, value2, ... valuen

These fields ultimately become input to another library. There are a lot of fields in question (>4000), so users specify a subset of them. Here's the function I wrote that does that. Perhaps it could be dropped into the library with some modifications.
import Csv.Parser as P
import List.Extra exposing (transpose)
import Dict exposing (Dict)

getDict : String -> Result P.Problem (Dict String (List String))
getDict s =
    let
        -- split the first row of data as headers
        zipFirstRow : List (List a) -> Result P.Problem (List ( a, List a ))
        zipFirstRow lst =
            case lst of
                [] ->
                    -- TODO: add this case to P.Problem, or create a new error type
                    Err (P.NoHeaderRow)
                xs :: xss ->
                    Ok (List.map2 Tuple.pair xs (transpose xss))
    in
    s
    |> P.parse { fieldSeparator = ',' }
    |> Result.andThen zipFirstRow
    |> Result.map (Dict.fromList)

Personally, I actually parse values out into a custom type, which makes things more complicated as I have to wrap errors all over the place. It would be great if this library could handle it by passing a function! I believe it would need a (P.Problem -> error) parameter to handle this.
-- Define custom error type to hold all errors
type MyError
  = NoHeader
  | CsvParseError P.Problem
  | MyParserError (List (List MyTypeParseError))

type MyTypeParseError = ... -- Errors for turning a String into your type
type MyType = ... -- Whatever you need it to be

parseIntoMyType : List (String, List String) -> Result MyError (List (String, List MyType))
-- define this how you need it to be
-- I found it useful to write
-- (String, List String) -> Result (List MyTypeParseError) MyType
-- and then collect the errors into the variant
-- MyParserError (List (List MyTypeParseError))

getDict : String -> Result MyError (Dict String (List MyType))
getDict s =
    let
        -- split the first row of data as headers
        zipFirstRow : List (List a) -> Result MyError (List ( a, List a ))
        zipFirstRow lst =
            case lst of
                [] ->
                    Err NoHeader
                xs :: xss ->
                    Ok (List.map2 Tuple.pair xs (transpose xss))
    in
    s
    |> P.parse { fieldSeparator = ',' }
    |> Result.mapError CsvParseError
    |> Result.andThen zipFirstRow
    |> Result.andThen parseIntoMyType
    |> Result.map Dict.fromList


(PUP-8133) Lookup bolt executor for file_upload function

Commit b6a166974 changed how bolt related run_* functions lookup the executor,
but I forgot to apply the same change to the file_upload function.
/cc @hlindberg @thallgren

While this fixes the immediate issue, I ran into problems when trying to execute a plan using the file_upload function.
plan sample::single_task(String $nodes) {
  $node_array = split($nodes, ',')
  file_upload(
    '/Users/josh/work/bolt/src.txt', 'dst.txt', $node_array
  )
}

I updated bolt's vendored puppet to my feature branch, and I get the following error:
$ bx bolt plan run sample::single_task --modulepath ./spec/fixtures/modules nodes=wmfg6hcjse7suul.delivery.puppetlabs.net
bundler: failed to load command: bolt (/Users/josh/work/bolt/.bundle/bin/bolt)
NoMethodError: undefined method `module_for_type' for nil:NilClass
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/types/p_object_type_extension.rb:25:in `create'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/types/puppet_object.rb:18:in `_pcore_type'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/types/type_formatter.rb:359:in `string_PuppetObject'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/visitor.rb:48:in `block in visit_this'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/visitor.rb:42:in `each'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/visitor.rb:42:in `visit_this'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/visitor.rb:82:in `visit_this_0'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/types/type_formatter.rb:114:in `append_string'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/types/type_formatter.rb:751:in `block in append_hash'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/types/type_formatter.rb:740:in `each'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/types/type_formatter.rb:740:in `append_hash'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/types/type_formatter.rb:571:in `string_Hash'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/visitor.rb:48:in `block in visit_this'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/visitor.rb:42:in `each'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/visitor.rb:42:in `visit_this'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/visitor.rb:82:in `visit_this_0'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/types/type_formatter.rb:114:in `append_string'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/types/type_formatter.rb:364:in `string_PuppetObject'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/visitor.rb:48:in `block in visit_this'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/visitor.rb:42:in `each'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/visitor.rb:42:in `visit_this'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/visitor.rb:82:in `visit_this_0'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/types/type_formatter.rb:114:in `append_string'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/types/type_formatter.rb:52:in `string'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/types/type_formatter.rb:17:in `string'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/pops/types/puppet_object.rb:36:in `to_s'
  /Users/josh/work/bolt/lib/bolt/cli.rb:496:in `puts'
  /Users/josh/work/bolt/lib/bolt/cli.rb:496:in `puts'
  /Users/josh/work/bolt/lib/bolt/cli.rb:496:in `block (2 levels) in run_plan'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet_pal.rb:206:in `block in with_loaded_environment'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet/context.rb:65:in `override'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet.rb:263:in `override'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet_pal.rb:200:in `with_loaded_environment'
  /Users/josh/work/bolt/vendored/puppet/lib/puppet_pal.rb:99:in `in_tmp_environment'
  /Users/josh/work/bolt/lib/bolt/cli.rb:495:in `block in run_plan'

CLA signed by all contributors.

I merged this PR. Will create a second PR for the NoMethodError.

I'm not able to reproduce the NoMethodError using the latest from bolt and feature_puppet-tasks. @joshcooper, what branches were you using? Can you still reproduce?


enable pin_memory for DataLoaders

To avoid the creation of a cuda:0 context, I needed to make sure that the BackgroundConsumer thread had its cuda device context set to the correct GPU.

For my learning, what is cuda:0 context?

In torch, CUDA GPU0 is referred to as device="cuda:0" so I really meant a CUDA context for GPU 0. pin_memory will create a context for the current CUDA device which defaults to GPU 0. To avoid this, you can change the default device via torch.set_device(). The problem is the the default device is thread-specific, so new thread fallback to GPU 0 unless you call torch.set_device() on the thread.
Some more context:
https://github.com/pytorch/pytorch/issues/58626
https://github.com/pytorch/pytorch/issues/32424


Consistent package names

Currently have

GeodesicRendering.jl
GeodesicTracer.jl

Should have the same language in package names, e.g. GeodesicTracing.jl, or conversely GeodesicRenderer.jl.

Going to refactor the whole project into Gradus.jl very soon, so this can be closed.


fixed border on the right of viewport

(merge fast pls ;) )
mentiend in the discussion of
#11207
seen here

@akien-mga cherry picked another change into this pr.
I can remove It but I was really curious what happens when one of them gets merged...
maybe someone can also just tell me.
I want to know how it behaves, since that would allow me to cherry pick a relevant commit from another pr to get rid of potential merge conflicts.

So, should we merge this PR ?

@groud so this is basically not a really good idea...
I didn't notice yet, that cherry-picked commits DO have different hashes.
hm...
apparently rebase would work to get rid of the cherry picked commit... but therefore the other commit has to be in master before.
ill just remove it and then we can merge...

@ground fixed and ready to merge.


Theoretical and practical documentation for using transform matrices in Godot

I can't reply to #17949 either because it's closed and gives me "You can't comment at this time. "...
So, anyway, replying to @reduz there:

Working with transforms directly is clearly difficult for a lot of people. We should really document this better.

Yeah, I completely agree, we ideally need a document that covers 1) all practical use cases in a shallow language 2) a mathematical text for those who'd like to understand the shallow language.
I mentioned this multiple occasions, I do have an incomplete write up (it's a newly created rst file, so it doesn't conflict with any existing docs.), and it's been sitting like that for a while, I can clean it up a little and dump whatever I currently have as a PR this weekend. I can send further PRs toward making it a complete guide. Would that be acceptable?
About what I currently have:
The math part currently includes a very brief introduction to Lie groups and Lie algebras, representation theory , a review of SO(2)/U(1) isomorphism (real vs complex representation of), and finally SO(3)/SU(2)/Spin(3) (real vs complex), and there's also a section about parametrization of the group elements (axis-angle for general Lie groups, and Euler angles for SO(3)). I also have a bare-bones section about SLERP.
I know, it probably sounds too technical due to many mathematical terms, but I tried write it in a totally non-technical and self-contained way, such that someone who only knows high school math can understand it.
The practical use text is a bit weak right now; I have a couple of tables along the lines "use this if you want to achieve this", and a table which clarifies representation vs parametrization, but we can hopefully improve that based on feedback and issue reports like this.
So let me know if this makes sense to you or not.

The number of equations that I have is probably measured in hundreds so the problem is quite magnified

I'm quite looking forward to reading all this and freshen up a bit my math-fu, I've grown rusty :D

I mean each equation is rendered and stored as a new raster image file. So if you have 100 equations you'll need to commit those 100 files, and the file is the hash of the equation so whenever you change one you need to find the old file, remove it from git, and commit the new file which doesn't scale with the number of equations and is a terrible experience. It also looks blurry and doesn't scale on screen either.

I dealt with a similar system like 15 years ago and it sucked. Mathjax was a revolution in that regard

I mean each equation is rendered and stored as a new raster image file. So if you have 100 equations you'll need to commit those 100 files, and the file is the hash of the equation so whenever you change one you need to find the old file, remove it from git, and commit the new file which doesn't scale with the number of equations and is a terrible experience.

Ah I see what you mean. It's not a problem in our case as we only commit source files, Sphinx and ReadTheDocs take care of building the HTML (and here the PNGs) for each commit.
Still, if only for the better visual output I'm happy to switch to mathjax.

An alternative is katex but it doesn't look like they have a plug-in for it

Ah, I see. So they probably remove all the old files and do a fresh build every time.
Yeah, apart from the visuals, there's also the fact that an equation is supposed to be a text (which you can select/copy), not an image.

I'm quite looking forward to reading all this and freshen up a bit my math-fu, I've grown rusty :D

It'd be great to have some feedback, so please do tell about it if you read it this weekend. It'll hopefully make quaternions and SLERP boring and non-magical for any reader.


Group by not working

Why something like this always return 0 for the cnt?
    public IAsyncEnumerable<AreaSummary> GetAreaAsync() =>
          _connection.ReadAsync(@"
                    select Description,  count(*) as cnt
                  From A
                  inner join lookup L2
                  on A.areaID= L2.ID
	  Group by  L2.Description

				  order by L2.Description")
            .Select<AreaSummary>()

        .SelectAwait(async u =>
        {
            await Task.Delay(1);
            return u;
        });

and
public class AreaSummary
{
    public string Description { get; set; }

    public int sum { get; set; }

}

Thanks

Okay, got it figure out.  Have to change the sum to cnt to match with the sql statement
public class AreaSummary
{
public string Description { get; set; }
public int cnt { get; set; }

}

And you don't have to use lowercase to match property name since mapper ignores case. You can Use Cnt to make it more consistent.


Option to rotate webcam

I use octoprint with a webcam that has portrait format instead of vertical format, so it looks really weird when I toggle fullscreen. Heres a screenshot:

Would it be possible to add the option to rotate the webcam/video so it can fit the screen?

Hey @vordenken,
I wonder if going to OctoPrint's web UI under Settings --> Webcam & Timelapse and check Rotate webcam 90 degrees will help solve this. Not sure but worth a try. OctoPod follows the configuration that you have for the webcam in OctoPrint.
Thanks,
Gaston

Hi @gdombiak ,
I already use this feature to make my webcam show up in the right direction within octoprint. Here's what it looks like:

I think OctoPod also displays it "correct" in that matter, but because my webcam is in portrait mode, this big grey bars appear on the side. It would be awesome to rotate the webcam within OctoPod only, so it uses the whole screen.

I agree, my cam view acts exactly the same way.
If i tap it to enlarge it turns an gets letterboxes over 60% of the screen.
It should not rotate if the cam is set to rotate in octoprint.

This happens when i tap the videobox:


Injection of istio sidecar proxy not working

Bug description
Manual injection of sidecar proxy not working.  Can someone please help me with this?
[ec2-user@avd-4ece ~]$ kubectl get pod -n myapp-test
NAME                                     READY     STATUS       RESTARTS   AGE
myapp-deployment-57564dcfd5-zjmjw   0/2       Init:Error   6          6m8s

Affected product area (please put an X in all that apply)
[ ] Configuration Infrastructure
[ ] Docs
[ X ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
Manual injection of sidecar proxy should work.
Steps to reproduce the bug
 istioctl kube-inject -f myapp.yml | kubectl apply -f -

[ec2-user@avd-4ece ~]$ kubectl logs -f myapp-deployment-57564dcfd5-zjmjw -myapp-test -c istio-init
Invalid option: -z
/usr/local/bin/istio-iptables.sh -p PORT -u UID -g GID [-m mode] [-b ports] [-d ports] [-i CIDR] [-x CIDR] [-k interfaces] [-t] [-h]

  -p: Specify the envoy port to which redirect all TCP traffic (default $ENVOY_PORT = 15001)
  -u: Specify the UID of the user for which the redirection is not
      applied. Typically, this is the UID of the proxy container
      (default to uid of $ENVOY_USER, uid of istio_proxy, or 1337)
  -g: Specify the GID of the user for which the redirection is not
      applied. (same default value as -u param)
  -m: The mode used to redirect inbound connections to Envoy, either "REDIRECT" or "TPROXY"
      (default to $ISTIO_INBOUND_INTERCEPTION_MODE)
  -b: Comma separated list of inbound ports for which traffic is to be redirected to Envoy (optional). The
      wildcard character "*" can be used to configure redirection for all ports. An empty list will disable
      all inbound redirection (default to $ISTIO_INBOUND_PORTS)
  -d: Comma separated list of inbound ports to be excluded from redirection to Envoy (optional). Only applies
      when all inbound traffic (i.e. "*") is being redirected (default to $ISTIO_LOCAL_EXCLUDE_PORTS)
  -i: Comma separated list of IP ranges in CIDR form to redirect to envoy (optional). The wildcard
      character "*" can be used to redirect all outbound traffic. An empty list will disable all outbound
      redirection (default to $ISTIO_SERVICE_CIDR)
  -x: Comma separated list of IP ranges in CIDR form to be excluded from redirection. Only applies when all
      outbound traffic (i.e. "*") is being redirected (default to $ISTIO_SERVICE_EXCLUDE_CIDR).
  -o: Comma separated list of outbound ports to be excluded from redirection to Envoy (optional).
  -k: Comma separated list of virtual interfaces whose inbound traffic (from VM)
      will be treated as outbound (optional)
  -t: Unit testing, only functions are loaded and no other instructions are executed.

Using environment variables in $ISTIO_SIDECAR_CONFIG (default: /var/lib/istio/envoy/sidecar.env)

Deployment Event log:

Name:               myapp-deployment-57564dcfd5-zjmjw
Namespace:          myapp-test
Priority:           0
PriorityClassName:  <none>
Node:               ip-10-24-79-104.ec2.internal/<IP_ADDRESS>
Start Time:         Thu, 22 Aug 2019 13:07:41 -0400
Labels:             app=myapp
                    pod-template-hash=57564dcfd5
Annotations:        sidecar.istio.io/inject=true
                    sidecar.istio.io/status={"version":"3936b28ea132ab7ecf98777374e289c5db57e64849528c28dc4de59dead46b10","initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-certs...
Status:             Pending
IP:                 <IP_ADDRESS>
Controlled By:      ReplicaSet/myapp-deployment-57564dcfd5
Init Containers:
  istio-init:
    Container ID:  docker://b5552d86cc9874f22915c0e7c3c6cca79c7afce76c5387bc7d1670b3c0ced620
    Image:         ecr:istio-proxy-init-1.2.4
    Image ID:      docker-pullable://....
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/local/bin/istio-iptables.sh
      -p
      15001
      -z
      15006
      -u
      1337
      -m
      REDIRECT
      -i
      *
      -x

      -b
      8001,8000
      -d
      15020
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 22 Aug 2019 13:09:12 -0400
      Finished:     Thu, 22 Aug 2019 13:09:12 -0400
    Ready:          False
    Restart Count:  4
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:        10m
      memory:     10Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-b2qcl (ro)
Containers:
  myapp:
    Container ID:
    Image:         ecr:myapp-5bcbbc0
    Image ID:
    Ports:         8001/TCP, 8000/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      python
      -u
      myapp.py
      --grpc-port
      8000
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  1000Mi
    Requests:
      cpu:      500m
      memory:   1000Mi
    Readiness:  tcp-socket :8000 delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      vertical-config  ConfigMap  Optional: false
    Environment:       <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-b2qcl (ro)
  istio-proxy:
    Container ID:
    Image:         ecr:istio-proxyv2-1.2.4
    Image ID:
    Port:          15090/TCP
    Host Port:     0/TCP
    Args:
      proxy
      sidecar
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --configPath
      /etc/istio/proxy
      --binaryPath
      /usr/local/bin/envoy
      --serviceCluster
      myapp.$(POD_NAMESPACE)
      --drainDuration
      45s
      --parentShutdownDuration
      1m0s
      --discoveryAddress
      istio-pilot.istio-system:15011
      --zipkinAddress
      zipkin.istio-system:9411
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --connectTimeout
      10s
      --proxyAdminPort
      15000
      --concurrency
      2
      --controlPlaneAuthPolicy
      MUTUAL_TLS
      --dnsRefreshRate
      300s
      --statusPort
      15020
      --applicationPorts
      8001,8000
      --controlPlaneBootstrap=false
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   128Mi
    Readiness:  http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
    Environment:
      POD_NAME:                          myapp-deployment-57564dcfd5-zjmjw (v1:metadata.name)
      POD_NAMESPACE:                     myapp-test (v1:metadata.namespace)
      INSTANCE_IP:                        (v1:status.podIP)
      HOST_IP:                            (v1:status.hostIP)
      ISTIO_META_POD_NAME:               myapp-deployment-57564dcfd5-zjmjw (v1:metadata.name)
      ISTIO_META_CONFIG_NAMESPACE:       myapp-test (v1:metadata.namespace)
      ISTIO_META_INTERCEPTION_MODE:      REDIRECT
      ISTIO_META_INCLUDE_INBOUND_PORTS:  8001,8000
      ISTIO_METAJSON_ANNOTATIONS:        {"sidecar.istio.io/inject":"true"}

      ISTIO_METAJSON_LABELS:  {"app":"myapp"}

    Mounts:
      /etc/certs/ from istio-certs (ro)
      /etc/istio/proxy from istio-envoy (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-b2qcl (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  istio-envoy:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  Memory
  istio-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio.default
    Optional:    true
  default-token-b2qcl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-b2qcl
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age              From                                   Message
  ----     ------     ----             ----                                   -------
  Normal   Scheduled  2m               default-scheduler                      Successfully assigned myapp-test/myapp-deployment-57564dcfd5-zjmjw to ip-10-24-79-104.ec2.internal
  Normal   Created    2m (x4 over 2m)  kubelet, ip-10-24-79-104.ec2.internal  Created container
  Normal   Started    2m (x4 over 2m)  kubelet, ip-10-24-79-104.ec2.internal  Started container
  Warning  BackOff    1m (x7 over 2m)  kubelet, ip-10-24-79-104.ec2.internal  Back-off restarting failed container
  Normal   Pulling    1m (x5 over 2m)  kubelet, ip-10-24-79-104.ec2.internal  pulling image "ecr:istio-proxy-init-1.2.4"
  Normal   Pulled     1m (x5 over 2m)  kubelet, ip-10-24-79-104.ec2.internal  Successfully pulled image "ecr:istio-proxy-init-1.2.4"

Version (include the output of istioctl version --remote and kubectl version)
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"753b2dbc622f5cc417845f0ff8a77f539a4213ea", GitTreeState:"clean", BuildDate:"2018-12-06T01:33:57Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.10-eks-2e569f", GitCommit:"2e569fd887357952e506846ed47fc30cc385409a", GitTreeState:"clean", BuildDate:"2019-07-25T23:13:33Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
client version: 1.2.4
citadel version: 1.2.4
galley version: 1.2.4
ingressgateway version: 94746ccd404a8e056483dd02e4e478097b950da6-dirty
pilot version: 1.2.4
policy version: 1.2.4
policy version: 1.2.4
sidecar-injector version: 1.2.4
telemetry version: 1.2.4

How was Istio installed?
https://github.com/istio/installer
NAME                                     READY     STATUS    RESTARTS   AGE
istio-citadel-bcc5cc6cf-db2kh            1/1       Running   0          7h20m
istio-galley-55669498c-bjtsd             2/2       Running   0          7h12m
istio-ingressgateway-d48c8cb9f-47s29     1/1       Running   0          5h10m
istio-pilot-5f775b764c-8ww7l             2/2       Running   0          6h52m
istio-policy-84b675694-4c58q             2/2       Running   0          6h50m
istio-policy-84b675694-n7g7q             2/2       Running   0          144m
istio-sidecar-injector-d95d9dccb-jwdrt   1/1       Running   0          3h6m
istio-telemetry-6c74f64b7c-frv2g         2/2       Running   0          6h41m
istio-tracing-78cc9cc55-rng2k            1/1       Running   0          6h42m

Environment where bug was observed (cloud vendor, OS, etc)
AWS EKS 1.12.7
.

I think it is because -z option is new and not in Istio 1.2

okk so I used latest image of istio proxy it and now manual injection works. But somehow automatic injection is not working. Any idea?
[ec2-user@avd-4ece ~]$ kubectl label namespace myapp-test istio-injection=enabled --overwrite

Injector pod logs:

2019-08-23T07:40:03.604108Z     info    AdmissionReview for Kind=/v1, Kind=Pod Namespace=myapp-test Name= (myapp-deployment-744f4bccd4-***** (actual name not yet known)) UID=38321851-c579-11e9-b45b-1286acc18274 Rfc6902PatchOperation=CREATE UserInfo={system:unsecured  [system:masters system:authenticated] map[]}
2019-08-23T07:40:03.604137Z     debug   Object: {"kind":"Pod","apiVersion":"v1","metadata":{"generateName":"myapp-deployment-744f4bccd4-","creationTimestamp":null,"labels":{"app":"myapp","pod-template-hash":"744f4bccd4"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"myapp-deployment-744f4bccd4","uid":"06b7acc3-c579-11e9-b45b-1286acc18274","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-b2qcl","secret":{"secretName":"default-token-b2qcl"}}],"containers":[{"name":"myapp","image":"ecr:myapp-5bcbbc0","command":["python","-u","myapp.py","--grpc-port","8000","--vertical-name","myapp_current"],"ports":[{"name":"admin","containerPort":8001,"protocol":"TCP"},{"name":"api","containerPort":8000,"protocol":"TCP"}],"envFrom":[{"configMapRef":{"name":"vertical-config"}}],"resources":{"limits":{"cpu":"500m","memory":"1000Mi"},"requests":{"cpu":"500m","memory":"1000Mi"}},"volumeMounts":[{"name":"default-token-b2qcl","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"readinessProbe":{"tcpSocket":{"port":8000},"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0},"status":{}}
2019-08-23T07:40:03.604143Z     debug   OldObject:
2019-08-23T07:40:03.604168Z     debug   Sidecar injection policy for myapp-test/myapp-deployment-744f4bccd4-***** (actual name not yet known): namespacePolicy:enabled useDefault:true inject:false required:true sidecar.istio.io/proxyImage:(unset) readiness.status.sidecar.istio.io/periodSeconds:(unset) readiness.status.sidecar.istio.io/failureThreshold:(unset) traffic.sidecar.istio.io/excludeInboundPorts:(unset) traffic.sidecar.istio.io/kubevirtInterfaces:(unset) sidecar.istio.io/inject:(unset) status.sidecar.istio.io/port:(unset) traffic.sidecar.istio.io/includeOutboundIPRanges:(unset) sidecar.istio.io/rewriteAppHTTPProbers:(unset) sidecar.istio.io/interceptionMode:(unset) readiness.status.sidecar.istio.io/initialDelaySeconds:(unset) traffic.sidecar.istio.io/includeInboundPorts:(unset) sidecar.istio.io/status:(unset) traffic.sidecar.istio.io/excludeOutboundIPRanges:(unset) traffic.sidecar.istio.io/excludeOutboundPorts:(unset) readiness.status.sidecar.istio.io/applicationPorts:(unset)
2019-08-23T07:40:03.607939Z     info    AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"ecr:istio-proxy-init-master-daily","command":["/usr/local/bin/istio-iptables.sh","-p","15001","-z","15006","-u","1337","-m","REDIRECT","-i","*","-x","","-b","8001,8000","-d","15020"],"resources":{"limits":{"cpu":"100m","memory":"50Mi"},"requests":{"cpu":"10m","memory":"10Mi"}},"imagePullPolicy":"Always","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"runAsUser":0,"runAsNonRoot":false}}]},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"ecr:istio-proxyv2-1.2.4","args":["proxy","sidecar","--domain","$(POD_NAMESPACE).svc.cluster.local","--configPath","/etc/istio/proxy","--binaryPath","/usr/local/bin/envoy","--serviceCluster","myapp.$(POD_NAMESPACE)","--drainDuration","45s","--parentShutdownDuration","1m0s","--discoveryAddress","istio-pilot.istio-system:15011","--zipkinAddress","zipkin.istio-system:9411","--proxyLogLevel=warning","--proxyComponentLogLevel=misc:error","--connectTimeout","10s","--proxyAdminPort","15000","--concurrency","2","--controlPlaneAuthPolicy","MUTUAL_TLS","--dnsRefreshRate","300s","--statusPort","15020","--applicationPorts","8001,8000","--controlPlaneBootstrap=false"],"ports":[{"name":"http-envoy-prom","containerPort":15090,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"fieldPath":"status.podIP"}}},{"name":"HOST_IP","valueFrom":{"fieldRef":{"fieldPath":"status.hostIP"}}},{"name":"ISTIO_META_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"ISTIO_META_CONFIG_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"ISTIO_META_INTERCEPTION_MODE","value":"REDIRECT"},{"name":"ISTIO_META_INCLUDE_INBOUND_PORTS","value":"8001,8000"},{"name":"ISTIO_METAJSON_LABELS","value":"{\"app\":\"myapp\",\"pod-template-hash\":\"744f4bccd4\"}\n"}],"resources":{"limits":{"cpu":"2","memory":"1Gi"},"requests":{"cpu":"100m","memory":"128Mi"}},"volumeMounts":[{"name":"istio-envoy","mountPath":"/etc/istio/proxy"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs/"},{"name":"default-token-b2qcl","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"readinessProbe":{"httpGet":{"path":"/healthz/ready","port":15020},"initialDelaySeconds":1,"periodSeconds":2,"failureThreshold":30},"imagePullPolicy":"Always","securityContext":{"runAsUser":1337,"runAsGroup":1337,"readOnlyRootFilesystem":true}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-envoy","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-certs","secret":{"secretName":"istio.default","optional":true}}},{"op":"add","path":"/spec/securityContext","value":{}},{"op":"add","path":"/metadata/annotations","value":{"sidecar.istio.io/status":"{\"version\":\"6cab5b52710d2bd9d9277823632b68b2948a53c2f9386ae0ae1dc0ee9254fd95\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}}]

injector logs look fine, general with injector issues they fail to even reach the injector and don't show up in logs. I assume you are seeing pods not being created? in this case I would do kubectl describe on the replicaset and it should give you more info

@howardjohn Thanks for pointing that out. Getting below error in replicaset:
Conditions:
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate
Events:
  Type     Reason        Age               From                   Message
  ----     ------        ----              ----                   -------
  Warning  FailedCreate  8s (x16 over 2m)  replicaset-controller  Error creating: Internal error occurred: failed calling admission webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-control.svc:443/inject?timeout=30s: service "istio-sidecar-injector" not found

I have everything installed in istio-system namespace but somehow the config is pointing to "istio-control".
I have installed Istio using below:
https://github.com/istio/installer
Can you please let me know from where is this config picked up?

I don't think it should do that anymore maybe you have an old config still
around. It's a mutating webhook config
On Sat, Aug 24, 2019, 12:26 PM Siddharth<EMAIL_ADDRESS>wrote:

@howardjohn https://github.com/howardjohn Thanks for pointing that out.
Getting below error in replicaset:
Conditions:
Type             Status  Reason

ReplicaFailure   True    FailedCreate
Events:
Type     Reason        Age               From                   Message

Warning  FailedCreate  8s (x16 over 2m)  replicaset-controller  Error creating: Internal error occurred: failed calling admission webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-control.svc:443/inject?timeout=30s: service "istio-sidecar-injector" not found
I have everything installed in istio-system namespace but somehow the
config is pointing to "istio-control".
I have installed Istio using below:
https://github.com/istio/installer
Can you please let me know from where is this config picked up?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/16481?email_source=notifications&email_token=AAEYGXIFKCKEGUGT5X55X3LQGGDPZA5CNFSM4IOXDHSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5CGBFY#issuecomment-524574871,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEYGXP3TO6UZYK4BZ5ORI3QGGDPZANCNFSM4IOXDHSA
.

@howardjohn Below is the config:
---
# Source: istio-autoinject/templates/mutatingwebhook.yaml

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: istio-sidecar-injector
  labels:
    app: sidecar-injector
    release: istio-system
webhooks:
  - name: sidecar-injector.istio.io
    clientConfig:
      service:
        name: istio-sidecar-injector
        namespace: istio-system
        path: "/inject"
      caBundle: ""
    rules:
      - operations: [ "CREATE" ]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    failurePolicy: Fail
    namespaceSelector:
      matchLabels:
        istio-injection: enabled
---

Not sure what is wrong here. I am using latest code:
https://github.com/istio/installer/tree/master/istio-control/istio-autoinject/templates

yeah I suspect you have 2 of them in your cluster though, that one and one from some older install or something. Try kubectl get MutatingWebhookConfiguration

@howardjohn Yup you are right. I can see now that mutating webhook config is independent of namespace and that's why an old config was still around. Injection works perfectly now. Thanks for the help. Closing the issue.

hello I am facing same issue can someone help me


Wrong charset for transactions table on mysql

Actual behavior
The charset of the transactions table doesn't accept transactions with special characters. This means that transactions like this, don't get picked up by Lisk-Core because of ł character.
Expected behavior
The transactions table should store transactions that contain special chars like ł. In
Steps to reproduce
Just run Lisk Service in docker with MySQL on docker.
Which version(s) does this affect? (Environment, OS, etc...)
commit c8360a8
How I (temporarily) solved it?
I fixed it by replacing the default charset from latin1 to utf8mb4_unicode_ci . This is the query ALTER TABLE transactions CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Dear @nerdvibe ,
Thank you for highlighting the issue.
Our team had earlier identified and fixed this issue as part of the PR https://github.com/LiskHQ/lisk-service/pull/436 and would be made available with the next beta.
Hence, closing this issue.


Provide option to allow webpack-dev-server

Add new npm script which will allow for live-reload via webpack-dev-server
While I was able to get this initially running, the live-reload was not reflecting the new changes in the browser

Live reload now added via webpack-dev-server, nice work @gah-boh Thanks!


placementVals.find(), never return null

Hi
This line is buggy, bc. find never return null value.
Its return undefined or value<T> of T[] array
https://github.com/mattlewis92/positioning/blob/7bafeb0a53b5f7695fa82e81a72000d70560670d/src/positioning.ts#L210
I suggest replace this line with
if (!placementVals.find(val => val.search('^' + obj) !== -1)) {

Please can you send a PR to the ng-bootstrap repo, as this is just a copy of the sources from there: https://github.com/ng-bootstrap/ng-bootstrap/blob/master/src/util/positioning.ts#L210
Thanks! 😄


Python: Bump Python project version to 1.6.0 for a release.

Motivation and Context
Bump Python project version to 1.6.0 for a release.

Description
Bump Python project version to 1.6.0 for a release.

Contribution Checklist

[X] The code builds clean without any errors or warnings
[X] The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
[X] All unit tests pass, and I have added new tests where possible
[X] I didn't break anyone :smile:

Python Unit Test Overview

Tests
Skipped
Failures
Errors
Time

2370
1 :zzz:
0 :x:
0 :fire:
1m 2s :stopwatch:


Plugin does not work in version 4.3 / 4.4

After upgrading to Camunda Modeler version 4.3 or 4.4, the plugin is not working. Any suggestions for updating?
Plugin works up to version 4.2.

Hi @passarela,
since this is a monorepo containing many projects, could you please be more specific about which sub-project your issue is about and what is the behavior?
Cheers,
Falko

Hi @falko
I refer to the Plugin
https://github.com/camunda-consulting/code/tree/master/snippets/camunda-modeler-plugins/camunda-modeler-plugin-usertask-generatedform-preview
Where it does not work in the current versions of modeler 4.3 and 4.4

@felix-mueller I think back in the days you wrote the generated form preview plugin. Do you have any idea on how to make it work for newer Modeler versions?

I faced the same problem and found the related issue:
https://github.com/camunda/camunda-modeler/issues/1992
Seems they are currently working on it.


Why don't publish to the marketplace?

https://developer.github.com/marketplace/actions/publishing-an-action-in-the-github-marketplace/

I'm curious about this too. Is there a maintenance cost that's holding it back (update deps, test, re-publish, after each change). Or is it just a low priority task? Could the community help with it?

Repos with multiple actions can't be published in the marketplace according to the docs at https://developer.github.com/marketplace/actions/publishing-an-action-in-the-github-marketplace/.

Publishing an action to the GitHub Marketplace requires that the action lives in your public repository's root directory and it exists alone in that repository without other actions.

Thank you for that explanation! I hope future updates to the marketplace will support monorepos!

What do you say about this:
https://github.com/marketplace/actions/net-core-cli
The fork:
https://github.com/baruchiro/github-actions

Note that there is also actions/setup-dotnet which installs the .NET Core SDK on the machine instead of running it in a Linux docker container. This allows for running unit tests natively on Windows/Linux/Mac in one build matrix.

The majority of the actions here are moved to accommodate the new changes.
All of them are available in the marketplace.
You can check out the new actions here: https://github.com/azure/actions


Make servlet URI configurable

There are two servlets for json and protobuf messages which URIs are hardcoded.

Done.