feat: create an Array3D column from a list of lists of arrays of dimension 1

Is your feature request related to a problem? Please describe.
It is possible to create an Array2D column from a list of arrays of dimension 1. Similarly, I think it might be nice to be able to create a Array3D column from a list of lists of arrays of dimension 1.
To illustrate my proposal, let's take the following toy dataset t:
import numpy as np
from datasets import Dataset, features

data_map = {
    1: np.array([[0.2, 0,4],[0.19, 0,3]]),
    2: np.array([[0.1, 0,4],[0.19, 0,3]]),
}

def create_toy_ds():
    my_dict = {"id":[1, 2]}
    return Dataset.from_dict(my_dict)

ds = create_toy_ds()

The following 2D processing works without any errors raised:
def prepare_dataset_2D(batch):
    batch["pixel_values"] = [data_map[index] for index in batch["id"]]
    return batch

ds_2D = ds.map(
    prepare_dataset_2D,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array2D(shape=(2, 3), dtype="float32")})
)

The following 3D processing doesn't work:
def prepare_dataset_3D(batch):
    batch["pixel_values"] = [[data_map[index]] for index in batch["id"]]
    return batch

ds_3D = ds.map(
    prepare_dataset_3D,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array3D(shape=(1,  2, 3, dtype="float32")})
)

The error raised is:
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
[<ipython-input-6-676547e4cd41>](https://localhost:8080/#) in <module>()
      3     batched=True,
      4     remove_columns=ds.column_names,
----> 5     features=features.Features({"pixel_values": features.Array3D(shape=(1, 2, 3), dtype="float32")})
      6 )

12 frames
[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1971                 new_fingerprint=new_fingerprint,
   1972                 disable_tqdm=disable_tqdm,
-> 1973                 desc=desc,
   1974             )
   1975         else:

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    518             self: "Dataset" = kwargs.pop("self")
    519         # apply actual function
--> 520         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    521         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    522         for dataset in datasets:

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    485         }
    486         # apply actual function
--> 487         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    488         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    489         # re-apply format to the output

[/usr/local/lib/python3.7/dist-packages/datasets/fingerprint.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    456             # Call actual function
    457
--> 458             out = func(self, *args, **kwargs)
    459
    460             # Update fingerprint of in-place transforms + update in-place history of transforms

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in _map_single(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2354                                 writer.write_table(batch)
   2355                             else:
-> 2356                                 writer.write_batch(batch)
   2357                 if update_data and writer is not None:
   2358                     writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py](https://localhost:8080/#) in write_batch(self, batch_examples, writer_batch_size)
    505             col_try_type = try_features[col] if try_features is not None and col in try_features else None
    506             typed_sequence = OptimizedTypedSequence(batch_examples[col], type=col_type, try_type=col_try_type, col=col)
--> 507             arrays.append(pa.array(typed_sequence))
    508             inferred_features[col] = typed_sequence.get_inferred_type()
    509         schema = inferred_features.arrow_schema if self.pa_writer is None else self.schema

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib._handle_arrow_array_protocol()

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py](https://localhost:8080/#) in __arrow_array__(self, type)
    175                     storage = list_of_np_array_to_pyarrow_listarray(data, type=pa_type.value_type)
    176                 else:
--> 177                     storage = pa.array(data, pa_type.storage_dtype)
    178                 return pa.ExtensionArray.from_storage(pa_type, storage)
    179

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Can only convert 1-dimensional array values

Describe the solution you'd like
No error in the second scenario and an identical result to the following snippets.
Describe alternatives you've considered
There are other alternatives that work such as:

def prepare_dataset_3D_bis(batch):
    batch["pixel_values"] = [[data_map[index].tolist()] for index in batch["id"]]
    return batch

ds_3D_bis = ds.map(
    prepare_dataset_3D_bis,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array3D(shape=(1, 2, 3), dtype="float32")})
)

or
def prepare_dataset_3D_ter(batch):
    batch["pixel_values"] = [data_map[index][np.newaxis, :, :] for index in batch["id"]]
    return batch

ds_3D_ter = ds.map(
    prepare_dataset_3D_ter,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array3D(shape=(1,  2, 3), dtype="float32")})
)

But both solutions require the user to be aware that data_map[index] is an np.array type.
cc @lhoestq as we discuss this offline :smile:

Hi @SaulLu, thanks for your proposal.
Just I got a bit confused about the dimensions...

For the 2D case, you mention it is possible to create an Array2D from a list of arrays of dimension 1
However, you give an example of creating an Array2D from arrays of dimension 2:

the values of data_map are arrays of dimension 2
the outer list in prepare_dataset_2D should not be taken into account in the dimension counting, as it is used because in map you pass batched=True

Note that for the 3D alternatives you mention:

In prepare_dataset_3D_ter, you create an Array3D from arrays of dimension 3:

the array data_map[index][np.newaxis, :, :] has dimension 3
the outer list in prepare_dataset_3D_ter is the one used by batched=True

In prepare_dataset_3D_bis, you create an Array3D from a list of list of lists:

the value of data_map[index].tolist() is a list of lists
it is enclosed by another list [data_map[index].tolist()], thus giving a list of list of lists
the outer list is the one used by batched=True

Therefore, if I understand correctly, your request would be to be able to create an Array3D from a list of an array of dimension 2:

In prepare_dataset_3D, data_map[index] is an array of dimension 2
it is enclosed by a list [data_map[index]], thus giving a list of an array of dimension 2
the outer list is the one used by batched=True

Please, feel free to tell me if I did not understand you correctly.

Hi @albertvillanova ,
Indeed my message was confusing and you guessed right :smile: : I think would be interesting to be able to create an Array3D from a list of an array of dimension 2.
For the 2D case I should have given as a "similar" example:

data_map_1D = {
    1: np.array([0.2, 0.4]),
    2: np.array([0.1, 0.4]),
}

def prepare_dataset_2D(batch):
    batch["pixel_values"] = [[data_map_1D[index]] for index in batch["id"]]
    return batch

ds_2D = ds.map(
    prepare_dataset_2D,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array2D(shape=(1, 2), dtype="float32")})
)


Can argument be as another aliase?

Question about usage of argument:
https://github.com/SimGus/Chatette/wiki/Generation-modifiers#argument
Example was provided in link:
~[greetings$NAME]
   Hi $NAME
   Hello $NAME!

I would like to use instead of $NAME another alias:
~[NAME]
  John
  Elvis

Following usage
~[greetings$~[NAME]]
cause SyntaxError: Invalid token
Is there a way to use alias as argument?

Hello!
There is currently no way to do that, though it's a feature I found myself needing as well.
I worked on implementing such a thing (there might still be a branch on this repo with what I tested out), but it was not clear what the best syntax for that would be: I think we'll agree ~[alias$~[some other alias]] is quite hard to read.
I'm still trying to figure out a good syntax, along with other questions (should I allow more than a single alias as an argument? should I allow nested arguments, i.e. an alias as an argument of another alias which is itself an argument to a third one? etc). Once I have an answer, I'll implement it (I'll try to do this shortly).
For the moment, you will need to write your template files in a different way, to avoid the need of an alias argument.
There are actually a few (arguably ugly) workaround you can use:

Use the alias directly instead of an argument:

~[greetings]
   Hi ~[NAME]
   Hello ~[NAME]!

instead of
~[greetings$NAME]
   Hi $NAME
   Hello $NAME!

Break your alias into a beginning and end, and use the alias (or whatever template) in between:

~[greetings-start]
   Hi
   Hello
~[greetings-end]
   !
   , how are you?
~[greetings]
   ~[greetings-start] ~[NAME] ~[greetings-end]

instead of
~[greetings$~[NAME]]
   Hi ~[NAME]!
   Hi ~[NAME], how are you?
   Hello ~[NAME]!
   Hello ~[NAME], how are you?

I know that none of those workarounds is fully satisfying, which is why I'm still willing to implement such a feature.
I hope this helps!

Hi @SimGus , thank you a lot for your quick reply!

Actually in my task I need to use nested alias. I mean structure like this:

~[NAME]
  ~[FIRST_NAME] ~[SURNAME]

~[FIRST_NAME]
  John
  Bill

~[SURNAME]
  Smith
  Brown

I think it will be also useful to add functionality to support entity as argument as well :

@[NAME]
  John
  Elvis

@[FIRST_NAME]
  John
  Bill

@[SURNAME]
  Smith
  Brown

~[greetings$NAME]
   Hi $NAME
   Hello $NAME!

So for different intents it can be used as:
%[intent_name]
    ~[greetings$@[NAME]]

%[intent_first_name]
    ~[greetings$@[FIRST_NAME]]

%[intent_surname]
    ~[greetings$@[SURNAME]]

Hi!
I see the kind of use case you have, it's quite similar to what I tried to do a few months back.
I still need to think a little bit more to find a good syntax for this, but once it's done I'll start implementing it.
I'll hit you up when I update the code, but I must warn you this is a quite large modification to the current codebase, so this might take time.
Thanks for your feedback :)


Flow cannot find property of subclass

Given this code (see also on flow.org/try: https://flow.org/try/#0PQKgBAAgZgNg9gdzCYAoVBjGBDAzrsAQTAG9UwwA7AJwC4qBXAWwCMBTa8sLjOS3AC7UGGAXGpgAFDQCUpLhQEALAJa4AdDTABeKtQDcXAL6oTmHPjAAhMGwAeAtpQAmBYmQpQ4ceoOorKAHMuHj4-ETEJaWoAGjAvODkPCjBcBgAHDmiZQxSwZTV1BJ1471ywIwpTdCw8Ais8NnkKFUcmekJQ-iEI8SlWtnaiJIV81Q0BphLJ8pMzWssGiXtHF3rG5rBebuFRPslJ+isRvLTM6gO2nOMQihYcKRlfIQDAzYpqNgEGakoxwsmRTKNyMQA)
/* @flow */

class A {
  nr: number

  constructor (nr) {
    this.nr = nr;
  }
}

class B extends A {
  foo: string

  constructor (nr, foo) {
    super(nr);
    this.foo = foo;
  }
}

class Base {
  item: A
  constructor (item: A) {
    this.item = item;
  }
}

class Bar extends Base {
  constructor (item: B) {
    super(item);
  }

  bla (): string {
    return this.item.foo;
  }
}

It reports the foo property within the bla-method as not found:
33:     return this.item.foo;
                         ^ property `foo`. Property not found in
33:     return this.item.foo;
               ^ A

I would expect that flow can infer foo as being a string, because the item is given in the constructor as being a type of B (which is a subclass of A). Also redeclaring item: B in class Bar leads to different errors.
This makes inheritance pretty hard to implement using flowtype.

item is explicitly annotated as A, so it's always going to be A. What you can do is make Base generic:
class Base<T> {
  item: T
  constructor (item: T) {
    this.item = item;
  }
}

In this case T is not a concrete type and is going to be inferred.

Thanks for the response.
That works around my problem. But to understand it: Without generics, it was never suppose to work?
If that's the case, you can close this issue.

Without generics, it was never suppose to work?

No, it wasn't.


[Vite] TailwindCSS 2.2 JIT no longer tracks updated Svelte files

Since 2.2 (tailwindlabs/tailwindcss#4514) the JIT engine switches to using the "tracking" context by default instead of "watching" context, which requires support in Vite. There's an issue about this filed at tailwindlabs/tailwindcss#4683 but I figure it's good to track this here as well.
Current workarounds:

Use TAILWIND_MODE=watch env var, which is ugly and not cross-platform without cross-env, and apparently doesn't trigger HMR
Start a separate tailwind process with npm-run-all
Downgrade tailwind to 2.1

Proper Fix:
https://github.com/vitejs/vite/commit/6eaec3ab74d126310d93f8a93f8577bed1c3f474 should in theory fix this, although I can confirm that the CI failing tests on Windows is not a fluke, I can replicate it on my machine.

Putting
const mode = process.env.NODE_ENV;
const dev = mode === "development";
process.env.TAILWIND_MODE = dev ? "watch" : "build";

I just want to mention that with vite 2.4+, tw JIT does not properly work without the mentioned code above. (Freshly installed sveltekit example with tailwind)

Putting
const mode = process.env.NODE_ENV;
const dev = mode === "development";
process.env.TAILWIND_MODE = dev ? "watch" : "build";

I just want to mention that with vite 2.4+, tw JIT does not properly work without the mentioned code above. (Freshly installed sveltekit example with tailwind)

svelte-add still includes this code for that reason

Ah ok, I somehow assumed that vite 2.4 should fix it 🙈


Bug:

PHP Version
8.0
CodeIgniter4 Version
4.1.5
Which operating systems have you tested for this bug?
Linux
Which server did you use?
apache
Database
MySQL 5.6
What happened?
See nothing using “production” because all errors and warnings blocked with error_reporting(0): set to zero.
Steps to Reproduce
Forced error and nothing showing in log files.
Expected Output
Expected errors and warnings to be shown in log file.
Anything else?
Raised detailed thread on forum with optional testing code and results:
https://forum.codeigniter.com/thread-80546.html

I can't reproduce. Or am I misunderstand something?
Add error to the controller.
--- a/app/Controllers/Home.php
+++ b/app/Controllers/Home.php
@@ -6,6 +6,6 @@ class Home extends BaseController
 {
     public function index()
     {
-        return view('welcome_message');
+        return view('welcome_message')
     }
 }

$ php spark serve

Navigate http://localhost:8080.
See the log:
CRITICAL - 2021-11-15 19:20:11 --> syntax error, unexpected token "}", expecting ";"
#0 /Users/kenji/work/codeigniter/CodeIgniter4/system/Autoloader/Autoloader.php(241): CodeIgniter\Autoloader\Autoloader->includeFile('/Users/kenji/wo...')
#1 /Users/kenji/work/codeigniter/CodeIgniter4/system/Autoloader/Autoloader.php(219): CodeIgniter\Autoloader\Autoloader->loadInNamespace('App\\Controllers...')
#2 [internal function]: CodeIgniter\Autoloader\Autoloader->loadClass('App\\Controllers...')
#3 /Users/kenji/work/codeigniter/CodeIgniter4/system/CodeIgniter.php(789): class_exists('\\App\\Controller...', true)
#4 /Users/kenji/work/codeigniter/CodeIgniter4/system/CodeIgniter.php(397): CodeIgniter\CodeIgniter->startController()
#5 /Users/kenji/work/codeigniter/CodeIgniter4/system/CodeIgniter.php(318): CodeIgniter\CodeIgniter->handleRequest(NULL, Object(Config\Cache), false)
#6 /Users/kenji/work/codeigniter/CodeIgniter4/public/index.php(37): CodeIgniter\CodeIgniter->run()
#7 /Users/kenji/work/codeigniter/CodeIgniter4/system/Commands/Server/rewrite.php(43): require_once('/Users/kenji/wo...')
#8 {main}

If you want to see all errors, you have to change the Boot file:
https://github.com/codeigniter4/CodeIgniter4/blob/0406780029a8f0e14fedb2eb6ac2aeda7b87f128/app/Config/Boot/production.php#L11
error_reporting(E_ALL);

Regret the delay, if I received a notification it was lost among all the other notifications.
I downloaded the latest CI4 Ver: 4.1.5 and noticed that the app/Config/Boot/production.php no longer has error_reporting(0)? I unsuccessfully searched for the Change Log update and could not find it?
Anyway my beef was that Production error_reporting used to be set to ZERO and the suggestion was to use CU_DEBUG to select PHP default error_logging by using the following changes,  notified in the forum Request:
https://forum.codeigniter.com/thread-80546.html
I have just tested the following proposal :
system/Debug/Exceptions -> public function initialize()
    {
      if( CI_DEBUG )  {
        set_exception_handler([$this, 'exceptionHandler']);
        set_error_handler([$this, 'errorHandler']);
        register_shutdown_function([$this, 'shutdownHandler']);
      }//endif;
    }

A memory saving spin-off of using the above was appreciable:
.env
  # DEFAULT production
  # CI_ENVIRONMENT = production
	# production  ==> 741,720 bytes - memory_get_usage()
	# production  ==> 746,424 bytes - memory_get_peak_usage()
  # CI_ENVIRONMENT = development
	# development ==> 831,792 bytes - memory_get_usage()
	# development ==> 836,496 bytes - memory_get_peak_usage()

app/Config/Boot/production.php was never changed with regard to the error_reporting() mask.
https://github.com/codeigniter4/CodeIgniter4/blob/8c7f70188e2bb7f19e6c7fcbbadb4dcb94c1cbc7/app/Config/Boot/production.php#L11
So it was never error_reporting(0); to begin with.


Has anyone gotten the MMU code to work on raspberry pi 4?

I've gotten most everything else working on raspberry pi 4, but so far can't get it to boot up when turning on the MMU.

Sent with GitHawk


create template for mentor profile

To create template for mentor profile and creativity way
It allow different types of color based themes displaying mentors profiles as user selections.
Showing the white background as default theme.

You mean that mentors can choose its own template for their profile ?
Not a static one but a dynamic one depending on mentor's chooses?

You mean that mentors can choose its own template for their profile ?
Not a static one but a dynamic one depending on mentor's chooses?

yes correct @sf3ris


Deserialization Relationship null (related to issue #70)

Hi, first for all, sorry for my english.
We are using your library to deserialization a json model that has null value relationships, (code below), a feature that is possible as of the latest version
"relationships": {
    "team": {
      "data": null
    }
}

We got an error (NPE) on method JsonApiEntityModelDeserializer#createRelationship because, according to that case, var "entry" is null. "data" atributte is null so there are no id or type attributes. Maybe is it need to be checked in the code or do we need to do something to prevent the library from trying to parse that case?
Thank you

Thanks for reporting this, I'll take a look at asap. What is your use case for creating an entity that does not belong to a team. Rather than specifying the relationship with "data":null why not delete the relationship part from the body?

We wanted to comply with the JSON:API specification which requires that if the relationship (to one) exists and there are no elements within it, the only valid relationship object is "data: null".
We also didn't want to manipulate the json file we received by removing parts of it. We just want to simplify the process of parsing the json file into our entity model by using the library. Another team, from the same project, is responsible for generating the json file.
Maybe for now, we should remove that part of the json file, to move forward with our project.
Thanks for your time.


⚠ Processors has degraded performance

In d6b155e, Processors ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 179 ms

Resolved: Processors performance has improved in 07518b5.


Feature/different deploy path

serve images from CDN
rewrite image paths prior to circle ci upload
standardize on ../assets/ syntax for images destined for retrieval from CDN

@denis-yuen I can't add you as a reviewer, even if remove you as an assignee -- I guess because you created the PR. Please take a look.

@denis-yuen I can't add you as a reviewer, even if remove you as an assignee -- I guess because you created the PR. Please take a look.

LGTM, but GitHub is probably right in this case.
I'm biased.
I do see 02e40a8 making it to github properly

I'm going to fix line 181. Curious what you are thoughts are on line 174.

To maybe be a bit more clear:

I actually like all builds to be available although maybe we'll need to clean them out after a while (maybe all builds that aren't feature branches?)
For the eventual CI/CD, I don't think we'll need a IAM role editing cloudfront anymore.
Rather, compose_setup should just set the UI fragment hash (i.e. https://gui.dockstore.org/<fragment-hash>/) correctly and that fragment-hash should be prominently displayed in the UI for debugging rather than a simple "develop" which changes over time

2. For the eventual CI/CD, I don't think we'll need a IAM role editing cloudfront anymore.
   Rather, compose_setup should just set the UI fragment hash (i.e. `https://gui.dockstore.org/<fragment-hash>/`) correctly and that fragment-hash should be prominently displayed in the UI for debugging rather than a simple "develop" which changes over time

I was thinking we should eventually (in 1.8/1.9) move to "full Cloudfront", but maybe not. Maybe we discuss in tomorrow's meeting.


Raindrops::Linux methods expect array

Hi,
I've faced a following error when using unicorn instrumentation.
2019-04-19 16:18:11 +0900 Starting prometheus exporter on port 9394
Prometheus Exporter Failed To Collect Unicorn Stats undefined method `map' for "/path/to/sock":String
Did you mean?  tap

I think that these methods expect array as an arg.
ref.
https://github.com/tmm1/raindrops/blob/1c18fd9c13f95fef6bcbdc0587d38886fa8e9064/lib/raindrops/linux.rb#L37
https://github.com/tmm1/raindrops/blob/1c18fd9c13f95fef6bcbdc0587d38886fa8e9064/ext/raindrops/linux_inet_diag.c#L574
Currently @listener_address is String, so I changed it to pass as an array.

@siebertm any feedback here? is this just relevant to socket connection?

TBH, we're not using unix sockets and this might have been slipped through. Definite :+1: from me!

cool, thanks @aiharak !


DNS probe module: let user set recursion desired bit

Hello, I would like to do DNS probes that have the dns_rd bit set to 0. Currently, it is always set to 1 (see https://github.com/zmap/zmap/blob/main/src/probe_modules/module_dns.c#L190).
I have make a fork of the project in which I've changed this value, however, this means that I have to recompile each time I want to change the value of this bit. Instead, I had ther idea to make this configurable to the user and i wanted to ask you if that would be a desirable change to zmap in your opinion. I'd offer to you to implement this change, however, I am not yet sure on how exactly to realize it. For example, it is conceivable to change the syntax to the value of --probe-args. Currently. users can specify questions like --probe-args="A,google.com;A,example.org". I can imagine allowing e.g. --probe-args="A,google.com;A,example.org;rd" to explicitly desire recursion and e.g. --probe-args="A,google.com;A,example.org;nrd" to explicitly set this bit to 0.
Do you think that this would be a worthwhile change to the project? Or do you see this change breaking backwards-compatibility? I'd be thankful for any feedback.

We could also add a new "probe-type" that's something like "A:rn". It's all a bit kludgy since we're passing it in via a string we're parsing in C, but I'm supportive. It would be nice to do it in a way that's backwards compatible if we can (e.g., if not specific, it still works and has the same behavior as today).

We could also add a new "probe-type" that's something like "A:rn". It's all a bit kludgy since we're passing it in via a string we're parsing in C, but I'm supportive. It would be nice to do it in a way that's backwards compatible if we can (e.g., if not specific, it still works and has the same behavior as today). Any chance that you'd be willing to submit a PR that adds the proposed functionality?

I have submitted a Draft PR under #764. I am not sure whether or not more documentation needs to be updated.

Thanks for doing that!! Left some notes on help on that PR!


dependency absinthe version

Failed to use "absinthe" because
  mix.exs specifies ~> 1.3.1

mix.exs specifies ~> 1.3.1 to   mix.exs specifies ~> 1.3.0  run

Can you display the full error message?


Add parallel computing support for solve DEA when using EnvelopDEA

A simple multiprocessing function is added, test_envelop_dea.py, _dea.py, _envelope_solver.py, dmu.py are modified.
The main change is below:
class EnvelopeSolver(BaseSolver):
    ......
    def apply(self) -> List[EnvelopResult]:
        if self.n_jobs <= 1:
            return [self._solve_problem(j) for j in range(self.DMUs.N)]  # older version
        else:
            # faster than problem.solve(pulp.PULP_CBC_CMD(threads=self.n_jobs))
            pool = multiprocessing.Pool(self.n_jobs)

            problem_processes = []
            for j in range(self.DMUs.N):
                problem_processes.append(pool.apply_async(self._solve_problem, args=(j,)))

            pool.close()
            pool.join()

            results = []
            for j in range(self.DMUs.N):
                results.append(problem_processes[j].get())

            return results

And we can solve the problem more faster when the number of DMUs are great:
import numpy as np
from Pyfrontier.frontier_model import EnvelopeDEA
from time import time

start_time = time()
dea = EnvelopeDEA(frontier="CRS", orient="in", n_jobs=2)
print(time() - start_time)

dea.fit(inputs, outputs)
dea.result

@ImLaoBJie
I appreciate your assistance. I have learned a great deal through the PR.
I will definitely make use of your idea and release the new version as soon as possible!


Fix deployment to IBM Cloud

This fix updates the deployment to use a toolchain where the parameters can be configured by the user as required by slackin server during deployment.
This closes #375 and closes #186.

@rauchg Ready for review. Pls. note that when testing that will need to add "&branch=fix-ibm-cloud-deployment" to the deployment link. The deployment link will be shown when you click to deploy on "IBM Cloud".
The URL would then become something like this: https://console.bluemix.net/devops/setup/deploy?repository=https://github.com/hickeyma/slackin&branch=fix-ibm-cloud-deployment.
This workaround is  needed  because the deployment files are in my clone and branch and not yet merged into the slackin repo and master branch.

+1 LGTM - Nice job.

Have received 2 +1's from @henrynash and through email from @rauchg ("Added you to the repo. I trust that you'll put it into great shape. Thank you").
@evilrabbit do you have any comments?


[SPARK-17402][SQL] separate the management of temp views and metastore tables/views in SessionCatalog

What changes were proposed in this pull request?
In SessionCatalog, we have several operations(getTableMetadata, tableExists, renameTable, dropTable) that handle both temp views and metastore tables/views. They can save some lines of code for some commands that need to deal with both temp views and metastore tables/views, but also introduce bugs for other commands, because the operation names say nothing about temp views and are very likely to be misused:

DataFrameWriter.saveAsTable/CREATE TABLE USING will fail if a same-name temp view exits
Catalog.dropTempView may drop metastore table mistakenly
ALTER TABLE RECOVER PARTITIONS/LOAD DATA/TRUNCATE TABLE/SHOW CREATE TABLE should report "table not found" instead of "temp view is not supported", if a same-name temp view exists, because these commands don't need to deal with temp views.

In some commands we support temp views mistakenly without mentioning it in document: ShowTablePropertiesCommand, ShowColumnsCommand, Catalog.listColumns.
Mixing the handling of temp views and metastore tables/views also makes it harder to implement thread-safe operations. e.g. AlterViewAsCommand checks isTemporaryTable first then createTempView, which is not atomic. Most temp view related operations in SessionCatalog hold a lock on the SessionCatalog object, which is unnecessary.
This PR separates the management of temp views and metastore tables/views in SessionCatalog, any commands that need to deal with temp views should explicitly call temp view related operations in SessionCatalog, to fix existing bugs and prevent future bug like this.
How was this patch tested?
existing tests and 3 new tests.

cc @yhuai @gatorsmile @liancheng

Test build #64943 has started for PR 14962 at commit ebed732.

also cc @srinathshankar  , it will be very easy to implement global temp view after this

Test build #64943 has finished for PR 14962 at commit ebed732.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class TempViewAlreadyExistsException(name: String)
class TempViewManager

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64943/
Test FAILed.

Test build #64973 has started for PR 14962 at commit 8eb6c42.

Test build #64973 has finished for PR 14962 at commit 8eb6c42.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class TempViewAlreadyExistsException(name: String)
class TempViewManager

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64973/
Test FAILed.

Test build #64984 has started for PR 14962 at commit 7314a33.

Test build #64987 has started for PR 14962 at commit 0075f9d.

Test build #64984 has finished for PR 14962 at commit 7314a33.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class TempViewAlreadyExistsException(name: String)
class TempViewManager

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64984/
Test FAILed.

Test build #64987 has finished for PR 14962 at commit 0075f9d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class TempViewAlreadyExistsException(name: String)
class TempViewManager

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64987/
Test PASSed.

Found a common bug in the following ALTER TABLE commands:
    | ALTER TABLE tableIdentifier (partitionSpec)?
        SET SERDE STRING (WITH SERDEPROPERTIES tablePropertyList)?     #setTableSerDe
    | ALTER TABLE tableIdentifier (partitionSpec)?
        SET SERDEPROPERTIES tablePropertyList                          #setTableSerDe
    | ALTER TABLE tableIdentifier ADD (IF NOT EXISTS)?
        partitionSpecLocation+                                         #addTablePartition
    | ALTER VIEW tableIdentifier ADD (IF NOT EXISTS)?
        partitionSpec+                                                 #addTablePartition
    | ALTER TABLE tableIdentifier
        from=partitionSpec RENAME TO to=partitionSpec                  #renameTablePartition
    | ALTER TABLE tableIdentifier
        DROP (IF EXISTS)? partitionSpec (',' partitionSpec)* PURGE?    #dropTablePartitions
    | ALTER VIEW tableIdentifier
        DROP (IF EXISTS)? partitionSpec (',' partitionSpec)*           #dropTablePartitions
    | ALTER TABLE tableIdentifier partitionSpec? SET locationSpec      #setTableLocation
    | ALTER TABLE tableIdentifier RECOVER PARTITIONS                   #recoverPartitions

We need to issue an exception when the tableType is VIEW. This is not introduced by this PR. Should we fix it here? or create a separate PR?

The biggest external change is in the table resolution order. Now, three common SessionCatalog functions tableExists, getTableMetadata and lookupRelation are not using the same order. tableExists and getTableMetadata do not check the existence of temp table, but lookupRelation always first checks the temp table. This inconsistency could be risky. That means, we allow users to create a table with the same name like a temp table. When we query the table, we will get the temp view at first.
In addition, we have review all the places that are using these three functions. For example, the analyzer rule PreWriteCheck is using both lookupRelation and tableExists. We will have a bug after this PR.

Found a common bug in the following ALTER TABLE commands:

let's fix them in separated PRs.

In addition, we need to carefully review all the places that are using these three functions. For example, the analyzer rule PreWriteCheck is using both lookupRelation and tableExists. We will have a bug after this PR.

yea, we should review them carefully(I have walked through all of them). PreWriteCheck does have a bug, but not caused by this PR. I'm thinking about if we should also separate lookUpRelation to a metastore version and a temp view version.

Sorry, I did not explain it clearly. In the analyzer rule PreWriteCheck, the target metastore table exists and thus tableExists returns true, but the following catalog.lookupRelation actually returns a temporary table. Thus, it could report a wrong error or miss some errors.

the target metastore table exists and thus tableExists returns true, but the following catalog.lookupRelation actually returns a temporary table.

isn't it true before this PR?

Test build #65100 has started for PR 14962 at commit cec6b3e.

Test build #65101 has started for PR 14962 at commit 01659dc.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65100/
Test FAILed.

Test build #65101 has finished for PR 14962 at commit 01659dc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65101/
Test FAILed.

Test build #65102 has started for PR 14962 at commit a8ce27f.

Test build #65102 has finished for PR 14962 at commit a8ce27f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65102/
Test FAILed.

Found another issue we need to fix.
    val tableName = "tab1"
    withTable(tableName) {
      Seq((1, 2), (3, 4)).toDF("i", "j").createOrReplaceTempView(tableName)
      sql(s"select * from $tableName where i < 2")
        .write.mode(SaveMode.Overwrite).saveAsTable(tableName)
      sql(s"select * from $tableName").show()
    }

Before this PR, the results are correct
+---+---+
|  i|  j|
+---+---+
|  1|  2|
+---+---+

After this PR, the results is incorrect.
+---+---+
|  i|  j|
+---+---+
|  1|  2|
|  3|  4|
+---+---+

the target metastore table exists and thus tableExists returns true, but the following catalog.lookupRelation actually returns a temporary table.

Before this PR, yeah, it behaves the same. After this PR, we expect catalog.lookupRelation returns a metastore table. However, it returns a temporary table. Thus, we might misse the detection.

    val tableName = "tab1"
    withTable(tableName) {
      Seq((1, 2), (3, 4)).toDF("i", "j").createOrReplaceTempView(tableName)
      sql(s"select * from $tableName where i < 2")
        .write.mode(SaveMode.Overwrite).saveAsTable(tableName)
      sql(s"select * from $tableName").show()
    }

@gatorsmile , why it's an issue? In this case, we create a temp view called tab1 with 2 rows first. Then we create a table also called tab1 with 1 row. Finally in select we use the name tab1, which refers to the temp view, it has 2 rows.

Test build #65153 has started for PR 14962 at commit e0d2427.

@gatorsmile It's actually another bug in previous code. When we use DataFrameWriter.saveAsTable with overwrite mode, we may mistakenly drop a same-name temp view. I have updated the PR description to include it.

Test build #65153 has finished for PR 14962 at commit e0d2427.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65153/
Test FAILed.

Merged build finished. Test FAILed.

uh... yeah, that is another existing bug. : ) Let me write a test suite for all the DDL statements when the temp view with the same name exists. We might hit more bugs...

wow... we are allowed to insert data into temporary views...

Test build #65186 has started for PR 14962 at commit f421355.

Test build #65186 has finished for PR 14962 at commit f421355.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65186/
Test PASSed.

Merged build finished. Test PASSed.

Is it possible to first have a PR to fix the bugs?

Maybe update this PR now? The only issue left is the atomicity issues. It sounds like we do not need to backport the fix, since the temporary view is session-specific. Right? We have to be very careful when implementing the global temporary views.

I'll update this PR after the global temp view

closing it as most of them were fixed in other PRs


pytest integration take 1

had a hell of a time figuring out a bug with Faker (dependency of factory_boy) see aa53067
currently interested in contributing to Faker as a part of the pycascades sprints this weekend (if possible)


Replace references to KeyVault secrets with KeyVault references

Description
Replace direct references to Keyvault secrets
References

@cla-bot check


ddev push pantheon is not working

Is there an existing issue for this?

[X] I have searched the existing issues

Run a Diagnostic and Paste Link Here
Running bash [-c /var/folders/yv/355khbsx7bdgzjqm_ysk2fn00000gn/T/test_ddev.sh]
OS Information: Darwin masons-MacBook-Pro.local 21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:22 PDT 2022; root:xnu-8020.121.3~4/RELEASE_X86_64 x86_64
User information: uid=501(mason) gid=20(staff) groups=20(staff),101(access_bpf),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),701(com.apple.sharepoint.group.1),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh),400(com.apple.access_remote_ae)
DDEV version:  ITEM             VALUE
 DDEV version     v1.19.2
 architecture     amd64
 db               drud/ddev-dbserver-mariadb-10.3:v1.19.2
 dba              phpmyadmin:5
 ddev-ssh-agent   drud/ddev-ssh-agent:v1.19.0
 docker           20.10.14
 docker-compose   v2.3.3
 docker-platform  docker-desktop
 mutagen          0.12.0
 os               darwin
 router           drud/ddev-router:v1.19.0
 web              drud/ddev-webserver:v1.19.2
======= DDEV global info =========
Global configuration:
instrumentation-opt-in=true
omit-containers=[]
mutagen-enabled=false
nfs-mount-enabled=false
router-bind-all-interfaces=false
internet-detection-timeout-ms=750
disable-http2=false
use-letsencrypt=false
letsencrypt-email=
table-style=default
simple-formatting=false
auto-restart-containers=false
use-hardened-images=false
fail-on-hook-fail=false
required-docker-compose-version=
use-docker-compose-from-path=false
no-bind-mounts=false

======= DOCKER info =========
docker location: lrwxr-xr-x  1 root  admin  54 Jan 28 13:36 /usr/local/bin/docker -> /Applications/Docker.app/Contents/Resources/bin/docker
Docker Desktop Version: Docker Desktop for Mac 4.8.2 build 79419
======= Mutagen Info =========
======= Docker Info =========
Docker platform: docker-desktop
Using docker context: default (unix:///var/run/docker.sock)
docker-compose: v2.3.3
Using DOCKER_HOST=unix:///var/run/docker.sock
Docker version: 20.10.14
Able to run simple container that mounts a volume.
Able to use internet inside container.
Docker disk space:
Filesystem                Size      Used Available Use% Mounted on
overlay                  58.4G      9.6G     45.8G  17% /

unable to create project at project root '/Users/mason/desktop/symfony': project root /Users/mason/desktop/symfony does not exist
Container ddev-associationofmetaphysicalstores-db  Removed
Container ddev-associationofmetaphysicalstores-dba  Removed
Container ddev-associationofmetaphysicalstores-web  Removed
Network ddev-associationofmetaphysicalstores_default  Removed
Project associationofmetaphysicalstores has been stopped.
The ddev-ssh-agent container has been removed. When you start it again you will have to use 'ddev auth ssh' to provide key authentication again.
Existing docker containers:
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
Creating a new ddev project config in the current directory (/Users/mason/tmp/tryddevproject-31618)
Once completed, your configuration will be written to /Users/mason/tmp/tryddevproject-31618/.ddev/config.yaml

Found a php codebase at /Users/mason/tmp/tryddevproject-31618
Project type has no settings paths configured, so not creating settings file.
Configuration complete. You may now run 'ddev start'.
Network ddev_default created
Starting tryddevproject-31618...
Container ddev-ssh-agent  Started
ssh-agent container is running: If you want to add authentication to the ssh-agent container, run 'ddev auth ssh' to enable your keys.
Pushed mkcert rootca certs to ddev-global-cache/mkcert
Network ddev-tryddevproject-31618_default  Created
Container ddev-tryddevproject-31618-dba  Started
Container ddev-tryddevproject-31618-db  Started
Container ddev-tryddevproject-31618-web  Started
Container ddev-router  Started
Project type has no settings paths configured, so not creating settings file.
Project type has no settings paths configured, so not creating settings file.
Successfully started tryddevproject-31618
Project can be reached at https://tryddevproject-31618.ddev.site https://<IP_ADDRESS>:63176
+ curl --fail -I http://tryddevproject-31618.ddev.site
HTTP/1.1 200 OK
Server: nginx/1.20.1
Date: Fri, 10 Jun 2022 16:54:42 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding

+ '[' 0 -ne 0 ']'
+ set +x
Thanks for running the diagnostic. It was successful.
Please provide the output of this script in a new gist at gist.github.com
Running ddev launch in 5 seconds

Current Behavior
following the docs https://ddev.readthedocs.io/en/stable/users/providers/pantheon/
I get an error when running "ddev push pantheon"

Expected Behavior
files must be uploaded to the pantheon server
Steps To Reproduce
No response
Anything else?
No response

I had no trouble testing this with Drupal 9 and current ddev v1.19.3 (there haven't been any changes since November 2021).
$ ddev push pantheon -y
Authenticating...
 [notice] Logged in via machine token.
Uploading database...
Wrote database dump from project 'ddev-pantheon-push' database 'db' to file /Users/rfay/workspace/ddev-pantheon-push/.ddev/.downloads/db.sql.gz in gzip format.
+ set -eu -o pipefail
+ ls /var/www/html/.ddev
+ pushd /var/www/html/.ddev/.downloads
+ terminus remote:drush ddev-pantheon-push.dev -- sql-drop -y
 [warning] This environment is in read-only Git mode. If you want to make changes to the codebase of this site (e.g. updating modules or plugins), you will need to toggle into read/write SFTP mode first.
Warning: Permanently added '[appserver.dev.d32c631e-c998-480f-93bc-7c36e6ae4142.drush.in]:2222,[<IP_ADDRESS>]:2222' (RSA) to the list of known hosts.

 // Do you really want to drop all tables in the database pantheon?: yes.

 [notice] Command: ddev-pantheon-push.dev -- drush sql-drop [Exit: 0]
+ gzip -dc db.sql.gz
+ terminus remote:drush ddev-pantheon-push.dev -- sql-cli
 [warning] This environment is in read-only Git mode. If you want to make changes to the codebase of this site (e.g. updating modules or plugins), you will need to toggle into read/write SFTP mode first.
 [notice] Command: ddev-pantheon-push.dev -- drush sql-cli [Exit: 0]
Database uploaded
Uploading files...
+ set -eu -o pipefail
+ ls /var/www/html/web/sites/default/files
+ drush rsync -y @self:%files @ddev-pantheon-push.dev:%files

 // Copy new and override existing files at
 // dev.d32c631e-c998-480f-93bc-7c36e6ae4142@appserver.dev.d32c631e-c998-480f-93bc-7c36e6ae4142.drush.in:files/.
 // The source is /var/www/html/web/sites/default/files/?: yes.

Files uploaded
push succeeded.

If you have more information, or are not using drupal (this is completely drupal-centric), please let me know. If you want to give me access to your project, share it with me at<EMAIL_ADDRESS>and give me permission to destroy it with a ddev push. Otherwise, I don't have much I can offer.
As I said in Discord, I recommend that you manually upload your files or use terminus to upload them.

can you describe the process you went through? did you install Drupal on pantheon or on localhost?
are you on macOS monterey?

I tested this on macOS Monterrey; It's drupal 9, same code on local as pushed to pantheon.
You might see if drush works on your local system. Does ddev drush st work?

No response for 5 days, so I'm going to close this for now. If you want to share the project with me, I'll help you chase it, the info is in https://github.com/drud/ddev/issues/3905#issuecomment-1152753318 - Closing for now, happy to continue the conversation here or elsewhere.

Finding the same issue:
`Authenticating...
[notice] Logging in via machine token.
[notice] Found a machine token for MY_EMAIL.
[notice] Logged in via machine token.
Skipping database push.
Uploading files...

set -eu -o pipefail
ls /var/www/html/web/sites/default/files
drush rsync -y @self:%files @MY_UUID.dev:%files

In HostPath.php line 88:
Site alias @MY_UUID.dev not found.
push failed: Failed to exec set -x   # You can enable bash debugging output by uncommenting
set -eu -o pipefail
ls ${DDEV_FILES_DIR} >/dev/null # This just refreshes stale NFS if possible
drush rsync -y @self:%files @${project}:%files
on web: exit status 1 `
Noticing this difference:
your output for the file command has this:
Uploading files...
+ set -eu -o pipefail
+ ls /var/www/html/web/sites/default/files
+ drush rsync -y @self:%files @ddev-pantheon-push.dev:%files
@ddev-pantheon-push.dev:%files

instead of what myself, and the OP had in:
@MY_UUID.dev:%files
So for some reason, (maybe WSL command line quirk, who knows,) Your rsync command is being constructed with a placeholder of ddev-pantheon-push instead of the site UUID.

Please open a new issue rather than commenting on an old closed one. Thanks. If you are able to collaborate on this come over to Discord, https://discord.gg/hCZFfAMc5k and we can do live debugging of it.


Spanish translation

I would like to collaborate in the translation into Spanish? It is already done?

We already have spanish translations for Qtum Core, if you spot a problem in them though then please make an issue or pull request


Add support of Karate

Karate can be integrated in ara as it is based on cucumber.
However, the report display is broken while we display it in the tool.
Some spaces are missing between keywords and it's not really readable.


Feature: associations

Added Classroom, Book and Rental classes.
Implemented relationships between classes.

Hello @nachosala89, sorry for replying late. I have seen you've done a splendid job refactoring the code.


Merge if statements with the enclosing one

Sonar cloud flagged issues - https://sonarcloud.io/project/issues?resolved=false&rules=java%3AS1066&severities=MAJOR&id=open-metadata_OpenMetadata

Hey! Can I work on this?

hi @andrefpoliveira Yes please take it over

Fixed by https://github.com/open-metadata/OpenMetadata/pull/7851


Loan issues

While creating receipts every month, if user pay less than actual amount to pay, the extra amount is not adding correctly for the next month issue is fixed.

Coverage remained the same at 87.108% when pulling a9d049ba20b9f9f297f3684108830745cc73bf76 on swethanaretla:loan_issues into 1510c97433537aacddbbf76cb5ce4ef8af0c40b6 on MicroPyramid:master.


Expose first byte enums

We need to access the network type and address kind inside Yoroi so I'm exposing this functionality. This is a breaking change but it's simple enough that hopefully you don't mind

Nice PR, thanks!


Request to add Content-Length automatically for request with body

Some servers, such as elasticsearch, will not accept POST request, if it doesn't have Content-Length field. Adding these value manually by couting number of characters is a hard work. Consider to add it automatically, or let user add it in header, with {{content_length}} stand for bytes length.

@tuyen-at-work In the meantime, have you found a way to add this manually?

@muya How do you handle variable replacement within the req.getBody() response?


Lambda now fails

I updated to your latest code but I still get the "There was a problem with the requested skills response" from Alexa.
Looking at Lambda logs I get the following
No JSON object could be decoded: ValueError
Traceback (most recent call last):
File "/var/task/main.py", line 32, in lambda_handler
login()
File "/var/task/main.py", line 95, in login
response = requests.get(login_uri()).json()
File "/var/task/requests/models.py", line 808, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib64/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Any suggestions whats gone wrong please?

I, too, have the same problem which cropped up today. I suspect that the API interface changed again. What's the best way to debug it now?

Same here.  Code stopped working this morning for me as well.  :(

Found the problem -- login api has changed pretty significantly (GET changed to POST, token handling needed, APP_ID changed, etc). Don't have the time to fix it right now but will take a look later. Here's a reference project that already figured out the changes:  https://github.com/Einstein42/myq-garage

Great find @bhusang.  I had a feeling that this was going to change somehow because the GET method poses a security issue since the password is being transmitted unencrypted.  This is a step towards the right direction for Chamberlain, but yes unfortunately for us, we will now have to modify or codes again.  But as long as we know what the new endpoints are.  I am confident that we will be able to figure it out.  Thanks for sharing your discovery.

Thanks all, updated based @arraylabs discoveries. The Chamberlain updates are a pain, but as @tigerbrain said it's a step in the right direction towards a real app.

works now just keep symbols in user/pass

Should I be changing my username and password for myq due to the mentioned security issue?

Hi @nimustan, I don't think you really have to, but if you don't feel comfortable about your password being exposed before.  Of course, you are free to do that.  I was personally ok with it cause I figured that nobody knows where I live anyway so they probably won't care about trying to hack into my garage.

Thanks @jbnunn for putting in the fixes so quickly and thank you again @arraylabs for saving us from having to miss out on this wonderful luxury.

I'm now getting a different error, I'm sure the username and password are correct and I use date encoder from the instructions to alter is.  The log now says

17:25:06
START RequestId: b118cd08-f3a3-11e6-bf9a-b1a8e2b0abd0 Version: $LATEST

17:25:07
'SecurityToken': KeyError Traceback (most recent call last): File "/var/task/main.py", line 31, in lambda_handler login() File "/var/task/main.py", line 99, in login myq_security_token = auth['SecurityToken'] KeyError: 'SecurityToken'

17:25:07
END RequestId: b118cd08-f3a3-11e6-bf9a-b1a8e2b0abd0

17:25:07
REPORT RequestId: b118cd08-f3a3-11e6-bf9a-b1a8e2b0abd0	Duration: 1111.90 ms	Billed Duration: 1200 ms Memory Size: 128 MB	Max Memory Used: 24 MB

17:25:07
START RequestId: b1ea266f-f3a3-11e6-b172-1dafaa018718 Version: $LATEST

17:25:08
'SecurityToken': KeyError Traceback (most recent call last): File "/var/task/main.py", line 31, in lambda_handler login() File "/var/task/main.py", line 99, in login myq_security_token = auth['SecurityToken'] KeyError: 'SecurityToken'

17:25:08
END RequestId: b1ea266f-f3a3-11e6-b172-1dafaa018718

17:25:08
REPORT RequestId: b1ea266f-f3a3-11e6-b172-1dafaa018718	Duration: 884.67 ms	Billed Duration: 900 ms Memory Size: 128 MB	Max Memory Used: 24 MB

I update the code changes that were published yesterday, however I am getting an error that is referencing the ASK skill id. I have double checked that the ID is correct. I am not sure what I am doing wrong. Any help would be appreciated.
START RequestId: 143c252c-f3a8-11e6-9996-e913311612ab Version: $LATEST
'session': KeyError
Traceback (most recent call last):
File "/var/task/main.py", line 24, in lambda_handler
if event['session']['application']['applicationId'] != "amzn1.ask.skill.229939cb-c632-4b39-b57a-fcd3beca4c79":
KeyError: 'session'

@CelticWebSolutions, FYI, I also experienced your error last night as I was editing my code to implement the changes @jbnunn implemented.  Before I experienced the problem, I was able to actually call the login() procedure multiple times with success.  So when I started getting a failed message, I decided to just call it a night thinking that maybe Chamberlain was updating their system which is causing my calls to fail already.  It was a good decision because this morning, I tried the exact same code again with success.  So I don't know if this is the exact situation for you, but just wanted you to know that it may be a problem with Chamberlain's new API and not your code?  Just a thought.

Thanks @tigerbrain, maybe I'll just leave it till tomorrow morning and see what happens then 👍

Hey; Thanks so much for updating this so quickly.
I am seconding what @SConaway alludes to.  I had previously encoded my email and password.  With the new API, those encoded strings, the login was failing.  I just passed in the actual email and password directly, and it works now.

@davedyk The PR from @tigerbrain works wonderfully.

Hi All,
Just to be clear, now that Chamberlain's new API has been updated to use a POST instead of a GET.  The username and password must no longer be encoded which is consistent with @SConaway and @davedyk's discovery.  I will create another pull request to make sure that the step by step instructions are updated accordingly.
@CelticWebSolutions, was your issue resolved after waiting for one day without doing anything?
tigerbrain

Hi Tigerbrain,
No it wasn't but I just read that people were not bothering with the @ encoding and simply using the @ in the email and it was working for them.  Just changed mine to have just a  straight forward @ and mine now works again too :)


SignalR full docs availability on MS Docs???

Hello SignalR Doc team;
When will you release full docs on SignalR Core 2.1 docs? Will they show up at 2.1 RTM?
Based on the intro video at Build, there are tons of new (an more complex) in the new Core version that is beyond the old docs on SignalR.
Thanks!
..Ben

We're working on them.

Thank you Sir!

We can keep this open. I'll see if @rachelappel  and the SignalR folks can at least let you know what's next.

That would be great Rick!

Some upcoming docs:
-Configuration
-Messagepack
-Connection lifetimes and reconnection inf
-Groups
-Security
-TypeScript integration
-Migration from .NET

Excellent Rachel;
When do you think these will take place?
Thanks!

They're in process now, and some have been merged and will be published soon.

Thank you!!!


Master

Init admin

Why are you trying to merge this with my master branch??

Sorry, surely, it was my mistake.
пт, 3 Фев 2017 г., 18:41 Thomas Vaeth<EMAIL_ADDRESS>
Why are you trying to merge this with my master branch??
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/thomasvaeth/trophy-jekyll/pull/19#issuecomment-277280502,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJxz2MYN8VpYLubSmrLwCq3X1fIZWikxks5rY0qcgaJpZM4L2Ns1
.


Hedera node does not start

Executing the following steps:

npm install @hashgraph/hedera-local -g
hedera start
yields the following error on the container for the consensus node

Cannot reproduce.


Enable KafkaChannel controller

Fixes #

Proposed Changes

Enable controller, webhook
Added some e2e tests that are either reusing test suites from eventing-kafka, or that are copy&pastes (for now)
Reply and other "advanced" delivery mechanisms are not implemented yet, so, no tests for them
Run conformance tests (except for the the things mentioned above)
Run e2e tests (except for the the things mentioned above)
No auth tests yet --> I will create separate Cronjob and GH actions for the auth tests

Release Note

Docs

/test pull-knative-sandbox-eventing-kafka-broker-upgrade-tests

/test pull-knative-sandbox-eventing-kafka-broker-unit-tests

I opened an issue for the unit test failure: https://github.com/knative-sandbox/eventing-kafka-broker/issues/1626

Please ignore the code style failure. It complains about things in vendor. ...that's strange.
 golangci-lint run ./...                                                                                ─╯
pkg/eventshub/event_info_store.go:203:20: Error return value of `wait.PollImmediate` is not checked (errcheck)
        wait.PollImmediate(retryInterval, retryTimeout, func() (bool, error) {
                          ^
pkg/eventshub/receiver/receiver.go:172:16: Error return value of `m.Finish` is not checked (errcheck)
        defer m.Finish(nil)
                      ^
pkg/environment/namespace.go:33:2: ineffectual assignment to nsSpec (ineffassign)
        nsSpec, err := c.CoreV1().Namespaces().Get(context.Background(), mr.namespace, metav1.GetOptions{})
        ^
pkg/environment/namespace.go:43:3: ineffectual assignment to nsSpec (ineffassign)
                nsSpec, err = c.CoreV1().Namespaces().Create(context.Background(), nsSpec, metav1.CreateOptions{})

=== CONT  TestSingleBinaryEventForChannel/InMemoryChannel-messaging.knative.dev/v1
    creation.go:59: Creating channel &TypeMeta{Kind:InMemoryChannel,APIVersion:messaging.knative.dev/v1,}-e2e-singleevent-channel-binary
    creation.go:67: Failed to create "InMemoryChannel" "e2e-singleevent-channel-binary": the server could not find the requested resource
    creation.go:72: Failed to create "InMemoryChannel" "e2e-singleevent-channel-binary": the server could not find the requested resource

        creation.go:205: Failed to create v1 broker "broker": Internal error occurred: failed calling webhook "webhook.eventing.knative.dev": Post "https://eventing-webhook.knative-eventing.svc:443/defaulting?timeout=10s": dial tcp <IP_ADDRESS>:443: connect: connection refused

Interesting...
I see these things in webhook logs when I check locally:
kafka-webhook-eventing-7896d95bdd-8jcfg kafka-webhook-eventing {"level":"error","ts":"2021-12-21T08:56:49.422Z","logger":"kafka-webhook-eventing.DefaultingWebhook","caller":"controller/controller.go:566","msg":"Reconcile error","knative.dev/pod":"kafka-webhook-eventing-7896d95bdd-8jcfg","knative.dev/traceid":"7e5ce7e6-f630-4808-b5e9-ee57ec0bbe42","knative.dev/key":"defaulting.webhook.kafka.eventing.knative.dev","duration":0.000018928,"error":"secret \"kafka-webhook-eventing-certs\" is missing \"ca-cert.pem\" key","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20211206113427-18589ac7627e/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20211206113427-18589ac7627e/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20211206113427-18589ac7627e/controller/controller.go:478"}
kafka-webhook-eventing-7896d95bdd-8jcfg kafka-webhook-eventing {"level":"info","ts":"2021-12-21T08:56:49.422Z","logger":"kafka-webhook-eventing","caller":"leaderelection/context.go:151","msg":"\"kafka-webhook-eventing-7896d95bdd-8jcfg_4a9e326a-be04-411f-a71c-cbef57482710\" has started leading \"kafka-webhook-eventing.validationwebhook.00-of-01\"","knative.dev/pod":"kafka-webhook-eventing-7896d95bdd-8jcfg"}

kafka-webhook-eventing-7896d95bdd-8jcfg kafka-webhook-eventing E1221 08:56:50.683826       1 reflector.go:138] k8s.io/client-go@v0.21.4/tools/cache/reflector.go:167: Failed to watch *v1.Broker: unknown (get brokers.eventing.knative.dev)

Rebased now...

Aaah, probably we need KafkaChannel here: https://github.com/knative-sandbox/eventing-kafka-broker/blob/main/control-plane/config/200-webhook/100-webhook-cluster-role.yaml#L93

/test pull-knative-sandbox-eventing-kafka-broker-integration-tests
=== FAIL: test/e2e_new TestBrokerDeletedRecreated (408.61s)
I1221 13:25:05.174917   61521 request.go:668] Waited for 1.198156968s due to client-side throttling, not priority and fairness, request: GET:https://<IP_ADDRESS>/api/v1/namespaces/test-wqytpxwo

/test pull-knative-sandbox-eventing-kafka-broker-integration-tests
Strimzi operator imagePullBackoff

/test pull-knative-sandbox-eventing-kafka-broker-upgrade-tests
More image pull issues:
my-cluster-zookeeper-0                      0/1   ImagePullBackOff

/test pull-knative-sandbox-eventing-kafka-broker-integration-tests

/retest
Retest after quay down


stated example with Trained models not working due to SparQL exception

Hi @daniilsorokin ,
I tried to run the example with trained models and after debugging through the codes it seems to me the SPARQL queries are throwing exception.  I put a flag when the query throws an exception and it seems to throw each time.
PREFIX g:<http://wikidata.org/>
PREFIX e:<http://www.wikidata.org/wiki/entity/>
PREFIX rdfs:<http://www.w3.org/2000/01/rdf-schema#>
PREFIX skos:<http://www.w3.org/2004/02/skos/core#>
PREFIX schema:<http://schema.org/>

SELECT DISTINCT ?e2 ?matchedlabel ?label WHERE
{
  {
    {GRAPH <http://wikidata.org/wiki/statements> {?e2 e:P1549s/e:P1549v "%demonym"@en}}
    UNION
    {VALUES ?labelpredicate {rdfs:label skos:altLabel}
     GRAPH <http://wikidata.org/wiki/terms> {?e2 ?labelpredicate ?matchedlabel. ?matchedlabel bif:contains "'Obama is a president'"@en}
     FILTER (lang(?matchedlabel) = "en")
    }
  }
FILTER EXISTS {{ GRAPH g:statements {{ ?e2 ?p ?v }} }}
FILTER (NOT EXISTS {{GRAPH g:instances {{?e2 rdf:type e:Q17442446}} }} &&
        NOT EXISTS {{GRAPH g:instances {{?e2 rdf:type e:Q15474042}} }} &&
        NOT EXISTS {{GRAPH g:instances {{?e2 rdf:type e:Q18616576}} }} &&
        NOT EXISTS {{GRAPH g:instances {{?e2 rdf:type e:Q5707594}} }} &&
        NOT EXISTS {{GRAPH g:instances {{?e2 rdf:type e:Q427626}} }} &&
        NOT EXISTS {{GRAPH g:instances {{?e2 rdf:type e:Q16521}} }} &&
        NOT EXISTS {{GRAPH g:instances {{?e2 rdf:type e:Q11173}} }}
       )
BIND (STRLEN(?matchedlabel) as ?len)
{
  GRAPH <http://wikidata.org/wiki/terms> { ?e2 rdfs:label ?label }
  FILTER ( lang(?label) = "en" )
}
} ORDER BY ?len LIMIT 100

At certain loop the code tries to submit a query like the above and it threw an error saying "Query is malformed: QName 'bif:contains' uses an undefined prefix"  (try it on Wikidata query service)..
I am not a SPARQL expert .. can you help me where it is going wrong?
----------------update------------------------------
added PREFIX bif:http://www.openlinksw.com/schemas/bif#
now it throws a server error
The root cause  com.bigdata.rdf.sparql.ast.QuadsOperationInTriplesModeException: Use of WITH and GRAPH constructs in query body is not supported in triples mode

Hi,
you need to install your own local Wikidata endpoint using the instructions from here: https://github.com/UKPLab/coling2018-graph-neural-networks-question-answering/blob/master/WikidataHowTo.md
Unfortunately, the generated queries do not work with the official Wikidata SPARQL endpoint, since they use a different RDF scheme. We have been using an alternative mapping of Wikidata information to RDF from the Wikidata Toolkit.
Best,
Daniil

Hi @daniilsorokin, thanks for open sourcing the code. I downloaded the provided Wikidata Virtuoso file, changed the paths to appropriate paths, and run ./virtuoso-opensource-install-<IP_ADDRESS>/bin/virtuoso-t -f -c {provided virtuoso.ini}. It seemed to be running well.
However, when I try to use it through link_mention function in entity-linking/core/candidate_retrieval.py, it doesn't seem to be working. In particular, this line always gives the error:
<urlopen error [Errno 111]> Connection refused
so it looks like virtuoso server is not properly working. Is there any possible reason for that, or is there any better way to debug this error?
Thanks very much!

Hi!
This looks weird. Can you access the your local endpoint through the browser http://localhost:8890/sparql ?
What URL string exactly did you put in the config file here: https://github.com/UKPLab/starsem2018-entity-linking/blob/165ea17f963ca5557c9e15f0cb71ccc101947d10/configs/trainmodel_vector_model.yaml#L82 ?


Stacked rounded bar chart with positive and negative and rounded on top and bottom.

I can't seem to find a complete solution for this and I also wonder why it isn't an option included in the library? Anyways, could someone help me out with this? I need to create a stacked bar chart with both positive and negative values. Rounded only the top and bottom parts. Also, some data points could be 0.
Here is the closest I came up with from various examples.
https://jsfiddle.net/uffo/5oty49r3/6/
And here is what I need to achieve

Any help would be much appreciated :)

This would be a nice thing to have in the library in the future. For non stacked bars it's fairly trivial as its just some rounding at the end of the bar that is not the base.
In terms of doing this cleanly, I think it would be best to have some sort of corner radius added to each bar element by the controller. The controller may have a better idea of where in the stack a bar is.

Hi @uffou,
I have same issue for dataset with 0 at end.
please help if you found any solution.
Thank you in advance for any advice.
Jitendra

Using the built-in borderRadius property in v3 coupled with the fact that nearly everything is scriptable starting in v3.0.0-beta.11, this is quite easy to do now.
https://jsfiddle.net/7sh342wj/


Techdebt: Sure - Organizations

Removing sure: #6035
pytest-ified, by adding fixtures and parametrization (instead of the calling out to functions) and making it more dry
focus on the actual testing needed, not test because we can
grouping tests into classes, to give a bit more structure
grouping tests from 1 big file into 5 main areas
uncovered some missing test cases
removed unused code

PS: noting how the various exceptions returned differ in structure re message and code. Did not investigate if this is how it should be. Using the various parametrize setups makes this clearer.

@hans-d I realize the work that has gone into this, and thank you for that. There is undoubtedly space to optimize our testing strategy here.
However, there is no way for me to meaningfully review a PR this large. I will happily look at a PR that only removes the sure-dependency. If we really want to improve the structure, let's look at that separately.


Obscuring traffic

Does this also obscure any traffic I 'might' do to torrents and similair sites. Bit (loads) confused.

If your VPN is at home and you are not at home, all traffic between you and the server will be wrapped in OpenVPN. So what this means: if you were to torrent something (using the technology itself isn't illegal btw), the ISP of the internet connection at home would still be able to see what you are torrenting, but the place you are at can only see you send OpenVPN traffic. Please reply if you have further questions, I am closing this but will reopen if needed.


ARROW-7405: [Java] ListVector isEmpty API is incorrect

Related to ARROW-7405.
Currently isEmpty API is always return false in BaseRepeatedValueVector, and its subclass ListVector did not overwrite this method.
This will lead to incorrect result, for example, a ListVector with data [1,2], null, [], [5,6] should get [false, false, true, false] with this API, but now it would return [false, false, false, false].
This change implements isEmpty(int index) for a ListVector that will return true if the index is an empty list or a null value, and false otherwise.

Related to ARROW-7405.
Currently isEmpty API is always return false in BaseRepeatedValueVector, and its subclass ListVector did not overwrite this method.
This will lead to incorrect result, for example, a ListVector with data [1,2], null, [], [5,6] should get [false, false, true, false] with this API, but now it would return [false, false, false, false].

For [1,2], null, [], [5,6], it should return [false, true, true, false]?
as in most scenarios, null should be considered empty?

Related to ARROW-7405.
Currently isEmpty API is always return false in BaseRepeatedValueVector, and its subclass ListVector did not overwrite this method.
This will lead to incorrect result, for example, a ListVector with data [1,2], null, [], [5,6] should get [false, false, true, false] with this API, but now it would return [false, false, false, false].

For [1,2], null, [], [5,6], it should return [false, true, true, false]?
as in most scenarios, null should be considered empty?

Thanks, I was also not sure about this and I have no strong insistence on this, let's see other's comments~

For [1,2], null, [], [5,6], it should return [false, true, true, false]?
as in most scenarios, null should be considered empty?

It might be worth a quick e-mail to the mailing list to see if anybody is used it at the expected behavior.  Given that is has been broken for this long, I'm not sure it makes too much of a difference. Either way we should be clarified in the parent class at least.
I'm not entirely sure the two should be conflated (we might want to throw an exception instead of returning a boolean) but I would need to think about this some more.

For [1,2], null, [], [5,6], it should return [false, true, true, false]?
as in most scenarios, null should be considered empty?

It might be worth a quick e-mail to the mailing list to see if anybody is used it at the expected behavior. Given that is has been broken for this long, I'm not sure it makes too much of a difference. Either way we should be clarified in the parent class at least.
I'm not entirely sure the two should be conflated (we might want to throw an exception instead of returning a boolean) but I would need to think about this some more.

ok, just opened a discuss :)

LGTM, just some minor things to cleanup

Thanks, done.

merged to master, thanks @tianchen92 !


Standard shuffle option does not work for Spotify playlists calls

Hi all,
i use the phoniebox to call Spotify Playlist only. No local music files are stored on my phoniebox.
My problem is that the standard shuffle function for a stored card does not work. Spotify starts with shuffle off every time although the shuffle option is switched on for the stored card. Because of that fact I always start the web interface after the phoniebox is up and running to switch shuffle on at the Spotify page of the phoniebox.
Does anyone know that problem and can give some hints to fix it?

I have tested shuffle only for local files. Would be surprised if it worked for Spotify out of the box :)

okay thx for your explanation.


[FEATURE REQ] AzureCliCredential and all new credentials need to be added to docs

Library or service name.
Azure Identity
Is your feature request related to a problem? Please describe.
Cannot find docs on how to use AzureCliCredential or any of the new credentials, i.e. VSCodeCredential, etc
azsdke2e

Moving to August milestone


ROS open_manipulator

http://cafe.naver.com/openrt/18175

@routiful Plz check it!

I commented OpenManipulator is released


[READY] Fix meta data issue

Problem
The metadata keys for an object are just flat key => value pairs and can't contain a nested hash.
Solution
Each key that is should be passed back as a header through the broker is prepended with head_. These keys are taken out of the metadata, the prefix stripped and capitalised.

:cake: but please raise issue for moving to config.


Move documentation to be user journey focused, not OTel concepts focused

Hi all,
To set some context, I work on (amongst other things) Agones, which has used OpenCensus in Go for a number of years, with the primary use case being Metrics (we don't really do trace right now). We have the understanding that we'll need to eventually move to OTel if we want to have a long term supported observability library.
First I want to preface this by recognising all the awesome work you do for OTel, so apolgies up front for being critical, but please know it's with the intent of improvement.
A while back (April) I looked at how to do the migration from Open Census, and I got really stuck on the onboarding process (especially compared to what I was previously used to with Open Census documentation). I ended up chatting with a few people at Kubecon EU about it, and rightly so - they suggested I provide some feedback here! So here I am.
The TL;DR I found is that I found that Open Census's documentation is focused around what I want to do (i.e. do I want to export metrics? Do I want to log something, do I want to implement tracing?), whereas it seems to me like OTel is focused around  how does OTel work (what are the concepts, what is a collector, what is an exporter?), which for onboarding creates quite a large barrier to entry -- as I need to understand OTel before I can do anything, rather than learning how to do something, and using that as the vehicle to learn the underlying concepts.
I wrote this friction log back in April, so hopefully it's still relevant (looking at the docs, still seems to be the case), but this was the struggle I was finding, and eventually gave up on.
Use case: I want to create metrics in Go (language doesn't actually matter, but it's what i know best), and export it to Google Cloud Monitoring (Also doesn't quite matter here either), and I've never used OTel before.

I go to https://opentelemetry.io/ and click on Docs (https://opentelemetry.io/docs/)
I read that page, and look for metrics, I see nothing in the navbar, but I do see "metrics" on the docs landing page at the top ("...such as traces, metrics, logs"), I'll click "metrics" then!
I get a section on "Reliability & Metrics" - cool. I know what metrics are, but I need to do a thing, and this isn't explaining that yet. Okay, I see another link in that section on "metrics", let's see where that goes.
That link goes to exactly where I am. Why is it a link? Well that's frustrating. Okay, now I'm on my own to work out my own adventure.
Next stop, I see on the navbar there's a "getting started" section. Awesome! I love Quickstarts, that will get me going on my journey!
I click "Getting Started and I get two options: "Dev" and "Ops". To be honest, I've no idea what this necessarily means, but I'm mostly sure I'm "dev", so let's go there.
"This is the getting-started page for you if: You develop software." oh nice, I am in the right place! Awesome.
I'm just trying to get something done, so I'm going to skip the "What is OpenTelemetry?, How can I instrument dependencies without touching their code?, How can I instrument my application manually?" - also it gives me a bunch of terms I'm not really ready to understand yet (exporters? propogators? resources? I've no idea what that is).
"Next, you can deep dive into the documentations for the language you are using:" -- awesome - this is what I want. So I'm going to click "Go". I click that, nice, I see the status, etc. Let's go into "Getting Started".
"Welcome to the OpenTelemetry for Go getting started guide! This guide will walk you through the basic steps in installing, instrumenting with, configuring, and exporting data from OpenTelemetry" - love it, that's what I want to do.
Simple Fibonacci demo, fantastic.
Got to Trace Instrumentation. Uh oh, not what I want to do. Skipping past it, scrolling down...
I get into exporters, resources... but nothing on metrics. Uh oh. (If I dig through Java, same thing, I can check other languages but I expect the same).
Okay, now I'm on my own again, and I don't know where to go. When in doubt -- search? (https://opentelemetry.io/search/?q=Metrics)
Searching "Metrics" gets me to the top result of: https://opentelemetry.io/docs/concepts/signals/metrics/ - which.... is useful, but doesn't let me get anything done.
Let's try searching again, and just clicking everything that comes up, because I'm out of ideas at this point.
I end up at https://opentelemetry.io/docs/reference/specification/metrics/semantic_conventions/ and https://opentelemetry.io/docs/reference/specification/metrics/api/ and https://opentelemetry.io/docs/reference/specification/metrics/sdk/ seems to sort of help, but maybe I'll come back to this, since I see some concepts possibly explained?
Okay, last idea! Let's see if there are examples I can look at! Let's go to https://github.com/open-telemetry
I find https://github.com/open-telemetry, nice.. let's see if there is a Go repo...
https://github.com/open-telemetry/opentelemetry-go good stuff, I see there is an example repository, we drop in there...
Okay, so the examples are based on.... what product they integrate with, but not the end user case. prometheus is usually used for metrics, so let's see what's in there! (https://github.com/open-telemetry/opentelemetry-go/blob/main/example/prometheus/main.go)
Oh hey! I got lucky! I see metrics!
This gives me a starting point, but now I have to go search through the Otel docs and I assume the Go API reference for this code to work out what each bit is, and how it all works together, but now I'm tired, and I don't know if I want to. I'm tempted to just reach for platform specific tools because it's an easier onboarding path, and even if the docs aren't great there, it's probably a wider community, so I can find more resources.

To reiterate my hight level point:   I found that the docs and onboarding process for Otel are written from the perspective of someone who knows what OTel is. It didn't seem to me to be written from the perspective of someone who has a user journey they are going on. This is where OC's docs where excellent -- they were user first, and that made them great for getting things done.
My strong suggestion would be to restructure the docs, into something that is close to what Open Census has:

Quckstarts, broken down by use case (metric, trace, logging), per language
A Tracing top level section
A Metrics top level section
A logging top level section
A section on installation and operation
etc etc.

Happy to discuss more, but wanted to pass the feedback on in hope that it's useful, and we can get more people (myself included!) onto OTel!

Hey @markmandel, thank you very much for sharing your journey with our docs. While it hurts reading some of what you wrote, you are spot on and we are well aware of most of the issues you called out. Let me try to address some of them here, but before going into that let me put a few things upfront:

We had some discussions on "journey focused" documentation already, that's why you find the "Devs" and "Ops" Getting Started, it's a work in progress change that's mainly stuck because of the second point
The docs across languages are fairly inconsistent right now. That has some historic reasons of very big differences in stability across the spec and across those languages. With metrics & logs being stable now, we are at a tipping point and started to fix that.
While we discussed some major reworks of the docs in the past a few times, we always came to the conclusion that we will not do a big project redoing everything that's planned to take 3 months and takes 3 years, instead we do incremental change, this is SLOW but at the end will let to something
Finally, we (the docs team) are fairly limited in size, so we are eagerly hoping for people to help us out to improve our documentation (yes that's an implicit invite to help :-) although I guess you have your own hand full of work for Agones and Quilkin, etc)

Again, I appreciate your feedback, I just wanted to fill in the context, now on some of your individual points:

A while back (April) I looked at how to do the migration from Open Census

The migration guides (living in the spec) have been created in April and we announced sunsetting OpenCensus in May, so you might find some more help here: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/opencensus.md

I go to https://opentelemetry.io/ and click on Docs (https://opentelemetry.io/docs/)

It looks like we need to make it more obvious that going through the "Getting Started" (for Devs & Ops) is the way to go if you want to take action

I get a section on "Reliability & Metrics" - cool. I know what metrics are, but I need to do a thing, and this isn't explaining that yet. Okay, I see another link in that section on "metrics", let's see where that goes.

Oh... that shouldn't happen.

I click "Getting Started and I get two options: "Dev" and "Ops". To be honest, I've no idea what this necessarily means, but I'm mostly sure I'm "dev", so let's go there.

We went back and forth with that and still didn't hit the nail apparently 😆 ... Looks like we need to rework that.

also it gives me a bunch of terms I'm not really ready to understand yet (exporters? propogators? resources? I've no idea what that is).

Looks like we need some concept pages for these as well.
From there on, let me comment on the Go docs: Metrics for go are in the process of becoming stable (see here for more details https://github.com/orgs/open-telemetry/projects/34 and check in with the go community). As outlined before this is a common pattern across the documentation ( I don't call it "problem" or "issue" on purpose, since it's just a matter of fact that OpenTelemetry is graduating and that means that some things are just not there yet)
See our docs also on that point: https://opentelemetry.io/docs/instrumentation/go/manual/#creating-metrics

To reiterate my hight level point: I found that the docs and onboarding process for Otel are written from the perspective of someone who knows what OTel is. It didn't seem to me to be written from the perspective of someone who has a user journey they are going on.

I do not disagree with that, only can point out the reasons as I have done above

This is where OC's docs where excellent -- they were user first, and that made them great for getting things done.
My strong suggestion would be to restructure the docs, into something that is close to what Open Census has:

For sure a place we can learn from (or even copy over some existing content)

Quckstarts, broken down by use case (metric, trace, logging), per language

The "Getting Started" / Quickstart is focused around automatic instrumentation right now (for those languages that have one), where you get all of that in one shot (if available). For languages which do not have automatic instrumentation (which go is an instance of, although there is some auto instrumentation in the works), we have no consistent idea yet, but it will be probably one guide having all signals (traces, logs, metrics) done together.

A Tracing top level section
A Metrics top level section
A logging top level section

Looking at the OpenCensus pages, this very closely looks like our concept pages on those topics (https://opentelemetry.io/docs/concepts/signals/) (minus SOME code examples in the OC docs, minus a bunch of docs that are just not there yet, e.g. for logs)

A section on installation and operation
etc etc.

Happy to discuss more, but wanted to pass the feedback on in hope that it's useful, and we can get more people (myself included!) onto OTel!

Your feedback is very useful. I reiterate that because I am very sure that some of my feedback on your feedback sounds defensive and not appreciative, so I apologies for that.

Yeah, I think there's two distinct problems here:

Metrics for Go are still very new. They suffered a lot of stability issues for over a year and so we intentionally didn't document anything because stuff was breaking all the time. It was just an absolute slog to implement for the Go SIG and they got it done, but there's docs gaps now.
We don't have a good quickstart-like experience at the top level. There's sorta that on a per-language level, for some languages, but you also need to know to get in there.

I think (1) is much more directly solvable.
I think (2) could be tricker from an IA standpoint. Since we need to consider operator workflows there's no good single quickstart unless we divide it up by Developer/Operations. And even then, that stratification can be challenging because someone in an operations role may indeed also have to add instrumentation for a language.

Thanks for the measured response and commentary!

Your feedback is very useful. I reiterate that because I am very sure that some of my feedback on your feedback sounds defensive and not appreciative, so I apologies for that.

Not at all! I think we're all on the same page here trying to get to a good place 👍🏻

While we discussed some major reworks of the docs in the past a few times, we always came to the conclusion that we will not do a big project redoing everything that's planned to take 3 months and takes 3 years, instead we do incremental change, this is SLOW but at the end will let to something

This totally makes sense, and I figured it was what was going to happen 👍🏻

Metrics for Go are still very new. They suffered a lot of stability issues for over a year and so we intentionally didn't document anything because stuff was breaking all the time. It was just an absolute slog to implement for the Go SIG and they got it done, but there's docs gaps now.

I totally get the impetus (and I've definitely done this myself! 😄 ) - what I feel would make things less generally confusing/frustrating (at least in the short term) as a new user is to be super clear on what is documented and isn't (particularly for each language) - ideally at a high level, to make choosing which navigation path to go down.
I hate to keep using OC here 😄 but if I look at the Quickstarts for different languages:

I can immediately see that C# doesn't have a quickstart for Metrics. While it may be ideal to have it long term, it short circuits any attempt to go into a Quickstart assuming that what I need might be there, since I know it's not.
This could be done in the OTel docs as a structure of the top level nav of the Quickstarts (where I think you might end up eventually anyway?) a change to the titles ("Getting Started with Tracing"), and/or maybe even a header on each quickstart stating which area is being covered.
So TL;DR: my recommendation -- just be super clear about what isn't documented as much as what is.

We don't have a good quickstart-like experience at the top level. There's sorta that on a per-language level, for some languages, but you also need to know to get in there.
I think (2) could be tricker from an IA standpoint. Since we need to consider operator workflows there's no good single quickstart unless we divide it up by Developer/Operations

This is where I would come back to structuring by "what do I want to do?" - as an end user, I've no idea if I'm dev or ops -- but I do know what I want to do. Is it create a trace? Deploy some kind of collector / draw pretty graphs traces etc" - that's far easier for me to navigate than anything else. And to agree with your point, the lines between dev and ops blur all the time 😄 it's too hard to know which side someone thinks they are on at any given moment.

This could be done in the OTel docs as a structure of the top level nav of the Quickstarts (where I think you might end up eventually anyway?) a change to the titles ("Getting Started with Tracing"), and/or maybe even a header on each quickstart stating which area is being covered.

I don't think we can easily split out the "Getting Started" by Signal: for some languages you get all of them through the Getting Started in one shot (see Java for example). I also struggle with this because it would add yet another level into the documentation, bloating everything even more.
I see your point that we need to do a better job on showing what is available in which language, we can think about a few ways on improving here, although this is a temporary issue as we hope to have traces, metrics & logs with all major languages done eventually: there is no reason for a language to not implement them all.

This is where I would come back to structuring by "what do I want to do?" - as an end user, I've no idea if I'm dev or ops -- but I do know what I want to do. Is it create a trace? Deploy some kind of collector / draw pretty graphs traces etc" - that's far easier for me to navigate than anything else. And to agree with your point, the lines between dev and ops blur all the time 😄 it's too hard to know which side someone thinks they are on at any given moment.

We are already giving you a set of things you want to do:

Automatic Instrumentation (all the signals for you for free)
Manual Instrumentation (Create a Trace, create a Metric, create a log)
Collector Documentation

Have a look at the Java Docs, which are ones of the most complete ones here:
https://opentelemetry.io/docs/instrumentation/java/manual/
The right site navigation bar gives you some of what you are looking for.
Yes, we could improve things by splitting out sections into pages, but I am currently reluctant to do that, since it adds a LOT more complexity to the docs while things are so much in a flux right now.


Workflow and InfoPath scanner updates

get a workflow action to Flow action mapping
Add a workflow analysis component that gets the used workflow actions and then calculates a “Upgradability score”. This will help customers assess to what extend their workflows could be implemented using Flow
Make "usage" information as rich as possible (last WF exection, usage of the site hosting the WF) ==> make an activity score?
Add InfoPath web part usage to the InfoPath inventory

Done


Migrate components from radium to styled-components

Components using radium-grid:

[ ] Auth
[ ] Cart
[ ] Contact
[ ] EditPage
[ ] Home/index
[ ] InfoPage/EditInfoPage
[ ] InfoPage/InfoPageView
[ ] Loader
[ ] MyDsc
[ ] OauthCallback
[ ] Page
[ ] Plasmids/PlasmidDetail
[ ] Plasmids/PlasmidTable
[ ] Plasmids/index
[ ] SearchBar
[ ] ShoppingCart/index
[ ] Strains/StrainDetail
[ ] Strains/StrainTable
[ ] Strains/index
[ ] TableLoader
[ ] editor/InlineEditor
[ ] App

Radium and radium-grid have been removed, and the app now uses styled-components and rebass. Current todos:

[] Make sure each component is mobile responsive
[] Remove custom CSS/SCSS
[] Fix panels
[] Fix header/navbar formatting

Components to test for mobile responsiveness w/ rebass:

[ ] Home (& its child components)
[ ] Editable pages
[ ] Strain catalog
[ ] Plasmids catalog
[ ] Shipping & payment forms
[ ] Shopping Cart

Need to remove the extra line from the plasmid and strain detail pages.
Is there a reason that the table content on these pages is split into two data arrays? It seems like it might be easier to just put all of the content into one array, then map over them to autogenerate a table. That would solve the blank lines problem, too.

Aside from the issue noted above, the migration is pretty much complete. Most pages have been entirely converted to Styled Components, with the only real Bootstrap remnants left being the stylized buttons.

Buttons and forms have been converted to SC. Now working on replacements for the table-responsive classes.

Every last scrap of Bootstrap has been removed and converted to SC. Will close this issue after merging into develop

develop now up to date with 100% styled-components