javadoc + binary xent scale by two

Build triggered. sha1 is merged.

Build triggered. sha1 is merged.

@AlexDBlack, please review and make changes as necessary.
Currently we have p,1-p hard coded into the binary cross entropy loss function and which is also accounted for in softmax. So the gradients for softmax+binarycross entropy gets a *2 scaling but works fine for binary cross entropy + sigmoid etc. Doesn't really matter for training as it's a constant scaling but should fix or clarify for users?
Also clarified in javadoc that binary cross entropy only works with one hot encoded labels of size two. And to use negative log likelihood or multiclass cross entropy in other cases.

"Labels are assumed to take one hot values of size two [0,1] or [1,0]"
This is definitely not the case.
BinaryXENT has to support arbitrary sized labels (not restricted to size 2; could be 1, could be N>2); in general it won't be one-hot either.


Update CI to use 0.35.1

Set master as 1.0.0-dev 🎉
Fix the changelog date
Bump distribution-scripts required to build 1.0.0-dev

There we go, time to merge #9507 first :)


Workspaces Not Saving or Opening Properly

Running Coder in docker on a mac and tested on Centos VM

Description
When I try to create or work in a workspace it is opening like a folder instead of a file. The workspace name shows up for me still as "Untitled" and the workspace below it the has the name of the workspace I tried to open. This causes terminal etc to index in the wrong spots etc.
If i try to save a workspace to create a new one, I see the save issues.
Steps to Reproduce

File
Save Workspace as
Save File enter a path: /home/test
The workspace now shows up properly as "test"
Try to add a folder to the workspace
The workspace name now changes to "Untitled", with the name of the workspace now in the folder spot. This breaks indexing and plugins etc.

This also causes Project plus to not work as without any root or folders defines it gets a URI error.

I believed that the workspaces functionality is not implemented yet.

Any news on when this will be solved?
We are still on version 1.604-vsc1.32.0, because we rely on opening multiple folders.

Duplicate of #196.

I was evaluating for my school's computer lab, but will drop it and moved on as work-space functionality is not working: having the ability to have more than one workspace. Code-server can take off (especially in schools teaching environment if this issue can be resolved.


Remove unwanted css

Provide new plugin for to remove unwanted css in material-ui / css

Please use the feature request template provided to guide you in submitting a feature requrest.

Duplicate of #14961


Dockerfile Build fail

https://github.com/easegress-io/easegress-portal/blob/main/rootfs/Dockerfile#L46
COPY --from=builder /app/public ./public
the public directory doesn't seem to exist

public is a empty folder in repo. it seems git not add it, we will fix that. you can add a simple empty folder public in repo to fix it now.


vendor: update backupstore

Depends on https://github.com/longhorn/backupstore/pull/112
Longhorn/longhorn#5628

@derekbit this will change nfs mount to use soft mode, but we didn't tackle https://github.com/longhorn/longhorn/issues/5662 yet. Probably we should draft this backport first until 5662 has been tackled?

@derekbit this will change nfs mount to use soft mode, but we didn't tackle longhorn/longhorn#5662 yet. Probably we should draft this backport first until 5662 has been tackled?
Wait this is the update to longhorn-engine instead, so it should be fine.

https://github.com/longhorn/longhorn/issues/5662 depends on https://github.com/longhorn/longhorn/issues/5628, so need to update this first.


Update phpdevtools to v4, phpcsfixer to v3 with Updated composer.lock

Questions
Answers

Description?
composer update  is shown as below

Type?
improvement

BC breaks?
no

Deprecations?
no

Fixed ticket?
No issue at all

Sponsor company
Your company or customer's name goes here (if applicable).

How to test?
Automatic checks pass, no more php-cs-fixer v2 deprecated warning. In theory this PR does NOT modify the blockwishlist behavior, it only updates composer dependencies. Please verify blockwishlist works as usual.

Updating dependencies
Lock file operations: 15 installs, 20 updates, 4 removals

Removing paragonie/random_compat (v2.0.18)
Removing symfony/debug (v3.4.42)
Removing symfony/polyfill-php70 (v1.17.0)
Removing symfony/polyfill-php72 (v1.17.0)
Locking composer/pcre (1.0.1)
Upgrading composer/semver (1.5.1 => 3.3.2)
Upgrading composer/xdebug-handler (1.4.2 => 2.0.5)
Upgrading doctrine/annotations (v1.4.0 => 1.14.3)
Locking doctrine/deprecations (v1.0.0)
Upgrading doctrine/lexer (1.0.2 => 2.1.0)
Upgrading friendsofphp/php-cs-fixer (v2.16.3 => v3.4.0)
Upgrading nikic/php-parser (v3.1.5 => v4.15.4)
Upgrading php-cs-fixer/diff (v1.3.0 => v2.0.2)
Locking prestashop/autoindex (v2.1.0)
Upgrading prestashop/header-stamp (v1.2 => v2.3)
Upgrading prestashop/php-dev-tools (v3.4 => v4.3.0)
Locking psr/cache (1.0.1)
Locking psr/container (1.1.1)
Locking psr/event-dispatcher (1.0.0)
Upgrading psr/log (1.1.3 => 1.1.4)
Upgrading squizlabs/php_codesniffer (3.5.5 => 3.7.2)
Upgrading symfony/console (v3.4.42 => v5.4.22)
Locking symfony/deprecation-contracts (v2.5.2)
Upgrading symfony/event-dispatcher (v3.4.42 => v5.4.22)
Locking symfony/event-dispatcher-contracts (v2.5.2)
Upgrading symfony/filesystem (v3.4.42 => v5.4.21)
Upgrading symfony/finder (v3.4.42 => v5.4.21)
Upgrading symfony/options-resolver (v3.4.42 => v5.4.21)
Upgrading symfony/polyfill-ctype (v1.17.0 => v1.27.0)
Locking symfony/polyfill-intl-grapheme (v1.27.0)
Locking symfony/polyfill-intl-normalizer (v1.27.0)
Upgrading symfony/polyfill-mbstring (v1.17.0 => v1.27.0)
Locking symfony/polyfill-php73 (v1.27.0)
Locking symfony/polyfill-php80 (v1.27.0)
Locking symfony/polyfill-php81 (v1.27.0)
Upgrading symfony/process (v3.4.42 => v5.4.22)
Locking symfony/service-contracts (v2.5.2)
Upgrading symfony/stopwatch (v3.4.42 => v5.4.21)
Locking symfony/string (v5.4.22)
Writing lock file

Thanks @leemyongpakvn


Error writing Spacing Between Slices DICOM tag?

I’m writing a DICOM volume after some image processing, and to do that I’m following the code in the example.
Other than the tags modified in the example, I also want to chage the Spacing Between Slices tag (0018,0088), but once the Dicom images are written, the tag is always set to 1.
Here is an extract of the code (after having loaded a sitk image, img, and its metadata, reader):
# Initialize writer object
writer = sitk.ImageFileWriter()

# Use the study/series/frame of reference information given in the meta-data
# dictionary and not the automatically generated information from the file IO
writer.KeepOriginalImageUIDOn()

# Copy all the DICOM tags from the first slice
tags_to_copy = reader.GetMetaDataKeys( 0 )

# Get spacing between slices (DICOM tag "0018|0088")
spacing = img.GetSpacing()[2]

# Create list of tuples (tag, values), and add the ones to modify
series_tag_values = [
                            (k, reader.GetMetaData(0, k))
                            for k in tags_to_copy
                            if reader.HasMetaDataKey(0, k)] + \
                        [   # Tags to modify #
                            ("0018|0088", str(spacing)),    # Spacing Between Slices
                         ]

# Save DICOM slices
for i in range( img.GetDepth() ):
        image_slice = img[:, :, i]

        # Tags shared by the series.
        for tag, value in series_tag_values:
            image_slice.SetMetaData(tag, unidecode.unidecode(value))

        # Slice specific tags.
        #   Instance Creation Date
        image_slice.SetMetaData("0008|0012", time.strftime("%Y%m%d"))
        #   Instance Creation Time
        image_slice.SetMetaData("0008|0013", time.strftime("%H%M%S"))
        #   Instance Number
        image_slice.SetMetaData("0020|0013", str(i))
        #   Image Position (Patient)
        image_slice.SetMetaData("0020|0032", '\\'.join(
            map(str, volume.TransformIndexToPhysicalPoint((0, 0, i)))))
        # In-Stack Position Number
        image_slice.SetMetaData("0020|9057", str(i))

        # Write to the output directory and add the extension dcm, to force writing
        # in DICOM format.
        writer.SetFileName( os.path.join( saving_directory, "IM-" + str(i).zfill(4) + '.dcm') )
        writer.Execute(image_slice)

If I print the SpacingBetweenSlices tag just before the writer execution,
by image_slice.GetMetaData("0018|0088"), I get the correct spacing, but once the whole volume is written the spacing between slices tag is always changed to 1.
I have the same problem even if I leave the tag empty using:
# Create list of tuples (tag, values), and add the ones to modify
series_tag_values = [
                            (k, reader.GetMetaData(0, k))
                            for k in tags_to_copy
                            if reader.HasMetaDataKey(0, k)] + \
                        [   # Tags to modify #
                            ("0018|0088", "",    # Spacing Between Slices
                         ]

Do you think I might be doing something wrong, or is there a bug in the writing of the tag?

For DICOM, I believe SimpleITK computes the spacing base on the distance between neighboring slices automatically.  So I don't think you can just set the value yourself.

But none of my spacing values correspond to 1, so even if it was calculating it itself, there would be something wrong I guess

Hello @wwaleww,
The DICOM tag 0018|0088 is not used by ITK when writing the output. It either relies on the slice origin or if tag 0020|0032 is in the meta-data dictionary will use that.
To help us debug the issue:

Please run the example code using your image as input and see that the gaussian blurred output is as expected (correct sizes, spacing, all slices are in the same series etc.), this will indicate if the issue is with the code or something unexpected with the data.
Before writing the output DICOM slices print their origins and spacing, per slice, see if each slice is a 3D image which it should be. The expected spacing is (dx,dy,1) and origin should be (x,y,z).

Hello @zivy ,
So I tried what you suggested:

Using the example code I load my image, which has spacing (dx, dy, dz) and origin (x, y z). None of those values are = 1. I then write the DICOM volume using the example code (the 0018|0088 tag is not explicitly modified). When I load the image rewritten by SimpleITK, and I call for image.GetSpacing() and image.GetOrigin(), the SimpleITK image spacing and origin are correctly recovered as (dx, dy, dz) and (x, y, z). But in the metadata the tag 0018|0088 is changed to 1.

If I print the origin and spacing per slice I get a 3D image, with expected spacing (dx, dy, dz) and origin (x,y,z).

The only thing that is wrongly written is the 0018|0088 tag. In my case it can be problematic because the metadata are not only seen by ITK users, but also by radiologists, so even if ITK handles the image spacing correctly independently of the 0018|0088 tag, I would like to print the tag correctly.
But then I ask myself: where does ITK take the 3D spacing from if not from 0018|0088? Because I see, if I interpret correctly the code, that 2D spacing is taken from the PixelSpacing 0028|0030 tag, but what about the third dimension? I don't see how it can reconstruct it only from 0020|0032, which just represents the Origin point, but carries no info about spacing.
Thanks for your answer!

Hello @wwaleww,
No magic with the third dimension. The important thing to understand is that the 2D images are actually positioned in 3D (using image position patient + direction cosine matrix, section C.<IP_ADDRESS> in the DICOM standard). If you have multiple images and assume that they are equally spaced you can compute the spacing from that information, which is what ITK does (see code here).
With respect to setting the value for 0018|0088, that should probably be a feature request on the ITK issue tracker. It makes sense that ITK will ignore this entry in the meta-data dictionary as it can be arbitrary and different from the actual image spacing, but it is a reasonable expectation that this tag be written correctly in the DICOM header.

Thanks a lot for the explanation @zivy, and I'll follow your suggestion!


Make the Early Access banner less tall

Closes: #848

@n1tranquilla Merge if you think it safe to deploy.


dockertools: fix buildLayeredImage nix-store permissions

Motivation for this change
Fix bug in dockerTools.buildLayeredImage that causes non-root containers to be not runnable.
Pardon my python, but I was unsure how - in the prettiest way - to hook into the existing logic around /nix/store dir-creation.
example docker error message:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/nix/store/3avafdzgh694z5m3rgw2vvlgsfz2gmkl-coredns-1.6.6/bin/coredns\": stat /nix/store/3avafdzgh694z5m3rgw2vvlgsfz2gmkl-coredns-1.6.6/bin/coredns: permission denied": unknown.

The cause of the error is that /nix and /nix/store has wrong permissions inside the final image, i.e.:
bash-4.4# ls -dl /nix/store/
drw-r--r-- 32 root root 34 Jan  1  1970 /nix/store/

The top-level directories lack world execute permissions, which, is ok as long we run the image as root, but fails when the image is run as an unprivileged user.
Steps to reproduce

cherry-pick 4fc255e from my branch onto master
docker load <$(nix-build . -A dockerTools.examples.bashLayeredWithUser)
docker run --rm -it -u somebody bash-layered-with-user /bin/echo "it works"

Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[X] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

cc @LnL7 @utdemir

@utdemir sounds good to me

@roberth Valid point! I applied your suggestion just now using the fancy github feature. :) I'll squash those extra commits into my test-commit in a sec.

@ofborg test docker-tools

Ofborg is happy too! Thanks :)


[Bug]: Scores in the retrieved nodes are in reversed order in the Weaviate integration

Bug Description
Hello,
I was using the retriever from a vector store index that has been initialized from a Weaviate collection. I noticed that the retrieved nodes have scores in reversed order: the first (most relevant) node, has score equals to zero and as we move to the least relevant nodes, the score increases.
We found in the code that LlamaIndex performs subtraction 1 - score, where score is the score that the Weaviate returns. But the Weaviate now, returns similarity score instead of distance. I think that only in vector (instead of hybrid) search, the distance can be returned instead of similarity (see here). You can use the code I provide below (from a Jupyter Notebook) in order to see the scores that LlamaIndex gives and the scores that Weaviate returns.
Version
llama-index==0.10.53
llama-index-vector-stores-weaviate==1.0.0
weaviate-client==4.6.5
Steps to Reproduce
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext, Document
from llama_index.vector_stores.weaviate import WeaviateVectorStore
from llama_index.core.vector_stores import VectorStoreQuery
from llama_index.core.schema import TextNode

from llama_index.embeddings.text_embeddings_inference import TextEmbeddingsInference
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.callbacks import CallbackManager, TokenCountingHandler
from llama_index.core.node_parser import SimpleNodeParser

import weaviate
import os

from transformers import AutoTokenizer, AutoModel
import tiktoken
import requests
from IPython.display import Markdown, display

# In[ ]:
# Embeddings initialization: OpenAI
embed_model = OpenAIEmbedding(model="text-embedding-3-small", api_key=os.environ.get("OPEN_AI_API_KEY"))
tokenizer = tiktoken.encoding_for_model("text-embedding-3-small").encode

# In[ ]:
tokenizer_obj = tokenizer
# The chunk_size must be compatible with the sequence length of the embed_model_obj that is used.
chunk_size = 450
chunk_overlap = 50
# Initialize a node parser that we will use in the documents parsing.
# First initialize the TokenCountingHandler with our tokenizer and the CallbackManager with our token counter.
# And then the node parser.
token_counter_handler = TokenCountingHandler(tokenizer=tokenizer_obj)
callback_manager = CallbackManager([token_counter_handler])
node_parser = SimpleNodeParser.from_defaults(chunk_size=chunk_size,
                                                  chunk_overlap=chunk_overlap,
                                                  callback_manager=callback_manager)

# In[66]:
client = weaviate.connect_to_local()

# In[127]:
# Now that the collection is already created we just connected to it.
vector_store = WeaviateVectorStore(
    weaviate_client=client, index_name="Test"
)

# In[128]:
vector_store_index = VectorStoreIndex.from_vector_store(vector_store=vector_store,
                                                        embed_model=embed_model,
                                                        transformations=[node_parser],
                                                        show_progress=True)

# In[100]:
def get_wikipedia_article_text(title):
    url = "https://en.wikipedia.org/w/api.php"
    params = {"action": "query", "format": "json", "prop": "extracts", "explaintext": True, "titles": title}
    response = requests.get(url, params=params).json()
    page = next(iter(response["query"]["pages"].values()))
    return page.get("extract", "Article not found.")

python_doc_text = get_wikipedia_article_text("Python (programming language)")
lion_doc_text = get_wikipedia_article_text("Lion")
lion_paragraph = lion_doc_text[:1000]

# In[25]:
python_doc = Document(doc_id='1',
                      text=python_doc_text,
                      metadata={
                           "title_of_parental_document": "Python_(programming_language)",
                           "source": "https://en.wikipedia.org/wiki/Python_(programming_language)"
                       })

# In[101]:
lion_doc = Document(doc_id='2',
                    text=lion_paragraph,
                    metadata={
                       "title_of_parental_document": "Lion",
                       "source": "https://en.wikipedia.org/wiki/Lion"
                   })

# In[104]:
vector_store_index.insert(document=python_doc)
vector_store_index.insert(document=lion_doc)

# In[129]:
retriever = vector_store_index.as_retriever(similarity_top_k=10,
                                            vector_store_query_mode="hybrid",
                                            alpha=0.5)
nodes = retriever.retrieve("What is lion?")

# In[131]:
# Always the retriever returns a list of nodes in decsending order based on the score (most relevant chunks going first in the list).
# But why here the most relevant chunk has a zero score?
for node in nodes:
    print(node.text)
    print()
    print(node.score)
    print("__________________________________________________________________________________________________________")
    print("__________________________________________________________________________________________________________")

print([node.score for node in nodes])
# The scores are: [0.0,  0.9217832833528519,  0.9288488179445267,  0.9365298748016357,  0.937725093215704,    #0.9396311119198799,  0.9409564286470413,  0.9446112886071205,  0.9455222226679325,  0.9476451091468334]

# In[108]:

# Code to query Weaviate without LlamaIndex.
query = "what is lion?"
query_vector = embed_model.get_query_embedding(query=query)

# In[121]:
# query_vector
from weaviate.classes.query import MetadataQuery

# In[123]:
from weaviate.classes.query import MetadataQuery
response = collection.query.hybrid(
    query=query,
    vector=query_vector,
    return_metadata=MetadataQuery(distance=True,
                             certainty=True,
                             score=True,
                             explain_score=True),
    alpha=0.50,
    limit=10,
)
x=[]
for o in response.objects:
    print(o.properties)
    print()
    print(o.metadata)
    print("_______________")
   x.append(o.metadata.score)

print(x)

# Scores from weaviate:
# [1.0,  0.08082851022481918,  0.07240726053714752,  0.07034952938556671,  0.0660715326666832,  #0.06566160172224045,  0.06270790100097656,  0.05426621064543724,  0.05242578685283661,  0.05184878036379814]

Relevant Logs/Tracbacks
No response

@terilias Seems like you have a decent understanding of the issue. Feel free to open a PR 👍🏻

Hi @logan-markewich , thanks, I would like to give it a try! I have found some more issues in the Weaviate integration and I was thinking to post them for reference, each in a separate thread. Is that ok?

@terilias go for it!

any updates on this?

Sorry, I have really delayed fixing it as I suggested! Please give me some days as I am intend to fix it!

Hi @logan-markewich, after so much delay I finally created a PR. I hope it is ok (this is my first ever PR).


Env variable to disable/enable namespace isolation policy

LGTM.  You'll want to update the calico-docker docs to reflect this as well.
https://github.com/projectcalico/calico-docker/blob/master/docs/kubernetes/KubernetesIntegration.md


Kindly explain under which circumstances the access_token needs to be stored and used

For Phoenix application action authorizations or when calling Google API only?

@acrolink an access_token will need to be used anytime you need to make a request to an OAuth2 provider's API (i.e. Google).
Storage of the access_token is outside the concern of this library.


Does Edge Runtime use alternative vm instead of node:vm?

Questions & Answers
Recently I'm interested in how does Edge Runtime work.
I investigate some code in edge-runtime/vm and I found this code below.
import type { CreateContextOptions } from 'vm'
import { createContext, runInContext } from 'vm' // is it same as 'node:vm'? or alternative something?

But what I confused is where does vm module came from.
In document, Edge Runtime doesn't use Node Runtime. so I think it's not equal to node:vm.
Can you explain about vm that edge runtume use?
Context of your question
Help us to understand the mental roadmap we need to follow to reach the same question.
I want to know what type of vm edge runtime use.
does it use vm-browserify ? or alternative something instead of node:vm ?

Hello, it's the same 🤠

@Kikobeats
Thanks for replying! But I have one more question sadly..
That means Edge Runtime is running on Node Runtime?
So it cannot use Node API because Edge runtime evaluate code in it's own sandbox environment?
Then how can we use Edge runtime in Edge server? Does that server need nodej
Oh... I think I have tons of misunderstanding about this. But can you explain more if you have some time?

When you run vercel dev or next dev, it's spawning a Node.js process that uses this project.
This project is simulating the Edge Runtime constraints you are going to experience when you deploy a Vercel Function
We are using Node.js for development, but you are not using it when you deploy to real infrastructure.
Hope this clarifies a bit!


A small percentage of DEP runs are stopping before end of this year.

For example, huc12:<PHONE_NUMBER>02, fpath: 82 stops after 2014.

The reason is that the rotation codes after 2014 are all U (undeveloped).  Will leave this open until we can determine a formal solution to this situation.

This generally remains a problem :( but very isolated.

Quantifying this, 1,508/1,501,075 OFES, so 0.1%


Authentication does not work

I am testing this against my Reolink C1-Pro, with username "admin" and password "password". I can get the low bitrate video stream by running:
./openRTSP -4 -I <IP_ADDRESS> -u admin password rtsp://<IP_ADDRESS>:554/h264Preview_01_sub > video.mp4

I can also get the full quality stream by swapping sub for main.
I have now tried using myRtspClient, and setting the same username and password in the code, but I am unable to start the streaming:
# ./complete_example rtsp://<IP_ADDRESS>:554/h264Preview_01_sub
Start play rtsp://<IP_ADDRESS>:554/h264Preview_01_sub
Then put video data into test_packet_recv.h264
RTSP/1.0 200 OK
CSeq: 1
Date: Mon, May 21 2018 21:05:28 GMT
Public: OPTIONS, DESCRIBE, SETUP, TEARDOWN, PLAY, PAUSE, GET_PARAMETER, SET_PARAMETER

Select Timeout
CheckAuth: error
DoDESCRIBE error
#

Searching trough the code, I cannot see the username and password being used much. It seems to me  that it is not being used at all really, as the MakeMd5DigestResp calls are all missing the username and password. Is this supposed to be working?
Best regards

It seems that after building everything from scratch, this works. Not quite sure what was wrong, but git clean and make made a difference. Sorry for the noise.


#3678 Update comment input box in the top  of comments

Fixes #
Describe the changes you have made in this PR -

[x] Updated the Add new comment box above the old comments
[x] Tests written for new code(and old code if feasible)
[x] Linter and other CI checks pass

Note: Please check Allow edits from maintainers. if you would like us to assist in the PR.

Hi @aadarshahebsingh
Thanks for raising the PR
Please assign the issue #3678 to yourself first.


docs: add doc comments for git source and friends

What does this PR try to resolve?
More doc comments for GitSource
How should we test and review this PR?
cargo doc --document-private-items --no-deps --open

Then proofread it!
Additional information
More follow-ups for other source implementations are coming.
I added some TODO(refactor) and will tackle them after this gets merged.

@bors r+

:pushpin: Commit 9a29b8b45c235a85ad2fea33a14ad167bb03b8d6 has been approved by epage
It is now in the queue for this repository.

:hourglass: Testing commit 9a29b8b45c235a85ad2fea33a14ad167bb03b8d6 with merge 6956c04ec9c7d916392a85aff9c47dfb934e6ade...

:sunny: Test successful - checks-actions
Approved by: epage
Pushing 6956c04ec9c7d916392a85aff9c47dfb934e6ade to master...

:sunny: Test successful - checks-actions
Approved by: epage
Pushing 6956c04ec9c7d916392a85aff9c47dfb934e6ade to master...

:eyes: Test was successful, but fast-forwarding failed: 422 Changes must be made through a pull request.


Fix #14 invalid kibana_hostname

Fix #14 by setting the kibana_hostname equal to the ES endpoint rather
then then aws_elasticsearch_domain.kibana_endpoint.
aws_elasticsearch_domain.kibana_endpoint is the complete URL to the
endpoint including the path /_plugin/kibana/ which makes for an invalid
CNAME.

/codefresh run test

/rebuild-readme

@maximmi thanks for looking at this.
Do you have doc for how to enable access for your bot?

@maximmi I rebased on upstream master and manually ran the make readme and force pushed. Thanks for your help.

/codefresh run test

@maximmi checking back here if you need anything from me?

@igable here is new release: https://github.com/cloudposse/terraform-aws-elasticsearch/releases/tag/0.10.0


Fix252

Fixes #252

There's a new subVI in FromClusterToTagBusScript that I'm not sure of the purpose:

It looks like it's just a pass through. Did you mean to include it?
Otherwise the change looks good to me.

@theSloopJohnB Fixed

Looks good!

I merged in the latest from master at GitHub's request - it has a VI not touched in this branch. I'll wait for the build to finish and then merge.

Build six failed because a file was still open when LabVIEW tried to delete it. I opened the build machine and saw that VIPM was running. I closed it and re-ran the build - success! I'll merge the change now.


#386: Addressbook transaction details

Depends on #399
Closes #386
Closes #389
Description

Display known addresses from the address book on the transaction details
Also let add or edit entrys within the transaction details

Looks like this:
User has the address on the address book

User does not have the address on the address book

Travis automatic deployment:
https://pr407--safereact.review.gnosisdev.com

Travis automatic deployment:
https://pr407--safereact.review.gnosisdev.com

Travis automatic deployment:
https://pr407--safereact.review.gnosisdev.com

Travis automatic deployment:
https://pr407--safereact.review.gnosisdev.com

Travis automatic deployment:
https://pr407--safereact.review.gnosisdev.com

Travis automatic deployment:
https://pr407--safereact.review.gnosisdev.com

is this ready to be reviewed?

Lets wait until #399 is merged so we don't have to iterate a lot

Travis automatic deployment:
https://pr407--safereact.review.gnosisdev.com

You can review this now @mikheevm

Travis automatic deployment:
https://pr407--safereact.review.gnosisdev.com


Use globalThis with polyfill to get global object

Currently this library uses eval() to get the global object, but this does not work when using a CSP policy not allowing eval.
There is a stage 3 ECMAscript proposal to create a globalThis object to achieve the same result, so lets use it and polyfill it when needed
using https://github.com/ljharb/globalThis

I tested this PR with my app (browser only) and it still works. Tests are passing.


[Bug]: codespace fail to run idpbuilder create

What is your environment, configuration, and command?
env : codespace
command :  idpbuilder create --protocol http
--host ${CODESPACE_NAME}-8080.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN}
--port 8080 --use-path-routing
What did you do and What did you see instead?
err message
Additional Information. Logs.
failed to ensure docker network: command "docker network create -d=bridge -o com.docker.network.bridge.enable_ip_masquerade=true -o com.docker.network.driver.mtu=1500 --ipv6 --subnet fc00:f853:ccd:e793::/64 kind" failed with error: exit status 1

@csantanapr

https://github.com/kubernetes-sigs/kind/issues/3748


Handle port forward rule check for vpc and non vpc Isolated networks

Description
Port forwarding rules not checked properly on Isolated networks (non-VPC)
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[X] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

How Has This Been Tested?
Created Isolated network and added port forward rules for the Instance created in that network
Create an Instance in a tier of a VPC network and  ran the health check from UI and cmk with performfreshchecks as true
All tests passed for both the routers

@blueorangutan package

@andrijapanicsb a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1045

@blueorangutan test

@andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

tested. it works fine.

thx @weizhouapache, that's one thing less for me to test :))
Thx

Trillian test result (tid-1239)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 56268 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3963-t1239-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Smoke tests completed. 83 look OK, 0 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File


Fix bug with specifying parents

In https://github.com/alphagov/content-tagger/pull/319 we switched the taxon system to use "edition" level links. But we forgot to port over the reading of the links. Turns out that if you ask the publishing-api for links, the edition level links won't be returned.
By fixing this bug we can also remove a lot of stubbing!
https://trello.com/c/8YH5l9kF

Closing because I'm reverting the original PR.


[WFLY-14194] Driver processor: only remove drivers explicitly registe…

…red by a module
https://issues.redhat.com/browse/WFLY-14194
With this fix:
a. JSTL TCK tests pass
b. drivers from driver modules are being correctly unregistered
so this let's the TCKs pass but TBH I'm not sure if now we are preventing all leaks: f.e. wouldn't TCK deployment cause a leak?
OTOH even if it would and it is correct I think we cannot do much about it

In case it helps, org.jboss.as.jpa.processor.PersistenceUnitServiceHandler is an example of attaching information (PersistenceAdaptorRemoval) to the DeploymentUnit during deployment that can later be referenced during undeployment for performing cleanup.

so this let's the TCKs pass but TBH I'm not sure if now we are preventing all leaks: f.e. wouldn't TCK deployment cause a leak?
OTOH even if it would and it is correct I think we cannot do much about it

Are you saying that any application that directly references a DataSource, the JDBC driver may be registered with the DriverManager and will not be unregistered from the DriverManager?
If yes, I wonder if we should have a JCA management operation that helps:

List all registered JDBC drivers
Unregister a particular JDBC driver

Good discussion, although let's not let it block getting this merged if what is here is valid (LGTM), if what we're talking about is an RFE, or even if it's a bug if it's a minor one.
What's here plus the earlier change this amends seems like a solid piece of work on its own. The drivers the DUP instantiates in deploy() are cleaned up in undeploy().
If we have other cases of server provided services instantiating drivers from deployments and not cleaning them up, that's good to look into.
Application code that's directly instantiating drivers and not cleaning them up is an application that has its own CL leak. Being helpful about that sounds more like an RFE. It seems like such an app would be better off doing their own cleanup, say in a @PreDestroy method vs having their poor admins be responsible for invoking a management op.

Thanks, @tadamski !

Thanks a lot for the quick fix :+1:  @tadamski

@scottmarlow That seems a good idea. I will create a JIRA for that.


Sync txs from blockchain

because of db issues, feed might get corrupted.
once a day when app is in background re-read user txs from the block he joined. (ie restart the  wallet tx listeners from an old block)
for new users when they join create property "joinedAtBlock" and save the current block.
for existing users if joinedAtBlock doesnt exists use block 6400000

@mauriguz I followed the steps that in the PR I didn't find the 'Start sync of txs from blockchain'  and 'Sync of tx from blockchain finished successfully' messages in the console

@sirpy I've tested fix that was pushed but I didn't find the 'Start sync of txs from blockchain' and 'Sync of tx from blockchain finished successfully' messages
https://www.screencast.com/t/qB6x7J8uu
Checked on QA 1.7.0
Desktop: Windows 10 // Google Chrome 86.0.4240.75

it should be working. you need to go to background tab. also in console
type userStorage.userProperties.getAll()
and look for the lastSyncTX variable

@sirpy the lastTxSyncDate var is displayed

Checked on  QA 1.7.1-2
Desktop: Windows 10 // Google Chrome  86.0.4240.75


feat: implement serde support

Fix #1

Whoa! Thanks a lot!
Just a minor nitpick: we should not enable serde std feature by default, ideally serde should be included without default features, and then we could use cpf with serde on a no_std program. When we enable ``cpf stdfeature flag then we can enableserde std` feature aswell.

Whoa! Thanks a lot!
Just a minor nitpick: we should not enable serde std feature by default, ideally serde should be included without default features, and then we could use cpf with serde on a no_std program. When we enable cpf std feature flag then we can enable serde std feature aswell.

Thanks for the quick response. Completely agree. It's fixed now.

Amazing work! Thanks!


placeholder tile color

While tiles are being fetched the canvas has a somewhat grey color by default as in this issue. Is the color customizable in vector map tiles as well?

when using TileLayerMode.vector tiles that are not loaded are not displayed at all, so the colour of the parent widget to the map will show through

What about for TileLayerMode.raster?


Allow for negative counter values

There is no version constant to change in the go library right?

I dont think so, I think Go does the versioning via tags, right?

yes but in some projects we had constants anyway so the library could use it in things like user agent. Couldn't remember if this was one of those projects.

Oh, i see. This project doesn't have that, thanks for bringing it up. I usually always forget the version number changes :roll_eyes:


Create Quality Profiles and Quality Profile Groups with Recyclarr

Is there an existing issue for this?

[X] I have searched the existing issues

Feature Description
To implement the TRaSH guidance, you need to manually configure Quality Profiles, group and order Qualities, and ensure cutoff scores, desired languages, etc... are all configured correctly.  This setup is error prone and tedious to keep up-to-date when you have multiple Radarr/Sonnar instances and want to have "!" versions of profiles to keep the two instances in sync.
The proposal is to add "Quality Profile" and "Quality Group" definition to the yml config.
The suggested behavior:

The quality profiles are "declarative".  Meaning the groups for the quality profile will be erased and re-created per the Recyclarr config.  This is similar behavior toreset_unmatched_scores and delete_old_custom_formats.  If a property is not defined in the quality_profile config (e.g. language), then the property will not be updated or, when creating, will be set to the Radarr/Sonarr default is. The quality profile itself must not be deleted as that would change the quality profile's id which would break lists and other functions.
If a quality profile doesn't exist then it will be created.
Any quality profiles that not defined in the recyclarr configs will not be deleted or updated.

Example:

radarr:
  quality_groups:
    - name: "Bluray|WEB-1080p"
      qualities:
        - "Bluray-1080p"
        - "WEBRip-1080p"
        - "WEBDL-1080p"
        - "Bluray-720p"
    - name: "WEB 720p"
      qualities:
        - "WEBRip-720p"
        - "WEBDL-720p"

  quality_profiles:
    - name: "SQP-1"
      language: Original
      upgrades_allowed: True
      upgrade_until: "Bluray|WEB-1080P"
      minimum_cf_score: 10
      upgrade_until_cf: 10000
      qualities:
        - Bluray|WEB-1080p
        - WEB 720p

@rcdailey and I discussed that early on and we felt that the groups could be referred to by multiple quality profiles so it made sense to split them up as concepts.
Another option might be to move the quality groups to outside of the "service" definition (e.g. to a top level) that way the same quality groups could be used by different services (e.g. radarr-4k & sonarr-4k could use the same). Not sure though, what do you think?

A few thoughts:
We might also want to provide a way to enable ungrouped qualities as well. I'd expect the "groups" list to represent the ordering of qualities you see in the UI while editing a quality profile. Anything not mentioned in the list won't be checked. Items in the list will be sorted to the very top in the same order they appear under groups.
Obviously this means groups probably isn't the right name:
radarr:
  quality_groups:
    - name: "Bluray|WEB-1080p"
      qualities:
        - "Bluray-1080p"
        - "WEBRip-1080p"
        - "WEBDL-1080p"
        - "Bluray-720p"
    - name: "WEB 720p"
      qualities:
        - "WEBRip-720p"
        - "WEBDL-720p"

  quality_profiles:
    - name: "SQP-1"
      language: Original
      upgrades_allowed: True
      upgrade_until: "Bluray|WEB-1080P"
      minimum_cf_score: 10
      upgrade_until_cf: 10000
      qualities:
        - Bluray|WEB-1080p
        - WEB 720p
        - HDTV-720p

The above is almost identical to your original example, except groups is qualities instead. It accepts either fixed quality names (as they are known by the service) or custom group names, and uses this lookup order:

Names are first checked for matches in the quality_groups list.
If no match, then check against the hard-coded list of qualities provided by the service.
If no match, error.

The example above uses HDTV-720p directly, which is not a group. And I'd expect that to work. The only thing I don't like about this is that it's a bit confusing. How do you know what's a group or not from the array itself? I'm not sure if that's a big deal, though.

@rcdailey and I discussed that early on and we felt that the groups could be referred to by multiple quality profiles so it made sense to split them up as concepts.

I'd add to this that it's also an attempt to reduce nesting. Indentation can get out of control with some of these configurations. Breaking things out not only makes them reusable but also keeps things easier to read (structurally).

Another option might be to move the quality groups to outside of the "service" definition (e.g. to a top level) that way the same quality groups could be used by different services (e.g. radarr-4k & sonarr-4k could use the same). Not sure though, what do you think?

I think I'd solve this problem with YAML includes. The topic of having includes has come up a few times in the past. I've always avoided doing the work because I strongly feel that the UI should be doing this. Basically, in the UI I'd expect groups to be centralized (or: not specific to any one service type) and reference them anywhere. The resulting YAML spit out by the GUI application would duplicate the data in the necessary spots.

The example above uses HDTV-720p directly, which is not a group. And I'd expect that to work. The only thing I don't like about this is that it's a bit confusing. How do you know what's a group or not from the array itself? I'm not sure if that's a big deal, though.

What would happen in the event that you call your quality group "HDTV-720p" as well? That's possible in the UI - would you have to include a check to see if the config has defined a group with that name, and use that if it is, instead of the standalone quality?

I'd add to this that it's also an attempt to reduce nesting. Indentation can get out of control with some of these configurations. Breaking things out not only makes them reusable but also keeps things easier to read (structurally).

Fair enough. To me it seems more logical nesting it as part of a profile but I understand the reasons for not wanting to do it that way, but...

I think I'd solve this problem with YAML includes. The topic of having includes has come up a few times in the past. I've always avoided doing the work because I strongly feel that the UI should be doing this. Basically, in the UI I'd expect groups to be centralized (or: not specific to any one service type) and reference them anywhere. The resulting YAML spit out by the GUI application would duplicate the data in the necessary spots.

...but saying that, I like this approach. This I think is actually more consisten with the CFs themselves; Custom Formats are completely external in their composition, so when you add one to your config file you're saying 'pull this external thing into my config file'. Having an external file or files containing quality groups would conceptually be very similar.

The example above uses HDTV-720p directly, which is not a group. And I'd expect that to work. The only thing I don't like about this is that it's a bit confusing. How do you know what's a group or not from the array itself?

Agree that it should be: qualities and not groups in the quality_profile.  That was my mistake and I've updated the original post.   The logic could be "for each quality, see if there is a group with that name and, if not, then assume it's a standalone quality".
If you still wanted the quality to appear but not be enabled (because you want to control how manual search results are displayed), then you could include it and there could be an optional enabled property when the quality is referenced in the quality_profile.
Proposed example (and sorry in advance @rcdailey  that it adds more indentation):
  quality_profiles:
    - name: "SQP-1"
      language: Original
      upgrades_allowed: True
      upgrade_until: "Bluray|WEB-1080P"
      minimum_cf_score: 10
      upgrade_until_cf: 10000
      qualities:
        - Bluray|WEB-1080p
        - WEB 720p
        - HDTV-720p
           enabled: false

What would happen in the event that you call your quality group "HDTV-720p" as well? That's possible in the UI - would you have to include a check to see if the config has defined a group with that name, and use that if it is, instead of the standalone quality?

It would use the group without question. Direct access to the quality (instead of the group) in this case would be impossible. However, I'd expect that to not matter since presumably you have that quality within the group you're referencing. If not, then there's a naming issue and I would choose different names.

...but saying that, I like this approach. This I think is actually more consistent with the Custom Formats themselves; Custom Formats are completely external in their composition, so when you add one to your config file you're saying 'pull this external thing into my config file'. Having an external file or files containing quality groups would conceptually be very similar.

I'm open to reconsidering include directives in the YAML but it's not at the top of my priority list sadly.

Agree that it should be: qualities and not groups in the quality_profile. That was my mistake and I've updated the original post. The logic could be "for each quality, see if there is a group with that name and, if not, then assume it's a standalone quality".

I wouldn't necessarily call this your "mistake". I didn't go back and read our old Discord conversation, but this might just be a new corner case I didn't think of. In either case, I appreciate you giving this so much thought!

If you still wanted the quality to appear but not be enabled (because you want to control how manual search results are displayed), then you could include it and there could be an optional enabled property when the quality is referenced in the quality_profile.
Proposed example (and sorry in advance @rcdailey that it adds more indentation):
  quality_profiles:
    - name: "SQP-1"
      language: Original
      upgrades_allowed: True
      upgrade_until: "Bluray|WEB-1080P"
      minimum_cf_score: 10
      upgrade_until_cf: 10000
      qualities:
        - Bluray|WEB-1080p
        - WEB 720p
        - HDTV-720p
          enabled: false

I don't think that's valid YAML If you're using a mapping, all elements must be key value pairs; so you'd have to do something like this instead:
         - name: HDTV-720p
           enabled: false

But honestly, I don't like that: It's too verbose and cluttered. That's why I proposed the order of precedence in the lookup semantics. I'm leaning towards making these kinds of corners a user-naming issue and have them fix it. It surely isn't that difficult to select more unique names for groups.

Doh! You are correct, I thought there was a way to have a default key but I don't think so.
I was trying to capture another case.  What I learned recently is that even if a quality is not enabled for a profile, it still affects the weighting of the manual search results.  So people may want to include disabled qualities and specify their order.  Let me think on it.

affects the weighting of the manual search results

Can you explain this in greater detail?

Manual search results are shown in order of the listed qualities/quality groups regardless of whether they're selected or not, and then by custom format score within each quality/quality group.
It's also worth bearing in mind that it can affect what files are downloaded. Let's say you already have a 1080p file in your movie folder. You set up a new profile, and only have Bluray-720p selected, and ordered below all the 1080p qualities in the list. In this scenario no file will ever be downloaded, because the existing file beats the highest selected quality, even though the 1080p quality is itself not selected.

Manual search results are shown in order of the listed qualities/quality groups regardless of whether they're selected or not, and then by custom format score within each quality/quality group.

So the way I was thinking this would be implemented in Recyclarr is as follows. For any group or quality in the YAML:

Enable it (check the box)
Sort it to the top of the list

For example, let's say I have four qualities: a, x, y, z. And by default, Radarr sorts them like this:

a
x
y
z

Now, in my YAML I only specify x and z in this order:

z
x

Here is what would happen:

z and x get checked (enabled)
a and y get unchecked (disabled)
Any enabled items get moved to the top. All unchecked items appear at the bottom
Checked items are sorted in the order shown in the YAML, while still remaining above unchecked items.

The final list would look like this:

[x] z
[x] x
[ ] a
[ ] y

Given this, I don't think your first concern is an issue. Because if manual search results appear based on the order of the qualities and not their checked/unchecked status, the ordering is still managed by Recyclarr in such a way that the ones you care about (i.e. the checked qualities/groups) will appear on top above the unchecked.

It's also worth bearing in mind that it can affect what files are downloaded. Let's say you already have a 1080p file in your movie folder. You set up a new profile, and only have Bluray-720p selected, and ordered below all the 1080p qualities in the list. In this scenario no file will ever be downloaded, because the existing file beats the highest selected quality, even though the 1080p quality is itself not selected.

You said you sorted 720p below 1080p, so I think this behavior is expected. Aren't you telling Radarr that you prefer 1080p over 720p? Why would it replace the 1080p one with a 720p copy in this case?
One other thing you're leaving off is the cutoff quality which can also impact this behavior (well, it would prevent the search in the first place IIRC).
At the end of the day I don't see how the logical ordering I described above would cause an issue here. User is still in full control of whether or not 1080p qualities are above or below 720p.

Manual search results are shown in order of the listed qualities/quality groups regardless of whether they're selected or not, and then by custom format score within each quality/quality group.

So the way I was thinking this would be implemented in Recyclarr is as follows. For any group or quality in the YAML:
* Enable it (check the box)

* Sort it to the top of the list

For example, let's say I have four qualities: a, x, y, z. And by default, Radarr sorts them like this:
* `a`

* `x`

* `y`

* `z`

Now, in my YAML I only specify x and z in this order:
* `z`

* `x`

Here is what would happen:
1. `z` and `x` get checked (enabled)

2. `a` and `y` get unchecked (disabled)

3. Any enabled items get moved to the top. All unchecked items appear at the bottom

4. Checked items are sorted in the order shown in the YAML, while still remaining above unchecked items.

The final list would look like this:
* [x]  `z`

* [x]  `x`

* [ ]  `a`

* [ ]  `y`

Given this, I don't think your first concern is an issue. Because if manual search results appear based on the order of the qualities and not their checked/unchecked status, the ordering is still managed by Recyclarr in such a way that the ones you care about (i.e. the checked qualities/groups) will appear on top above the unchecked.

It's also worth bearing in mind that it can affect what files are downloaded. Let's say you already have a 1080p file in your movie folder. You set up a new profile, and only have Bluray-720p selected, and ordered below all the 1080p qualities in the list. In this scenario no file will ever be downloaded, because the existing file beats the highest selected quality, even though the 1080p quality is itself not selected.

You said you sorted 720p below 1080p, so I think this behavior is expected. Aren't you telling Radarr that you prefer 1080p over 720p? Why would it replace the 1080p one with a 720p copy in this case?
One other thing you're leaving off is the cutoff quality which can also impact this behavior (well, it would prevent the search in the first place IIRC).
At the end of the day I don't see how the logical ordering I described above would cause an issue here. User is still in full control of whether or not 1080p qualities are above or below 720p.

Agreed on all points - I didn't think any of these things were issues, was just trying to describe @voltron4lyfe 's comment in some more detail.
The second comment about 1080p/720p and selection/non-selection isn't really anything to do with recyclarr I suppose, it was just tangentally related. I'd seen discussion elsewhere this week that a user could hypothetically already have a movie file, set up a profile expecting an 'upgrade' based on CF scores, and then not see anything happening even though an 'unwanted' quality being unchecked. You're right that cut-off quality factors into this too.
Anyways, in short, I agree these aren't really issues, and even if they were then your ordering proposal would make them non-issues in any event.

I 100% appreciate bringing up all the points you did. I'm not really trying to "shoot down" issues; rather I like to talk through them and make sure an idea is robust enough to not have those problems. It was good to be able to think through things. Thanks!

Absolutely, and it hasn't come across as such 🤘

i think some good suggestions and ideas are being raised here.
simply put, i think this kind of data structure really resembles the user experience in sonarr/radarr, and this is a good thing. :)
the main thing i would add is a way to assign CF scores to QPs more succinctly and intuitively, i.e. in a one-QP-to-many-CF relationship. like is how it appears in the sonarr/radarr GUI.
here's my crack at it:
---
radarr:
  quality_groups:
    Bluray|WEB-1080p:
      qualities:
        - Bluray-1080p
        - WEBRip-1080p
        - WEBDL-1080p
        - Bluray-720p
  quality_profiles:
    SQP-1:
      language: Original
      upgrades_allowed: true
      upgrade_until: Bluray|WEB-1080P
      minimum_cf_score: 10
      upgrade_until_cf: 10000
      qualities:
        Bluray|WEB-1080p:
        WEB 720p:
        HDTV-720p:
          enabled: false
      custom_formats:
        e0c07d59beb37348e975a930d5e50319: 10 # CF score of 10 applied to QP
        9d27d9d2181838f76dee150882bdc58c: # null value -> use `trash_score`
        c2863d2a50c9acad1fb50e53ece60817: 200

i know this is different than the existing way to define CFs (e.g. CFs might be specified multiple times now), but hear me out.
you could probably leave the old data structure in there (radarr: instance: custom_formats) for legacy reasons. or just for people who don't want to use recyclarr to set up QPs and only want it for CFs. however, it could also be possible to use the new structure above with a dummy QP to specify CFs.
with this new structure, a unique set of CFs could first be generated and synced to radarr as usual, and then the next routine would be to set up the QPs and scores.
thanks

Hey everyone. I've been actively working on this the past week or two and I'm really close to being done. I have hit a little wrinkle, though, so I wanted to share the problem and solutions I've thought of here to see if anyone has an opinion. Given the number of people that use Recyclarr now, I don't feel right making decisions unilaterally. My choices aren't always the best ones, so I really value feedback to help guide my decision making process.
This is likely going to be a long read so I apologize in advance. I hope you all will bear with me on the level of detail!
My original design for listing qualities was to only list the qualities a user wants to enable. I believe this to offer the best user experience for editing YAML, because users only need to be concerned with what they care about. Disabled qualities would be inferred by those missing from your list of qualities in YAML. So for example, take this YAML:
qualities:
  - Bluray-1080p
  - Bluray-720p
  - DVD

This would yield ONLY those 3 qualities being checked in your quality profile. The rest would be unchecked.
This introduces a problem: How do we order the disabled qualities?
For those that don't know, order of qualities in a profile is important, regardless of whether or not that quality is enabled or disabled. When Radarr calculates whether or not a particular movie has met cutoff, it does so by comparing the index of the quality of your current file to the index of the quality of your cutoff. In the example YAML above, if our cutoff is Bluray-1080p and your current file for a movie is Bluray-720p, then cutoff has not been met because 720p is ordered below 1080p. If I put 720p above 1080p, then cutoff is considered met because 1080p is now below 720p.
Because we aren't requiring users to specify all of the qualities in a profile, we need to have some intelligent method of ordering them so that the user doesn't get confusing cutoff behavior. I'll share a few ideas I've thought of below, along with the pros/cons.
Enabled Sorted to Top
Using the same YAML example previously shown, any explicitly specified qualities (which would be enabled) are sorted to the top of the qualities list. For example:

Notice that the order of enabled qualities reflects what is in the YAML. Recyclarr would not touch the disabled qualities. The downside to this option is that it's possible to get unwanted downgrades of your movies. For example, let's say I already grabbed a 1080p release for a movie and I no longer want Bluray-1080p, so I remove it from my YAML:
qualities:
  # Comment out the quality below
  # - Bluray-1080p
  - Bluray-720p
  - DVD

Next time you sync, your profile qualities now look like this:

Bluray-1080p was disabled and moved below the enabled qualities. In addition, the cutoff would need to be set to either Bluray-720p or DVD. In either case, Radarr will replace your 1080p file with a lower-quality release because the 1080p quality got moved down the list, thus putting it below any cutoff you could set.
Enabled Sorted to Bottom
Similar to the previous approach, this one would sort those qualities to the bottom of the list:

Unlike sorting to the top (which had the downside of causing unwanted downgrades), this one has the inverse problem: Not upgrading when you want. For example:
Say you have cutoff set to Bluray-720p. Radarr grabbed a 720p release for a movie, which met cutoff. Now in your YAML, you remove Bluray-720p and set cutoff to Bluray-1080p:
qualities:
  - Bluray-1080p
  - DVD

What you get in Radarr looks like this:

In this situation, the 720p file will never be upgraded to 1080p, because the Bluray-720p quality is placed above your new cutoff (Bluray-1080p). Radarr will think that your cutoff is met, no matter which enabled quality you find in search results.
Explicitly List All Qualities
This solution requires a user to list all qualities in their YAML. If any are missing, Recyclarr shows an error (with a list of missing qualities) and will not sync your quality profile until all missing qualities are added.
 We use key/value pairs to indicate whether a quality profile is enabled or not, since we can no longer infer that based on whether one is present in the list or not. false means it is a disabled quality, true means it is enabled.
Obviously this YAML is contrived and doesn't represent what the final version would look like. It is intended to convey an idea only.
As far as Radarr is concerned, this idea has no downsides. It does however have a poor user experience.
Closing Thoughts
My personal view on all of these solutions:

Sorting To Top: I feel like the side effects of this one are the most severe because it can cause downgrades of existing files without any user intervention. I personally think this one is not an option at all.
Sorting To Bottom: While I don't like that this one could cause users to not get upgrades, it's at least non-destructive: A user keeps the file they have unless they make a change to fix it so they can get upgrades again. I'll go over some ideas to solve this later.
Explicit Listing of Qualities: This one is the "safest" option, but also the worst YAML user experience. It becomes really tedious for users to add all the qualities and maintain that list.

Sorting Workarounds
Whether qualities get sorted to the top or the bottom, we can add the ability for a user to explicitly disable qualities so that they do not have to remove them. This enables them to still control the order of those qualities in the list. So going back to this example:
qualities:
  - Bluray-1080p
  - Bluray-720p
  - DVD

We can convert each of these to objects, and add properties to enable or disable them. I'll also add a quality group to the above example, to demonstrate the reason for the more complicated syntax:
qualities:
  - name: Bluray-1080p
  - name: Bluray-720p
    enabled: false
  - name: WEB 720p
    qualities:
      - WEBRip-720p
      - WEBDL-720p
  - name: DVD

In the above YAML:

Bluray-720p is "unchecked" (disabled) because enable: false. However, it remains between Bluray-1080p and WEB 720p because we didn't have to delete it from the YAML simply to disable it.
name always identifies a quality unless groups: is provided, which turns it into a quality group. The point of showing this is that by using an array of objects (instead of a list of strings), this enables us to customize each quality in different ways.

I think I favor this solution + sort to bottom the most so far. It still has some drawbacks, mainly that users can delete qualities and not be aware of the ordering issue which would, worst case, cause upgrades to not happen. But the tradeoff is that the user experience in YAML is really fantastic (concise). It also explains pretty clearly what you want to happen. The drawbacks don't concern me too greatly because they are non-destructive.
I know I've typed a boat load but hopefully everyone has made it to the end. I'd love to read any feedback.

I like the "Alternate Sort to Bottom" also.  I believe that it offers the most flexibility and usability for a majority of the end user base. I can hear the screams already if it was implemented the other way with Sorted to Top, and all of a sudden all of their media is downgraded when they just wanted the future releases in that lower format.

How about a new mandatory variable: quality_sorting. You could have:

quality_sorting: top - puts selected qualities to the top of the list, in the specified order
quality_sorting: bottom puts selected qualities to the top of the list, in the specified order
quality_sorting: maintain keeps selected qualities in their existing list position, ignoring specified order
Or something along those lines? This is similar to what I mentioned the other day, but slightly more developed (for better or worse).

thanks for your excellent work.
no more names
before directly answering your question, i would like to express a desire to move away from using a name keys in general (not just for qualities) because of unnecessary verbosity. instead of creating a mapping just for name (e.g. name: Bluray-1080p), could we just use:

when options are not needed: a scalar serving as the name, but also accept an empty mapping with the key serving as the name
when open options are needed: a mapping with the key serving as the name

e.g. this yaml taken from your "Alternative Sort to Bottom" in prev post:
qualities:
  - name: Bluray-1080p
  - name: Bluray-720p
    enabled: false
  - name: WEB 720p
    qualities:
      - WEBRip-720p
      - WEBDL-720p
  - name: DVD

instead:
qualities:
  - Bluray-1080p # alternatively `- Bluray-1080p:`
  - Bluray-720p:
      enabled: false
  - WEB 720p:
      qualities:
        - WEBRip-720p
        - WEBDL-720p
  - DVD: # alternatively `- DVD`

basically, at every sequence node: if scalar, name = scalar value; if mapping: name = mapping key. mixing and matching within such a sequence would be acceptable. this is concise yet robust and expressive, and it is reminiscent of the docker compose yaml spec (which i admire).
alternative sort to bottom
i am in favor of "Alternative Sort to Bottom". sort to bottom should be the default, implicit assumption. loop over the qualities sequence, and if an instance of enabled: false is encountered (enabled: true is default/implicit), then switch to explicit parsing mode ("Explicitly List All Qualities") where you throw an error if there are missing qualities.
reusable quality groups?
qualities is a property of <instance>.quality_profiles, right?
previously, it was suggested that quality groups might be defined externally for reuse (or shall we say recyclarring?) in multiple quality profiles. perahps you'd have <instance>.quality_groups, and if a group referenced in <instance>.quality_profiles.<profile>.qualities doesn't exist, you'd get an error. has that idea been abandoned? or if you are planning on doing it, are you planning on making it possible to define a quality group either internally or externally to a quality profile (or if a group is defined both externally and internally, use the internal definition)? i don't feel super strongly one way or another. just curious.

@ftc2 Thank you for your detailed comment. You addressed a lot of points. I want to provide some responses below.

before directly answering your question, i would like to express a desire to move away from using a name keys in general (not just for qualities) because of unnecessary verbosity.

Personally, I have never seen any application that supports YAML use a syntax where sequences support a mixture of scalars and mappings. You cited docker compose as a reference, but even there I have not seen it. Can you send me an example of a legitimate use case for this mixture in syntax?
Supporting variant-style types, while probably possible, is a lot of work. So this being "unnecessary verbosity" isn't strictly true. My YAML parsing library, YamlDotNet, has mechanisms to support it but it requires implementing parsing code at a low level. I stick to predictable (and yes, sometimes verbose) object models which requires a consistent syntax. In this case, it means that sequences should represent a set of uniform types. That means if I have a sequence of mappings, every element of that sequence needs to be a mapping in order to make YamlDotNet happy.
And while I don't do this today, serializing YAML back out becomes equally as painful when there's a lot of custom objects. Trust me, I ran into this already, and spent a lot of time refactoring my YAML parsing code to simplify the serialization/deserialization logic so that I can let YamlDotNet do more of the tedious stuff.
And my last argument against your suggestion would be this: With both of your competing YAML examples visible on your monitor, take 5-10 steps away from your screen. Just far enough that you can still make out the shape of the YAML code but can't read it. Do you notice much of a difference? I don't. All of this to say: I don't see much of a reduction in the YAML itself with your proposed example.
I really dislike saying ideas are bad. And I'm certainly not trying to do that here. Part of the reason I asked for more real-world examples is to see if I can open up to the idea a bit more. Just because I've never seen it in the wild doesn't mean it isn't perfectly valid and commonplace. The extra coding is undeniable, though, unfortunately.
I'm not shooting your idea down completely. At the very least, I think we should continue to discuss it and work something out. How do you feel about continuing this specific discussion over on #188?

if an instance of enabled: false is encountered (enabled: true is default/implicit), then switch to explicit parsing mode ("Explicitly List All Qualities") where you throw an error if there are missing qualities

You lost me here. So you're saying that if the user specifies any quality as disabled, then they must add every other missing quality, too?

qualities is a property of .quality_profiles, right?

That's right; I left it out to keep my examples a little more concise.

previously, it was suggested that quality groups might be defined externally for reuse (or shall we say recyclarring?) in multiple quality profiles. perahps you'd have .quality_groups, and if a group referenced in .quality_profiles..qualities doesn't exist, you'd get an error. has that idea been abandoned? or if you are planning on doing it, are you planning on making it possible to define a quality group either internally or externally to a quality profile (or if a group is defined both externally and internally, use the internal definition)? i don't feel super strongly one way or another. just curious.

You are right to bring this up. I should have explicitly commented on this.
At the moment, I'm considering abandoning the idea of reusable quality groups.
After giving it some thought, I wondered if there really were opportunities to reuse quality groups between profiles. I imagine that, in most cases, profiles will define groups in unique ways. I guess what would be really helpful here is a real-world example, if anyone has one. If no one can't come up with a use case for it, I don't see a reason to add the complexity.
The other motivation for it was to reduce indentation but honestly it doesn't look too bad in the "alternative sort to bottom" examples.

How about a new mandatory variable: quality_sorting. You could have:

quality_sorting: top - puts selected qualities to the top of the list, in the specified order
quality_sorting: bottom puts selected qualities to the bottom of the list, in the specified order
quality_sorting: maintain keeps selected qualities in their existing list position, ignoring specified order
Or something along those lines? This is similar to what I mentioned the other day, but slightly more developed (for better or worse).
Edit: you could even have quality_sorting: explicit, which then requires the user to manually specify every quality as you detailed above, but at least then the user is opting into that choice.

Given everything I wrote about how some of these options are more error prone than others, do you honestly see a scenario where you'd recommend a user use quality_sorting: top over quality_sorting: bottom? Do you see yourself recommending a change to the sorting behavior period? And in what situations would this be useful?

Given everything I wrote about how some of these options are more error prone than others, do you honestly see a scenario where you'd recommend a user use quality_sorting: top over quality_sorting: bottom? Do you see yourself recommending a change to the sorting behavior period? And in what situations would this be useful?

I had a use case in my head a minute ago, but then realized it was dumb and why anyone would do that is beyond me.  But I would think that the end user would just change the profile the movie is matched to with the Quality profile configuration that they need.  So going from 1080p to 720p, move from Bluray|WEB-1080p to I'm too cheap for storage so I'm reducing it to 720p.  So, I don't believe that there is any valid use case to enable this kind of functionality.

After giving it some thought, I wondered if there really were opportunities to reuse quality groups between profiles. I imagine that, in most cases, profiles will define groups in unique ways. I guess what would be really helpful here is a real-world example, if anyone has one. If no one can't come up with a use case for it, I don't see a reason to add the complexity.

I can come up with at least one use case for this.  I have multiple instances of Radarr/Sonarr, one of them is standard and one is UHD.  They have a connector so that if I put into a specific profile it will get a Bluray|WEB-1080p version and a Remux|Bluray|IMAX-E|2160p version.  So I keep the Bluray|WEB-1080p and the Bluray|WEB-1080p! versions synced just for that reason.

How about a new mandatory variable: quality_sorting. You could have:

quality_sorting: top - puts selected qualities to the top of the list, in the specified order

quality_sorting: bottom puts selected qualities to the bottom of the list, in the specified order

quality_sorting: maintain keeps selected qualities in their existing list position, ignoring specified order

Or something along those lines? This is similar to what I mentioned the other day, but slightly more developed (for better or worse).

Edit: you could even have quality_sorting: explicit, which then requires the user to manually specify every quality as you detailed above, but at least then the user is opting into that choice.

Given everything I wrote about how some of these options are more error prone than others, do you honestly see a scenario where you'd recommend a user use quality_sorting: top over quality_sorting: bottom? Do you see yourself recommending a change to the sorting behavior period? And in what situations would this be useful?

Yes, two:

Manual searches are, by default, presented in the same order as the quality list. With desired qualities at the bottom of the list, users would have to scroll to the bottom of the list of manual search results to see the releases they want to grab (or set a filter).

When a user deliberately wants to replace 'higher quality' releases. Example, someone has a library of LQ remuxes, and has decided to replace them with HQ encodes or WEBDLs. Sorting to the bottom means this wouldn't happen.

The other options I suggested I'm not bothered about, but I believe there are valid reasons for both top sorting and bottom sorting.

At the very least, I think we should continue to discuss it and work something out. How do you feel about continuing this specific discussion over on https://github.com/recyclarr/recyclarr/issues/188?

sounds good. i'm short on time today, but hopefully i can reply in the near future.
i'll quickly add that i guess it's two separate issues:
1. not using a name:  mapping anymore and
2. allow the mixing and matching scalars and mappings in a sequence (and letting a scalar stand in for an empty mapping when options aren't needed)
i actually don't care all that much about the second issue but will try to address your questions about it for fun later.
what do you think of the first issue?

if an instance of enabled: false is encountered (enabled: true is default/implicit), then switch to explicit parsing mode ("Explicitly List All Qualities") where you throw an error if there are missing qualities

You lost me here. So you're saying that if the user specifies any quality as disabled, then they must add every other missing quality, too?

yes, you understood correctly, but i think that i actually misunderstood your proposal, so never mind. i see no benefit to requiring a full definition with all qualities in any case. the user can fully spec it if they want, and if not, just move what is spec'd to the bottom.

After giving it some thought, I wondered if there really were opportunities to reuse quality groups between profiles.

imo it's really not that important. i am in favor of abandoning it in the sake of simplicity.

what do you think of the first issue?

I think you've split up the two talking points pretty nicely. I am not opposed to the first idea at all. In fact, I already do this in my YAML for instance names under sonarr: and radarr:. I think there are likely other opportunities to simplify key/value pairs this way. Your example of doing trash_id: score in quality profiles is another good example of a place we could do this. It's all case by case.
I'm happy to discuss this further in the other issue, as I mentioned before. No rush.

Yes, two:

Manual searches are, by default, presented in the same order as the quality list. With desired qualities at the bottom of the list, users would have to scroll to the bottom of the list of manual search results to see the releases they want to grab (or set a filter).
When a user deliberately wants to replace 'higher quality' releases. Example, someone has a library of LQ remuxes, and has decided to replace them with HQ encodes or WEBDLs. Sorting to the bottom means this wouldn't happen.

The other options I suggested I'm not bothered about, but I believe there are valid reasons for both top sorting and bottom sorting.

This is excellent. Thank you for the examples! Here are my thoughts:

Do you expect users to change the sorting order of qualities in their profile either permanently or momentarily just to make search results appear at the top? Given the downsides we've talked about for top-sorted qualities, it seems like there are competing goals here: Either we want "safe" cutoff behavior or better sorting for manual search results. For manual searches, how do CF scores affect the sorting? I'm honestly curious about this. Also, users are able to click the columns in manual search results to sort by whatever they want. Given this, how strongly do you feel about this one?
Even if we both dismiss the first point, I think this one has enough merit to justify the change. I really appreciate you sharing this one. I didn't even consider it at first.

I think at this point, for adding a sorting option, I'd like to add it to the roadmap. I want to get "basic" quality profile syncing done first. Then I want to come back and add enhancements. This will also give people time to get a feel for quality profile syncing without a sorting override. I think this will serve as a useful UAT period as well.

I can come up with at least one use case for this. I have multiple instances of Radarr/Sonarr, one of them is standard and one is UHD. They have a connector so that if I put into a specific profile it will get a Bluray|WEB-1080p version and a Remux|Bluray|IMAX-E|2160p version. So I keep the Bluray|WEB-1080p and the Bluray|WEB-1080p! versions synced just for that reason.

I really appreciate this example and I agree its a valid one. Unfortunately, it's a corner case. By that I just mean that there's probably a small number of users that would benefit. That doesn't mean I won't do it, but like the idea yammes proposed to control sort order, it'll probably be an enhancement implemented in a later phase. I don't think it would be too hard to implement, though.

I think based on the discussion so far, this is the plan:

Phase 1: Implement quality syncing by using the "alternative sort to bottom" solution.
Phase 2: Implement a quality_sort initially with just top and bottom.
Phase 3: Support for reusable quality groups

Between each phase I will cut a release.
If this plan is missing anything, please let me know. Thanks again for all the discussion! I'm really happy with where we landed.

1. Do you expect users to change the sorting order of qualities in their profile either permanently or momentarily just to make search results appear at the top? Given the downsides we've talked about for top-sorted qualities, it seems like there are competing goals here: Either we want "safe" cutoff behavior or better sorting for manual search results. For manual searches, how do CF scores affect the sorting? I'm honestly curious about this. Also, users are able to click the columns in manual search results to sort by whatever they want. Given this, how strongly do you feel about this one?

I'd advise someone to permanently pull their wanted qualities to the top of the list, in top-down order of preference, and that advice is usually consistent from others who offer support. Manual searches are (I think) shown in the same order that is used to prioritise a grab - so, quality trumps all, then CF score next.

2. Even if we both dismiss the first point, I think this one has enough merit to justify the change. I really appreciate you sharing this one. I didn't even consider it at first.

Without having any evidence to back it up, I'd say a reasonable proportion of guide users already have a media library, then find their way to the guides. After implementing one of the guide profiles, at least some of those users decide to omit one of the higher qualities in favour of 'HQ' releases that are lower in the default quality list.

The more I mull it over, the less I feel that bottom-sorting is the right choice. Yes, it is more likely to lead to 'destructive' behaviour, i.e. more file deletion-replacements, but it does closer mirror how I interpret the *arr GUI to be used. I can't recall ever seeing a recommendation for GUI users select the qualities they want, then drag them to the bottom of the list. Going from what we've discussed so far, I'd be in favour of an 'Alternative Sort To Top': e.g.:
qualities:
  - name: Bluray-1080p
    enabled: false
  - name: Bluray-720p

Results in:

Or:
qualities:
  - name: Bluray-1080p
  - name: Bluray-720p
  - name: WEB 720p
    qualities:
      - WEBRip-720p
      - WEBDL-720p
    enabled: false
  - name: DVD

Results in:

I'd advise someone to permanently pull their wanted qualities to the top of the list, in top-down order of preference, and that advice is usually consistent from others who offer support. Manual searches are (I think) shown in the same order that is used to prioritise a grab - so, quality trumps all, then CF score next.

I just want to make sure one of my other points doesn't get missed: I was able to sort by clicking columns. Doesn't this make the quality ordering pretty unimportant? Or is there something else I'm missing?

The more I mull it over, the less I feel that bottom-sorting is the right choice

What led you to change your mind? If it's with regards to how search results are sorted when doing manual searches, I feel pretty confident that users being able to click headers to sort by that field makes that a non-issue. Maybe there are other reasons?

I just want to make sure one of my other points doesn't get missed: I was able to sort by clicking columns. Doesn't this make the quality ordering pretty unimportant? Or is there something else I'm missing?

I'd have to do some more testing, but I think that clicking a column sorts purely by the metric you've chosen - so if you click the CF score column, it'll rank everything purely by CF score and not consider the quality order that you've set in your profile.

What led you to change your mind? If it's with regards to how search results are sorted when doing manual searches, I feel pretty confident that users being able to click headers to sort by that field makes that a non-issue. Maybe there are other reasons?

The first reason is the possibilty of encountering both issues I described. I don't personally feel that it's 'risky' to expect a user to read the documentation and configure their config files accordingly (I'm under no illusion that plenty of users, as is the case now, won't), to ensure upgrade are grabbed when a user would expect them to be whilst avoiding unwanted deletions . There's even less risk when it comes to templates. The second is more of a gut feeling that it's the opposite way to how most users would set their sort order in the application, or be advised to set their sort order in the application. I accept there's nothing empirical about a gut feeling, and that perhaps my views on this aren't shared, which is of course absolutely fine.

That's not exactly what we're talking about here. What we're talking about are the consequences of deleting a previously enabled quality and how those consequences change depending on the sorting algorithm we use. I don't think I'd ever tell a user to sort all qualities to the top or bottom of the list when editing through the GUI, because the GUI has all the qualities listed and when they uncheck a quality, it keeps its order in the list. The fact that a GUI exists makes there far less challenges to deal with compared to a YAML file.
The only reason we choose to sort to the top or the bottom is to both 1) allow us to not have to fully list every quality and 2) have some deterministic behavior for qualities we delete from the list, as opposed to setting enabled: false.
The whole idea here is how do we prevent users from shooting themselves in the foot, while at the same time having a good UX for the YAML.

I've told users on several occasions to sort all their checked qualities to the top of the list, specifically to combat both the issues I described. I've also seen situations where a user no longer wants a certain (higher) quality in their profile, so unchecks it, and assumed that will work. It won't, unless they move the other checked qualities higher than it.
Another thing to consider - if a user hasn't been using suitable file naming before finding Recyclarr and the guides (and is starting from scratch), then their files will get the default assigned quality which is HDTV-1080p for 1080p files. None of the guide profiles recommend HDTV-1080p, so therefore would always be above any guide-recommended bottom-sorted qualities, meaning potentially no upgrades would be grabbed at all.
I accept that bottom-sorting is 'safer', but we should anticipate users reaching out for support in some circumstances when upgrades are not being grabbed as they expect.

Thanks yammes. All of your arguments make sense. I think this discussion proves, at the very least, that we need a quality_sort option right off the bat. As to what to default it to if a user didn't provide it, I'm inclined to go with top despite most of the replies here being in favor of bottom. The motivation for choosing top as the default wouldn't be based on risk but on exactly what you mentioned: What people recommend from a support perspective.
I'll continue to give this some more thought, but I think at this point the only change to my earlier bullet-point list would be combining phase 2 into phase 1.

I'm going to give it some more thought too - unwanted deletions are, well, unwanted, so maybe there's something else we've not stumbled upon yet to give users some protection.

Quality profile sync is implemented in commit ce338e24f3a1ca97232c59cf378f144069986c6a
This will be in the next release of Recyclarr.


Errors in the log when creating a site using the Events bp

Describe the issue
There are errors in the log when creating a site using the Events bp.
Steps to reproduce

Create a site in Studio using the Events bp from the marketplace
Watch the catalina log and notice the errors

[ERROR] 2019-10-16T13:48:35,867 [http-nio-8080-exec-3] [dependency.RegexDependencyResolver] | Unexpected error resolving dependencies for site: events path: /site/components/section/speakers.xml
[ERROR] 2019-10-16T13:48:35,879 [http-nio-8080-exec-3] [dependency.RegexDependencyResolver] | Unexpected error resolving dependencies for site: events path: /templates/web/section/speakers.ftl
[ERROR] 2019-10-16T13:48:35,900 [http-nio-8080-exec-3] [dependency.RegexDependencyResolver] | Unexpected error resolving dependencies for site: events path: /config/studio/content-types/component/speakers/form-definition.xml

Expected Behavior
There should be no errors when creating a site using the Events bp
Logs
Here's a complete log for when the site was created using the Events bp from the marketplace
https://gist.github.com/alhambrav/9adb9fdc487fcfb20d304de1b3572d33
Specs
Version
Studio Version Number: 3.1.4-SNAPSHOT-31df36
Build Number: 31df362f307966926633b458806695069f844f77
Build Date/Time: 10-16-2019 10:01:14 -0400
OS
OS X
Browser
Chrome browser

Closing the ticket.  Errors are gone after fix to create site as an orphan in Studio.


Subdivide OSjs.API namespace

@RyanRiffle mentioned in #53

Also it would also be a good idea to divide the API into more namespaces. The process helpers like launch, kill, getProcess, getProcesses etc. could be in API.Process and so on and so forth.

I think this is a really good idea.

Input ?

Sounds great to me. Also maybe provide an api directory inside javascript. Then say for OSjs.GUI provide src/javascript/API/gui.js which would hold all OSjs.GUI.* functions
That way they are easy to relate from the loaded scripts e.g. using developer tools and the source code.

Also maybe provide an api directory inside javascript.

These functions are so small that I think it would be more appropriate to rather create api.js with all said functions, and leave core.js for the base classes and protected global events etc (that cannot be called via API)

That sounds good also.

Moved to #49


Passing fabiojdf tests to PyTest and more

1 - passing some tests to PyTest / test_water_saturation.py
2 -Indonesia and Fertl tests (added) / test_water_saturation.py
3 - version modification to >=3.8.8 / setup.py
to run those tests simply execute "pytest" on terminal

Also, there is something odd in Indonesia. I made some personal tests with it and get values like 4.17 or something. I'll leave some warnings here.


[Bug] Failed to open a jpg file in mergeEditor

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version:

OS Version: Codespace

Steps to Reproduce:

clone https://github.com/zWingz/dev.git
git checkout image-a
git checkout image-b
git merge image-a

@zWingz Are you still seeing this issue?
If so, do you see it on desktop or only in codespaces?

@mjbvz Both at desktop and codespaces

@mjbvz
The image can be opened in CodeSpaces

But it can't opened by SCM after merge the image-a branch


Bower

Amazing library, thanks guy!
Please, add your lib to bower registry http://bower.io/ http://bower.io/docs/creating-packages/

thanks george. I've added a bower package at the current version as 'nlp_compromise', but am not quite sure what i'm doing. can you check it out?
bower info nlp_compromise
cheers

I'm fixed path in bower.json - https://github.com/spencermountain/nlp_compromise/pull/19


getting null value on tracket.get_slot()

----stories.md file-----------------
Generated Story -5208991511085841103
goodbye
utter_goodbye
export
Generated Story -5208991511085841103
greet
utter_greet
goodbye
utter_goodbye
export
story_jehanzaib
greet
utter_greet
inform -utter_ask_brand
inform{“name”:“China Bistro”} -slot{“name”:“China Bistro”}
action_brand
goodbye
utter_goodbye
story_jehanzaib_001
greet
utter_greet
inform -utter_ask_brand
inform{“name”:“Pranzo”} -slot{“name”:“Pranzo”}
action_brand
goodbye
utter_goodbye
story_jehanzaib_002
greet
utter_greet
inform -utter_ask_brand
inform{“name”:“Lalqila”} -slot{“name”:“Lalqila”}
action_brand
goodbye
utter_goodbye
story_jehanzaib_004
greet
utter_greet
inform -utter_ask_brand
inform{“name”:“Kaybees”} -slot{“name”:“Kaybees”}
action_brand
goodbye
utter_goodbye
story_jehanzaib_005
greet
utter_greet
inform -utter_ask_brand
inform{“name”:“EGO”} -slot{“name”:“EGO”}
action_brand
goodbye
utter_goodbye
story_jehanzaib_006
greet
utter_greet
inform -utter_ask_brand
inform{“name”:“Oh My Grill”} -slot{“name”:“Oh My Grill”}
action_brand
goodbye
utter_goodbye
story_jehanzaib_007
greet
utter_greet
inform{“name”:“Krispy2GO”} -slot{“name”:“Krispy2GO”}
action_brand
goodbye
utter_goodbye
story_jehanzaib_007
greet
utter_greet
inform -utter_ask_brand
inform{“name”:“Kaybees”} -slot{“name”:“Kaybees”}
action_brand
goodbye
utter_goodbye
New Story
greet
utter_greet
inform
utter_ask_brand
inform
action_brand
---------------domain.yml------------------------ session_config: session_expiration_time: 0.0 carry_over_slots_to_new_session: true intents:
greet
inform
goodbye entities:
name slots: name: type: unfeaturized responses: utter_greet:
text: Hello! How can I help? utter_goodbye:
text: Talk to you later.
text: Bye bye :frowning: utter_ask_brand:
text: What is your favourite brand? actions:
utter_greet
utter_ask_brand
utter_goodbye
action_brand
---------------data.json------------------------------ { “rasa_nlu_data”: { “common_examples”: [ { “text”: “Hello”, “intent”: “greet”, “entities”: [] }, { “text”: “goodbye”, “intent”: “goodbye”, “entities”: [] }, { “text”: “What is your favourite brand name?”, “intent”: “inform”, “entities”: [ { “start”: 23, “end”: 27, “entity”: “name”, “value”: “brand” } ] },
{
"intent": "greet",
"text": "hey"
},
{
"intent": "greet",
"text": "hello"
},
{
"intent": "greet",
"text": "hi"
},
{
"intent": "greet",
"text": "heya"
},
{
"intent": "greet",
"text": "howdy"
},
{
"intent": "greet",
"text": "hey there"
},
{
"intent": "goodbye",
"text": "bye"
},
{
"intent": "goodbye",
"text": "bye bye"
},
{
"intent": "goodbye",
"text": "see ya"
},
{
"intent": "goodbye",
"text": "see you later"
},
{
"intent": "inform",
"text": "What's do you want to eat?"
},
{
"text": "Is Facebook your favourite brand?",
"intent": "inform",
"entities": [
{
"start": 3,
"end": 10,
"entity": "name",
"value": "Facebook"
}
]
},
{
"text": "Show me what's is the best discount in China Grill",
"intent": "inform",
"entities": [
{
"start": 38,
"end": 49,
"entity": "name",
"value": "China Grill"
}
]
},
{
"text": "I wonder what is the most demanded brand here?",
"intent": "inform",
"entities": [
{
"start": 35,
"end": 39,
"entity": "name",
"value": "brand"
}
]
},
{
"intent": "inform",
"text": "what is the top rated brand product?"
},
{
"intent": "inform",
"text": "Tell me the top 5 brands"
},
{
"text": "Is Gloria Jeans Coffee favourite brand?",
"intent": "inform",
"entities": [
{
"start": 3,
"end": 21,
"entity": "name",
"value": "Gloria Jeans Coffee"
}
]
},
{
"text": "I am going to visit China Bistro. Dont know what are the favourite items there.",
"intent": "inform",
"entities": [
{
"start": 20,
"end": 31,
"entity": "name",
"value": "China Bistro"
}
]
},
{
"text": "My friend are going to eat burger from burger lab. Do you have any idea what are the deals out there?",
"intent": "inform",
"entities": [
{
"start": 39,
"end": 48,
"entity": "name",
"value": "burger lab"
}
]
},
{
"text": "Show me the deals of Tavern Gill, please",
"intent": "inform",
"entities": [
{
"start": 21,
"end": 31,
"entity": "name",
"value": "Tavern Gill"
}
]
},
{
"text": "What are the deals of Tavern Grill",
"intent": "inform",
"entities": [
{
"start": 22,
"end": 33,
"entity": "name",
"value": "Tavern Grill"
}
]
},
{
"text": "I need discount for Rochester Cafe and Grill restaurant",
"intent": "inform",
"entities": [
{
"start": 20,
"end": 43,
"entity": "name",
"value": "Rochester Cafe and Grill restaurant"
}
]
},
{
"text": "Oh, sorry, tell me the most demanded menu of Tandoor",
"intent": "inform",
"entities": [
{
"start": 45,
"end": 51,
"entity": "name",
"value": "Tandoor"
}
]
},
{
"text": "I want food from LAZZO De Cafe",
"intent": "inform",
"entities": [
{
"start": 17,
"end": 29,
"entity": "name",
"value": "LAZZO De Cafe"
}
]
},
{
"text": "Is Beezer Bytz open now",
"intent": "inform",
"entities": [
{
"start": 3,
"end": 13,
"entity": "name",
"value": "Beezer Bytz"
}
]
},
{
"text": "My Favourite brand is China Bistro",
"intent": "inform",
"entities": [
{
"start": 22,
"end": 33,
"entity": "name",
"value": "China Bistro"
}
]
},
{
"intent": "inform",
"text": "I need to know the brand detail?"
},
{
"intent": "inform",
"text": "Kaybees"
}
],
"regex_features": [],
"lookup_tables": [],
"entity_synonyms": []
} }
----actions.py-----------------------
from future import absolute_import from future import division from future import unicode_literals
from rasa_core_sdk import Action from rasa_core_sdk.events import SlotSet #key 8c96f578e5b37e9083158c1f6eb8bf57
class ActionBrand(Action): def name(self): return ‘action_brand’
def run(self,dispatcher,tracker,domain):
from apixu.client import ApixuClient
import requests
brand_response = requests.get(
	"http://<IP_ADDRESS>:9001/api/brand/get?city_id=5d6fafacdd5a444cf6925cf2&customer_id=53616c7465645f5f6c986d50cabe1890102dc607dc988e30db0859d4a8211bf604d6d524d26f693228b24a18b2fb715f")

brand_response_json = brand_response.json()
print(brand_response.status_code)

name = tracker.get_slot('name')

if name == None:
	print("name not coming")
else:
	print(name)

ls = []
for i in range(0, len(brand_response_json["data"])):
	#if name in i:
		brand_name = brand_response_json["data"][i]["name"]
		ls.append(brand_name)

if ls.__contains__(name):
	response = """The brand which you have selected is {} exists in our application""".format(name)
	print(name)
else:
	response = """The brand which you have selected is {} doesn't exists in the application""".format(name)
	print(name)

print(name)
dispatcher.utter_message(response)

#return []
return [SlotSet('name',name)]

----- I am getting null on “name” field. Please help me

Thanks for the issue, @amn41 will get back to you about it soon!
You may find help in the docs and the forum, too 🤗


Don't let rails overriding #to_json break mocking requests

This change forces the gem to call the ruby verison of to_json, rather than Rails' overridden version which encodes special characters and causes ERB not to be evaluated correctly. Kinda messy, but it works.

before:

after:


Feature: reset all progress

Currently, I have to manually unselect the qns that I have completed one by one for spaced repetition practice. Would be great if there's a button to reset all progress back to zero :)

@therollingambit please share the commit/pr for this fix


identify text citation given on the project landing page

Hi we are doing a demo presentation here :)
We have found that CWL (Common Workflow Language) has their citation claimed on the project landing page:
https://www.commonwl.org/
However we do not get the citation results as the homepage intends:
http://citeas.org/cite/CWL
Thanks for looking into it!

Hello! As of right now CiteAs is not set up to follow language such as 'use this citation' or detect a citation format on a web page and parse it. So with CWL CiteAs worked as intended and found the DOI that was referenced in the citation, and used that to build a new citation which is similar to what the authors intended.
Finding author-intended citations is something we will continue to look into.

Hello! As of right now CiteAs is not set up to follow language such as 'use this citation' or detect a citation format on a web page and parse it. So with CWL CiteAs worked as intended and found the DOI that was referenced in the citation, and used that to build a new citation which is similar to what the authors intended.
Finding author-intended citations is something we will continue to look into.


Fixed get-ocp yaml to generate ocp install manifests and ignition files

I did not clean up the results so you could try your playbook to create the bootstrap.

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket: