Fixed code review items. Details are in the description

1)For all 7 notebooks, I added copyright wordings, rewrote titles, subtitles, and added extra descriptions and instructions .
2)Remove demo folder
3)Rewrote readme
4)Updated resources for setting up search engines, vector store, and cloud functions
5)Checked grammars and spellings using Pycharm spelling plugins

Does this need to be added to the repo? Please ignore if it has to be a zip file.
genai-on-vertex-ai/developer_productivity_with_genai/utilities/bank_of_anthos_codebase.zip

For this one, it needs to be a zip because this is exactly the codebase I used in the vector store. If users want to replicate exactly what I did, they need to upload this codebase to a bucket and chunk them from there.
This repo is from a GCP open source project - bank of anthos. The reason I didn't point users to that public repo is because the original code repo failed many times when I chunked them to embeddings and I fixed a few things to make it work. This zip has all the fixes I did.


fix(transformer/arrow-function): handle unicode when capitalizing property name

Previously was assuming property name starts with an ASCII character. If it started with a multi-byte unicode char, &property[1..] would panic.

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#7312
#7311  👈 (View in Graphite)
#7310
#7234
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Merge activity

Nov 17, 12:01 AM EST: The merge label '0-merge' was detected. This PR will be added to the Graphite merge queue once it meets the requirements.


[Question] Cannot initialize pod from values.yaml

kubectl logs airflow-dev-scheduler-55cfb858cd-gv8mb -n airflow -c airflow-scheduler
Error from server (BadRequest): container "airflow-scheduler" in pod "airflow-dev-scheduler-55cfb858cd-gv8mb" is waiting to start: PodInitializing
Name:         airflow-dev-scheduler-5d879b58c9-559gk
Namespace:    airflow
Priority:     0
Node:         apache-cluster-worker/<IP_ADDRESS>
Start Time:   Tue, 16 Feb 2021 16:40:59 +0000
Labels:       app=airflow
              component=scheduler
              pod-template-hash=5d879b58c9
              release=airflow-dev
Annotations:  checksum/config-env: 0b85322f8fb9394816b10ae3ab9d773e7122bf7eefab906409dc2bcb22729a3a
              checksum/config-git-clone: 060905034a27d7ab32ced419cf3184f1edeb4d12230230bf337a5d51f8b23a17
              checksum/config-scripts: 67349ba4354d46dd6b7fe5b9aa0ce9c0b696fa0aa6b6ec3f1062bf83224cf780
              checksum/config-variables-pools: 51c903cf270e9a3a4945737106d3899e90ee28a48e8eab2967018cda91c5ef01
              checksum/dags-git-ref: c18ccde4e0d067bd9f9b40448692398a1419a0d874e00d376955a267a3a6ac76
              checksum/secret-connections: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
              cluster-autoscaler.kubernetes.io/safe-to-evict: true
Status:       Pending
IP:           <IP_ADDRESS>
IPs:
  IP:           <IP_ADDRESS>
Controlled By:  ReplicaSet/airflow-dev-scheduler-5d879b58c9
Init Containers:
  git-clone:
    Container ID:  containerd://b3497a4cc514dffb475aa1336a14e0b1c6340223ed07eca2b6f735097923d0be
    Image:         alpine/git:latest
    Image ID:      docker.io/alpine/git@sha256:1f6d3a3e26e7452fab3e144d9dd3f11712b2f8dffa4b2094918bf4cd874e7841
    Port:          <none>
    Host Port:     <none>
    Command:
      /home/airflow/git/git-clone.sh
    Args:
<EMAIL_ADDRESS>      release/DEV
      /dags/dags
      bitbucket.example.com
      22
      id_rsa
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    128
      Started:      Tue, 16 Feb 2021 17:42:26 +0000
      Finished:     Tue, 16 Feb 2021 17:42:26 +0000
    Ready:          False
    Restart Count:  17
    Environment Variables from:
      airflow-dev-env  ConfigMap  Optional: false
    Environment:
      DATABASE_PASSWORD:  <set to the key 'postgresql-password' in secret 'airflow-dev-postgresql'>  Optional: false
    Mounts:
      /dags from dags-data (rw)
      /home/airflow/git from git-clone (rw)
      /keys from git-clone-secret (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from airflow-dev-token-f7scc (ro)
Containers:
  git-sync:
    Container ID:
    Image:         alpine/git:latest
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /home/airflow/git/git-sync.sh
    Args:
<EMAIL_ADDRESS>      release/DEV
      /dags/dags
      bitbucket.example.com
      22
      id_rsa
      60
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      airflow-dev-env  ConfigMap  Optional: false
    Environment:
      DATABASE_PASSWORD:  <set to the key 'postgresql-password' in secret 'airflow-dev-postgresql'>  Optional: false
    Mounts:
      /dags from dags-data (rw)
      /home/airflow/git from git-clone (rw)
      /keys from git-clone-secret (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from airflow-dev-token-f7scc (ro)
  airflow-scheduler:
    Container ID:
    Image:         apache/airflow:1.10.12-python3.6
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/dumb-init
      --
    Args:
      bash
      -c
      true \
       && mkdir -p /home/airflow/.local/bin \
       && export PATH="/home/airflow/.local/bin:$PATH" \
       && echo "*** upgrading airflow db..." \
       && airflow upgradedb || airflow db upgrade \
       && echo "*** adding Airflow variables..." \
       && airflow variables -i /home/airflow/variables-pools/variables.json \
       && echo "*** adding Airflow pools..." \
       && airflow pool -i /home/airflow/variables-pools/pools.json \
       && echo "*** running scheduler..." \
       && exec airflow scheduler -n -1

    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       exec [python -Wignore -c import os
os.environ['AIRFLOW__CORE__LOGGING_LEVEL'] = 'ERROR'
os.environ['AIRFLOW__LOGGING__LOGGING_LEVEL'] = 'ERROR'
from airflow.jobs.scheduler_job import SchedulerJob
from airflow.utils.net import get_hostname
import sys
job = SchedulerJob.most_recent_job()
sys.exit(0 if job.is_alive() and job.hostname == get_hostname() else 1)
] delay=300s timeout=1s period=30s #success=1 #failure=5
    Environment Variables from:
      airflow-dev-env  ConfigMap  Optional: false
    Environment:
      DATABASE_PASSWORD:  <set to the key 'postgresql-password' in secret 'airflow-dev-postgresql'>  Optional: false
    Mounts:
      /home/airflow/scripts from scripts (rw)
      /home/airflow/variables-pools/ from variables-pools (rw)
      /opt/airflow/dags from dags-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from airflow-dev-token-f7scc (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      airflow-dev-scripts
    Optional:  false
  dags-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  git-clone:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      airflow-dev-scripts-git
    Optional:  false
  git-clone-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  airflow-git-keys
    Optional:    false
  variables-pools:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      airflow-dev-variables-pools
    Optional:  false
  airflow-dev-token-f7scc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  airflow-dev-token-f7scc
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                  From     Message
  ----     ------   ----                 ----     -------
  Warning  BackOff  57s (x304 over 65m)  kubelet  Back-off restarting failed container

From log, what setting was not configured? This is for same for web, worker and scheduler all the same PodInitializing error

@scheung38 can you please try with version 8.0.0 of the chart and see if there is still an issue?

@scheung38 any updates?


At @Scheduled logic to transmission code

Addresses https://www.pivotaltracker.com/story/show/187125884
This will be easier to review once https://github.com/codeforamerica/mdpilot/pull/106 is in, as it builds off of that work.

Looks good to me. Manually checked and saw transmissions getting sent and files making it to GDrive!


Add input option for Model Name in GoogleVertexAIEmbedding

Adaptation of multi-lingual support of Vertex AI Embedding, which is published Google Could Next on last week! 🎉

@HenryHengZJ thank you for your comment!
I edited as your suggestion!


Blank line placement in example

The current example has
Give back your heart
{blank line}
{blank line}
Desire is a lovestruck ladykiller

It should be
Take your soul from your heart
{blank line}
Give back your heart
{blank line}
Desire is a lovestruck ladykiller

since it should return the the value after exiting the while loop

You're absolutely right - fix incoming.
Also, you can include code blocks in issues by enclosing them in triple backticks
like this

:)

I think there's even more issues. The actual FizzBuzzing part of the example involves nested blocks, there's the main Until my world is Desire block and the 3 nested If blocks. However there is no documentation for using nested blocks. If I were to make a suggestion it would be to bring back the reserved and and use it such that each loop would only execute the line below it (such as If statements do in Java) as well as any lines below it that start with and up until there is a line that doesn't begin with and. So the issue you pointed out would be alleviated because the while loop would only execute the next line (Take your soul from your heart) and wouldn't move on the the Give statement until the while statement was broken because there's no and in front of the Give.

@N-Masi How does that work with nested blocks, though? If you have leading Ands, there's no way to close an inner block without closing the outer block.

@wolfgang42 You know that's a good point and after thinking about it for a bit and looking at Dylan's fix I think his idea of just using blank lines to denote the end of blocks works way better than mine due to what you just pointed out. Though I wonder if you could replace the blank line with some sort of phrase such as That's the end of that or something else more poetically inspired.


Set Array: no matching function for call

Hi!
I've some error when trying to use database.set<object_t> to send arrays.
Trying the example "Set.ino"
object_t arr;
arr.initArray();
writer.join(arr, 4, object_t(1), object_t(2), object_t(string_t("test")), object_t(boolean_t(true)));
database.set<object_t>(aClient, "/test/arr", arr2); // this is firebase.cpp:117
Fails as well with
In file included from .pio/libdeps/m5stick-c/FirebaseClient/src/FirebaseClient.h:41,
from include/main.h:15,
from src/firebase.cpp:1:
.pio/libdeps/m5stick-c/FirebaseClient/src/database/Database.h: In instantiation of 'bool Database::set(AsyncClientClass&, const String&, T) [with T = object_t]':
src/firebase.cpp:117:66:   required from here
.pio/libdeps/m5stick-c/FirebaseClient/src/database/Database.h:389:104: error: no matching function for call to 'Database::storeAsync(AsyncClientClass&, const String&, object_t&, async_request_handler_t::http_request_method, bool, AsyncResult*, NULL)'
return storeAsync(aClient, path, value, async_request_handler_t::http_put, false, &result, NULL);
In file included from .pio/libdeps/m5stick-c/FirebaseClient/src/FirebaseClient.h:41,
from include/main.h:15,
from src/firebase.cpp:1:
.pio/libdeps/m5stick-c/FirebaseClient/src/database/Database.h:881:10: note: candidate: 'template bool Database::storeAsync(AsyncClientClass&, const String&, const T&, async_request_handler_t::http_request_method, bool, AsyncResult*, AsyncResultCallback, const String&)'
bool storeAsync(AsyncClientClass &aClient, const String &path, const T &value, async_request_handler_t::http_request_method mode, bool async, AsyncResult *aResult, AsyncResultCallback cb, const String &uid)
.pio/libdeps/m5stick-c/FirebaseClient/src/database/Database.h:881:10: note:   template argument deduction/substitution failed:
In file included from .pio/libdeps/m5stick-c/FirebaseClient/src/FirebaseClient.h:41,
from include/main.h:15,
from src/firebase.cpp:1:
.pio/libdeps/m5stick-c/FirebaseClient/src/database/Database.h:389:104: note:   candidate expects 8 arguments, 7 provided
return storeAsync(aClient, path, value, async_request_handler_t::http_put, false, &result, NULL);
Any ideias?
Cheers!

You can update the library.


[B13] No recipe for Arcane Stone in the Thaumonomicon

I guessed the recipe just based on what it was in TC4, but couldn't find it anywhere at all in the book, and clicking on it in the other book recipes doesn't show it like it does for other components.

Yeah there really isn't. Just double-checked, the Stone is mentioned in a number of places but clicking on it from the Thaumonomicon yields no recipe page. I was convinced that was wrong, it's not :)

@Geethebluesky  just as confirmation can you post the screen shot of your Thaumonomicon or do you not have ThamicJEI installed?

I do have Thaumic JEI, but you want me to screenshot.... what exactly?
What's missing is the "click here for recipe" in the Thaumonomicon, it never appears below any reference to Arcane Stone, and clicking does nothing. Onlgly way to see a recipe for it is to search it in JEI.

Hover over each of the Arcane Stone blocks in that recipe. Does it show you how to make the Arcane Stone itself?

@Geethebluesky  Your right ... I don't see the Arcane Stone recipe anywhere in the Thaumonomicon.

Sometimes, you can just click on the block in a recipe to find that recipe

For the block

Thaumic JEI: https://minecraft.curseforge.com/projects/thaumic-jei
Fixes it

You don't understand. We're saying Thaumcraft itself should be providing a recipe for this in the Thaumonomicon.
Some people do not use JEI at all, it doesn't make sense for them to not have access to the recipes the mod uses anyways.

I was replying to Zsashas


Implement Commenting on Tables

Since we are updating the on-disk storage format, is there anything special we need to do?


Added rollout.revisionHistoryLimit

This PR allows setting .spec.revisionHistoryLimit for selected controllers.

Could you add a test-case for rollout.revisionHistoryLimit to charts/generic-service/ci?

Could you add a test-case for rollout.revisionHistoryLimit to charts/generic-service/ci?

Yes, I added a test.


WIP: Spike - Adding forms API to designer

Merged as github token requires being run on the default branch


Experiment/components instances highlight

Component Experiment #706
How to try it?
Try highlighting and selecting component children on the canvas. Uses feature flag Component Children Highlights.
Editing is not included in this PR, the elements can be only selected.
To try it on a different project please add ?branch_name=experiment/components-instances-highlight to the end of the URL.
🍕 https://utopia.pizza/p/4c23662a-hissing-repair/?branch_name=experiment/components-instances-highlight

Ok, so I've just been playing with this and extended the sample project in a slightly evil manner - see this link.
First reaction is overwhelmingly positive - this addresses a huge part of the problem, certainly on the canvas.
Beyond that, I struggled with these things the most

Even just bringing in one component makes the navigator rather unwieldy. With multiple, the navigator quickly becomes pretty an endless (and endlessly indented) list. In itself perhaps not such a big deal (have you seen. my Sketch files?) but...
... it makes it hard to understand when you're going from "inside the main component" to "inside elsewhere".
It's similarly hard to understand "where you are", especially if the component you're editing is rather small and well-structured.
I haven't yet worked out what the difference is between selecting a component vs the first rendered child of one but... it doesn't seem like a big deal?

However! I think the split-pane navigator could address a huge part of this, by making me "jump" between the top pane (main component I'm editing, i.e. the one in the scene) that only renders what's in the render function of MainComponent and bottom pane (secondary component I've clicked inside of, including its render function outline). Empty otherwise. Focus of "selected element" moves from one panel to the other.
Alternatively, perhaps a more sophisticated "unfolding / folding" approach could help here, where we keep one navigator. If you select something from a secondary component directly on the canvas, we could then show only the render tree of the component in question (styled differently in the navigator).

Merged into https://github.com/concrete-utopia/utopia/pull/727


Implement caching using Spring's @Cacheable

ok to test

ok to test

ok to test

I think if:

The per-method caches can be collapsed into per-entity caches using custom key generation
CacheHelper renamed to EntityServiceWrapper

Then it is good to go.  There was some other small nitpicks (rename stateDataService) and we should discuss the move to ehcache.

Now using single cache (per return type) and broadcasting cache evict to other nodes. Merging later this morning.


Fix: Update Table Component's HTML Parser To Not Break When Encountering HTML Comments

Jira
http://vjira2:8080/browse/BDS-1665
Summary
Fixes the <bolt-table> web component JavaScript to better handle HTML comments in the initial HTML when initially rendering.
Details
HTML comments are a different node type (different from text nodes + normal DOM nodes) which wasn't included in the original parser logic for this.

Broken Schema Table (Left) vs Fixed (Right)

How to test

[ ] Review code changes
[ ] Confirm new Jest tests added (which include checking the component rendering when HTML comments are in the original content) are passing

@sghoweri , could you add steps to reproduce the original issue?  I don't see it on https://boltdesignsystem.com/pattern-lab/?p=viewall-components-button, which I believe is the page your screenshot is from.  I also tried adding an HTML comment to a table cell locally and nothing broke.
Also, https://fix-handle-html-comments-in-table-compone.boltdesignsystem.com/pattern-lab/?p=viewall-components-button does not show vertical lines between cell as in your "Fixed" screenshot.  Should it?

From @sghoweri on Slack (for the benefit of other reviewers):

the easiest way to replicate the issue is to pull down the latest code on master, wrap a  around the schema props table: https://github.com/bolt-design-system/bolt/blob/master/docs-site/src/components/pattern-lab-utils/schema-docs.twig#L110 to see the error

I didn't get any of my questions answered-- but I guess if @danielamorse is happy then this is "Ready to Merge" ¯_(ツ)_/¯

PR was released with v2.8.4-canary.bde01a443.0

PR was released with v2.9.0


react-msal-bot gulp bundle issue

Category

[ ] Question
[x] Bug
[ ] Enhancement

Expected or Desired Behavior
I'm trying to package up the extension for deployment into the app catalog.
Observed Behavior
When I run gulp bundle I get the following warning/error:
Warning - [webpack] 'dist':
./node_modules/adaptivecards/lib/adaptivecards.js
(Emitted value instead of an instance of Error) Cannot find source file '../src/adaptivecards.ts': Error: Can't resolve '../src/adaptivecards.ts' in 'C:/Users/name/Downloads/sp-dev-fx-extensions-master/samples/react-msal-bot/spfx/node_modules/adaptivecards/lib'
NonErrorEmittedError: (Emitted value instead of an instance of Error) Cannot find source file '../src/adaptivecards.ts': Error: Can't resolve '../src/adaptivecards.ts' in 'C:/Users/name/Downloads/sp-dev-fx-extensions-master/samples/react-msal-bot/spfx/node_modules/adaptivecards/lib'
there are other errors along similar lines such as:
./node_modules/adaptivecards/lib/card-elements.js
(Emitted value instead of an instance of Error) Cannot find source file '../src/card-elements.ts': Error: Can't resolve '../src/card-elements.ts' in 'C:/Users/name/Downloads/sp-dev-fx-extensions-master/samples/react-msal-bot/spfx/node_modules/adaptivecards/lib'
NonErrorEmittedError: (Emitted value instead of an instance of Error) Cannot find source file '../src/card-elements.ts': Error: Can't resolve '../src/card-elements.ts' in 'C:/Users/name/Downloads/sp-dev-fx-extensions-master/samples/react-msal-bot/spfx/node_modules/adaptivecards/lib'
Steps to Reproduce

get latest of https://github.com/SharePoint/sp-dev-fx-extensions/tree/master/samples/react-msal-bot

go to the /spfx folder

run the following:

npm install

gulp clean

gulp build

gulp bundle

error message. I  originally got this error when I was updating my spfx extension to 1.6 and had all kinds of errors. So I got the original code from the repo and tried building from scratch and have been receiving these errors and I'm currently blocked by this issue.

Thanks,
Simon

I am also getting the same error,  any updates on the issue?

Hey @omega1119 did you ever resolve this issue?
I'll go ahead and close the issue because it is a couple years old, but feel free to re-open it if you need more help.


Shipperctl installs management and application CRDs independently

Up until now, shipper has been installing CRD objects to management
clusters only and this was a part of shipperctl clusters setup management command. As we're splitting shipper in management and
application components, these components need their own CRDs to
operate on the relevant cluster.
Therefore, an updated list of CRD groups:

Management

Application
Release
Cluster
RolloutBlock

Application

InstallationTarget
CapacityTarget
TrafficTarget

Folow-up to: https://github.com/bookingcom/shipper/pull/292
Signed-off-by: Oleg Sidorov<EMAIL_ADDRESS>

The build predictably failed as we test against 2 distinct clusters: kind-mgmt and kind-app. The PR could be put on hold and rebased once targetObjects are migrated to app clusters.

Implemented in #306


Fix marshaling of null field values

Fixes https://github.com/temporalio/cli/issues/607
Fix handling of nil value in custom JSON Marshaler.
Evidence this is correct
PR contains a new test case that fails without the fix
Potential risks
Could break anything that uses this JSON Marshaler, e.g. printing payloads by CLI.

This had already been fixed at https://github.com/temporalio/api-go/pull/159.


AFNetworkActivityLogger embed nasted framework

Due to the Run Script carthage copy-frameworks the framework embed nasted framework which prevent from using it in application deployed in app store.
[Transporter Error Output]: ERROR ITMS-90205: "Invalid Bundle. The bundle at 'MyApp.app/Frameworks/AFNetworkActivityLogger.framework' contains disallowed nested bundles."
[Transporter Error Output]: ERROR ITMS-90206: "Invalid Bundle. The bundle at 'MyApp.app/Frameworks/AFNetworkActivityLogger.framework' contains disallowed file 'Frameworks'."

have the same problem. any suggestions how to get around this?

Fork the repo and fix it yourself as noone seems to care anymore. Remove the build phase as framework should not embed other frameworks, it causes nested framework, only final app embed them all.


unable to return type from parse_macro_input

Code
use syn::parse::{Parse, ParseBuffer};
use syn::parse_macro_input;
use proc_macro2::TokenStream;

#[cfg(test)]
mod test {
    use super::*;

    #[test]
    fn test_compile() {}
}

fn return_prototype() -> String {
    let ts = TokenStream::new();

    parse_macro_input!(ts as Prototype).0
}

pub struct Prototype(pub String);

impl Parse for Prototype {
    fn parse(input: &ParseBuffer) -> syn::Result<Self> {
        Ok(Prototype { 0: "".to_string() })
    }
}

Cargo.toml
[lib]
proc-macro = true

[dependencies]
proc-macro2 = "1.0.*"
syn = { version = "1.0.*", features = ["full", "extra-traits"] }
quote = "1.0.*"

I was expecting my code should compile, but it fails with this message:
error[E0308]: mismatched types
  --> src\lib.rs:19:5
   |
19 |     parse_macro_input!(ts as Prototype).0
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `proc_macro::TokenStream`, found struct `proc_macro2::TokenStream`
   |
   = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0308]: mismatched types
  --> src\lib.rs:19:5
   |
16 | fn return_prototype() -> String {
   |                          ------ expected `std::string::String` because of return type
...
19 |     parse_macro_input!(ts as Prototype).0
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |     |
   |     expected struct `std::string::String`, found struct `proc_macro::TokenStream`
   |     help: try using a conversion method: `$crate::export::TokenStream::from(err.to_compile_error()).to_string()`
   |
   = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to 2 previous errors; 1 warning emitted

I think this is behaving correctly; https://docs.rs/syn/1.0.34/syn/macro.parse_macro_input.html calls out that it must be called from inside a function that returns proc_macro::TokenStream.
For parsing from proc_macro2::TokenStream inside a function that does not return TokenStream you would probably want to use syn::parse2. https://docs.rs/syn/1.0.34/syn/parse/index.html#the-synparse-functions

Thanks @dtolnay, this worked :)


Drop unneeded deps on OSX (icu, libiconv, xz, zlib)

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.


Use inheritance between optics

Currently, Optics have a few methods asX, e.g. asTraverse, asOptional on Lens to "upcast" optics.
Instead, we could introduce inheritance between optics, e.g.
trait Lens[S, T, A, B] extends Optional[S, T, A, B] with Getter[S, A]

Then we can deprecate all the asX methods.

There is an issue with overriding modifyF
https://scastie.scala-lang.org/XVL7KRPEQo6wfRy3GgtUpw

Please do! 🙏

I made some progress but unfortunately it doesn't look good.
As @tpolecat told me in gitter, Scala doesn't accept narrowing on inputs https://gitter.im/scala/scala?at=602fd83b9ba13e72e43d7ea1
We could get around that by defining two version of modifyF for Lens, one that takes a Functor and one that takes an Applicative and write one in terms of the other. This trick works if all the arguments are within the same parentheses but it doesn't work if they are split over several parameterer lists :(
https://scastie.scala-lang.org/GPj1KhiBQnSsoXCACTLEVw

Since there is only one method affected:  modifyF, maybe we could consider tto remove the overload and do:
trait Lens[S, T, A, B] {
  def foo[F[_]: Functor    ](update: A => F[B])(from: S): F[T]
  def bar[F[_]: Applicative](update: A => F[B])(from: S): F[T]
}

but I am not sure how to call foo and bar

modifyF -> Functor and modifyA -> Applicative

Inheritance is overloading some methods because they have slightly different shape for optics with 2 or 4 type parameters, see https://github.com/optics-dev/Monocle/pull/1082.

Implemented in #1088


WebM Opus rate is 0 on big endian machines

nestegg_track_audio_params() returns params->rate == 0 on big endian machines for Webm.
Noticed this in Firefox, but this also happens with a master-checkout of nestegg and make && make check:
=========================================
   libnestegg 0.1git: ./test-suite.log
=========================================

# TOTAL: 1
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: test/regress
==================

--- ./test/media/detodos.webm.ok    2019-08-19 15:42:07.776490667 +0200
+++ /tmp/tmp.smhiBL2lWA 2019-08-19 17:26:50.942396884 +0200
@@ -1,7 +1,7 @@
 1<PHONE_NUMBER> 20832 1
 1 3 1 0
 7ee6872f42345d2f717a88b33668ff11cc6d0a25 19
-48000.000000 1 16 6500000 80000000
+0.000000 1 16 6500000 80000000
 0 1 0 8 0 8362763d1961389c12879b7e2f3b27c9ba1b42cf 37 [...]
FAIL test/regress.test (exit status: 1)

Thanks for reporting this!

I tried the patch on my machine. Now the unittests work, as well as Firefox!
Thank you very much for that quick fix!

Thanks for testing and reporting back, that's much appreciated.


Questions to Chapter 2

Hi Aurelien,
I have some questions concerning the GridsearchCV and RandomsearchCV. Firstly, concerning the RandomsearchCV. what do the scipy.stats.reciprocal and scipy.stats.expon do exactly?. Why is important to specify an n_iter ? what does n_jobs do?
secondly, what could be possible reasons for my Gridsearch running for a very long while?
Thirdly, what roles do this backend play because i ran my Gridsearch with C having values of up to 300,000 and i had an error concerning the backend. How could this be solved?
1)The code below just keeps running :
processed = full_pipeline.fit_transform(practice_set)

#training model
from sklearn.svm import SVR
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import GridSearchCV
svm_reg = SVR(gamma='auto')
predictor = svm_reg.fit(processed,practice_set_labels)
predictions = svm_reg.predict(processed)
mean_squared_error = mean_squared_error(y_true=practice_set_labels,y_pred=predictions)
rmse = np.sqrt(mean_squared_error)

#cross validation
#score = cross_val_score(svm_reg,processed,practice_set_labels,scoring='neg_mean_squared_error',cv=10)
#cv_score = np.sqrt(-score)

def display(score):
    print('scores\n',score)
    print(score.mean())
    print(score.std())
params = [{'kernel':['linear','rbf'],'C':[1.,3.,30.,100.]}]
grid_search = GridSearchCV(svm_reg,param_grid=params,scoring='neg_mean_squared_error',cv=5)
grid_search.fit(processed,practice_set_labels)
print(grid_search.best_score_)

2)Running this one which is identical to that of the Jupyter notebook:
processed = full_pipeline.fit_transform(practice_set)

#training model
from sklearn.svm import SVR
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import GridSearchCV
svm_reg = SVR(gamma='auto')
predictor = svm_reg.fit(processed,practice_set_labels)
predictions = svm_reg.predict(processed)
mean_squared_error = mean_squared_error(y_true=practice_set_labels,y_pred=predictions)
rmse = np.sqrt(mean_squared_error)

#cross validation
#score = cross_val_score(svm_reg,processed,practice_set_labels,scoring='neg_mean_squared_error',cv=10)
#cv_score = np.sqrt(-score)

def display(score):
    print('scores\n',score)
    print(score.mean())
    print(score.std())

param_grid = [
        {'kernel': ['linear'], 'C': [10., 30., 100., 300., 1000., 3000., 10000., 30000.0]},
        {'kernel': ['rbf'], 'C': [1.0, 3.0, 10., 30., 100., 300., 1000.0],
         'gamma': [0.01, 0.03, 0.1, 0.3, 1.0, 3.0]},
    ]

svm_reg = SVR()
grid_search = GridSearchCV(svm_reg, param_grid, cv=5, scoring='neg_mean_squared_error', verbose=2, n_jobs=4)
grid_search.fit(processed, practice_set_labels)

ERROR MESSAGE:
[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
exception calling callback for <Future at 0x2274f53ca90 state=finished raised BrokenProcessPool>
joblib.externals.loky.process_executor._RemoteTraceback:
'''
Traceback (most recent call last):
  File "C:\Users\fritz\AppData\Local\Programs\Python\Python37\lib\multiprocessing\queues.py", line 109, in get
    self._sem.release()
OSError: [WinError 6] Das Handle ist ungültig

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\externals\loky\process_executor.py", line 391, in _process_worker
    call_item = call_queue.get(block=True, timeout=timeout)
  File "C:\Users\fritz\AppData\Local\Programs\Python\Python37\lib\multiprocessing\queues.py", line 111, in get
    self._rlock.release()
OSError: [WinError 6] Das Handle ist ungültig
'''

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\externals\loky\_base.py", line 625, in _invoke_callbacks
    callback(self)
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\parallel.py", line 309, in __call__
    self.parallel.dispatch_next()
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\parallel.py", line 731, in dispatch_next
    if not self.dispatch_one_batch(self._original_iterator):
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\parallel.py", line 759, in dispatch_one_batch
    self._dispatch(tasks)
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\parallel.py", line 716, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\_parallel_backends.py", line 510, in apply_async
    future = self._workers.submit(SafeFunction(func))
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\externals\loky\reusable_executor.py", line 151, in submit
    fn, *args, **kwargs)
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\externals\loky\process_executor.py", line 1022, in submit
    raise self._flags.broken
joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.
FEHLER: Der Prozess "37772" wurde nicht gefunden.
FEHLER: Der Prozess "32116" wurde nicht gefunden.
FEHLER: Der Prozess "37652" wurde nicht gefunden.
FEHLER: Der Prozess "7540" wurde nicht gefunden.
joblib.externals.loky.process_executor._RemoteTraceback:
'''
Traceback (most recent call last):
  File "C:\Users\fritz\AppData\Local\Programs\Python\Python37\lib\multiprocessing\queues.py", line 109, in get
    self._sem.release()
OSError: [WinError 6] Das Handle ist ungültig

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\externals\loky\process_executor.py", line 391, in _process_worker
    call_item = call_queue.get(block=True, timeout=timeout)
  File "C:\Users\fritz\AppData\Local\Programs\Python\Python37\lib\multiprocessing\queues.py", line 111, in get
    self._rlock.release()
OSError: [WinError 6] Das Handle ist ungültig
'''

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:/Users/fritz/PycharmProjects/Hello/Chap-2_Excercise.py", line 110, in <module>
    grid_search.fit(processed, practice_set_labels)
  File "C:\Users\fritz\Hello\lib\site-packages\sklearn\model_selection\_search.py", line 688, in fit
    self._run_search(evaluate_candidates)
  File "C:\Users\fritz\Hello\lib\site-packages\sklearn\model_selection\_search.py", line 1149, in _run_search
    evaluate_candidates(ParameterGrid(self.param_grid))
  File "C:\Users\fritz\Hello\lib\site-packages\sklearn\model_selection\_search.py", line 667, in evaluate_candidates
    cv.split(X, y, groups)))
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\parallel.py", line 934, in __call__
    self.retrieve()
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)
  File "C:\Users\fritz\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\_base.py", line 432, in result
    return self.__get_result()
  File "C:\Users\fritz\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\externals\loky\_base.py", line 625, in _invoke_callbacks
    callback(self)
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\parallel.py", line 309, in __call__
    self.parallel.dispatch_next()
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\parallel.py", line 731, in dispatch_next
    if not self.dispatch_one_batch(self._original_iterator):
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\parallel.py", line 759, in dispatch_one_batch
    self._dispatch(tasks)
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\parallel.py", line 716, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\_parallel_backends.py", line 510, in apply_async
    future = self._workers.submit(SafeFunction(func))
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\externals\loky\reusable_executor.py", line 151, in submit
    fn, *args, **kwargs)
  File "C:\Users\fritz\Hello\lib\site-packages\joblib\externals\loky\process_executor.py", line 1022, in submit
    raise self._flags.broken
joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.

Process finished with exit code 1

Hi @FritzPeleke ,
For the reciprocal and expon distributions, please see my explanations in #282 .
Regarding n_iter, in theRandomizedSearchCV class this determines the number of times the model will be trained (times the number of folds). For example, if n_iter=10 and cv=3, then the model will be trained 30 times. The larger n_iter is, the more likely you are to end up with a good set of hyperparameters. But also the more times the model will be trained, so it will take a lot of time.
If Gridsearch runs for a very long time, this is probably because (1) the hyperparameter space is large (i.e., there are many combinations of hyperparameters to evaluate), and/or (2) training the model once takes a long time, perhaps because the dataset is large.
n_jobs tells Scikit-Learn the number of jobs that will run in parallel for this task. If n_jobs=4, for example, then Scikit-Learn will use up to 4 CPU cores in parallel to train the model, which can speed up training by a factor of (up to) 4. If you set n_jobs=-1, it means "use all available CPU cores". If you are having joblib errors, try using n_jobs=1 to avoid parallelism. If this fixes the problem, then it means there's a bug in your joblib installation and you will need to fix it if you want to run jobs in parallel: I believe there are issues with Windows+Anaconda+joblib, please check with anaconda or joblib (I don't use Windows, so I can't help you on this).
Hope this helps!

Hi Aurelien,
What would you suggest to a person who is totally new to all of this? Do you recommend a person to first learn python then read the book? I am currently in Kenya and would like guidance in understanding and mastering the skills in machine learning.
Please help.

Hi @mathewsmutethia ,
You're embarking on a great journey, welcome to Machine Learning! 👍

Indeed, the very first thing you will need to master is Python 3 and programming in general. There are several books for that, and online tutorials, such as the official tutorial.
Next you need to know some college-level math: some basic linear algebra (What is a vector? what is a matrix? How do you multiply two matrices?) and have some basic understanding of calculus (not necessarily very advanced), especially derivatives and partial derivatives. You can check out my tutorial notebook for linear algebra. And this Khan Academy YouTube playlist to learn calculus.
Lastly, you need to know the main Scientific Python libraries, in particular NumPy, Pandas and Matplotlib. You can check out the tutorials I did for these.

I hope this helps! There's a lot to learn, so it will take some time, but there's nothing super difficult if you take your time. Be patient and never give up, and you'll make it, don't worry!
I hope this helps,
Aurélien

Hi @ageron ,
I found a way around the joblib problem. The dask-ml module performs the n_jobs and skips the joblib problem encountered with sklearn.  All one needs to do is install and import the GridsearchCV from dask-ml module and the rest of the code is exactly like sklearn. Its GridsearchCV is said to be faster than that of sklearn. Hope this helps others who have the same problem with windows.
from dask_ml.model_selection import GridSearchCV

Hi @ageron
Took your advice. The journey looks long, but I'll hopefully get there on time.
Regards,
Mathews

Hi ageron,
could you explain more about teh income category, I don't understand it what it is and how to apply it?
Thanks,
Jason

Hi @jasonjoe2019 ,
Thanks for your question. Suppose we know that the median income is a very important feature to predict the median housing price in a district. Then it is important to ensure that the train set, the validation set and the test sets are representative of the overall distribution of incomes. For example, if 10% of all districts in California have a median income between 30k$ and 40k$ (just making up some numbers here), then you want to ensure that the training set, the validation set and the test set all have as close as possible to 10% districts with a median income in this range.
It's a bit like in political surveys: if you suspect that votes depend strongly on, say, the voter's gender, then you need to ensure that the sample of people you include in your survey have the same ratio of men and women as in the overall population. For example, if you know that 51% of all voters are female, and your survey sample includes 1,000 people, then you want 510 females and 490 males in your sample. This is called stratified sampling.
Gender is a categorical feature (there is a discrete amount of possible values), so it's easy to ensure that the distributions match. But income is continuous (there's an infinite amount of possible values), so it's harder to ensure that the income distribution in the sets closely matches the overall income distribution. One approach is to bucketize the income feature and use the resulting bucketized feature (which is categorical) for stratified sampling. For example, incomes from 0 to 15k$ will be assigned to bucket 1, incomes from 15 to 30k$ will be assigned to bucket 2, incomes 30 to 45k$ will be assigned to bucket 3, incomes 45 to 60k$ will be assigned to bucket 4, and incomes higher than 60k$ will be assigned to bucket 5. Then we can estimate the proportion of incomes in the overall population for each bucket, and then ensure that the training set, validation set and test set have the same proportions (or as close as possible).
To create the bucketized feature described above, we can write:
housing["income_cat"] = pd.cut(housing["median_income"],
                               bins=[0., 1.5, 3.0, 4.5, 6., np.inf],
                               labels=[1, 2, 3, 4, 5])

Here's another approach that gives the exact same result (it's much less elegant, but that's what I used in the 1st edition of my book):
housing["income_cat"] = np.ceil(housing["median_income"] / 1.5)
housing["income_cat"].where(housing["income_cat"] < 5, 5.0, inplace=True)

The first line divides the income by 1.5 and rounds the result up. For example, if the income is 4 (which represents $40,000 since incomes are already divided by 10,000 in the dataset), then the income category will be 4/1.5=2.67 rounded up, which is 3.
The second replaces any income greater than 5 with 5. The where() method confused many readers: it says "if income category < 5 then keep it, or else replace it with 5" (many readers thought it was saying the opposite).
You can easily check that these two approaches are equivalent, but the cut() approach is much clearer.
I hope all of this is clear and that you'll enjoy the book!

Hi ageron,
Thanks for your great help!
The explanation is very clear, it's absolutely very useful.
Thanks,
Jason

Hi Ageron,
For the Handling text and categorical attributes part of the book.
after tranforming the category data to the 2D binary array, how to use it to prepare the training data.
I think other training data is not 1D dimension data, thus how to combine the original number training data and the new 2D binary array, then fit model?
Thanks,
Jason

Hi @jasonjoe2019 ,
Thanks for your question. Scikit-Learn estimators expect the input data to be a 2D array, with one row per instance, and one column per feature. Suppose your original input data has two numerical features and one categorical feature (for example, the weekday):
[
  [1.5, 2.0, "monday"],
  [2.5, 1.7, "wednesday"],
  [2.5, 1.7, "monday"],
  ...
]

The categorical feature (the weekday in this case) can be simply converted to a number directly, for example mapping "monday" to 0., "tuesday" to 1., and so on:
[
  [1.5, 2.0, 0.],
  [2.5, 1.7, 2.],
  [2.5, 1.7, 0.],
  ...
]

For some categorical features which have a natural order, that's fine. For weekdays, it's probably all right. But for other categorical features, it's not ideal, since there's no natural order. For example, suppose we're looking at a new article's category, which can be either "financial", "fashion", "international", and "tech". Then there's no natural order between these categories, and we typically prefer a one-hot encoding (or else the model would wrongly assume that some categories are similar when in fact they're not). For example, "financial" will be mapped to [1., 0., 0., 0.], "fashion" will be mapped to [0., 1., 0., 0.], and so on. Assuming the original dataset looks like this:
[
  [1.5, 2.0, "fashion"],
  [2.5, 1.7, "fashion"],
  [2.5, 1.7, "financial"],
  ...
]

will become this:
[
  [1.5, 2.0, 0., 1., 0., 0.],
  [2.5, 1.7, 0., 1., 0., 0.],
  [2.5, 1.7, 1., 0., 0., 0.]
  ...
]

Does this help?

So one categorical feature is transformed into several binary features?
Thanks.

Yes, when doing one-hot encoding, that's exactly right. 👍
Now there are other ways to encode categories. One of the most popular is to use an embedding matrix. This is just a matrix containing float values, with one row per category and an arbitrary number of columns (it's a hyperparameter you can tweak). For example, assuming the embeddings vectors are 3D (i.e., there are 3 columns), and there are 4 categories (the same as earlier), then we may have an embedding matrix like this, for example:
[
    [0.5, 0.3, 0.1],   # financial
    [0.1, 0.7, 0.3],   # fashion
    [0.3, 0.7, 0.1],  # international
    [0.9, 0.9, 0.2]  # tech
]

In this case, each category is simply represented by the corresponding embedding vector, so assuming the original dataset is:
[
  [1.5, 2.0, "fashion"],
  [2.5, 1.7, "fashion"],
  [2.5, 1.7, "financial"],
  ...
]

then resulting dataset is:
[
  [1.5, 2.0, 0.1, 0.7, 0.3],
  [2.5, 1.7, 0.1, 0.7, 0.3],
  [2.5, 1.7, 0.5, 0.3, 0.1],
  ...
]

Each category is just replaced with 3 features.
Now you may wonder how to define the embedding matrix. When training neural networks, the most common solution is to make it part of the model, so its values can we tweaked by gradient descent. This way, during training the embedding vectors will gradually change, hopefully in a way that results in the most useful embedding vectors possible, for example similar categories will likely have similar embedding vectors.
Hope this helps!

Thank you so much for your help!
I learned a lot.

Hey, I hope you can help me on this short question, (I have the german translation of your book)
in Chapter 2 at analysing the best model, you list the most importance features. then you say, that I can drop not significant features like "NEAR OCEAN", "ISLAND", ect.
These are categorial features. My solution would be that you change housing_cat to a binary category, which has 1 for INLAND and 0 for all other. Is that the right way? Or do you meant somthing diffrent?

Hi @Tim4497 ,
Thanks for your question. Yes, you got it, that could definitely be one way to do this. However, I was thinking of dropping the unimportant features after the categorical feature is one-hot encoded. The result is the same, but this solution makes it perhaps a bit easier to try different options (e.g., dropping just ISLAND, or dropping NEAR OCEAN & ISLAND, etc.).
I hope this makes sense. But your solution would work, so do whatever you feel more comfortable with.


raft: hide Campaign rules on applying all entries

solve #873

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Thanks @mrmiywj
Please sign the CLA first.

Hi @mrmiywj The CI can't compile the code.

CI still can't compile. 😭

@siddontang I'm not sure why it passed tests on linux but failed on OS X..

seems that we should merge https://github.com/pingcap/tikv/pull/917 first.

PTAL @hhkbp2

Hi @mrmiywj
We find that etcd may have a bug for this implementation, I have told @xiang90 already. we should wait them fix it first then continue.

Fine! We should wait them fix them!
On Mon, Aug 8, 2016 at 4:49 PM siddontang<EMAIL_ADDRESS>wrote:

Hi @mrmiywj https://github.com/mrmiywj
We find that etcd may have a bug for this implementation, I have told
@xiang90 https://github.com/xiang90 already. we should wait them fix it
first then continue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pingcap/tikv/pull/912#issuecomment-238177327, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHHT7AbfeToUZ7kP62fU5aUMVCDFflfnks5qdu2tgaJpZM4Jc-6D
.

@mrmiywj
etcd fixed https://github.com/coreos/etcd/pull/6129

Hmm, it seems that the test fails.
test raftstore::test_compact_log::test_node_compact_limit ... thread 'raftstore-1::test_node_compact_limit' panicked at 'attempt to subtract with overflow', src/raft/raft_log.rs:318 note: Run with RUST_BACKTRACE=1for a backtrace. thread 'raftstore-3::test_node_compact_limit' panicked at 'attempt to subtract with overflow', src/raft/raft_log.rs:318 thread 'raftstore-5::test_node_compact_limit' panicked at 'attempt to subtract with overflow', src/raft/raft_log.rs:318 thread 'raftstore-4::test_node_compact_limit' panicked at 'attempt to subtract with overflow', src/raft/raft_log.rs:318 thread panicked while panicking. aborting. error: Process didn't exit successfully:/root/tikv/target/debug/tests-57d16572e02bef4a (signal: 4, SIGILL: illegal instruction)

It seems the test fails.

test raftstore::test_compact_log::test_node_compact_limit ... thread 'raftstore-1::test_node_compact_limit' panicked at 'attempt to subtract with overflow', src/raft/raft_log.rs:318
note: Run with RUST_BACKTRACE=1 for a backtrace.
thread 'raftstore-3::test_node_compact_limit' panicked at 'attempt to subtract with overflow', src/raft/raft_log.rs:318
thread 'raftstore-5::test_node_compact_limit' panicked at 'attempt to subtract with overflow', src/raft/raft_log.rs:318
thread 'raftstore-4::test_node_compact_limit' panicked at 'attempt to subtract with overflow', src/raft/raft_log.rs:318
thread panicked while panicking. aborting.
error: Process didn't exit successfully: /root/tikv/target/debug/tests-57d16572e02bef4a (signal: 4, SIGILL: illegal instruction)

Does this test failure occur in your local machine?

Yep. It failed on the same test case.

So I think this is introduced in your changes, maybe we should fix it first.
seems that etcd doesn't handle overflow too. @xiang90

But I'm not sure which caused it..

you can use cargo test test_node_compact_limit  -- --nocapture and print the argument passing to slice or other places to see the overflow problem.

PTAL @hhkbp2

@mrmiywj
Seem that etcd ignores this overflow error. see https://github.com/coreos/etcd/issues/6143

Hi @mrmiywj,
There are 2 issues causing this PR fails to build:

function soft_state() should be deleted or marked as 'dead code' since the only usage of it in test case test_raw_node_start() is revised to None.

failure on test case test_node_compact_limit in ./tests/raftstore/test_compact_log.rs:149. For those test nodes of empty peer, there still is the relationship of raft log:
first index = last index + 1

and it will cause runtime panic in
https://github.com/mrmiywj/tikv/blob/hide-campaign-rules/src/raft/raft_log.rs#L318
when calling log.slice(6, 6) in raft.step() because first index is 6 and last index is 5.
A bug report is issued to etcd raft, and i will keep an eye on the bug fix. After that bug fix is merged, we could merge this PR.
cc @siddontang

@siddontang It passes tests on my linux machine now.

LGTM
PTAL @BusyJay @hhkbp2

LGTM

PTAL @BusyJay

Please format your code by using make all or make dev.

@BusyJay
I just format the code I wrote. There are some formatting problems in the base code.
BTW, the format script in the Makefile does not work... May I should issue it?

@BusyJay
Of course.

root@avalon:~/tikv# rustfmt --write-mode=diff src/raft/raft.rs
Diff in src/raft/raft.rs at line 1269:
self.prs = HashMap::with_capacity(meta.get_conf_state().get_nodes().len());⏎
for &n in meta.get_conf_state().get_nodes() {⏎
let next_idx = self.raft_log.last_index() + 1;⏎

       let matched = if n == self.id {⏎

           next_idx - 1⏎

       } else {⏎

           0⏎

       };⏎

       let matched = if n == self.id { next_idx - 1 } else { 0 };⏎
       self.set_progress(n, matched, next_idx);⏎
       info!("{} restored progress of {} [{:?}]",⏎
             self.tag,⏎

I think the output could not be caught by grep "Diff at line"

rest LGTM

@BusyJay https://travis-ci.org/pingcap/tikv/jobs/151487794


Update FontAwesome to include mastadon

Hi,
I wanted to add a link to mastadon but noticed the icon wasn't rendered correctly. It seems this is because the mastadon Icon is only added in version 5.0.11 of fontawesome https://fontawesome.com/icons/mastodon?style=brands while the theme stil uses 4.7.0 from two years ago https://github.com/nairobilug/pelican-alchemy/blob/master/alchemy/static/css/font-awesome.min.css#L2
It would be nice if the fontawesome version could be updated, thanks!

It would be even nicer to have the "real" font-awesome included, maybe, if possible as submodule. I patched my own fork but its kinda dirty. If there is interest, I can create a PR.

Please update font awesome.
I would like to use the "strava" icon but it's only awailable from version Version 5.8.0.
Thanks.

Hello, I've recently joined pelican-alchemy dev team and I am available to fix this issue. Unfortunately, I have no experience with Font Awesome, so I'm not sure how to proceed.
From what I've gathered we provide a copy of Font Awesome stylesheet and font files along with the theme. Our version is 4.7.0 and current upstream version is 5.11.2.
How did we generate the bundled version in the first place? Did we select one of "kits" that are mentioned on the FA website? Or did we use some repacking tool to select a subset of provided glyphs?
Version 5 is not backwards-compatible because of CSS class renaming, fa was split into fab and fas -  how do we handle user-specified ICONS? Do we assume them always to be brands, like suggested here?
PS: as a workaround for now you can refer to any new CDN hosted version of Font Awesome with THEME_CSS_OVERRIDES

I don't have much experience in FontAwesome myself... In efa3235e I merely followed the instructions on the FA website, and changed the fa class in the theme to the appropriate fab or fas. I think I did this exhaustively. Regarding user-specifed ICONS, I did assume that they would all be brands, because that matched my own use-case, but this indeed a debatable choice.
What's wrong with using the CDN? This lowers the burden on the server hosting the Pelican website, and this makes it more likely that the client reuses a cached version (from another website also using the CDN). And users preferring to host their own version could still use THEME_CSS_OVERRIDES, but they would be responsible of retrieving the files... ;-)

I agree on the benefits of CDN but I would rather have it as an opt-in instead of opt-out feature.
CDN is very much the first-world only solution. When every party involved behaves the way they are expected to - it works. When any of them do something unexpected - it doesn't and can't be made to work. I'm sure you've heard of the Great Firewall of China; India, Russia, some African countries are doing similar things. You never know which URL or IP might become inaccessible (e.g. mail.python.org is blocked in Russia - who would've thought, huh?) - but with your own web server you at least can do something about it.
Also, some users might be concerned about the privacy issue. Using CDN exposes all your visitors to tracking by the CDN owner, whom you can not control and can only trust (more or less blindly).

Fair enough.

Just FYI, I think #85 would address this issue, without needing a CDN.


ttx fails roundtripping NotoSans-VF.ttf

Get binary from:
https://drive.google.com/drive/folders/1uZSioduV6-7-n16eIRNT3xgggnIftxjx
ttx'ing back to binary fails.  Need to debug :(.

I’ll have a look today

I downloaded the font from the url above, run ttx back and forth with it and it completed successfully without errors.
I'm using fonttools master at current commit (efca91e7) and python 3.6.4 on macOS.
I'll try with 2.7.x. Can you paste the error you see?

Thanks I'll try again and report.

Yeah works for me as well. Should have been my faulty table repacker perhaps.


Redefining variable type

The following fails for at least Dart, but likely most other languages.
a = 1
a = 1.0

Yes - this is legal python, but not legal in a statically typed language.
I'm thinking we should have a linter which gives actionable feedback, so it could be rewritten as:
a = 1
a_float = 1.0

This is the crux of https://github.com/adsharma/py2many/issues/260
The py2many approach is to sacrifice compatibility for lower complexity and overall performance. I'm not sure if the broader python community accepts it.

Until an implementation is decided upon, it should trigger a NotImplementedError


tabs and forms setPristine issue

Bug description:
I have a form. After submitting I want to call $setPristine().
Error message: Cannot read property '$setPristine' of undefined
Link to minimally-working plunker that reproduces the issue:
http://plnkr.co/edit/cr9KWYT2i3VIuANJt2Uj?p=preview
The submitForm() reset the model and call $setPristine().
Version of Angular, UIBS, and Bootstrap
Angular: 1.5.3
UIBS: 1.3.2
Bootstrap: 3.3.6

@kukac7, the bug is in your code.  you are trying to reference the form from the wrong scope.


OnShow event we should get the position above or below the input of the calendar

When there isn't enough space below the input date or trigger element the calendare is shown above the trigger.
There isn't anyway to know where is placed the calendar.
My purpose would be to attach an arrow in CSS on the calendar's container and I need to know its position.
Can you add an attribute for example  calendarPostition: top on the event OnShow.
Or if it's not possible can you add an option on the public api like forceBelowCalendar: true to disable the changing position of the calendar and always force it to appear below the trigger element ?

Added in v2.0.5
https://litepicker.com/docs/options#position-205
@zellerbaptiste
in your case you can use position: 'bottom'.
I know the thread is old, perhaps in the future someone will need it.

I use 1.x, however upgrading to 2.x would require time, so I'll wait  till it matures and also date day click event is not added in new version, so no bigger need.
For 1.x workaround is just to overwrite default top class:
 top: 1559px !important;


Fix signature file being gzipped

Describe your changes
Issue ticket number and link
Checklist

[x] Is it a bug fix
[ ] Is a typo/documentation fix
[ ] Is a feature enhancement
[ ] It is a refactor
[ ] Created tests that fail without the change (if possible)
[ ] Extended the README / documentation, if necessary

will fix on our packages service


Ignore node_modules when linting

When re-using a node module, one does not have control over the packages while implementing the Rocketlet.
The dev-environment refusing the build due to linting errors, particularly in dependencies is not satisfying.
[09:53:41] 'lint-no-exit-ts' errored after 3.77 s
[09:53:41] Error in plugin 'gulp-tslint'
Message:
  /.../temporary-rocketlets-dev-environment/rocketlets/smarti/node_modules/http-status-enum/src/index.ts [43, 35]: Missing trailing comma.

Ah, so your folder structure is:
dev-env
  -> rocketlets
     -> smarti (which is a git repository)
        -> node_modules

Okay, so the rocketlets/**/node_modules needs to be ignored :+1:

Exactly. That's how I thought you intended a dev environment to be used.
Any other ideas, how it should be consumed?
Additionally, it was nice it the .gitignore ignored all the Rocketlets - except the samples. Don't know whether such expressions are supported though

Yeah it is, I just wanted to confirm it.
Yes, I am going to see about that as I think it is possible. :+1:


Why?

Why you not update your good program for about 4 months. Please rewrite BB for 0.13.1! We need transfer!

Busy with other projects, I may work on it some more to work with
blockserver.
On Feb 7, 2016 7:08 AM, "MrGenga"<EMAIL_ADDRESS>wrote:

Why you not update your good program for about 4 months Please rewrite BB
for 0131! We need transfer!
—
Reply to this email directly or view it on GitHub
https://github.com/jython234/BouncyBall/issues/10.


cumulative_num comparisons_from_blocking_rules_chart time

I've made a DuckDBLinker object with two blocking rules. Individually, I can count the number of comparisons for each blocking rule with the count_num_comparisons_from_blocking_rule function - for each of the blocking rules this takes less than 3 seconds. However I have not yet been able to successfully run the cumulative_num comparisons_from_blocking_rules_chart function as it takes an unreasonably long time to run (I have kept it going for well over 10 minutes with no output) with just those two blocking rules.
Has anyone else seen a similar issue, is this expected behaviour?
(on version 3.6.0)

Hey @James-Osmond - I haven't heard of this behaviour before. What size of data are you working with?

Hi @RossKen - this is a dataset with just over half a million rows, and 15 fields (not included uid). The blocking rules use 7 of these fields.
Unfortunately this isn't public data so I don't think I can provide a reproducible example!

Do you know how long your predict() step takes on this data?
cumulative_num comparisons_from_blocking_rules_chart uses a snippet of that code flow to produce the chart.

@ThomasHepworth I tried running predict() in the background out of interest, and gave up after 70 minutes I think. They weren't particularly good blocking rules, but I was just interested in seeing what the cumulative number of comparisons was with them, which in theory should have been quick enough to calculate, based on how long it took the code to calculate the number of pairs for each BR individually.

@James-Osmond I've had one thought on this - what form were your blocking rules?
Were they a sequence of "l.column1 = r.column1" statements? Or something more complex?
Related to issue #1188 - under the hood, splink hashes the combinations of columns in your blocking rules with a unique id that can be referenced, making it run much quicker. This only works when you have blocking rules that are "=" statements (on one or more columns). So, if you are doing something more complex, that could possible explain the slow down.

Hi @RossKen - one of the blocking rules DOES have a list filter in it. Wouldn't be the first time my use of those has caused me Splink troubles!
I thought I might have had issues even ignoring that one (but I'm not sure and don't have the data to hand so can't check). Another one is comparing columns to a VALUE, rather than to each other directly (l.sex='f' and r.sex='f'), would that cause a similar problem? Also got a few l.x=r.x OR (l.y=r.y AND l.z = r.z)s in there

So your comparisons of values should be fine (l.sex='f' and r.sex='f').
Your l.x=r.x OR (l.y=r.y AND l.z = r.z) may be causing some issues though. Can you try breaking them up into separate blocking rules and see what happens? E.g.
"blocking_rules_to_generate_predictions" [
        "l.x = r.x",
        "l.y = r.y AND l.z = r.z"
        ]

This should be quicker as I think it will only run "l.y = r.y and l.z = r.z" if "l.x = r.x" is not true, whereas you are currently running them every time.
I'm also not sure how performance "OR" is an operation, so that may help as well.

Not able to recreate this issue so closing


import CSV doesn't create collumns when filename contains "_"

Hallo,
I found one thing when importing the CSV files.
jdbc:relique:csv:C:\MA01Projects\WGWR\Import
When the filename contains underscore "_", I can browse data, but no column names are created. So I cannot make export, import these data. See attachment.
thanks,
J.Matous

System information:

Operating system (distribution) and version
DBeaver version
Java version
Additional extensions

Connection specification:

Database name and version
Driver name
Do you use tunnels or proxies (SSH, SOCKS, etc)?

Describe the problem you're observing:
Steps to reproduce, if exist:
Include any warning/errors/backtraces from the logs

Hello @jindrichmatous ,
Could you clarify what database, OS, DBeaver version are used?

Hallo,
Thanks for response.
Of cause, It is easy to change file name, but for other customers. It is about CSV import there is no database. I am sending versions of system and driver.
Regards,
J.Matous
From: uslss<EMAIL_ADDRESS>Sent: Saturday, May 11, 2019 10:51 PM
To: dbeaver/dbeaver
Cc: jindrichmatous; Mention
Subject: Re: [dbeaver/dbeaver] import CSV doesn't create collumns when filename contains "_" (#5839)
Hello @jindrichmatous https://github.com/jindrichmatous  ,
Could you clarify what database, OS, DBeaver version are used?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/dbeaver/dbeaver/issues/5839#issuecomment-491542907 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABPOG7FJJOUB337OGGVCEETPU4WRNANCNFSM4HJ36HCQ .  https://github.com/notifications/beacon/ABPOG7ERIVHF6H6RVAYOL6DPU4WRNA5CNFSM4HJ36HC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVGFS6Y.gif

I can't reproduce this. Underscore in file name doesn't affect anything.
Maybe you have some specific driver properties?
Could you send sample CSV file which is not recognized properly?

Hallo Serge Rider,
Thanks for the time. I run it on different machine and it works right.
So close it, it is not so important.
JM
From: Serge Rider<EMAIL_ADDRESS>Sent: Friday, May 17, 2019 8:31 PM
To: dbeaver/dbeaver
Cc: jindrichmatous; Mention
Subject: Re: [dbeaver/dbeaver] import CSV doesn't create collumns when filename contains "_" (#5839)
I can't reproduce this. Underscore in file name doesn't affect anything.
Maybe you have some specific driver properties?
Could you send sample CSV file which is not recognized properly?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/dbeaver/dbeaver/issues/5839?email_source=notifications&email_token=ABPOG7ENU6SCA2TG23K2H73PV32VNA5CNFSM4HJ36HC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVVQTWY#issuecomment-493554139 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABPOG7FFZHKSPG6SLJ524D3PV32VNANCNFSM4HJ36HCQ .  https://github.com/notifications/beacon/ABPOG7HSQ2LM3QRCW6HAFLLPV32VNA5CNFSM4HJ36HC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVVQTWY.gif


Emformer pretrained model

🚀 The feature
Thank you for releasing the Emformer implementation.
Do you plan to release also any pretrained model?
Motivation, pitch
The pretrained model would increase the adoption of the code for sure :-)
Alternatives
No response
Additional context
No response

+1

Hi @oplatek and @Tomiinek — thanks for the request! We currently have one pretrained Emformer-RNNT model trained on LibriSpeech available as a prototype feature; it is accessible via torchaudio.prototype.pipelines.EMFORMER_RNNT_BASE_LIBRISPEECH.
https://github.com/pytorch/audio/tree/main/examples/asr/librispeech_emformer_rnnt contains the training recipe we used to produce the model as well as a demo script (pipeline_demo.py) that shows how to use the pretrained model (see #2093 for a video of the demo in action).
Please let us know if you have any questions or feedback.

Thanks for your prompt reply! We will try it out :slightly_smiling_face:

Hi @hwangjeff, I do not see a conformer experimental pipeline (or regular pipeline). Does it mean there is no pretrained model for conformer, right?

That's right — at the moment, we don't have a pretrained model for Conformer available. That being said, we are actively working on building a LibriSpeech training recipe for it, and so we may release the corresponding pretrained model in the coming weeks.

Hi @oplatek and @Tomiinek — thanks for the request! We currently have one pretrained Emformer-RNNT model trained on LibriSpeech available as a prototype feature; it is accessible via torchaudio.prototype.pipelines.EMFORMER_RNNT_BASE_LIBRISPEECH.
https://github.com/pytorch/audio/tree/main/examples/asr/librispeech_emformer_rnnt contains the training recipe we used to produce the model as well as a demo script (pipeline_demo.py) that shows how to use the pretrained model (see #2093 for a video of the demo in action).
Please let us know if you have any questions or feedback.

Hi @hwangjeff - do you have an quick pointer on using pre-trained pipeline with torchserve?


Rbush 2.x compatibility

Hi;
I just installed Rbush latest version ( 2.0.1 ) as a dependency, and it's not working out of the box. Adding support for 2.x or bundling with the old version would be great.
Best,
Chris

Yes, it seems that the library is not compatible with last rbush version


Manual Pagination - useAsyncDebounce fn not a function

Using 7.0.0-rc.15
Describe the bug
Originally created a conversation on Spectrum but I believe since this is a bug it might be better suited to be on Github.
I'm running into a bug when trying to use useAsyncDebounce:
index.es.js:286 Uncaught (in promise) TypeError: fn is not a function
    at _callee$ (index.es.js:286)
    at tryCatch (runtime.js:45)
    at Generator.invoke [as _invoke] (runtime.js:271)
    at Generator.prototype.<computed> [as next] (runtime.js:97)
    at tryCatch (runtime.js:45)
    at invoke (runtime.js:135)
    at runtime.js:170
    at new Promise (<anonymous>)
    at callInvokeWithMethodAndArg (runtime.js:169)
    at AsyncIterator.enqueue [as _invoke] (runtime.js:192)

What I'm trying to accomplish:

Manual pagination that will take can use a search term and filters
Have a form that takes in a search term / filters.
When users submit that form, the pageIndex needs to be reset and new data needs to be requested from my API
When clicking over to the next page, the new data should be requested via api with the latest search term and filters

To Reproduce
For full steps on my exact use case and bug discovery (including my actual code) please visit the spectrum post. Will leave out my full code from the bug here.
I created a Sandbox for a simple recreation. The sandbox uses the pagination controlled example as it's base and simply adds the useAsyncDebounce as described here

Visit Sandbox
Open chrome console
You'll see the error in the chrome console

Expected behavior
I expect to not get an error and have a usable debounced function which will prevent pageIndex state being reset to 0 twice in order to avoid 2 consecutive data fetches.
Screenshots

Desktop (please complete the following information):

OS: Mac Mojave 10.14.1
Browser: Chrome
Version: 79.0.3945.79

I'll look into this ASAP. Thanks!

@tannerlinsley i was able to provide a hotfix in this pr https://github.com/tannerlinsley/react-table/pull/1796

There are actually deeper problems with this function beyond what you fixed. I'll be shipping a new useAsyncDebounce function to address both the issues you brought up and the other ones I just found. :)

any updates, @tannerlinsley?

This will be fixed in the next release.

This will be fixed in the next release.

Ah, I probably shoulda been more clear in my question. Was wondering what's the ETA on the next release? Just want to give my team notice as to around when to expect a fix. Thanks for your hard work!

I am running into the same problem.
Thank you for submitting the report @nicholaai and working on a new release @tannerlinsley.
I will wait for the new release before using useAsyncDebounce

Fixed in master, slated for release. Thanks!


3.0 doesn't have Vue.extend anymore.

Get error Uncaught TypeError: Vue.extend is not a function at v-idle.js:6
I'm presuming this is a "Doesn't work with Vue3" issue. Is that the case?
If so, are there any plans to upgrade the code?
Many thanks

I didn't yet, but looks promising!

@malekim first of all, thanks for the work your pulling off here. I wanna ask, if you could give an update about the progress status of including vue 3 support?

@MoeNeuron
Added vue3 support


Race condition fix for ConditionsCompilerProvider dictionary access

Avoids a possible NullReferenceException when checking for the existence of a key in a non-thread-safe dictionary.
Although the dictionary is only written to inside the lock, accessing it outside the lock while any number of other threads are potentially writing to it can cause a NullPointerException (eg: if the internal buckets are being resized, or are half-way through being written to).
Reads need to happen inside the lock, which can be solved by simply removing the 3 lines as shown here. (You could also achieve the same thing by using a ConcurrentDictionary instead and the GetOrAdd method to specify how to populate the entry when it's missing)
You can trigger this bug fairly easily by setting up a few threads which all call ConditionsCompilerProvider::GetMethodInfo in parallel with the same type/method name values.  I didn't add a test to the tests project, as it's not a deterministic thing, but I can send you some sample code to trigger it if you like.

Thank you for this, yes I appreciate if you can show me some code to reproduce it. as am not to familiar with multithreading issue here

Sample code to trigger the issue:
(As ConditionsCompilerProvider is internal, you can either create a console app in that assembly, or add [assembly: InternalsVisibleTo("Program")] to any declaration of namespace SqlKata.Compilers
Within the first hundred or so iterations, I usually manage to trigger a NullReferenceException in that dictionary access.  Let me know if you need any more info.
static void Main(string[] args)
{
	for (var i = 0; i < 1000; i++)
	{
	    Console.WriteLine($"test iteration {i}");
	    DoThreadingTest();
	}
}

private static void DoThreadingTest()
{
    const int numThreads = 4;

    var methods = typeof(SqlServerCompiler).GetRuntimeMethods().Select(m => m.Name).ToArray();
    var ccp = new ConditionsCompilerProvider(new SqlServerCompiler());

    var threads = Enumerable.Range(0, numThreads).Select(n =>
        new Thread(() => {
            var random = new Random();

            for (var i = 0; i < 100; i++){
                var methodName = methods[random.Next(methods.Length)];
                var methodInfo = ccp.GetMethodInfo(typeof(string), methodName);

                if (!methodInfo.Name.Equals(methodName))
                    throw new Exception($"{methodName} != {methodInfo.Name}");
            }
        })
    ).ToArray();

    foreach (var t in threads)
    {
        t.Start();
    }

    foreach (var t in threads)
    {
        t.Join();
    }
}

@ahmad-moussawi Do you know when the next nuget release will be?  I'd like to use this in production but it would be nice to know this bug is resolved.

@mcavanagh really appreciated, the stable release was planned to be in April, I didn't publish it yet, you can use the nightly build package for now, since the PR is merge now

@ahmad-moussawi Just a heads up - the latest nightly builds only seem to be appearing on MyGet, and not on Nuget (eg: latest pre-release on nuget is 2.0.2-ci-771 from January, whereas MyGet has 2.0.2-ci-776 from a April 20th.)  Is that intentional?


3M Health Information Systems - CGS Affected - HotFix Available

Submission Template
Please provide the following information.

Vendor Name: 3M Health Information Systems
Product Name: CGS
Version(s) affected
Status:  Affected
Update Available:  Yes https://support.3mhis.com/app/home - Must Login
Notes: HotFix available behind paywall: https://support.3mhis.com/app/account/updates/ri/5210
References
Last Updated:  Date of last update

For questions about choice for status, please see the information below.

Unknown - Status unknown.  Default choice.
Affected - Reported to be affected by CVE-2021-44228.
Not Affected - Reported to NOT be affected by CVE-2021-44228 and no further action necessary.
Fixed - Patch and/or mitigations available (see provided links).
Under Investigation - Vendor investigating status.

This product has been added to the repository. Thank you.


Rspec 3.1: uninitialized constant RSpec::Support

I tried running rspec-ontap with Rspec 3.1 and Rails 4.2, but got this error:
.../gems/rspec-core-3.1.7/lib/rspec/core/formatters/base_formatter.rb:1:in `<top (required)>': uninitialized constant RSpec::Support (NameError)
from .../gems/rspec-ontap-0.2.0/lib/rspec/ontap.rb:1:in `require'
from .../gems/rspec-ontap-0.2.0/lib/rspec/ontap.rb:1:in `<top (required)>'
from ...@global/gems/bundler-1.7.8/lib/bundler/runtime.rb:85:in `require'
from ...@global/gems/bundler-1.7.8/lib/bundler/runtime.rb:85:in `rescue in block in require'
from ...@global/gems/bundler-1.7.8/lib/bundler/runtime.rb:68:in `block in require'
from ...@global/gems/bundler-1.7.8/lib/bundler/runtime.rb:61:in `each'
from ...@global/gems/bundler-1.7.8/lib/bundler/runtime.rb:61:in `require'
from ...@global/gems/bundler-1.7.8/lib/bundler.rb:134:in `require'
from myapp/config/application.rb:15:in `<top (required)>'
from .../gems/spring-1.2.0/lib/spring/application.rb:82:in `require'
from .../gems/spring-1.2.0/lib/spring/application.rb:82:in `preload'
from .../gems/spring-1.2.0/lib/spring/application.rb:143:in `serve'
from .../gems/spring-1.2.0/lib/spring/application.rb:131:in `block in run'
from .../gems/spring-1.2.0/lib/spring/application.rb:125:in `loop'
from .../gems/spring-1.2.0/lib/spring/application.rb:125:in `run'
from .../gems/spring-1.2.0/lib/spring/application/boot.rb:18:in `<top (required)>'
from .rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from .rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from -e:1:in `<main>'

Is this gem still being maintained?

Yes --though lower priority since I do less Ruby coding these days. If you have a fix I will gladly merge. Otherwise I will take a look as soon as I can.

Thanks for your quick response!  I had never heard about Tap before today, and I just wanted to test it.  If I were a real user I would look into fixing it, but right now I can't prioritise it.
Thanks for your work!

Should all be fixed now. There were a lot of API changes in v3+!
(Btw, not sure the error you got was actually caused by rspec-ontap. Notice the error was in the base_formatter.rb file of rspec itself. But I imagine whatever it was has been fixed now too b/c things are running smooth on my end now with all the latest installs.)
BIG THANKS!!!!


[niconico] Support encrypted official videos

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

[ ] At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
[ ] Searched the bugtracker for similar pull requests
[ ] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[ ] I am the original author of this code and I am willing to release it under Unlicense
[ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[ ] Bug fix
[ ] Improvement
[ ] New extractor
[ ] New feature

Description of your pull request and other information
Explanation of your pull request in arbitrary form goes here. Please make sure the description explains the purpose and effect of your pull request and is worded well enough to be understood. Provide as much context and examples as possible.
ytdl-org#23824 .
I made this pull request before, but I was giving up because it didn't merge at all.
Recently, when I was looking at the issues of youtube-dl, I found out that a fork called youtube-dlc was made, and I made a pull request.
(Since it uses Google Translate, the sentence may be unnatural)

Will have a look at it ~tomorrow. The heartbeat can probably be implemented directly into the extractor or as another function.

I think that the implementation of heartbeat used the code of https://github.com/panhartstuff/youtube-dl/commit/ff1968568a32cbb6d64440075124faefdff287f6.
It may be possible to implement better. (I'm not familiar with Python...)
I think that ytdl-org#14582 etc. can be helpful.

Some collection
https://github.com/ytdl-org/youtube-dl/pull/23824
https://github.com/ytdl-org/youtube-dl/issues/14582
https://github.com/ytdl-org/youtube-dl/pull/16495
Hm. It seems trickier than I thought. In theory this should work and not affect any other extractors since you check for heartbeat_url in info_dict. Maybe it's just me but code structure doesn't feel natural. There are unfortunately multiple ways to handle this.

Like it is in this PR
This rather (imo) poor attempt https://github.com/ytdl-org/youtube-dl/issues/14582#issuecomment-360767112

but still got potential to be included into the extractor itself

extend the http downloader https://github.com/ytdl-org/youtube-dl/issues/14582#issuecomment-362788161

it's similar to the youtube_live_chat downloader YoutubeLiveChatReplayFD which seems like a good approach

Short term fix would be to actually merge this PR. However it's kind of bad practice to change the core like this.

As you said, I agree that it is best not to change the core as much as possible.
However, niconico's videos use both HTTP and HLS, of which the official encrypted videos use only HLS.
Also, as https://github.com/ytdl-org/youtube-dl/pull/16495/files#r189578412, this implementation cannot be used with other stream types such as HLS.
Therefore, implementations that extend HttpFD, such as the third YoutubeLiveChatReplayFD, will at least need to make further changes to the code in the comments.
If you don't want to change the core, it would be nice to improve the implementation that does the heartbeat in the second Extractor, but what about?

A lot of niconico improvements (this and others) were added in another fork: https://github.com/animelover1984/youtube-dl
I've tried the other implementations, and I couldn't find a better solution than this one.
 A combination of them trying to lock downloaders out and bad coding of the website in general require a lot of changes to the core of youtube-dl to get things working at all.
For example, adding the ability to download livestreams require websockets, which itself requires a pip package and python 3.7+.
This is probably the only change for this site that still allows youtube-dl to be run on python 2

A lot of niconico improvements (this and others) were added in another fork: https://github.com/animelover1984/youtube-dl
I've tried the other implementations, and I couldn't find a better solution than this one.
 A combination of them trying to lock downloaders out and bad coding of the website in general require a lot of changes to the core of youtube-dl to get things working at all.
For example, adding the ability to download livestreams require websockets, which itself requires a pip package and python 3.7+.
This is probably the only change for this site that still allows youtube-dl to be run on python 2

Big Thanks will have a look at it. There are websocket packages for python 2.7 tho. But I generally advise against python2 anyway. EOL will be January 1st 2020 so pretty soon.
I honestly don't know why this site tries to use such weird mechanisms. They can be reversed. DRM would likely be a better option but hard to implement in large scale and can be defeated at some point.
While looking at your code. Is websockets really needed for heartbeat? Those alternative options don't use websockets. Is it just out of convenience or actually required?

While looking at your code. Is websockets really needed for heartbeat? Those alternative options don't use websockets. Is it just out of convenience or actually required?

No, I was just a bit unclear. What I meant was that improvements related to this site that weren't this heartbeat issue involve websockets.

Supplement from me (I'm not good at English and I read using Google Translate, so I may have misunderstood the meaning).
First, you don't need WebSockets to download AES-128 encrypted videos. Also, sending heartbeats does not require WebSockets. The heartbeat is done with a regular HTTP request.
If you want to add a "niconico live" extractor, like the code at https://github.com/animelover1984/youtube-dl, you'll need a WebSocket. "Nico nico live" sends information via WebSocket and seems to be needed when downloading.
Currently, a tool called "Nico Live New Distribution Recording Tool (Tentative)" (https://github.com/guest-nico/nicoNewStreamRecorderKakkoKari) is often used to download "niconico live". For the specifications, https://github.com/guest-nico/nicoNewStreamRecorderKakkoKari/blob/master/siyou.md will be helpful (although all are in Japanese ...).
"Niconico video (ニコニコ動画)" and "niconico live (ニコニコ生放送)" are operated by the same company under the same "niconico (ニコニコ)" brand, but they use completely different systems.

Big Thanks will have a look at it. There are websocket packages for python 2.7 tho. But I generally advise against python2 anyway. EOL will be January 1st 2020 so pretty soon.

I don't understand why youtube-dl still supports Python2 beyond EOL ... Supporting both Python2 and Python3 makes development, especially support for complex sites like "niconico" more difficult. I will. (Python3 supports up to Python 3.2 released in 2011)
I think at least Python2 support should be discontinued.
About encrypted videos

I honestly don't know why this site tries to use such weird mechanisms. They can be reversed. DRM would likely be a better option but hard to implement in large scale and can be defeated at some point.

First, about encrypted videos.
Applies only to some official videos such as anime.
For the mechanism, see https://qiita.com/tor4kichi/items/91550a71119f3878bfba#comment-346de200d84c79876831 (Japanese).
The mechanism of encryption itself is common, but the URL of the decryption key can only be accessed once. If you access it more than once, you will not be able to decrypt the segment because it will return a fake key that cannot be decrypted.
In addition, you need to ping https://nvapi.nicovideo.jp/v1/2ab0cbaa/watch?t={tracking_id} before downloading the playlist. Otherwise, a fake key that cannot be decrypted will be returned, even if the decryption key has never been accessed.
Previously, no measures were taken against downloads, but it has been implemented since March 2019. However, even official videos may not be encrypted.
I presume that "niconico video" intentionally does not use DRM. This is because there is a license fee to use DRM (the operating company of "niconico" has been in the red until recent years), and browser support is limited.
In the fix / niconico-c branch, processing has been added to handle these.
About heartbeat
Next, about heartbeat.
The mechanism is https://qiita.com/tor4kichi/items/91550a71119f3878bfba#動画再生の�� % 8F% E3% 83% BC% E3% 83% 88% E3% 83% 93% E3% 83% BC% E3% 83% 88% E3% 82% 92% E8% A1% 8C% E3% 81% 86 Please see (Japanese).
Heartbeat is required for all videos that use the new Nico Nico Douga distribution system (DMC: Dwango Media Cluster), with or without encryption.
Old videos may use the old distribution system, but it seems that they will gradually move to the new distribution system.
The video viewing session itself can be obtained by analyzing the json data in the source of the viewing page. However, if you do not send a heartbeat, the viewing session will be closed after a certain amount of time.
Short videos can be downloaded while the viewing session is active, but long videos will fail to download in the middle because the viewing session will be closed during the download.
Unencrypted videos use HTTP, so youtube-dl's resume feature allows you to resume the download from the middle of the download. This means that you can run youtube-dl a few times to complete the download. However, it is very painful to keep running youtube-dl over and over again.
Encrypted videos use HLS and cannot be resumed because the decryption key is different for each viewing session.
In other words, in order to support encrypted videos, in addition to complying with the specifications described in "About encrypted videos", send heartbeats regularly during download to maintain the viewing session Must be.
If you implement the heartbeat function, in addition to being able to support encrypted videos, you will be able to download unencrypted videos with just one run of youtube-dl without resuming.
I don't understand why "niconico video" uses such a strange mechanism. Is it a countermeasure against downloads, or is it to reduce the load on traffic and servers ...

Haven't forgotten this PR.

This is great. The only problem I have is  uploader metadata is not fetched or some other problem —it becomes NA in output template. With the same output template,  aforementioned fork (https://github.com/animelover1984/youtube-dl) produces correct output.

Long time no see!
We are pleased that each other's repositories are back.
Currently it is being developed with "blackjack4494/yt-dlc", what will you do in the future?
I was temporarily developing with "tsukumijima/yt-dl", but since the original repository was restored, "tsukumijima/yt-dl" was archived.


Add support for image memory qualifiers

coherent, volatile, restrict, readonly, writeonly
Slang reports an error if I use any of them.
Simple compute-shader to reproduce the problem:
layout(set = 0, binding = 1, rgba32) uniform writeonly image2D gOutput;
layout(local_size_x = 16, local_size_y = 16, local_size_z = 1) in;
void main()
{
imageStore(gOutput, gl_LocalInvocationID.xy, vec4(1,1,1,1));
}

This will be easy enough to add. If you only need them to be parsed, and don't need reflection, you should be able to do it yourself as a workaround until I can get to it.
In source/slang/slang-stdlib.cpp, near the end, you will find these lines:
        sb << "__modifier(SimpleModifier)           flat;\n";
        sb << "__modifier(SimpleModifier)           highp;\n";

You should be able to do the same thing to add as many qualifiers as you like, that Slang will parse, ignore, and then spit back out. E.g.:
        sb << "__modifier(SimpleModifier)           coherent;\n";
        sb << "__modifier(SimpleModifier)           volatile;\n";
...

I'll try to do a more thorough survey of GLSL qualifiers soon.


Typo "Azure file storage"→"Azure File Storage"

https://docs.microsoft.com/en-us/azure/machine-learning/team-data-science-process/roles-tasks
#PingMSFTDocs

@hyoshioka0128 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Hi @hyoshioka0128 - As noted in https://github.com/MicrosoftDocs/azure-docs/pull/75843, internal style guidance replaced "Azure File Storage" with "Azure Files" as an Azure service name a couple years ago. The change in the pull request introduces incorrect capitalization, so we'll close the pull request. Thanks.


Please add a license

Please choose and add a license. It looks like a promising project.

hi,
indeed i should.
licensing has been a little confusing because it current links against z3, but i'm in the middle of untangling that. soon (tm) ;)


Make Dredd thread-safe

Use thread local storage to make Dredd thread safe, and add tests that use tsan to demonstrate this.

Addressed via #169 - except that tsan is not used in the CI.


How can I get pages count of docx file? Thanks!

Hi All:
I want to get pages count of docx file with python-docx.
But I can not find way in python-docx document.
Any help is appreciated!

There isn't a way to do it in python-docx.
Until the document is rendered for a certain page size, there's no way
to know where the page breaks are or how many pages there will be.
python-docx has no way to know if you're using A4 or US-Letter or legal
size pages, which will produce different page counts.  I sometimes get
different results with the same data depending on whether I'm on Windows
of macOS.
David
On 10/10/22 10:18 PM, fiveight58 wrote:

Hi All:
I want to get pages count of docx file with python-docx.
But I can not find way in python-docx document.
Any help is appreciated!
—
Reply to this email directly, view it on GitHub
https://github.com/python-openxml/python-docx/issues/1142, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AC7ECDNRXR6SJZ3U56QJSU3WCTMBBANCNFSM6AAAAAARB3L3ME.
You are receiving this because you are subscribed to this
thread.Message ID: @.***>

--
David K. Woods
Pronouns: he, him, his

I get it. Thank you!


skaffold 1.7.0

Updating package skaffold to release v1.7.0.
Release info
v1.7.0 Release - 04/02/2020
Highlights:

kustomize dependencies include environment files (#3720) #3721
Support globs in custom/buildpacks builder deps #3878

Linux
curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/v1.7.0/skaffold-linux-amd64 && chmod +x skaffold && sudo mv skaffold /usr/local/bin
macOS
curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/v1.7.0/skaffold-darwin-amd64 && chmod +x skaffold && sudo mv skaffold /usr/local/bin
Windows
https://storage.googleapis.com/skaffold/releases/v1.7.0/skaffold-windows-amd64.exe
Docker image
gcr.io/k8s-skaffold/skaffold:v1.7.0
Fixes:

Fix GCB build failure for multi-module Jib projects #3852
Fix possible nil dereference #3869
Fix console output for internal Jib tasks/goals #3880
Fix go test helper #3859

Updates & Refactors:

Better status check #3892
disable jib gradle in skaffold init by default #3906
Use new name for the linter’s cache #3894
Use less memory for linting #3888
Simplify Kaniko error message #3870
Wait for the logs to be printed #3877
Master Keychain #3865
Replace errors.Wrap with %w #3860
Show compilation errors #3866
Cobra context #3842
Format go test output with Go rather than bash and jq #3853

Design proposals:

Update debug-events design proposal status #3874

Docs updates:

Rework debug docs #3875
Fix of documentation issue #3266 microservices example is broken #3867
[docs] [release] fix firebase-tools version #3857
[examples] upgrade nodejs example dependencies #3858
Fix doc link to local cluster info #3856
upgrade hugo + small fixes #3854

Huge thanks goes out to all of our contributors for this release:

Balint Pato
Brian de Alwis
David Gageot
Dmitry Stoyanov
Nick Kubala
Nick Novitski
Tad Cordle
tejal29

https://github.com/fishworks/fish-food/pull/667 should fix build error.


Update npm deps

New version of Biome performs more comprehensive checks

Anytime!

@smorimoto I was under the impression that you'd be able to merge after my approval, apologies if that's not the case, I just merged. The change should go out with the next point release, if you think it makes sense to create a release from these changes please let me know.

Unfortunately, I'm not a GitHub employee... 😢

That can always be fixed, we're hiring: https://www.github.careers/careers-home/jobs :)


Support vsDivider SSR support

About
Support vsDivider SSR support.

perfect I will check to update, thank you very much for the contribution


Additional ScrollingFrame properties

Coverage increased (+0.02%) to 97.918% when pulling d71c2f2309256e9049bfc9e32ff7c7d144265f62 on CodeKingdomsTeam:scrollingframe-properties into b42d32f690e330a170b10561710fc1f081285e3d on LPGhatguy:master.

Thanks a bunch! Sorry I've been so behind on pull requests for Lemur.


feat: add login in e2e test

Signed-off-by: liuying<EMAIL_ADDRESS>
What type of PR is this?
/kind feature

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #211
Special notes for reviewers:

Does this PR introduced a user-facing change?

Additional documentation, usage docs, etc.:

Codecov Report

Merging #212 (ffd696b) into master (6e5832f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #212   +/-   ##
=======================================
  Coverage   46.82%   46.82%
=======================================
  Files           7        7
  Lines         598      598
  Branches      168      168
=======================================
  Hits          280      280
  Misses        245      245
  Partials       73       73


Goreleaser brew

@busser The issue is we can't. Just try it. If it fails, delete the last commits and release and retry.
You need to have the github token in your env variables that is all.

And yes tfautomv-maintainers should be added as maintainers or we could let anybody at Padok use it.
I'd go for the second option.