Istio-proxy connection drops during Istio version upgrades/downgrades

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description
We are experiencing significant connection drops in istio-proxy when upgrading or downgrading between major Istio versions. This issue was first encountered during an upgrade from 1.21 to 1.22, and we are currently observing similar behavior while testing upgrades and downgrades between 1.22 and 1.23.
Interestingly, we have not observed connection drops when upgrading between minor versions within the same major release (e.g., 1.22.2 to 1.22.6 or 1.23.1 to 1.23.3). However, downgrading from 1.23 to 1.22 consistently triggers the issue.
After enabling debug logs, we observed that istio-proxy drains all its listeners during these problematic upgrades/downgrades.
Please find attached the relevant metrics graphs illustrating the connection drops.

We are experiencing significant connection drops particularly for websocket services.
{"time":"2024-11-07T22:55:12.383604Z","level":"debug","scope":"envoy connection","msg":"current connecting state: true","caller":"external/envoy/source/common/network/connection_impl.h:98","thread":"140","ConnectionId":"884319"}
{"time":"2024-11-07T22:55:12.383611Z","level":"debug","scope":"envoy connection","msg":"connecting to <IP_ADDRESS>:8000","caller":"external/envoy/source/common/network/connection_impl.cc:1017","thread":"140","ConnectionId":"884319"}
{"time":"2024-11-07T22:55:12.383704Z","level":"debug","scope":"envoy connection","msg":"connection in progress","caller":"external/envoy/source/common/network/connection_impl.cc:1036","thread":"140","ConnectionId":"884319"}
{"time":"2024-11-07T22:55:12.383923Z","level":"debug","scope":"envoy connection","msg":"connected","caller":"external/envoy/source/common/network/connection_impl.cc:746","thread":"140","ConnectionId":"884319"}
{"time":"2024-11-07T22:55:12.383936Z","level":"debug","scope":"envoy pool","msg":"attaching to next stream","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:328","thread":"140","ConnectionId":"884319"}
{"time":"2024-11-07T22:55:12.383940Z","level":"debug","scope":"envoy pool","msg":"creating stream","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:182","thread":"140","ConnectionId":"884319"}

{"time":"2024-11-07T22:57:22.111256Z","level":"debug","scope":"envoy pool","msg":"draining busy client 884319 for cluster outbound|80||websockets.xxx.svc.cluster.local","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:441","thread":"140"}

{"time":"2024-11-07T22:57:22.115820Z","level":"debug","scope":"envoy connection","msg":"remote close","caller":"external/envoy/source/common/network/connection_impl.cc:714","thread":"140","ConnectionId":"884319"}
{"time":"2024-11-07T22:57:22.115829Z","level":"debug","scope":"envoy connection","msg":"closing socket: 0","caller":"external/envoy/source/common/network/connection_impl.cc:276","thread":"140","ConnectionId":"884319"}
{"time":"2024-11-07T22:57:22.116187Z","level":"debug","scope":"envoy pool","msg":"client disconnected, failure reason: ","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:495","thread":"140","ConnectionId":"884319"}
{"time":"2024-11-07T22:57:22.130772Z","level":"debug","scope":"envoy pool","msg":"destroying stream: 0 remaining","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:215","thread":"140","ConnectionId":"884319"}

Suspected EnvoyFilter Configuration
We have the following EnvoyFilters in our Istio-system namespace, which we suspect might be contributing to the connection drops, although we are not certain:
- apiVersion: networking.istio.io/v1alpha3
  kind: EnvoyFilter
  spec:
    configPatches:
    - applyTo: NETWORK_FILTER
      match:
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
      patch:
        operation: MERGE
        value:
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            tracing:
              client_sampling:
                value: x
              overall_sampling:
                value: x
              random_sampling:
                value: x
- apiVersion: networking.istio.io/v1alpha3
  kind: EnvoyFilter
  spec:
    configPatches:
    - applyTo: ROUTE_CONFIGURATION
      match:
        context: ANY
      patch:
        operation: MERGE
        value:
          response_headers_to_remove:
          - x-envoy-foo
## We have not migrated to Telemtry API yet(beacasue of disable_host_header_fallback support), so we are still using EnvoyFilter based istio stats
- apiVersion: networking.istio.io/v1alpha3
  kind: EnvoyFilter
  spec:
    configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_OUTBOUND # similar applied to SIDECAR_OUTBOUND GATEWAY
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: envoy.filters.http.router
      patch:
        operation: INSERT_BEFORE
        value:
          name: istio.stats
          typed_config:
            '@type': type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/stats.PluginConfig
            value:
              disable_host_header_fallback: true
              metrics:
              - dimensions:
- apiVersion: networking.istio.io/v1alpha3
  kind: EnvoyFilter
  spec:
    configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: SIDECAR_INBOUND # similar applied to SIDECAR_OUTBOUND GATEWAY
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.tcp_proxy
      patch:
        operation: INSERT_BEFORE
        value:
          name: istio.stats
          typed_config:
            '@type': type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/stats.PluginConfig
            value:
              disable_host_header_fallback: true
              metrics:
              - dimensions:
- apiVersion: networking.istio.io/v1alpha3
  kind: EnvoyFilter
  spec:
    configPatches:
    - applyTo: NETWORK_FILTER
      match:
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
      patch:
        operation: MERGE
        value:
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            strip_trailing_host_dot: true
- apiVersion: networking.istio.io/v1alpha3
  kind: EnvoyFilter
  spec:
    configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: ANY
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
      patch:
        operation: MERGE
        value:
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            max_request_headers_kb: xxx

Version
❯ istioctl version
client version: 1.22.2
control plane version: 1.22.4
data plane version: 1.22.4 (2353 proxies)

❯ kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.6

Additional Information
No response

Didn't you upgrade dataplane?
You can dump the listener configuration geenrated by both 1.23 and 1.22 and  then compare, should find out what diff they have

Hi @hzxuzhonghu, thanks for responding.
To clarify, we are only updating the istiod deployment and not making any changes to the data plane. The only change in the control plane configuration during the 1.22 to 1.23 upgrade was the removal of the enableProtocolSniffingForOutbound field from the mesh config - https://github.com/istio/istio/pull/51989.
We observed some differences in the istio-proxy listeners after upgrading to 1.23. but we couldn't identify a definitive cause for the connection drops.

The InitialReadBufferSize: 512 attribute was added to the envoy.filters.listener.tls_inspector listener filter.
The proxy100Continue: true attribute was added to the envoy.filters.http.router listener filter.
A filterChainMatch block, which included a prefixRanges definition, was removed.

Here are the relevant snippets from the listener configurations:
Added:
listenerFilters:
- name: envoy.filters.listener.tls_inspector
  typedConfig:
    '@type': type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector
    initialReadBufferSize: 512

- name: envoy.filters.http.router
 typedConfig:
   '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
maxRequestHeadersKb: 320
normalizePath: true
pathWithEscapedSlashesAction: KEEP_UNCHANGED
proxy100Continue: true

Removed
- filterChainMatch:
    prefixRanges:
    - addressPrefix: <IP_ADDRESS>
      prefixLen: 0

Yeah，these are the reasons cause listener drain, and all connection would be closed too

canary upgrade will help.

When XDS changes it (often) requires listener drains which terminate connections. Upgrading often changes the XDS, as we make improvements/features/etc.
Canary upgrade of the control plane (https://istio.io/latest/docs/setup/upgrade/canary/) solves this, since no proxy changes the XDS it gets -- only new proxies get the new config.


Change ant to gradle in tutorial

JMeter has been updated to use gradle for building instead of ant.
Some directories have been added meanwhile (in the src folder and elsewhere).
Try to reflect those changes in the tutorial.
Description
Update tutorial
Motivation and Context
Build system has been updated from ant to gradle. Tutorial should be using the new build system in their examples.
How Has This Been Tested?

Screenshots (if appropriate):
Types of changes

Bug fix (non-breaking change which fixes an issue)

Checklist:

[x] My code follows the code style of this project.
[x] I have updated the documentation accordingly.

Codecov Report

Merging #521 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #521      +/-   ##
============================================
+ Coverage     55.96%   55.96%   +<.01%
- Complexity     9960     9961       +1
============================================
  Files          1018     1018
  Lines         62738    62738
  Branches       7059     7059
============================================
+ Hits          35112    35113       +1
+ Misses        25161    25160       -1
  Partials       2465     2465

Impacted Files
Coverage Δ
Complexity Δ

...n/java/org/apache/jmeter/reporters/Summariser.java
86.25% <0%> (+0.76%)
19% <0%> (+1%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0565293...ca134f2. Read the comment docs.

Thanks for this PR.
LGTM, just need to update missing bolt folder.

yes good idea
On Friday, October 4, 2019, Vladimir Sitnikov<EMAIL_ADDRESS>wrote:

@vlsi commented on this pull request.
In xdocs/usermanual/jmeter_tutorial.xml
https://github.com/apache/jmeter/pull/521#discussion_r331461106:

@@ -110,12 +110,28 @@ The folders inside of src

+<!--
+  <dt>bom
I think we should approach bom in a different way: we should probably add
a section on "how to add a new third-party dependency to JMeter" and "how
to update dependency version".
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/apache/jmeter/pull/521?email_source=notifications&email_token=AAX3RKYSV237GF4IHYK7BRLQM4UDFA5CNFSM4I5DRNMKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCG5EOBY#discussion_r331461106,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAX3RK6LWDYISGP334IC3SLQM4UDFANCNFSM4I5DRNMA
.

--
[image: logo Ubik Ingenierie] https://www.ubik-ingenierie.com Philippe
Mouawad
Senior Performance Expert
320914981 <+33320914981> |<EMAIL_ADDRESS>[image: ubik-ingenierie.com] ubik-ingenierie.com
https://www.ubik-ingenierie.com | [image: <IP_ADDRESS>.81] <IP_ADDRESS>.81
<+33320914981> | [image: 23 rue du chemin de fer , 59100 , Roubaix] 23 rue
du chemin de fer, 59100, Roubaix
https://www.openstreetmap.org/#map=18/50.69454/3.16455


Create a dialouge box before the sync starts.

Problem description.
In the services app, when a user clicks the sync now button, it directly starts syncing which we can't stop by pressing the back button.

Solution
Create a dialogue box when the sync now button is clicked. The dialogue box will ask the user if they want to proceed or not. If the user clicks yes, the sync will start, else it will close the dialogue.
Advantage
It will allow users to avoid syncing because of a misclick.
If this is a valid issue I would like to work on it.

I think stopping sync when the back button is pressed while the sync is going on is a better option, even after implementing the dialog, the user cannot stop the sync if it starts

I think stopping sync when the back button is pressed while the sync is going on is a better option, even after implementing the dialog, the user cannot stop the sync if it starts

I thought about the same too, but I didn't want to mess up data while it starts syncing and stopping it suddenly. If we put a proper message in the dialogue box, the user can know what's going to happen when the button is clicked and can decide accordingly, if he/she wants to proceed or not.

@SilentCruzer thanks for bringing this up, since there are multiple ways to approach this I will need to consult with the PMC.

When I first installed the app and was exploring it, I clicked the sync button and it took a long time to download all the data. At that  moment I felt the need of having a cancel button. But I guess this problem is only faced by contributors like us. People using the app for real applications on daily basis would know the results of initiating a sync.
I personally feel that they might be annoyed by doing confirmation each time they want to sync. Moreover its only the first sync that takes much data usage and time subsequent syncs are lot faster.

In the interest of a timely feedback, the PMC/TSC had an email discussion about this issue. The general consensus was seems like it could be help sometimes but also adds a step to a frequent workflow. Note that some people have already requested shorter and easier syncing steps. The concerns about easier flow in logging in and syncing is the reason we have the GSoC project Improve ODK-X Services Login User-Interface.
One approach that would not add any required clicks is to allow cancellation within the first few seconds. The idea would be within the first few seconds the Services would show a message like "Preparing to sync" but it's just spinning for a few seconds to allow the user to cancel.

@wbrunette may I work on this issue?

@sanyud24 I was reviewing the issues assigned to you and you have several. Some of them are from a while ago. It would be nice if you removed yourself from the ones you are no longer working on. Then I will assign you this issue. Keep your self assigned to ones that you have completed and are under review.

Sure @wbrunette, I've unassigned myself from https://github.com/odk-x/tool-suite-X/issues/154, My PRs are under review in the other 2 issues


Getting TypeError - no implicit conversion of Pathname into String

I am trying to get Cocopods (local) work with artifactory. Below are the steps I have done.

Created Cocopods-local - repo
Deployed 'mypod.tar.gz' to local instance of jfrog artifactory pro.
Changed pod file.

use_frameworks!

ENV['COCOAPODS_DISABLE_STATS'] = 'true'

plugin 'cocoapods-art', :sources => [
 'cocoapods-local'
]

target 'Download Sample' do
   pod 'dowloadHud'
 target 'Download SampleTests' do
   inherit! :search_paths
 end
 target 'Download SampleUITests' do
   inherit! :search_paths
 end
end

When I run pod install, I am getting below error.

[!]Error installing dowloadHud

TypeError - no implicit conversion of Pathname into String
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/fileutils.rb:513:in `initialize'
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/fileutils.rb:513:in `exception'
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/fileutils.rb:513:in `raise'
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/fileutils.rb:513:in `block in mv'
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/fileutils.rb:1558:in `block in fu_each_src_dest'
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/fileutils.rb:1567:in `block in fu_each_src_dest0'
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/fileutils.rb:1565:in `each'

I encountered the same issue.
When I tried with pod install --verbose, I discovered that the pod was found but the unarchive operation seemed to fail. The pod install error was :
`Installing AppMonitoring (0.1.2)

Http download
$ /usr/bin/curl -n -f -L -o /var/folders/8f/wn5b73g536bc9r1x37qgw3pr0000gp/T/d20170329-79444-1tb7180/file.tgz
http://url.of.repo/api/pods/cocoapods-corporate/pod/pkg/AppMonitoring/0.1.2 --create-dirs
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100  482k  100  482k    0     0  2177k      0 --:--:-- --:--:-- --:--:-- 2183k
$ /usr/bin/tar xfz /var/folders/8f/wn5b73g536bc9r1x37qgw3pr0000gp/T/d20170329-79444-1tb7180/file.tgz -C /var/folders/8f/wn5b73g536bc9r1x37qgw3pr0000gp/T/d20170329-79444-1tb7180
[!] Error installing AppMonitoring
`

I tried the link used by the previous curl command and the download was successfull.
In my case it was because the uploaded tar.gz file did not contain the .git folder.
So to fix this, I just opened a terminal and went to the folder of my pod, then typed the command :
tar pczf MyPod.tar.gz .
Finally I uploaded my archive and update with pod repo-art update <repo_name>
After that pod install was succesfull.
Hope it'll help you.

@PinwheelAi  Can you check your spelling? You created Cocopods-local but in your podfile you wrote cocoapods-local.


Updated URL to fix failing action

Happy new year! Wishing you all the best.
Updated URL of blog from @lukemurraynz

I need to figure out why reddit is throwing a 403.. will update on this later 😃

Thank you John for the updates!


Stream and pipe resources with responseType: 'stream'

Describe the feature
With axios, we can stream and pipe resources like this:
import axios from 'axios'
import fs from 'fs'

const file = fs.createWriteStream('./assets/sample.jpg')

const { data } = await axios({
  url: 'https://samplesite.com/uploads/sample.jpg',
  method: 'GET',
  responseType: 'stream'
})
data.pipe(file)

How can we do it with ofetch? Any documentation around?
Additional information

[ ] Would you be willing to help implement this feature?

Hi. You can use same responseType: "stream" API with ofetch 👍🏼

@pi0
I am getting TypeError: response.pipe is not a function, my code:
import { ofetch } from 'ofetch';
import fs from 'fs';

const response = await ofetch('https://defly.io/img/add-skins.js', {
    responseType: "stream",
});

response.pipe(fs.createWriteStream('./cache/allSkins.json'));

same issue

unlike node-fetch, ofetch returns a web ReadableStream which has .pipeTo method and accepts also a web WritableStream.
This is a working example for Node.js 18+
import { ofetch } from "ofetch";
import fs from "node:fs";

const response = await ofetch("https://defly.io/img/add-skins.js", {
  responseType: "stream",
});

const fsStream = fs.createWriteStream("./add-skins.js");

response.pipeTo(
  new WritableStream({
    write(chunk) {
      fsStream.write(chunk);
    },
  })
);


Models should trigger a success event when saved.

I have a view that is responsible for displaying error and success.  I have another view that is used to manipulate the model.  When I save I need a 'success' event so that I can display a message in the view that is responsible for displaying messages.
This may benefit other backbone users.

+1 since it's an unobtrusive enhancement I see myself using. Maybe 'saved' instead of 'success' for the event name. Should also explore making this consistent across fetch and destroy. Destroy already triggers an event except it's not exclusive to the callback. Maybe model should both call success if it exists, and trigger the event. Or destroy should only trigger if there is no success.
I suspect I would get annoyed if model only triggered sometimes (in the event no success callback is passed) so I think doing both is probably the right decision. If you pass a callback, call it. But still trigger the event (like the current model.destroy())
And further, extend to collections fetch/create.

If this was added, I'd want it to always fire.

+1 from me

Take a look at the above commit -- you should be able to listen for sync, which will fire whenever a call has succeeded to make a change on the server-side ... creating, updating, or destroying a model.

I certainly think this is consistent.  However, I don't see the point?  Why not just bind a dom element to $.ajaxStop, to display "Successful Sync"?  This doesn't give the developer any knowledge of what happened other than they got a success status.  Was it a fetch, was it a save, was it destroyed?

This looks a lot like what was done with Backbone.wrapError, in which case this could dry it up.
Backbone.wrapSuccess = function(onSuccess, originalModel, options) {
  return function(model, resp) {
    if (onSuccess) {
      onSuccess(model, resp);
    } else {
      originalModel.trigger('sync', model, resp, options);
    }
  };
};

Or more generically something like this for both error and sucess.
Backbone.wrapAction = function(onAction, originalModel, options, eventName) {
  return function(model, resp) {
    var resp = model === originalModel ? resp : model;
    if (onAction) {
      onAction(model, resp, options);
    } else {
      originalModel.trigger(eventName, model, resp, options);
    }
  };
};

Then the eventName could just be passed in.

Unfortunately, the "what to do" bit of the success callback is different in every case. But if you have a DRYer commit, I'd love to take a pull request.


Uncaught TypeError: Cannot read property 'classList' of undefined

When not using lineStringCtrl, polygonCtrl, squareCtrl or markerCtrl this line throws the error in the title.
https://github.com/mapbox/gl-draw/blob/46d69fdbf6d83a7847133dbb9077995dba588f39/src/draw.js#L384-L387

done in https://github.com/mapbox/gl-draw/commit/c5a951db398d903ea59aa1387db511ccc92acf29

I don't think this is fixed. The issue is here https://github.com/mapbox/gl-draw/blob/master/src/draw.js#L389-L392 and appears when not using all controls when initiating Draw. Example:
    let draw = this.draw = glDraw({
      position: 'bottom-right',
      keybindings: true,
      geoJSON: [],
      controls: {
        square: true,
      }
    });

this.lineStringCtrl, this.polygonCtrl and this.markerCtrl will then be undefined and throws a TypeError when accessing classList.

@gstf oh no! I'll fix this as soon as possible.
Thank you :smile:

:+1: thanks!


As a developer, I want to one-time bulk fix HathiTrust excerpt page ranges from a spreadsheet so that we can pull correct page content when we reindex.

[x] ~adapt import script~ write script to update digital pages
[x] run against this spreadsheet

acceptance criteria

[ ] updates digital page range in database
[ ] creates log entry documenting the change
[ ] reindexes pages in Solr based on new range
[ ] appropriate/reasonable error handling and reporting

@mnaydan I'm writing the script to require a CSV that includes source_id, pages_orig, and new_pages_digital. I also prefer we do filtering on the CSV before using it with the script (it should only include rows for records we want updated). Does that sound ok to you?

For testing, I downloaded the first tab from Google Sheets, filtered out all the rows where the digital page range was marked as "correct", and renamed the "new digital range" to new_pages_digital.
In case it's useful, I used grep to filter out correct rows:
grep --invert ,correct, excerpt_updates.csv > excerpt_update_changes.csv

I didn't know that one of the rows was marked as "SUPPRESS" but it turned out to be useful for testing error handling (that record failed to save).

@rlskoeser that sounds good generally, but here's the needle in the haystack... there was at least one case where original page range changed (roman numerals) due to a typo I discovered in the original input. Will original page range need to match the database field exactly? Would fixing the typo in the database resolve the issue? If there's quick error handling for any matches NOT found that would be useful.

@mnaydan script reports on matches that are not found - I had one where the original page range was slightly different in the spreadsheet that in my copy of the database (which is probably out dated).  Updating the incorrect page range in the database would resolve that problem. I thought looking for exact match would be best.

@rlskoeser okay perfect, I'll update that page range in the database and if it reports on no matches then we are good.

I ran the script in staging with this CSV file as input (generated from the google doc version as noted above):
excerpt_update_changes.csv
Here is how I ran the script and the summary output:
$ ./manage.py adjust_excerpts /tmp/excerpt_update_changes.csv
Error saving mdp.39015036664038 (SUPPRESS): Can't parse chunk 'SUPPRESS'
No record found for source id uc1.c2641998 and pages_orig 32-33, 66

Updated 119 records. 0 unchanged, 1 not found, 1 error.

I ran this in staging without refreshing from production because I wanted it to have the recent-ish rsync changes. (I did not run rsync immediately before).
If it's helpful for testing, you could update the original pages for the not found record in the staging database and I can run this script again. I could also run rsync. At some point before we release, we may want to test the full set of steps we will be doing in production (perhaps after we fix excerpt ids): replicate production to staging, rsync, update excerpts.
If I run the script again with the same input, it recognizes that it doesn't need to make changes:
$ ./manage.py adjust_excerpts /tmp/excerpt_update_changes.csv
Error saving mdp.39015036664038 (SUPPRESS): Can't parse chunk 'SUPPRESS'
No record found for source id uc1.c2641998 and pages_orig 32-33, 66

Updated 0 records. 119 unchanged, 1 not found, 1 error.

@rlskoeser this is really helpful, thanks! Let me fix the errors in production and staging and then we can re-run... give me a moment.

@mnaydan I don't know! Probably not correctly, I didn't know that was a possible case. LMK what it should do - at a minimum I can make sure to filter out full works when we look for matches...

@rlskoeser OK good to know. It's not a full work, it just doesn't have any physical page numbers printed on the pages! I will just fix that one manually on the backend and then you shouldn't have to change anything with the script. I think it's the only blank.

@mnaydan worth checking what it did when you test the script; if it's the only excerpt from that volume it may have done the right thing.

@rlskoeser great minds! Ok, can we run it again? I am expecting the uc1.c2641998 error to be handled now, and 2 additional updated records.

Regenerated a test CSV from the google sheets (forgot to exclude the suppressed one) and ran again; here is the output:
$ ./manage.py adjust_excerpts /tmp/excerpt_update_changes_v2.csv
Error saving mdp.39015036664038 (SUPPRESS): Can't parse chunk 'SUPPRESS'

Updated 3 records. 117 unchanged, 0 not found, 1 error.

@rlskoeser yay! This is exactly what I expected. Do you want to close and track testing the full set of steps elsewhere?

@rlskoeser wait, I just saw your full set of acceptance criteria. I'm getting clearly bleary-eyed after a long week... let me test all those steps.

I spot checked a few records, everything looks great! I tested single page, changed range, unchanged/correct range, discontinuous page numbers, ark:/ IDs, the one blank original page range record... they all appear in the database and indexed how I would expect in all cases.


Animations

An animate: Bool property to animate the chart when it appears the first time. This has not been implemented yet, since it requires some time to make it working with the area fill. An animated mask should be the easiest solution.

Done it! Check my pull request ;)
#10

Any update on this?


Transform layered styles

Done :

remove @layer and adjust specificity of selectors
remove @layer and re-order @keyframes, @counter-style, ...
some refactoring

Known issues :

unable to handle layers having a different order in multiple @media rules
!important behaviour is lacking WPT tests and browser implementations are inconsistent

Not handled here :

revert
revert-layer

@oluoluoxenfree This PR contains a big chunk of work towards completing this plugin :)
Please ask any questions you might have about this change.

@oluoluoxenfree I've added some suggestions for code comments that hopefully answer your questions. Please let me know if these are not yet fully clear.

What are the next steps on this also?

If this batch of changes is clear and works for you then it can be merged.
In general I think only !important and revert|revert-layer still need to be handled.

@romainmenke is this ready to merge?


Import CA for Captive Portal

The script needs to be able to update/import the Letsencrypt CA, to support the Captive Portal better.

Just a nudge to myself to figure this out.
LetsEncrypt issued new intermediate CAs on June 6th, which broke all my "pinned" manually loaded R3 Intermediate certificates in pfSense.
https://letsencrypt.org/2024/04/12/changes-to-issuance-chains.html
And funnily enough, that is one of their goals, to switch Intermediates more often to make it painful for those that don't handle changing intermediate certs automatically.  They will be switching between 2 intermediate chains on each renewal, and switching to two different ones occasionally.  I guess good job LetsEncrypt for breaking my current setup :-)
So this script needs to grab the intermediate from the chain.pem file and load that as a new CA on each update.


Sparse files gets de-sparsified

After execution when output files are copied from the worker to the filesystem CAS directory, they lose their sparse file properties and get fully de-sparsified. This obviously increases disk usage, and the entire process of copying the file & hashing takes considerably more time than a simple move operation followed by a sha256sum calculation on a sparse file.
(byte-by-byte copy and not moved or copied via reflink/copy_file_range).
https://github.com/TraceMachina/native-link/blob/f989e612715a7fe645e69c4c78a50e9b7262ad17/config/examples/basic_cas.json
genrule(
  name = "create_sparse_file",
  outs = ["re_large_file"],
  cmd = "fallocate -l 5G $(OUTS) "
)

lukas@PC6061B ~/Downloads/turbo-cache $ filefrag -v tmp/turbo_cache/work/30d5d89de2794c8b81c8ceac8c02d0cc96085888988a72075d39e8b939202650/bazel-out/k8-fastbuild/bin/re_large_file
Filesystem type is: 9123683e
File size of tmp/turbo_cache/work/30d5d89de2794c8b81c8ceac8c02d0cc96085888988a72075d39e8b939202650/bazel-out/k8-fastbuild/bin/re_large_file is<PHONE_NUMBER> (1310720 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..   65535:<PHONE_NUMBER>..1863537599:  65536:             unwritten
   1:    65536..  196607:<PHONE_NUMBER>..1872581567: 131072:<PHONE_NUMBER>: unwritten
   2:   196608..  327679:<PHONE_NUMBER>..1881691071: 131072:<PHONE_NUMBER>: unwritten
   3:   327680..  458751:<PHONE_NUMBER>..1883106065: 131072:<PHONE_NUMBER>: unwritten
   4:   458752..  524287:<PHONE_NUMBER>..1883722687:  65536:<PHONE_NUMBER>: unwritten
   5:   524288..  589823:<PHONE_NUMBER>..1883807173:  65536:<PHONE_NUMBER>: unwritten
   6:   589824..  655359:<PHONE_NUMBER>..1890829397:  65536:<PHONE_NUMBER>: unwritten
   7:   655360..  720895:<PHONE_NUMBER>..2028295103:  65536:<PHONE_NUMBER>: unwritten
   8:   720896..  917503:<PHONE_NUMBER>..2029081535: 196608:<PHONE_NUMBER>: unwritten
   9:   917504.. 1114111:<PHONE_NUMBER>..2030654399: 196608:<PHONE_NUMBER>: unwritten
  10:  1114112.. 1310719:<PHONE_NUMBER>..2031178687: 196608:<PHONE_NUMBER>: last,unwritten,eof
tmp/turbo_cache/work/30d5d89de2794c8b81c8ceac8c02d0cc96085888988a72075d39e8b939202650/bazel-out/k8-fastbuild/bin/re_large_file: 11 extents found

File size of tmp/turbo_cache/data-worker-test/content_path-cas/7f06c62352aebd8125b2a1841e2b9e1ffcbed602f381c3dcb3200200e383d1d5-5368709120 is<PHONE_NUMBER> (1310720 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..   65535:<PHONE_NUMBER>..2035075504:  65536:
   1:    65536..  131071:<PHONE_NUMBER>..2035657566:  65536:<PHONE_NUMBER>:
   2:   131072..  196607:<PHONE_NUMBER>..2037253713:  65536:<PHONE_NUMBER>:
   3:   196608..  262143:<PHONE_NUMBER>..2037339071:  65536:<PHONE_NUMBER>:
   4:   262144..  327679:<PHONE_NUMBER>..2038778730:  65536:<PHONE_NUMBER>:
   5:   327680..  458751:<PHONE_NUMBER>..2047186766: 131072:<PHONE_NUMBER>:
   6:   458752..  655359:<PHONE_NUMBER>..2047431615: 196608:<PHONE_NUMBER>:
   7:   655360..  851967:<PHONE_NUMBER>..2047693759: 196608:<PHONE_NUMBER>:
   8:   851968..  858862:<PHONE_NUMBER>..2047733422:   6895:<PHONE_NUMBER>:
   9:   858863..  859572:<PHONE_NUMBER>..2048257992:    710:<PHONE_NUMBER>:
  10:   859573..  859584:<PHONE_NUMBER>..1548714359:     12:<PHONE_NUMBER>:
  11:   859585..  859589:<PHONE_NUMBER>..1548714162:      5:<PHONE_NUMBER>:
  12:   859590..  925125:<PHONE_NUMBER>..2047038399:  65536:<PHONE_NUMBER>:
  13:   925126..  990661:<PHONE_NUMBER>..2047824831:  65536:<PHONE_NUMBER>:
  14:   990662.. 1187269:<PHONE_NUMBER>..2053264318: 196608:<PHONE_NUMBER>:
  15:  1187270.. 1310719:<PHONE_NUMBER>..2055484921: 123450:<PHONE_NUMBER>: last,eof

@lukts30 We're looking into this one.

I'm trying to focus on the root-problem and how we can support this kind of feature without breaking other API requirements (like NFS filesystems or other filesystems/stores that are not sparse supported) and I want to pose the problem to you to see if this is the underlying issue:
Problem:
When the Worker & CAS are on the same machine, it requires a full copy of the data.
If it is the problem, possible solution:
What I will propose is to make the Worker aware of when it is uploading to a local filesystem CAS and hardlink the file if it is on the same filesystem. This would make the copy only cost a hardlink (which is extremely cheap).
As a side note to this problem, you may be interested to also use dedup_store and/or compression_store which can be used together to optimize filesystem disk size for files that are often the same between builds.

Thanks for looking into this issue.

When the Worker & CAS are on the same machine, it requires a full copy of the data.

That is an accurate summary of this issue.

Worker aware of when it is uploading to a local filesystem CAS and hardlink the file if it is on the same filesystem

Indeed hardlinking/moving files or "copying" via reflink/copy_file_range (on supported FS) all would similarly address the expensive copy operation.
I am wondering if the suggested hardlink approach would also work in a scenario where the Workers are distributed, but the storage for both the CAS and worker filesystems is managed through the same shared NFS/SMB file system share?

I am wondering if the suggested hardlink approach would also work in a scenario where the Workers are distributed, but the storage for both the CAS and worker filesystems is managed through the same shared NFS/SMB file system share?

NFS/SMB as a shared medium for distributed workers to write too is not currently supported. This is because we evict items and currently don't have the ability to have external sources notify of things changing in the FilesystemStore.
Instead of using NFS, the preferred model is to use a remote store and xfer the data through some kind of pipe (eg: TCP). The complexity of adding network filesystem support sounds like it'd be very difficult to write and possibly add a lot of technical debt.
Given that, we do currently plan on supporting a Fuse filesystem that will materialize the files on demand. This would allow compression of data over the network & deduplication, at the cost of latency. In the case of NFS I would suspect the latency to be about the same, so it "might" be what you are looking for.

Commit https://github.com/TraceMachina/nativelink/commit/a0788fabc1831714e39fa5047e0a385a2c62234f did not change anything in this regard right? Still copies and does not use move/hard link to relocate the file.

I tested again a bit and RE through nativelink is still noticeably slower then what I would hope for.
My testing involved using a buck2 BUILD file first building :random_data and a separate build of :instant_copy, which utilizes the cached file for a reflink copy (takes only 5ms).
But :instant_copy took 70 seconds to complete, with  more then 99% time spend "transferring" the file to the CAS. Observed operations involved reading the entire $OUT file at a rate of 300MB/s, followed by simultaneous reading and writing, each at 300MB/s. (monitored through iotop; does not include time/data for RE client to fetch the result artifact)

accumulated read: 20GB
accumulated written: 10GB

Because the file is read twice the speed is in practice only 150MB/s. But even the 300MB/s is rather slow compared to other tools.
For comparison both curl/wget can download a file from a local HTTP Server (caddy) at 800MB/s without any special parameter and the builtin HTTP download rule in buck2 even achieves 1.2GB/s by default.
Additionally as I configured the hash to be BLAKE3 I also ran b3sum with a single thread on the 10GB file and that took 4-6s (avg > 1.6GB/s read).
genrule(
    name = "random_data",
    out = "gen_large_file",
    cmd =  "dd if=/dev/urandom of=$OUT bs=1G count=10 iflag=fullblock",
)

genrule(
    name = "instant_copy",
    out = "gen_large_file",
    cmd =  "cp --reflink=always $(location :random_data) $OUT",
)

Yes, the commit I made was just lining things up to support the ability to support special filesystem calls like sparse copy.
Can you confirm that you compiled with --release (-o3'ish)? There is a significant performance difference if you don't compile in release mode.
I saw that you are using basic_cas.json, but can you confirm that you did not make any changes to it (that would impact these results)?

I am using a release build and the tests were done with commit https://github.com/TraceMachina/nativelink/commit/2a89ce6384b428869e21219af303c753bd3087b5 because of https://github.com/TraceMachina/nativelink/issues/665.
I used basic_cas.json only changed the output paths, set the hash to blake3 and tried removing the memory store but that did not change much.

I did some local testing and yes I do see it taking much longer than it should be.
Here's my local results:
real 24.834s = time dd if=/dev/urandom of=/tmp/dummy_foo bs=1G count=10 iflag=fullblock
real 3.31s   = cat ./dummy_foo | time b3sum --no-mmap --num-threads 1
real 8.040s  = time cp ./dummy_foo ./dummy_foo2

Nativelink results (modified source to capture timing):
24.854s = running dd if=/dev/urandom of=/tmp/dummy_foo bs=1G count=10 iflag=fullblock
47.4s   = computing hash
25.312s = uploading file (copy file)

If I change this line to 1Mib:
24.751s = running dd if=/dev/urandom of=/tmp/dummy_foo bs=1G count=10 iflag=fullblock
7.128s  = computing hash
10.7408 = uploading file (copy file)

I then wanted to see if I put the hashing function onto a different spawn (thread) than the spawn (thread) that reads the file contents how might it improve:
24.7814s = running dd if=/dev/urandom of=/tmp/dummy_foo bs=1G count=10 iflag=fullblock
3.5946s  = computing hash
10.677s  = uploading file (copy file)

I then checked to see if I put the uploading/copying part onto different spawns/threads if it would help:
24.8469s = running dd if=/dev/urandom of=/tmp/dummy_foo bs=1G count=10 iflag=fullblock
3.2696s  = computing hash
7.2525s  = uploading file (copy file)

So the obvious low hanging fruit here is to simply increase the default DEFAULT_READ_BUFF_SIZE to a larger value and possibly make it configurable in global config.
I'm torn on if we should support multi-spawn/thread in this section of code, since we intentionally try to not create any spawns per grpc connection in order to help keep users in a single thread. This keeps extreme parallelism in a fairly cooperative manner, otherwise one user could create millions of small files and use lots of threads computing digests and such, starving everyone else. I'll think about this one a bit more.

I can confirm that using a 1 MiB buffer helped significantly and brought down build times from 17m45.5s down to 10m50.7s (includes negligible few seconds for RE client to fetch artifacts).
Local builds take 6m52.4s.
Based on these findings I would recommend making DEFAULT_READ_BUFF_SIZE configurable.
Also is there a way to log how long execution, hashing & uploading took?

Yes. @aaronmondal or @blakehatch, do one of you want to make this DEFAULT_READ_BUFF_SIZE a global config (also need to do some code searching to see if there's other places that could use the same config).
As for optimizing the threading, I think we should optimize it. Right now we are already paying a very high cost because tokio actually uses the synchronous filesystem API in a different threadpool. Since this is already a cost, we should instead stream the data from the synchronous filesystem API. One of the big advantages to doing it this way is that we could easily wire this up to mmap later, which would give even higher throughput.
We do not currently separate out the hashing time vs other time in the worker/running_actions_manager... currently it's lumped into upload_results. We could separate this out I think. @aaronmondal this might an easy one for you :-)
As an FYI, @lukts30, we are currently working on some tooling around using ReClient/Chromium as a benchmark metric that will help us understand where we need to improve.

I spent a little more time on this. I have a local change I'll push up soon that fixes the hash time completely. It should bring the total hash time to be in parity with b3sum in single threaded mode.
I'm currently looking at optimizing upload time. In doing so, there's a very high chance I'll also implement it as a file move. This should make local execution overhead nearly zero.

How about this: https://github.com/allada/nativelink-fork/commit/cf25b316cf3e06f217d314200e26ec288eaa19d8
It still needs some cleanup and I decided to cleanup some code, so it'll be multiple PRs before it's in.
It appears rayon + mmap and just mmap is about the same speed on my computer, so I may disable multi-threading.
24.7308s = running dd if=/dev/urandom of=/tmp/dummy_foo bs=1G count=10 iflag=fullblock
0.56695s = computing hash
0.00005s = uploading file (copy file)

I did not use the rayon hashing, but if it helps your use case a lot, could you open a new ticket and we can possibly make it enabled via a config.


(PUP-7469) Add tests for runtime string conversion

This commit adds some well needed tests for the StringConverter method
string_PRuntimeType. It also removes some dead code from that method.

Transient appveyor failure, closing and reopening:
Finished in 9 minutes 18 seconds
21358 examples, 0 failures, 62 pending
Spec run exited with 1
Command exited with code 1
appveyor PushArtifact .\spec_order.txt

CLA signed by all contributors.

@joshcooper has told me this is OK to merge... so I'm merging it ;0


Nav project list items are transitioning jerkily

Also, not quite sure why the text position transforms as well.

Fixed


chore(jenkins): branch->branches

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@3a5f3c5). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff            @@
##             main      #154   +/-   ##
========================================
  Coverage        ?   100.00%
========================================
  Files           ?         1
  Lines           ?         8
  Branches        ?         1
========================================
  Hits            ?         8
  Misses          ?         0
  Partials        ?         0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3a5f3c5...226b84c. Read the comment docs.


Feature request : ghostnet

Please can you train ghostnet.
(i don't have the imagenet dataset)

@pfeatherstone It's not something I plan to tackle, based on my look at ghostnet a while back, it wasn't compelling for me vs other mobilenets and smaller efficientnet/mixnets. Using the training code here it would be straightforward for someone to try training a ghostnet.

added in #561


SUBMARINE-1163. Remove mlflow

What is this PR for?
Remove mlflow from submarine
What type of PR is it?
Feature
Todos

[x] - Remove mllfow from sdk
[x] - Update document

What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-1163
How should this be tested?
Screenshots (if appropriate)
Questions:

Do the license files need updating? No
Are there breaking changes for older versions? Yes
Does this need new documentation? Yes

There are some codes about mlflow in K8sSubmitter, should we need to remove these?

There are some codes about mlflow in K8sSubmitter, should we need to remove these?

We decide to only remove mlflow in python sdk, so the part of mlflow in K8sSubmitter will remain.


Question about React Native + iOS

Hello! Thank you for this wonderful library. Would this work with React Native + iOS, by any chance?

unfortunatly it doesnt work in IOS browser at the moment. But apple is working on supporting WebRTC - here
and i dunno react native

I know there are couple of native camera access modules for iOS for React Native. Not sure whether they can expose the video stream to common js though.

btw some people are trying to use phonegap to run it on IOS. apparently it would be possible. i hope this helps, i dunno phonegap either :)
https://twitter.com/astraccia/status/846494423735263232

Thanks for the really, really helpful replies! You guys really went above and beyond :D ... unlike most StackOverflow threads.
Appreciate it!

I've found the world's crappiest hack for making this work.

Code a react-native app--> then use the following:
1_ https://github.com/axemclion/react-native-cordova-plugin --> extends the ability to use Cordova plugins in React-Native.
2_ Pull in the following plugin: https://www.npmjs.com/package/cordova-plugin-iosrtc
This makes the following possible in React-Native on IOS --->
3_ https://github.com/jeromeetienne/AR.js
So in the most jerry-rigged, daisy-chained way, you could actually get this running in iOS...theoretically 😆

Who is game to try ?

I am actually going to attempt it! I'll see what I can report back :)

:( I want to get this working for students in iOS but this hack seems too hard for them :( I cannot believe this iOS problem, so silly

@armenr Status on that report? I'm excited to see what you found!

@armenr How are you making use of this library with react native


Add WebGPU backend build to CI

Add an optional CI task to build the WebGPU backend on the wasm32-unknown-unknown target.
Note: This is just a test to see if we can at least see the status of the WebGPU build in the Ci pipeline. It may not be worth merging until the WebGPU backend is more mature, since it may just add noise for now.

Looks trivially fixable so far?

Looks trivially fixable so far?

I'm happy to clean up this build if we'd like to add a webgpu build to the CI requirements. I'm not sure how mature that backend is, but I suppose ensuring it keeps building can't hurt...

I'm happy to clean up this build if we'd like to add a webgpu build to the CI requirements. I'm not sure how mature that backend is, but I suppose ensuring it keeps building can't hurt...

It's not mature at all. In fact, it's more of a burden right now... We need to make a decision - either support it on CI, or just carry the code as dead weight (for now!) without any builds/checks. I think the latter is easier, and it's ok until there starts to be any logic in it.

It's not mature at all. In fact, it's more of a burden right now... We need to make a decision - either support it on CI, or just carry the code as dead weight (for now!) without any builds/checks. I think the latter is easier, and it's ok until there starts to be any logic in it.

I'm going to close this for now--when the WebGPU backend becomes more mature, this PR should be trivially reproducible.
For now, I'll remove the extra noise this PR causes.

I'm going to close this for now--when the WebGPU backend becomes more mature, this PR should be trivially reproducible.
For now, I'll remove the extra noise this PR causes.


Link broken in Choosing a Compiler

"here" is supposed to link to the compiler section, but it is currently broken.

Good catch, will be fixed with #150.


Error: It appears that the downloaded file is corrupt

Steps to Reproduce

... cloned repository from within WSL Ubuntu in Windows 10,
... successfully cloned
... run flutter doctor and got  an error message

Logs

flutter doctor -v
Downloading Dart SDK from Flutter engine fee001c93f25a1e7258e762781a7361f122d29f5...
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100  120M  100  120M    0     0  3208k      0  0:00:38  0:00:38 --:--:-- 3158k
/mnt/c/Users/foncy/flutter/bin/internal/update_dart_sdk.sh: line 82: unzip: command not found
It appears that the downloaded file is corrupt; please try again.
If this problem persists, please report the problem at:
https://github.com/flutter/flutter/issues/new?template=ACTIVATION.md

flutter doctor -v
Downloading Dart SDK from Flutter engine fee001c93f25a1e7258e762781a7361f122d29f5...
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100  120M  100  120M    0     0  3208k      0  0:00:38  0:00:38 --:--:-- 3158k
/mnt/c/Users/foncy/flutter/bin/internal/update_dart_sdk.sh: line 82: unzip: command not found
It appears that the downloaded file is corrupt; please try again.
If this problem persists, please report the problem at:
https://github.com/flutter/flutter/issues/new?template=ACTIVATION.md

In your logs it says "unzip: command not found". Looks like the unzip util is not available on your machine, which is a requirement for Flutter: https://flutter.dev/docs/get-started/install/linux

I see, yes. I tried running the command in powershell and it worked. thanks.

Inside WSL:
sudo apt install unzip
flutter doctor
`


Add GoReportCard badge

@tpantelis @mangelajo please review

@gliptak thanks for your contributions lately. Nice to see the project getting some traction.
Tom is away for some days but will be back soon. Are you planning to use submariner? , if so, how? :)

@mangelajo I arrived here from Rancher. Looking into Kubernetes ecosystem components, but do not have all targets identified at this time.

Travis is acting up for different reasons, but I will merge since this is only doc-related.


Bump node

Codecov Report
Patch and project coverage have no change.

Comparison is base (8fc0fe7) 78.77% compared to head (9df308a) 78.77%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #438   +/-   ##
=======================================
  Coverage   78.77%   78.77%
=======================================
  Files          39       39
  Lines         928      928
  Branches      157      157
=======================================
  Hits          731      731
  Misses         85       85
  Partials      112      112

Flag
Coverage Δ

unittests
78.77% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Performing a Scan on Global Secondary Index

I am trying to scan on a GSI and I am able to do this in both on the AWS CLI and AWS console, but have been unable to figure out how to do this in Dynamoose.
Is there a way to do this? I saw this issue #136 regarding using the query with .using() feature, but it doesn't appear that .using() is supported in the scan operation.
My use case is similar to that described in the following link: Guidelines For GSI using Sparse Indexes

First sorry for the slow response.
Scan looks at the whole table, regardless of the index.  To search a GSI, you should use .query.

@brandongoode, thanks for your response. I was able to implement with a query.
 This returns only the entries from the table where the sparse index is populated and thus doesn't appear to scan the whole table.
Relevant AWS information regarding sparse index:
AWS Guidelines For GSI using Sparse Indexes
 AWS Guidelines For LSI using Sparse Indexes  <--- this link actually describes my use case most closely, in my first post I specified the wrong link. :)
BTW this project is pretty awesome - thank you!

@brandongoode, @fishcharlie. @expatiating is right. It is possible to scan secondary indexes in DynamoDB. Here's another link. It is explicitly stated in the last paragraph, "You can also perform Scan operations on a secondary index, in the same way that you would do so on a table." Has this been implemented in Dynamoose yet? It's a quite valuable feature.

@dolsem Maybe I don’t understand well enough. But what is the benefit of scanning vs querying on an index? I was always under the impression that query’s had better performance and the only reason you would use scan over query is if you didn’t have an index for the given attribute.
From my understanding there is no feature currently in Dynamoose that allows this feature. A PR would be awesome tho.

@fishcharlie if you have a sparse index, the scan operation might not be that costly, depending on your data model. Moreover, I don't think there's a way to query ALL elements with query. If your data is not separated into chunks by using hash key + range key (so you use only hash key), I believe your query will always return a single element, because you need to explicitly set the value of the hash key you're querying.

@dolsem But how would that be different than querying the table itself? I thought that all of the data from your standard table are replicated across to your indexes. So how would scanning an index produce different data than just scanning the table itself?

@fishcharlie & @dolsem glad there is renewed interest in this a year later! :)
@fishcharlie there are very efficient use cases to perform a sparse index scan.
For instance, you have an ordering system and want to display on the dashboard all unfulfilled orders based on a parameter openTimeIndex, which you've set up as a secondary index. When the orders come in you set openTimeIndex: <unix time stamp order came in> (so you can sort oldest to newest as opposed to a boolean) and after you fulfill the order you set the parameter to openTimeIndex: null. Then in any subsequent scan operations on the index will not return any of the orders that are set to null, i.e., the index is sparse!
And the nice thing about using a sparse secondary index to do my scan is that I don't need to know anything about the partition key nor the sort key; all I need to know is the index.
As mentioned last year, I managed to make a query work in it's place, but it required me to add a GSI that used a fairly static parameter (location id) as the partition key and then my openTimeIndex sparsely populated parameter as the sort key.
Happy coding! 👨💻 👩💻

If someone wants to try to implement this I'd be happy to review a PR.

um i am using query in place of scan using secondary index here is my schema `//Service Schema.
var dynamoose = require('dynamoose');
var ServiceSchema = new dynamoose.Schema({
id: {
type: String,
index:{
global: true,
name: 'ServiceIdIndex'
}
},
name : {
hashKey: true,
type: String
},
description: {
type: String,
default: "Once a company has access to this, he has access to all companies",
required: true
},
image:{
type: String
},
thumbnail : {
type: String
},
cost: {
type: Number
},
createdAt:{
type: Date,
default: Date.now
},
redirectUris: {
type:[String]
},
enabled: {
type: Boolean,
required: true,
default: false,
index: {
global: true,
name: "enabledGlobalIndex"
}
},
grants: {
type: [String]
}
});
ServiceSchema.method('getBaseFeature', async function () {
var {Feature} = require("./index");
var feature = await Feature.query("serviceId").eq(this.id).filter({type: "base"}).exec();
if (feature.length != 0)
{
return feature[0];
}
return false;
});
module.exports = ServiceSchema;and my query is hereawait Service.query("enabled").eq(true).exec();`


Cast from negative i16 to u16 crashes at run time

    let value = -1i16;
    let intermediate = value as! u16;

This code will compile, but crash when running the executable. Found when trying to run.
It only crashes if the value is negative.

That's intentional, the cast you've used is infallible, so if the cast is impossible to do, it will crash.

Aha, it was a crash i found in the jaktnesmonster project of @jntrnr.
I regretfully don't understand enough jakt to figure out the better way of doing this. I will close the issue.


Incorrect or old version published?

When installing package dynamodb-data-mapper with npm, the itemIdentifier -function seem to be missing the latest fix:
function itemIdentifier(marshalled, keyProperties) {
    var e_15, _a;
    var keyAttributes = [];
    try {
        for (var keyProperties_1 = tslib_1.__values(keyProperties), keyProperties_1_1 = keyProperties_1.next(); !keyProperties_1_1.done; keyProperties_1_1 = keyProperties_1.next()) {
            var key = keyProperties_1_1.value;
            var value = marshalled[key];
            key + "=" + (value.B || value.N || value.S);
        }
    }
// ...

However, when built from source code, the itemIdentifier the fix is there:
function itemIdentifier(marshalled, keyProperties) {
    var e_15, _a;
    var keyAttributes = [];
    try {
        for (var keyProperties_1 = tslib_1.__values(keyProperties), keyProperties_1_1 = keyProperties_1.next(); !keyProperties_1_1.done; keyProperties_1_1 = keyProperties_1.next()) {
            var key = keyProperties_1_1.value;
            var value = marshalled[key];
            keyAttributes.push(key + "=" + (value.B || value.N || value.S));
        }
    }
// ...

So, the fix is not released yet or the incorrect version was published to npm?

The published version is old.
I'm also waiting for the fix to be published.
@jeskew, can you publish a new version?

Sorry @ofekray, but I am no longer at Amazon and cannot publish a new version. There's some discussion on the current status of the project at https://github.com/aws/aws-sdk-js/issues/2637


[PE-D][Tester D] Name field does not accept symbols

Name does not accept symbols such as "," or "/" even though they are a valid part of names.

Labels: severity.Medium type.FunctionalityBug
original: jetlfj/ped#1

Seems like a duplicate of #222.
I propose that we close this issue


[release-4.14] OCPBUGS-36478: Set CRI-O version to match Kubernetes' version

/label backport-risk-assessed

  1293  Jul 03 04:18:19 el92-prel-upgrade-block-2minor-host1 ovs-ctl[1008]: ovsdb-server: (null): Invalid --user option openvswitch:hugetlbfs (user openvswitch is not in group hugetlbfs), aborting.
      1 Jul 03 04:18:19 el92-prel-upgrade-block-2minor-host1 ovs-ctl[953]: Starting ovsdb-server ... failed!
      2 Jul 03 04:18:19 el92-prel-upgrade-block-2minor-host1 systemd[1]: ovsdb-server.service: Control process exited, code=exited, status=1/FAILURE
      3 Jul 03 04:18:19 el92-prel-upgrade-block-2minor-host1 systemd[1]: ovsdb-server.service: Failed with result 'exit-code'.
      4 Jul 03 04:18:19 el92-prel-upgrade-block-2minor-host1 systemd[1]: Failed to start Open vSwitch Database Unit.
      5 Jul 03 04:18:19 el92-prel-upgrade-block-2minor-host1 systemd[1]: Dependency failed for Open vSwitch Forwarding Unit.
      6 Jul 03 04:18:19 el92-prel-upgrade-block-2minor-host1 systemd[1]: Dependency failed for Open vSwitch.
      7 Jul 03 04:18:19 el92-prel-upgrade-block-2minor-host1 systemd[1]: Dependency failed for Configures Open vSwitch for OVN (MicroShift).
      8 Jul 03 04:18:19 el92-prel-upgrade-block-2minor-host1 systemd[1]: icroshift-ovs-init.service: Job microshift-ovs-init.service/start failed with result 'dependency'.

/lgtm
/label cherry-pick-approved
/remove-label jira/invalid-bug

/label -jira/invalid-bug

/jira refresh

/label jira/valid-bug

/remove-label jira/invalid-bug

/label cherry-pick-approved


p-Key in form-field shows print-view

When entering data containing a "p" (eg. apple) into a form-field (eg. http://swiss.github.io/styleguide/pages/forms.html ) the print-view suddenly appears and it is not possible to use the form field as expected.
When disabling either (?!) vendors.js or main.js the error does not occur.
PS: what are vendor.js and main.js for?

This is a major issue. We'll remove this feature completely, as it is non-essential.

+1 just found the same bug. Fires everywhere not only textfields.
Do you know when you will release 2.5.3?

Commented in e35a1fe2273c02ad39bb68405b2804b78de96aa7. Needs to be removed completely.

b44094378fcf2dc57735727fefa3fab136a4e59b
aba7b43a024e6024e85d418495ed9f092394fb78

ok, looks good now.

Was tested in Chrome latest version, Firefox latest version und IE9. Now OK.


feat(Popover): use theme.colors.secondary for outline color

Overview

Popover: use theme.colors.secondary for outline color
gamut-icons: add vertical menu icon

for file navigator sandbox work - i needed a white outline for the popover
and a vertical menu

PR Checklist

[ ] Related to designs:
[x] Related to JIRA ticket: EGG-1134
[x] I have run this code to verify it works
[ ] This PR includes unit tests for the code change

📬Published Alpha Packages:@codecademy/gamut-icons@4.12.3-alpha.deb0d5.0@codecademy/gamut-kit@0.3.103-alpha.deb0d5.0@codecademy/gamut-labs@17.1.1-alpha.deb0d5.0@codecademy/gamut@32.18.8-alpha.deb0d5.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://60a2759f6a7f84120155e7e9--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-icons@4.12.3-alpha.d7403b.0@codecademy/gamut-kit@0.3.103-alpha.d7403b.0@codecademy/gamut-labs@17.1.1-alpha.d7403b.0@codecademy/gamut@32.18.8-alpha.d7403b.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://60a295a355da890d7d0f9154--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut-icons@4.12.3-alpha.3f3801.0@codecademy/gamut-kit@0.3.104-alpha.3f3801.0@codecademy/gamut-labs@17.1.2-alpha.3f3801.0@codecademy/gamut@32.18.9-alpha.3f3801.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://60a2a70c8110b10cd41b554b--gamut-preview.netlify.appDeploy Logs

🚀 Styleguide deploy preview ready!https://60a51ea99210060357a770b8--gamut-preview.netlify.appDeploy Logs


Amplify gets an appClientId from nowhere and now can't update the stack

Note: If your issue/bug is regarding the AWS Amplify Console service, please log it in the
Amplify Console GitHub Issue Tracker

Describe the bug
I'm developing an application using Amplify
Everything was doing fine, I have done some changes on my dev environment to include Social login, and it was working fine locally
Then when I tried to deploy using Amplify Console CD it was failing, after digging into it I found the solution here using a custom script for the amplify simplepush
Just to put this on context
After having everything working again, I was happy to push my changes to staging
So, I have changed my branch, checkout the staging environment and tried to push
And then I got stuck in an error raised saying that it can't find the AppClientID

Resource Name: XXXXXXXXXXX (AWS::Cognito::UserPoolClient)
Event Type: update
Reason: User pool client does not exist. (Service: AWSCognitoIdentityProviderService; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: YYYYYYYYYYYYYYYYYY
URL: https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/xxxxxxxxxxx

The URL goes to a The page you are looking for does not exist. page
The client Id, true, it doesn't exist, and I have no idea why it is trying to update it
So I looked at both
amplify/#current-cloud-backend/amplify-meta.json
and
amplify/backend/amplify-meta.json
Both contain a line of code like this (on the auth->output section):
"AppClientID": "XXXXXXXXXX"

The #current-cloud-backend is supposed to come from the cloud, so I'm not supposed to touch it, but I have no idea how it did get that code, the dev appClient is not this code either.
So, I tried changing the code to (on the amplify/backend/amplify-meta.json file):
"AppClientID": "MY-VALID-ID"
and then pushing again
But the error continues, and then the amplify/backend/amplify-meta.json was updated with the wrong id again
I thought about running amplify push -f, but I'm afraid it is gonna mess things up
Any idea what might be causing it and how to fix it?
Amplify CLI Version
4.19.0
To Reproduce

Resource Name: XXXXXXXXXXX (AWS::Cognito::UserPoolClient)
Event Type: update
Reason: User pool client does not exist. (Service: AWSCognitoIdentityProviderService; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: YYYYYYYYYYYYYYYYYY
URL: https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/xxxxxxxxxxx

Expected behavior
The correct Cognito UserPoolClient should be fetched and update accordingly
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: Mac
13.13.0

Additional context
Add any other context about the problem here.

"AppClientID" is the cognito user pool app client id, seems like it's from another environment, and not recognized in the staging backend env.
Could you describe in more details on what happened "So, I have changed my branch, checkout the staging environment and tried to push"?
What you can try is in the same local git branch, run amplify env checkout staging, and then run amplify status to make sure the auth is in the create status, and then run amplify push

I have looked at all my environments, none of them has this code on an appClientId of Cognito
I had run amplify pull --restore, and apparently this weird appClientId is already on the cloud somehow, because it shows up on my amplify-meta.json even after restore.
I had an authentication before, only not with social providers, so it won't be on create mode, only update.
If I go on my dev branch and run amplify env checkout stg I'm asked to input my oauth key/secrets for Facebook/Google login, after doing that running amplify status I do see

| Auth     | myApp             | Update    | awscloudformation |

Every time I try to push I get the error above, telling me the resourceId of the userPoolClient was not found, even after manually changing the value on the amplify-meta.json file

There is something really bad with my setup
After restoring my staging env with amplify env pull --restore
I decided to apply only my changes on the API, So I merged only a branch that I was working on changes on the API
Then I checked amplify status, which showed correctly that only the API has changed
Then tried to push it amplify api push
And I got a weird error

UPDATE_FAILED      authmyapp                 AWS::CloudFormation::Stack Tue May 12 2020 21:15:53 GMT+0200 (Central European Summer Time) Parameters: [userPoolGroupList, resourceNameTruncated, dependsOn] must have values

I tried to grep userPoolGroupList trying to make sense of this error message, but no success, no idea why it did even tried to do something on auth again if I was just pushing changes on the API.

I did:
aws cognito-idp describe-user-pool-client --user-pool-id <region-poolId> --client-id <INVALID ID> On all my user pools, none of them have this userpool client id
But everytime I do a pull restore my amplify-meta.json gets updated with: "AppClientID": "INVALID ID"
Something like
    "auth": {
        "myproject": {
            "service": "Cognito",
            "providerPlugin": "awscloudformation",
            "output": {
                "GoogleWebClient": "111111111.apps.googleusercontent.com",
                "AppClientSecret": "aaaaaaaaaaa",
                "UserPoolId": "region-pooId",
                "AppClientIDWeb": "VALID ID",
                "AppClientID": "INVALID ID",
                "FacebookWebClient": "2222222222",
                "IdentityPoolId": "region:Id",
                "IdentityPoolName": "myproject__env",
                "UserPoolName": "mypoolname"
            },
            "lastPushTimeStamp": "2020-05-13T20:48:29.797Z",
            "providerMetadata": {
                "s3TemplateURL": "https://s3.amazonaws.com/myproject-deployment/amplify-cfn-templates/auth/lexis-cloudformation-template.yml",
                "logicalId": "authmyproject"
            },
            "lastPushDirHash": "XXXXXXXXXXXXXX="
        }
    },

I tried going on full rad mode now and checked the S3TemplateURL that it shows on the amplify-meta file
I thought that maybe that is where it was getting the invalid id, maybe that was Amplify "point of truth"... so, I have changed the id there, crossed my fingers and did a pull....
Nope, back to the invalid one

Could someone help me with that? My deployment is blocked due to this issue
@UnleashedMind

What I believe has happened is the following:
I have added auth to my amplify project months ago
Eventually, I noticed one of the created clients were not being used, so I have deleted it from the console.
I had not updated the auth during months
Now that I have introduced the social authentication Amplify tried to update and because of hte client Id not existing anymore it can't.
Should have a way to make Amplify aware of this removal and not try to update it

@danielfranca - we're looking into this issue now. We need some more details from you. Are you able to go over this on a chat? Can you DM me on twitter to align on a good time? https://twitter.com/renebrandel

@brene Hey, thanks
Two things, I don't have Twitter, but for me, it works any time after 5:30 pm CEST
Can we use another tool? If there's no other way I can make a throwaway account just for that.
And, I have just created a ticket on AWS support :'( What is the best way to coordinate it, so you don't duplicate efforts?

@danielfranca you can alternatively join our discord community here: https://discord.gg/jWVbPfC to chat with me 1:1 (AFAIK, you don't need to sign-up for it and can just join as guest)

@danielfranca
Could you zip up your amplify folder and send it to us?
<EMAIL_ADDRESS>

@UnleashedMind Just did :)

Should I try the hack we discussed, changing the deployment bucket, or should I wait a bit more?

bad luck, your anti-virus engine didn't like the attachment:
The message (MID 758444620) could not be fully scanned by the anti-virus engine.

And, This message timed out during scanning.

Actions taken:
    Message quarantined

Maybe I can send it over Discord?

I'm sending you an email

@UnleashedMind
I've tried the hack

Went to the S3 Deployment Bucket
Downloaded the #current-cloud-backend.zip
Replaced the wrong clientId with the correct one
Uploaded it
Replaced the appClientId with the correct one on my local amplify/backend
Double checked that there is no more reference to this Id elsewhere
Tried to push it again

As soon as the push starts running I can see that the clientId is set back to the wrong one on my local backend (not in the cloud one)
And then I get the same error again
I've uploaded my amplify folder to Firefox Send, as soon as I get your email I'll email back with the link/password.

I'd workaround it creating a new environment, copying everything on my DynamoDb Tables to the new ones, copying my S3 bucket storage, and re-creating the users.
This was not ideal, of course, but as it is still not in production it could be done without much disruption. However, I would like to understand this issue to avoid this happening on a production environment, which would lead to way worse consequences to the users.

Because the Amplify CLI uses CloudFormation to create the AWS resource, and If you delete a resource from an AWS CloudFormation stack, then you must remove the resource from your AWS CloudFormation template. Otherwise, your stack fails to update, and you get an error message.
For this particular case, you removed the User Pool App Client,  which was depended on by multiple other resources, so removing it from the template became very complicated. And the CLI has the pull operation before push, which will revert your local changes and make things even harder to recover.
So, try not to manually update or delete the resources created by the Amplify CLI, specially in the production environment, it can be very hard to recover.

@danielfranca
Are you still blocked?  Let us know what else we can help.

I understand that, but mistakes happen and would be nice if we can import external changes into the CloudFormation stack, or even do some kind of dry run in order to sync the environments.
I'm not blocked anymore, because I have created a new environment.

@danielfranca thanks for the update, let us know if you need any more assistance.


feat(uint): widen accepted range of types on try_from_bytes_le

(Feel free to close this PR if the change is not wanted/needed—I know there's no issue associated to this)
Was just tinkering with the crate for a project, and saw that the type definition for bytes could be widened a bit so it can also accepted full vectors rather than just slices.

hello! thanks for opening the PR :)
can you give me an example you have in mind?
I'm not opposed to making this more flexible but if you just want to use a vector v instead here, you can just &v
e.g.
fn foo(a: &[u8]) {
    dbg!(a);
}

fn main() {
    let b = vec![1,2,3];
    foo(&b);
}

np sure! it's mainly to also be able to pass Strings as well, e.g
fn foo(a: impl AsRef<[u8]> + std::fmt::Debug) {
    dbg!(a);
}

fn main() {
    let b = String::from("0xcafe");
    foo(&b);
}```

it's pretty minor for sure—but just add a little bit of extra ergonomics :).

this is a personal preference but in this case I'd rather the caller just use String::as_bytes
I assume there is a slight compile-time penalty to using impl Foo over a more concrete type and don't feel that this change in ergonomics justifies the increased cost on everyone else using this method

Actually, in this case the impl Trait syntax is zero cost, as it's just syntactic sugar for generics. That being said, happy to close if it's your personal preference to make the user explicitly cast the string, which i find reasonable imo!

right and I would be very surprised if the compile-time cost for generics was the same as for the concrete type

haha makes sense—read too fast and didn't notice you were talking about compile time. np, thank you!


Converting the project into PCL

Making the project from NET45 into a PCL so it can be used with all the .NET profiles.

Hi Igor. Does this pull request specifically target PCL only? If so, would you mind instead forking the repo and applying the change there, so that this repo remains supporting .NET 4.5.


Enable Hot Push Code

Since you enable downloading new content from an external source to the app, I wonder if you can update the main.jsbundle with this approach.
You define the location of the main.jsbundle in the AppDelegate.m. So I imagine if you trigger a download you may be able to update it? But I guess the current location of main.jsbundle is in a secure spot, so you have to point the AppDelegate.m to your existing Documents folder. In some cases that might be security problem.
As an inspiration two plugins to enable hot pushing code in Cordova:
https://github.com/markmarijnissen/cordova-app-loader
https://github.com/mathieudutour/cordova-plugin-hotpushes based on
https://github.com/phonegap/phonegap-plugin-contentsync
Both plugins check for new versions via a json file on the host server and only trigger updates when there is a newer version. Also the contentsync plugin provides different states do identify the current download status.
TL;DR: Enable hot push code to the app:

version check of main.jsbundle
download new main.jsbundle
replace old main.jsbundle / change location of main.jsbundle in AppDelegate.m to reflect update
reload app when update is done to roll out updated main.jsbundle

This should be done in another module.


Add footnote about colors being in-gamut (i.e. components in [0,1])

Based on the interesting discussion in https://github.com/AcademySoftwareFoundation/OpenPBR/issues/149, it seems worth adding this side note:

@anderslanglands and @KelSolaar Would appreciate your thoughts on whether this change makes sense.

It might also be:
"... all RGB colours are in-gamut of the working space, i.e., have components in the $[0, +\inf]$ range"

"In-gamut" of what: We need to specify the container
Why limiting to 1]? Colour emission should not be restricted so we might need a case for albedo et al. and another one for emission.

@KelSolaar

It might also be: "... all RGB colours are in-gamut of the working space, i.e., have components in the $[0, \infty]$ range"

OK, so in your view "in-gamut" (of the working color space of the shader) simply means that the color components are non-negative?

"In-gamut" of what: We need to specify the container

Can you clarify what you mean by container? We say in the spec that "A color3 value is associated with a color space ACEScg by default)" and that "The assumed color space of all the color parameters .. by default .. is assumed to be ACEScg".
So "in-gamut" is implied to mean in-gamut of that color space (or ACEScg if unspecified). Is that still under-specified in your view?

Why limiting to 1? Colour emission should not be restricted so we might need a case for albedo et al. and another one for emission.

Yes right, so emission color should not be restricted to [0,1], but it seemed like according to our earlier discussion  we had agreed that for all the other colors, which represent albedos or fractional tints, they should be restricted to [0,1] (regardless of the color space). Do you still agree with that?

OK, so in your view "in-gamut" (of the working color space of the shader) simply means that the color components are non-negative?

Yes, e.g., HDR values are considered in-gamut of their colourspace for example.

Is that still under-specified in your view?

I would certainly be very specific, there is an infinity of gamuts and it does not cost much to add.

Do you still agree with that?

Yes!

Sorry can you clarify what you mean by “there is an infinity of gamuts”?
We say the color space must be specified (or is ACEScg by default). How can we be more specific that that?

We are splitting hairs here but "gamut" by definition (Oxford Languages) is "the complete range or scope of something", what I'm saying is that the notes does not explicitly specifies it.

Note that we assume that all RGB colors are in the specified color space's gamut, i.e. have components in the $[0, +inf]$ range.

This is better because the "something" is specified explicitly in the sentence right next to "gamut" and you do not have to reach the end of the sentence to understand what is being said.
In theory, the i.e., portion of the sentence could be omitted and the sentence would be explicit by itself, I could only write:

Note that we assume that all RGB colors are in-gamut.

@jstone-lucasfilm Note, don't merge this one yet. There needs to be some changes to the way we describe the color ranges in the spec.

I modified this as follows:

@jstone-lucasfilm can be merged?


[agent/plugin] Fix config variable usage bug

Please answer these questions before submitting pull request

Why submit this pull request?

[x] Bug fix

[ ] New feature provided

[ ] Improve performance

Related issues

Bug fix

Bug description.

How to fix?

New feature or improvement

Describe the details and related test reports.

FYI @aderm

ACK


FEDORA PACKAGING: Possible false positive OpenSSL warning from rpmlint

bpfman.x86_64: W: crypto-policy-non-compliance-openssl /usr/sbin/bpfman SSL_CTX_set_cipher_list
As far as I can tell, rpmlint is only checking for the presence of the symbol, not whether it's actually called in the code, so this is, as far as I can tell, a false positive. I've seen this in other packages too, and I was never able to find any code path that actually hit that function from OpenSSL. But it would be good to check.


Strong performance loss compared to 2014 version

I just updated from a much earlier version (I believe the version I was using was this one: https://github.com/altmany/export_fig/commit/26617824f56b55264a4712326cf2511783b99d16) to the latest version. However, the exporting of two Figures into a single PDF files takes 30s all together. I ran the profiler and there are three parts where time is lost at:

10s @ [status, message] = ghostscript(options) called in the eps2pdf
10s @ eps_remove_background(tmp_nam, 1 + using_hg2(fig)) called in eps_remove_background
10s @  pj.Return = generateGraphicsOutput('HG', container, s) in  C:\Program Files\MATLAB\R2015b\toolbox\matlab\graphics\private\alternatePrintPath.m called in print2eps

Any suggestions what might be the issue? The old version maybe took 2 or 3 seconds to do the exact same thing. Let me know if you need additional information.

This is a direct side-effect of Matlab's new graphics system (HG2, starting in R2014b). This results in a huge EPS file, which takes ages to prepare and to process.
Try to use the painters renderer and/or a non-vectorized image format (e.g., PNG or JPG).

Thank you so much, using painters did the job! I don't quite understand why it doesn't do it automatically as it says in the export_fig.m description "The default value (opengl for bitmaps, painters % for vector formats) generally gives good results"? Any ideas there?

It's not so simple: painters do not properly render patches and transparencies, so if these are detected in the figure then OpenGL is used by default. After all, in most cases accuracy is more important than performance. You can always override the default export renderer if you wish.


non-human omia disease 3 out of 6

Resolving conflicts leads to weird diffs. Closing and restarting.


some bug with reducer

Do you want to request a feature or report a bug?
a bug
What is the current behavior?
this plugin can not save data when refresh
If the current behavior is a bug, please provide the steps to reproduce.
createPersistedState({
      reducer(state) {
        return {
          phone: state.user.phone,
        };
      }
})

when refresh, the data will lossed

https://jsfiddle.net/wmf8zvf8/131/
What is the expected behavior?
when i use reducer and refresh, the data will not lossed
If this is a feature request, what is motivation or use case for changing the behavior?

This is not a bug with the plugin, but with your setup. Please provide more information about your setup, otherwise this issue will be closed.

This is not a bug with the plugin, but with your setup. Please provide more information about your setup, otherwise this issue will be closed.

i only set reducer , the code is above, state.useris my vuex module,state.user.phoneis key in my usermodule,  when i refresh brower, the data will be reset.    but if i don't set reducer, all data will saved, if refresh brower, all data will not be reset,  so  how to use reducer correctlly?


got SIGSEGV using in Spark

I tried to use your implementation in Spark and got that

A fatal error has been detected by the Java Runtime Environment:
SIGSEGV (0xb) at pc=0x000000010d75d414, pid=34100, tid=48387
JRE version: Java(TM) SE Runtime Environment (8.0_51-b16) (build 1.8.0_51-b16)
Java VM: Java HotSpot(TM) 64-Bit Server VM (25.51-b03 mixed mode bsd-amd64 compressed oops)
Problematic frame:
V  [libjvm.dylib+0x55d414]
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
An error report file with more information is saved as:
hs_err_pid34100.log
Compiled method (nm)   10145 2030     n 0       sun.misc.Unsafe::getObject (native)
total in heap  [0x000000010e7b98d0,0x000000010e7b9c28] = 856
relocation     [0x000000010e7b99f8,0x000000010e7b9a38] = 64
main code      [0x000000010e7b9a40,0x000000010e7b9c28] = 488
Compiled method (nm)   10145 2030     n 0       sun.misc.Unsafe::getObject (native)
total in heap  [0x000000010e7b98d0,0x000000010e7b9c28] = 856
relocation     [0x000000010e7b99f8,0x000000010e7b9a38] = 64
main code      [0x000000010e7b9a40,0x000000010e7b9c28] = 488
If you would like to submit a bug report, please visit:
http://bugreport.java.com/bugreport/crash.jsp

hs_err_pid34100.txt

@AzatFazulzyanov Thank you for the report. Could you share more details please. Which version do you use? How do you create and use Bloom filter in Spark. I would appreciate a small repro.
Please be aware that capturing or broadcasting the variable won't work because the BF class doesn't "contain" the memory but a pointer to it. You need to use serialization/deserialization instead.

Oh, I tried to simply broadcast it, so it would not work, that is the point.

The BF class exposes def writeTo(out: OutputStream) and def readFrom[T](in: InputStream) APIs for serialization and deserialization respectively. Try to use them please.