Publishing gfx_app

cargo publish tries to build the package with all features, but we don't have a platform that supports all D3D11, Vulkan, and Metal at the same time.

When doing cargo publish --no-verify, I'm getting an error:

no known crate named gfx_window_metal

@liquidnight2 apologies! the bug is now re-opened.

How would I write a test project with the code from the instancing example, without using
gfx_app::Application ?

Technically, by starting off the triangle example and adding features you see (e.g. from the instancing one). But I do agree this is not convenient... Duplicating the code all over the examples would not be a good solution either, so I'm open to ideas, if you got any!

This is effectively blocked by https://github.com/Aatch/spirv-utils/pull/2

Published now! 🎆 🎆 🎆


Aliyun ACK cluster driver is not visible on Rancher for create a new cluster

What kind of request is this (question/bug/enhancement/feature request):
bug
Steps to reproduce (least amount of steps as possible):

Activate the cluster driver for Aliyun ACK (inside Tools/Drivers)
Go the page for  add new cluster to Rancher (<rancher-url>g/clusters/add/selectadd where the list of drivers/options lists for creating a new cluster)

Result:
The Aliyun ACK cluster driver is not visible, so not possible to add a cluster with this driver. But the Aliyun ACK driver is mark activate/download on the Tools/Driver page.
Other details that may be helpful:

I tried to activate all the drivers (nodes and cluster): everything is visible, except the Aliyun ACK driver
I tried reinstalling the driver, deactivating it, etc .: unable to see the driver

URL of the driver/version of the driver use: https://github.com/rancher/kontainer-engine-driver-aliyun/releases/download/v0.2.4/kontainer-engine-driver-aliyun-linux
API output about the driver:
{
	[...]
	"active": true,
	"actualUrl": "https://github.com/rancher/kontainer-engine-driver-aliyun/releases/download/v0.2.4/kontainer-engine-driver-aliyun-linux",
	"annotations": {
		"lifecycle.cattle.io/create.mgmt-kontainer-driver-lifecycle": "true"
	},
	"baseType": "kontainerDriver",
	"builtIn": false,
	"checksum": "",
	"conditions": [ {
			"lastUpdateTime": "2020-12-04T11:02:11Z",
			"status": "True",
			"type": "Downloaded"
		},
		{
			"lastUpdateTime": "2020-12-04T11:02:11Z",
			"status": "True",
			"type": "Installed"
		},
		{
			"lastUpdateTime": "2020-12-15T09:22:47Z",
			"status": "True",
			"type": "Inactive"
		},
		{
			"lastUpdateTime": "2020-12-15T09:22:55Z",
			"status": "True",
			"type": "Active"
		}
	],
	[...]
	"executablePath": "/management-state/bin/kontainer-engine-driver-aliyun",
	[...]
	"labels": {
		"cattle.io/creator": "norman"
	},
	"links": {
		"remove": "…/v3/kontainerDrivers/kd-wwbk2",
		"self": "…/v3/kontainerDrivers/kd-wwbk2",
		"update": "…/v3/kontainerDrivers/kd-wwbk2"
	},
	"name": "aliyun",
	"state": "active",
	"transitioning": "no",
	"transitioningMessage": "",
	"type": "kontainerDriver",
	"uiUrl": "",
	"url": "https://github.com/rancher/kontainer-engine-driver-aliyun/releases/download/v0.2.4/kontainer-engine-driver-aliyun-linux",
	[...]
}

Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): v2.5.1
Installation option (single install/HA): HA on an RKE cluster

This is the default for Rancher v2.5.1 (https://github.com/rancher/rancher/blob/v2.5.1/pkg/data/management/kontainerdriver_data.go#L63), did you change anything on the driver while testing?

Yes, I have tried to install another version from the driver repository (https://github.com/rancher/kontainer-engine-driver-aliyun/releases).
I tried with this version (https://drivers.rancher.cn/kontainer-engine-driver-aliyun/0.2.6/kontainer-engine-driver-aliyun-linux), I the same issue, nothing visible on the "add cluster" page.

Ok, I found the problem the UI link (js file) was not the correct URL on my config it's was from an old version of Rancher (so with an upgrade it breaks driver interface, but, weirdly, we don't have more information about that on the Rancher UI / API). And thanks for link to the code link to that!

@AEnguerrand

This website seems unsafe now


az cosmosdb collection - commands deprecated

"az cosmosdb collection" commands are deprecated.
Warning when these commands are used in cli
"This command has been deprecated and will be removed in a future release. Use 'cosmosdb sql container, cosmosdb mongodb collection, cosmosdb cassandra table, cosmosdb gremlin graph or cosmosdb table' instead."

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7d791e10-4181-cf15-1398-499eb0918c9c
Version Independent ID: 2e5f864d-29bd-da4f-b964-a87a39f988c0
Content: az cosmosdb collection
Content Source: latest/docs-ref-autogen/cosmosdb/collection.yml
Service: cosmos-db
GitHub Login: @rloutlaw
Microsoft Alias: routlaw

cosmosdb

Hi @pk422 yes this is correct. These commands are deprecated. Please use the API specific commands. Do you have any specific question for us ?

I don't understand the ask in this issue. Please open it again if you have any questions for us. @pk422

This 'az cosmosdb collection' is marked as deprecated, however where is the non-deprecated sql equivalent of this command?
I don't see a "az cosmosdb sql collection" command available to carry out queries/deletes on SQL API documents?
@MehaKaushik

The commands available in CLI module are intended to perform database and container level operations.
The data plane operations can not be done using these commands. You can use the cosmo v3 sdk for it (https://github.com/Azure/azure-cosmos-dotnet-v3)

Thanks @MehaKaushik, appreciate it!


Opening links with esteem mobile doesn't work when the app is open or running in the background.

Describe the bug
In the latest update of the app, I can now open links from Steemit, busy and steempeak with the app. I tried this out and it didn't work when the app was open. But, when I closed the app, the function worked.
To Reproduce
Ensure you have launched the app, and it is running in the background

Click on a steemit, busy, or steempeak link
Select option to open with Esteem
See error (The app launches, but the page of the link doesn't open)

Expected behavior
I expected that even though the app was open, when I open a link with it, the link should open.
Screenshots or Video
https://www.youtube.com/watch?v=U9GiUDPizmg
Test Environment

Device: Redmi Note 5 pro
OS: Android 8.1
Software version: Esteem mobile 2.0.9

Additional context

Is there a way to bypass boot screen?
Please make pincode security optional.

it is same issue with #590


[FEATURE] UI(Thread)+DX: Create reusable dialogs

Description
Just like how notistack works, we can create reusable dialogs the provides an imperative api to easily create dialog messages, without repeating code.
Screenshots
No response
Additional information
No response

does this help or is it something else that you wanted?

@saitama2812 I think this works as @mathiasayivor liked it, would you like to work on this?

Open for KWOC

Assign me this I can work on it. I can create a reusable dialog box which can contain props for title and message and can be used anywhere in the applicaiton

You can close this issue since it has already been resolved


Magento & Pagespeed. Cache images from wrong url

I just installed the mod_pagespeed extension on my apache 2.4.7.
The installation went well. I added this to the .htaccess file
############################################
mod_pagespeed

   ModPagespeed on
   ModPagespeedDisallow "*index.php/admin/*"
   ModPagespeedEnableFilters collapse_whitespace
   ModPagespeedEnableFilters combine_css
   ModPagespeedEnableFilters combine_javascript
   ModPagespeedEnableFilters elide_attributes
   ModPagespeedEnableFilters extend_cache
   ModPagespeedEnableFilters flatten_css_imports
   ModPagespeedEnableFilters inline_import_to_link
   ModPagespeedEnableFilters inline_css
   ModPagespeedEnableFilters inline_javascript
   ModPagespeedEnableFilters inline_preview_images
   ModPagespeedEnableFilters rewrite_javascript
   ModPagespeedEnableFilters move_css_above_scripts
   ModPagespeedEnableFilters rewrite_images
   ModPagespeedEnableFilters convert_jpeg_to_webp
   ModPagespeedEnableFilters convert_to_webp_lossless
   ModPagespeedEnableFilters insert_dns_prefetch
   ModPagespeedEnableFilters prioritize_critical_css
   ModPagespeedEnableFilters remove_comments
   ModPagespeedEnableFilters remove_quotes
   ModPagespeedEnableFilters rewrite_css
   ModPagespeedEnableFilters rewrite_domains
   ModPagespeedEnableFilters rewrite_style_attributes
#This caches scripts and images in your local storage, so to flush you cache, flush your local storage as well.
ModPagespeedEnableFilters local_storage_cache

But when i look at the /var/log/apache2/error.log i see a loot off errors of pagespeed
[Fri Jun 19 22:21:41.053945 2015] [pagespeed:error] [pid 12378] [mod_pagespeed <IP_ADDRESS>-7251 @12378] http://www.myurl.nl/catalog/product/cache/1/small_image/258x245/9df78eab33525d08d6e5fb8d27136e95/d/i/dino_large.jpg (connecting to:media):0: Error status=670005 (No address associated with hostname) serf_connection_create2 [Fri Jun 19 22:21:41.054031 2015] [pagespeed:warn] [pid 12378] [mod_pagespeed <IP_ADDRESS>-7251 @12378] Fetch failed to start: http://www.myurl.nl/catalog/product/cache/1/small_image/258x245/9df78eab33525d08d6e5fb8d27136e95/d/i/dino_large.jpg (connecting to:media)
Pagespeed tries to retrieve the images from the wrong location. Now it search for: http://www.myurl.nl/catalog/product/cache But this should be http://www.myurl.nl/media/catalog/product/cache
I haven't changed anything at the pagespeed configuration and all images work wel on the website.
I have magento 1.8.1 CE
Who can help me?

This looks familiar; pagespeed is trying to fetch //foo/bar/baz when it should be fetching /foo/bar/baz.  I think we had this before and I tried to reproduce with no success.

These reports have all had "connecting to:[first url segment]".


Updating Readme file

Updated "what, how and why" section, and also added user testing section, which presents the results of our user UI testing of the app.
:ballot_box_with_check: Definition of Done checklist

[x] Meaningful title and valuable description in pull request
[x] Code change tested manually if not covered by automated tests
[x] Pull request linked to one or more issues/stories
[x] Acceptance criteria of the linked stories satisfied
[x] Code change comprehended by those who approved it
[x] State of target branch maintained or improved after the code change

Are those the last changes or do we have more to come?

Yea we need hardware and software architecture part as well


test, build: Fix gopath creation

Fixes: #9338

@mkumatag Can you fix shellcheck warnings? https://travis-ci.org/coreos/etcd/jobs/346155752#L746

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@3239641). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #9356   +/-   ##
=========================================
  Coverage          ?   72.48%
=========================================
  Files             ?      361
  Lines             ?    30716
  Branches          ?        0
=========================================
  Hits              ?    22263
  Misses            ?     6832
  Partials          ?     1621

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3239641...adf4e05. Read the comment docs.

@gyuho fixed the shellcheck warning, PTAL?

Just added that ENV https://jenkins-etcd.prod.coreos.systems/job/etcd-ci-ppc64/.


Build error with changing the vendor and product ID (CON-763)

I have changed the vendor and product ID's for our product and tried building the project, which throws error.
Attaching the error log
deviceID_errors.txt
I have generated the Certs don't where to place exactly to solve this errors.

@SYESHASWINI Please share the details about the mfg_tool commands used and upload your sdkconfig file.

@dhrishi I have used chip cert commands for generating the certificates.
This is the sdk file:
sdk_cert.txt

@SYESHASWINI Please take a look at mfg_tool.
You have configured the VID/PID to non-test and still using the Test DAC provider and it is causing the problem. Test DAC provider only have test certs from VID:0xFFF1-0xFFF4 and PID:0x8000:0x8063.
And hence it is failing to compile. You will have to generate the manufacturing binary and flash it, then set the correct config options, build/flash firmware and that would work. All the info you need is in mfg_tool's readme.

I am closing this issue. Please refer the README and and it should help. Feel free to re-open in case of anything.

@shubhamdp I tried generating partition using the command
./mfg_tool.py -cn "My bulb" -v 0xFFF2 -p 0x8001 --pai
-k path/to/esp-matter/connectedhomeip/connectedhomeip/credentials/test/attestation/Chip-Test-PAI-FFF2-8001-Key.pem
-c path/to/esp-matter/connectedhomeip/connectedhomeip/credentials/test/attestation/Chip-Test-PAI-FFF2-8001-Cert.pem
-cd path/to/esp-matter/connectedhomeip/connectedhomeip/credentials/test/certification-declaration/Chip-Test-CD-FFF2-8001.der
where it throws the following error:
Traceback (most recent call last):
File "/home/ubuntu/esp-matter/esp-matter/tools/mfg_tool/./mfg_tool.py", line 630, in
main()
File "/home/ubuntu/esp-matter/esp-matter/tools/mfg_tool/./mfg_tool.py", line 623, in main
write_per_device_unique_data(args)
File "/home/ubuntu/esp-matter/esp-matter/tools/mfg_tool/./mfg_tool.py", line 360, in write_per_device_unique_data
generate_onboarding_data(args, int(row['Index']), int(chip_factory_get_val('discriminator')), int(row['PIN Code']))
File "/home/ubuntu/esp-matter/esp-matter/tools/mfg_tool/./mfg_tool.py", line 438, in generate_onboarding_data
if args.commissioning_flow == CommissioningFlow.Standard:
NameError: name 'CommissioningFlow' is not defined


Add Autoregressive distribution

Solving #198
This PR is not ready to be merged. Changes -

Added AR distribution in timeseries.py
Added timeseries in distributions/init.py

I am not sure if initial_state_prior can work with pymc4 distributions.

I am not sure if initial_state_prior can work with pymc4 distributions.

Not exactly. We could get it to work but I think that we should not have an initial_state_prior as a parameter that takes a Distribution. We should have an initial_state parameter that accepts a Tensor as input.

I am not sure if initial_state_prior can work with pymc4 distributions.

Not exactly. We could get it to work but I think that we should not have an initial_state_prior as a parameter that takes a Distribution. We should have an initial_state parameter that accepts a Tensor as input.

I will add that.
Also should we add another parameter order in AR distribution or is it fine to calculate order as done here.

Let's not use an order parameter.
About the failed build, you're having lint problems. To fix them, from the project's root directory you have to run make black. If you are running windows, you have to run python -m black pymc4 tests instead

The sample method seems to work correctly
>>> ar_model = AR('my-regressor', num_timesteps=5, coefficients=[-0.2, 0.8], level_scale=1)
>>> ar_model.sample()
<tf.Tensor: shape=(5, 1), dtype=float32, numpy=
array([[ 0.5804499 ],
       [ 0.02004898],
       [-0.62362194],
       [ 0.70075184],
       [ 1.396882  ]], dtype=float32)>
>>>
>>> pm.evaluate_model(ar_model)
(<tf.Tensor: shape=(5, 1), dtype=float32, numpy=
array([[ 0.5592842 ],
       [-0.6405227 ],
       [ 0.5006866 ],
       [-0.41925687],
       [ 0.03968433]], dtype=float32)>, SamplingState(
    untransformed_values: ['my-regressor']
    transformed_values: []
    observed_values: []
    distributions: ['AR:my-regressor']
    num_potentials=0
    deterministics: []
    posterior_predictives: []))


Stack became unresponsive after updated kube_service_addresses and kube_pods_subnet

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Environment:

Cloud provider or hardware configuration:
aws, us-east-1 region, 3 x etcd + 3 x master + 4 x worker + 2 bastin

$ kubectl get nodes -owide
NAME                            STATUS    ROLES     AGE       VERSION           EXTERNAL-IP   OS-IMAGE                                       KERNEL-VERSION   CONTAINER-RUNTIME
ip-10-250-65-145.ec2.internal   Ready     master    5d        v1.9.3+coreos.0   <none>        Container Linux by CoreOS 1632.3.0 (Ladybug)   4.14.19-coreos   docker://17.9.1
ip-10-250-66-47.ec2.internal    Ready     node      5d        v1.9.3+coreos.0   <none>        Container Linux by CoreOS 1632.3.0 (Ladybug)   4.14.19-coreos   docker://17.9.1
ip-10-250-74-104.ec2.internal   Ready     master    5d        v1.9.3+coreos.0   <none>        Container Linux by CoreOS 1632.3.0 (Ladybug)   4.14.19-coreos   docker://17.9.1
ip-10-250-79-15.ec2.internal    Ready     node      5d        v1.9.3+coreos.0   <none>        Container Linux by CoreOS 1632.3.0 (Ladybug)   4.14.19-coreos   docker://17.9.1
ip-10-250-87-55.ec2.internal    Ready     node      5d        v1.9.3+coreos.0   <none>        Container Linux by CoreOS 1632.3.0 (Ladybug)   4.14.19-coreos   docker://17.9.1
ip-10-250-89-182.ec2.internal   Ready     node      5d        v1.9.3+coreos.0   <none>        Container Linux by CoreOS 1632.3.0 (Ladybug)   4.14.19-coreos   docker://17.9.1
ip-10-250-90-6.ec2.internal     Ready     master    5d        v1.9.3+coreos.0   <none>        Container Linux by CoreOS 1632.3.0 (Ladybug)   4.14.19-coreos   docker://17.9.1

OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):

$ printf "$(uname -srm)\n$(cat /etc/os-release)\n"
Linux 4.14.19-coreos x86_64
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1632.3.0
VERSION_ID=1632.3.0
BUILD_ID=2018-02-14-0338
PRETTY_NAME="Container Linux by CoreOS 1632.3.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Version of Ansible (ansible --version):

$ ansible --version
ansible <IP_ADDRESS>
  config file = /home/ubuntu/code/stack/kubespray/ansible.cfg
  configured module search path = [u'/home/ubuntu/code/stack/kubespray/library']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Nov 20 2017, 18:23:56) [GCC 5.4.0 20160609]

Kubespray version (commit) (git rev-parse --short HEAD):
$ git rev-parse --short HEAD
4361ac6

Network plugin used:
cilium

Copy of your inventory file:
$ vi inventory/hosts
[all]
kubernetes-stack-master0 ansible_host=<IP_ADDRESS>
kubernetes-stack-master1 ansible_host=<IP_ADDRESS>
kubernetes-stack-master2 ansible_host=<IP_ADDRESS>
kubernetes-stack-worker0 ansible_host=<IP_ADDRESS>
kubernetes-stack-worker1 ansible_host=<IP_ADDRESS>
kubernetes-stack-worker2 ansible_host=<IP_ADDRESS>
kubernetes-stack-worker3 ansible_host=<IP_ADDRESS>
kubernetes-stack-etcd0 ansible_host=<IP_ADDRESS>
kubernetes-stack-etcd1 ansible_host=<IP_ADDRESS>
kubernetes-stack-etcd2 ansible_host=<IP_ADDRESS>

bastion ansible_host=*.*.*.*
bastion ansible_host=*.*.*.*

[kube-master]
kubernetes-stack-master0
kubernetes-stack-master1
kubernetes-stack-master2

[kube-node]
kubernetes-stack-worker0
kubernetes-stack-worker1
kubernetes-stack-worker2
kubernetes-stack-worker3

[etcd]
kubernetes-stack-etcd0
kubernetes-stack-etcd1
kubernetes-stack-etcd2

[k8s-cluster:children]
kube-node
kube-master

[k8s-cluster:vars]
apiserver_loadbalancer_domain_name="kubernetes-elb-stack-*********.us-east-1.elb.amazonaws.com"

Command used to invoke ansible:
$ ansible-playbook -i ./inventory/hosts ./cluster.yml -e ansible_ssh_user=core -e bootstrap_os=coreos -b --become-user=root --flush-cache -e ansible_user=core

Output of ansible run:

https://gist.github.com/brant4test/5df76d89bd1e7ea8c40eda4ef13ef19e
Anything else do we need to know:

https://gist.github.com/brant4test/da36498045f66701f0103aea2013a986
after updated the ip spaces of current cluster, as follows, and re-run ansible-playbook cluster.yml, the stack became unresponsive
kube_service_addresses: <IP_ADDRESS>/18
kube_pods_subnet: <IP_ADDRESS>/18

to
kube_service_addresses: <IP_ADDRESS>/18
kube_pods_subnet: <IP_ADDRESS>/18

then I tried these ways to delete pods, all failed.
1. to delete pods by helm;
2. to delete pods by namespaces;
3. to delete pods by restarting aws instance (masters and workers);

elaborate on the "1. to delete pods by helm;"
$ helm list
NAME        	REVISION	UPDATED                 	STATUS  	CHART          	NAMESPACE
falling-mule	1       	Wed Mar 14 05:36:09 2018	DEPLOYED	charts1-0.1.0	default
jumpy-squid 	1       	Wed Mar 14 05:35:51 2018	DEPLOYED	charts2-0.1.0   default
$ helm del --purge falling-mule

^C (over 4 mins)
$ helm list
NAME       	REVISION	UPDATED                 	STATUS  	CHART   	NAMESPACE
jumpy-squid	1       	Wed Mar 14 05:35:51 2018	DEPLOYED	charts2-0.1.0	default

Seems like charts1 has gone, but actually the namespace and pods of charts1 are still there, even no slightest change.

Should I reset your cluster with reset.yml playbook and or deploy anew?
or
Should I "terraform destroy" and "terraform apply"a new stack?
Is there support now for changing network IP space for a running cluster?
Thanks!

There is no support for changing network adresses for a running cluster.  Even if we could change the components, there is no way to migrate deployed pods to the new network plugin. This issue goes beyond the scope that Kubespray can provide.

thanks


Incorrect sign in scipy.spatial.transform.Slerp for orientation quaternions

Maybe I am just not getting Quaternions, but I think there is an incorrect sign when using Slerp with orientation quaternions (w=0).
I am trying to interpolate positions between two vectors (vector part in orientation quaternions) and get the fractions of the shortest rotation between them.
I compared the results to mathutils  and gohlke's transformations.py
This code shows the results for the three packages. Scipy uses scalar-last, the others use scalar-first notation.
import transformations as tf
import mathutils
from scipy.spatial.transform import Rotation
import numpy as np
from scipy.spatial.transform import Slerp

def xyzw_to_wxyz(q):

    return np.array([q[3], q[0], q[1], q[2]])

# scipy : scalar-last format
quat0 = np.array([1.0, 0.0, 0.0, 0.0]) # x-axis
quat1 = np.array([0.0, 1.0, 0.0, 0.0]) # y-axis

key_rots = Rotation.from_quat([quat0, quat1])
slerp = Slerp([0, 1], key_rots)
times = [0.0, 0.25, 0.5, 0.75, 1.0]

for fraction in times:

    s1 = tf.quaternion_slerp(xyzw_to_wxyz(quat0), xyzw_to_wxyz(quat1), fraction)

    quat_a = mathutils.Quaternion(xyzw_to_wxyz(quat0))
    quat_b = mathutils.Quaternion(xyzw_to_wxyz(quat1))

    print()
    print(fraction)
    print('transformations.py', s1)
    print('mathutils', quat_a.slerp(quat_b, fraction))
    print('scipy', slerp([fraction]).as_quat())

Output is
fraction 0.0
transformations.py [0. 1. 0. 0.]
mathutils Quaternion((0.0, 1.0, 0.0, 0.0))
scipy [[1. 0. 0. 0.]]

fraction 0.25
transformations.py [0.         0.92387953 0.38268343 0.        ]
mathutils Quaternion((0.0, 0.9238795042037964, 0.3826834559440613, 0.0))
scipy [[ 0.92387953 -0.38268343  0.          0.        ]]

fraction 0.5
transformations.py [0.         0.70710678 0.70710678 0.        ]
mathutils Quaternion((0.0, 0.7071067690849304, 0.7071067690849304, 0.0))
scipy [[ 0.70710678 -0.70710678  0.          0.        ]]

fraction 0.75
transformations.py [0.         0.38268343 0.92387953 0.        ]
mathutils Quaternion((0.0, 0.3826834559440613, 0.9238795042037964, 0.0))
scipy [[ 0.38268343 -0.92387953  0.          0.        ]]

fraction 1.0
transformations.py [0. 0. 1. 0.]
mathutils Quaternion((0.0, 0.0, 1.0, 0.0))
scipy [[ 6.123234e-17 -1.000000e+00  0.000000e+00  0.000000e+00]]

You can see that mathutils and transformations.py arrive at y for fraction 1.0. But scipy has a flipped sign and returns -y
Scipy/Numpy/Python version information:
1.2.0 1.15.2 sys.version_info(major=3, minor=5, micro=5, releaselevel='final', serial=0)

I am trying to interpolate positions between two vectors (vector part in orientation quaternions) and get the fractions of the shortest rotation between them.

Can't be sure on that, but it seems to me you are using the wrong tool to solve your problem.
Quaternions to represent rotations have the sign ambiguity (i.e. q and -q represent the same rotation) and we don't try to maintain any sign conventions inside Rotation, because it doesn't have a strong meaning and would only complicate the code.
Looks like geometric_slerp from #10380 is the right tool to interpolate unit vectors in 3D. You can use dev version of scipy to get it right now.
It might sound a bit unsatisfactory, but it's sort of complicated to achieve what you are asking for in Slerp and doesn't make a lot of sense from the Rotation design perspective.


[feature request] manual mini-batching and batch dimension scaling

In models with mixed levels of nesting (e.g. global_plate > local_plate_1 > local_plate_2 > ...), mibi-batching across different batch dimensions requires introducing proper scale factors for each batch dimension. Pyro handles these scale factors automatically if mini-batching is achieved via pyro.iarange(..., size=..., subsample_size=...) or pyro.iarange(..., size=..., subsample=...). The latter construct is flexible and allows arbitrary mibi-batching schemes, including big data situations where the full data tensor can not be loaded all at once.
Mini-batching, however, is often done manually and externally and not via pyro.iarange. In such cases, the appropriate scale factors must also be applied manually via poutine.scale. We are being consistent here: manual mini-batching? then manual scaling. However, most of the examples (DMM, VAE, ...) have little to no emphasis on this issue and neglect scaling altogether. While convergence is not a big deal while working with adaptive optimizers, neglecting the scale factors leads to wrong ELBO estimates.

[ ] Adding a word of caution to the examples about scale factors and/or throwing in poutine.scale when mini-batching manually to set a good precedent for the new users?

Good points.  Most of our examples don't do nested subsampling, but maybe we could add a manually-batched version of our LDA example or something similar?

@mbabadi agreed we could improve docs about subsampling. I'm inclined to recommend users use pyro.iarange(..., subsample=...) when any minibatching is done, as that clarifies the intention of the code. Do you know of any cases where minibatching cannot be done through pyro.iarange(..., subsample=...)?

Do you know of any cases where minibatching cannot be done through pyro.iarange(..., subsample=...)?

I think one use case is when we need to feed from a large dataset (i.e. calling data.cuda() at once will take a lot of GPU memory) for which the torch data loaders work great since they will spin up a thread of workers that will keep pulling off batches of data and transferring it to the GPU incrementally. We are using data loaders in our examples, but many of our datasets are probably small enough that they can be directly transferred in one shot.

@neerajprad The use case you suggest cannot be accomplished via iarange(..., subsample_size=...), but it can be accomplished via iarange(..., subsample=...) (that is the motivating use case behind the subsample kwarg).

The use case you suggest cannot be accomplished via iarange(..., subsample_size=...), but it can be accomplished via iarange(..., subsample=...) (that is the motivating use case behind the subsample kwarg).

Ahh, my bad. In that case, we should probably just change our examples to use subsample=, which will do the correct scaling.

@fritzo @neerajprad I also can not imagine what can not be accomplished by iarange(..., subsample=...)! A callable subsampler can take care of both incremental data loading and optionally sending the minibatch to CUDA. That would be great if you could simply encourage the usage of this motif in the examples.


rtfm: init at 0.2.2

Description of changes
It's a dash/docset reader with built in documentation for Crystal and GTK APIs. It's written in Crystal using GTK4 bindings.
https://github.com/hugopl/rtfm

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

@drupol, @SuperSandro2000 could you please review this PR? It's been 3 days and no one wants to :(

@hugopl could you help with one point? I generate gi docs using gi-docgen. The GDK/GTK docs have a dependencies block, that has hard-coded dependency URLs.
Is there a way to change Browse documentation link to local docsets?
  [dependencies."GObject-2.0"]
  name = "GObject"
  description = "The base type system library"
  docs_url = "https://docs.gtk.org/gobject/"

  [dependencies."Gio-2.0"]
  name = "Gio"
  description = "GObject Interfaces and Objects, Networking, IPC, and I/O"
  docs_url = "https://docs.gtk.org/gio/"

  [dependencies."cairo-1.0"]
  name = "cairo"
  description = "A 2D graphics library with support for multiple output devices"
  docs_url = "https://www.cairographics.org/manual/"
  ...

After force-pushing last commit, i noticed some checks were failed.
ofborg-eval — This PR does not cleanly list package outputs after merging.

 error: attribute 'devdoc' missing

       at /var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/mr-est/ofborg-evaluator-7/pkgs/applications/misc/rtfm/default.nix:67:20:

           66|
           67|     for file in "${gtk4.devdoc}"/share/doc/*; do
             |                    ^
           68|       ln -s "$file" "gtk-doc/$(basename "$file")"

It happens because gtk4.devdoc output depends in x11Support optional argument
outputs = [ "out" "dev" ] ++ lib.optionals x11Support [ "devdoc" ];

So, maybe i need to override x11Support = true?

So, maybe i need to override x11Support = true?

seems like it

So, maybe i need to override x11Support = true?

seems like it

Nothing happened, i don't know why and how to fix it

Is there a way to change Browse documentation link to local docsets?

I can reproduce this on Archlinux too and create an issue.

@sund3RRR would you mind in creating a PR for rtfm README with the instructions to install this NixOS package?

@sund3RRR would you mind in creating a PR for rtfm README with the instructions to install this NixOS package?

Sure, but these instructions will be useless until this PR is approved

Yeah, sure, I mean, when it gets approved/merged.


Applied fixes from StyleCI

This pull request applies code style fixes from an analysis carried out by StyleCI.

For more information, click here.

Current coverage is 100.00%

Merging #8 into master will not affect coverage as of b64e661

@@            master      #8   diff @@
======================================
  Files            2       2
  Stmts          100     100
  Branches         0       0
  Methods         20      20
======================================
  Hit            100     100
  Partial          0       0
  Missed           0       0

Review entire Coverage Diff as of b64e661

Powered by Codecov. Updated on successful CI builds.


provide standard reload mapping

This is more of a request than a bug. I cam across this question at stackoverflow:
http://stackoverflow.com/questions/8621247/fastest-way-to-make-vim-use-new-plugin-in-vim-bundles/8621357#8621357
As you can see I suggested to do something like this : map \u call pathogen#infect() to rescan the bundles directory.
Maybe you could provides a default mapping for this and mention somehting in the documentation ? Unless
this happens to be a bad idea for some reason ?

I've wanted something like this for a while. Two big caveats:

Calling pathogen#infect() a second time results in scanning not just the original runtime path, but also inside any bundles that were added the previous go around. With a lot of bundles, this is noticeably slow. Plus the path ends up in a weird order.
If Vim has already loaded, we need to manually activate plugins and filetype detection. This is not necessarily unfeasible, but I'm guessing there are edge cases hiding in there (things like #38).

@tpope have you thought a bit more regarding this? I'm currently using pathogen for my standalone plugin manager and it would be awesome not having to close the editor when you install a new plugin.


Cannot set output codepage

Steps to reproduce
Running chcp <codepage>, setting [Console]::OutputEncoding, or using PInvoke to call SetConsoleOutputCP should be setting the console output codepage. In Windows PowerShell this works but in PowerShell nothing is changed.
Add-Type -TypeDefinition @'
using Microsoft.Win32.SafeHandles;
using System;
using System.Runtime.InteropServices;

namespace RawConsole
{
    public class NativeMethods
    {
        [DllImport("Kernel32.dll")]
        public static extern uint GetConsoleCP();

        [DllImport("Kernel32.dll")]
        public static extern uint GetConsoleOutputCP();

        [DllImport("Kernel32.dll")]
        public static extern bool SetConsoleCP(
            uint wCodePageID);

        [DllImport("Kernel32.dll")]
        public static extern bool SetConsoleOutputCP(
            uint wCodePageID);
    }
}
'@

# Take note of the original value
[RawConsole.Nativemethods]::GetConsoleOutputCP()

# Set using one of these 3 options, the value can be anything just something that isn't currently set
[RawConsole.Nativemethods]::SetConsoleOutputCP(1257)
chcp 1257
[Console]::OutputEncoding = [Text.Encoding]::GetEncoding(1257)

# This should be updated with our new value
[RawConsole.Nativemethods]::GetConsoleOutputCP()

Note: You cannot use [Console]::OutputEncoding to get the value for this test as it caches the value on the first call. This is why I'm using the PInvoke call to ensure there isn't any caching shenanigans going on in the .NET layer.
Expected behavior
GetConsoleOutputCP() should output 1257.
Actual behavior
GetConsoleOutputCP() outputs the original value. This value is whatever the console output codepage was set to when pwsh.exe was started.
Environment data
Name                           Value
----                           -----
PSVersion                      7.1.0
PSEdition                      Core
GitCommitId                    7.1.0
OS                             Microsoft Windows 10.0.17763
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

Using conhost and not Windows Terminal here. This also only affects PowerShell and not Windows PowerShell.
Other notes
Calling SetConsoleCP() and GetConsoleCP() works just fine it's just the output codepage that doesn't seem to have any affect. The Win32 function returns $true which indicates nothing went wrong. I'm honestly unsure whether this is a conhost problem, .NET issue, or PowerShell problem. Just trying here first.
I've had a brief look at the OpenConsole code in the terminal repo. Ultimately SetConsoleOutputCP calls DoSrvSetConsoleOutputCodePage and there's nothing in there that jumps out to me to explain this behaviour. Of course I'm not actually using the OpenConsole.exe conhost but the one builtin to Windows and I'm 100% assuming the code is the same or at least not dramatically different. In any case it also doesn't explain why Windows PowerShell works but PowerShell does not.

Just to give a trivial example of when this can be a problem, my default code page is 437 and running this in pwsh will output ? because the character Ǿ isn't in that codepage.
$cmd = '[Console]::WriteLine([string][char]0x01FE)'  # Ǿ

$encCmd = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($cmd))
pwsh.exe -EncodedCommand $encCmd

Note: This is highly dependent on the child process and how it's outputting the data. In this case this replicates the behaviour when the child process would open the STDOUT stream as a file and write raw bytes based on the console cp, or even WriteConsoleA.
Adding more confusion to the mix running [Console]::OutputEncoding = [Text.Encoding]::UTF8 seems to actually affect the child process' .NET Console OutputEncoding but the native methods (chcp/PInvoke) do not.
$cmd = '[Console]::WriteLine([string][char]0x01FE)'  # Ǿ

# chcp 65001  # Does not work as well as the PInvoke SetConsoleOutputCP(65001) method above
[Console]::OutputEncoding = [Text.UTF8Encoding]::new($false)

$encCmd = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($cmd))
pwsh.exe -EncodedCommand $encCmd

It seems like there's some inheritance going on with .NET that I just don't understand yet. The raw console output CP that Win32 reports is still the original value but setting [Console]::OutputEncoding does affect child .NET processes.

@jborean93 I don't believe PowerShell is doing anything here.  Can you write a simple C# app w/o PowerShell and see if it repros?  At least we can determine if it's .NET doing something.

@jborean93, I haven't looked at the details yet, but I wonder if https://stackoverflow.com/a/65986012/45375 helps; in short running chcp  from inside PowerShell is ineffective, due to .NET's caching of encodings in the [Console] class.
With respect to what chcp reports, it is, perhaps surprisingly, [Console]::InputEncoding that matters.

Although perhaps a bit counter-intuitive, it does seem that InputEncoding does call SetConsoleCP.

@SteveL-MSFT I thought it was something up with .NET but if I was to compile this code (with the 5.0.1 sdk) and run it works without any problems:
using System;
using System.Runtime.InteropServices;

namespace myApp
{
    public class NativeMethods
    {
        [DllImport("Kernel32.dll")]
        public static extern uint GetConsoleCP();

        [DllImport("Kernel32.dll")]
        public static extern uint GetConsoleOutputCP();

        [DllImport("Kernel32.dll")]
        public static extern bool SetConsoleCP(
            uint wCodePageID);

        [DllImport("Kernel32.dll")]
        public static extern bool SetConsoleOutputCP(
            uint wCodePageID);
    }

    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine(NativeMethods.GetConsoleOutputCP().ToString());
            NativeMethods.SetConsoleOutputCP(65001);
            Console.WriteLine(NativeMethods.GetConsoleOutputCP().ToString());
        }
    }
}

If I was to run the equivalent (below) of that in PowerShell I get the same encoding (437) in my output instead of the one I set.
Add-Type -TypeDefinition @'
using System;
using System.Runtime.InteropServices;

namespace RawConsole
{
    public class NativeMethods
    {
        [DllImport("Kernel32.dll")]
        public static extern uint GetConsoleCP();

        [DllImport("Kernel32.dll")]
        public static extern uint GetConsoleOutputCP();

        [DllImport("Kernel32.dll")]
        public static extern bool SetConsoleCP(
            uint wCodePageID);

        [DllImport("Kernel32.dll")]
        public static extern bool SetConsoleOutputCP(
            uint wCodePageID);
    }
}
'@

[RawConsole.NativeMethods]::GetConsoleOutputCP()
[RawConsole.NativeMethods]::SetConsoleOutputCP(65001)
[RawConsole.NativeMethods]::GetConsoleOutputCP()

@jborean93, I haven't looked at the details yet, but I wonder if https://stackoverflow.com/a/65986012/45375 helps; in short, running chcp from inside PowerShell is ineffective, due to .NET's caching of encodings in the [Console] class.

It would be the case if I was using [Console]::OutputEncoding to check the value I set. In this case I'm going straight to the source with PInvoke to call GetConsoleOutputCP() and it's not being changed.

Although perhaps a bit counter-intuitive, it does seem that InputEncoding does call SetConsoleCP.

There are 2 cp APIs, SetConsoleCP which corresponds with the input code page and SetConsoleOutputCP which is the output codepage.

PowerShell does have the $OutputEncoding automatic variable which is set on initial state and can be set by assigning it a value.  However, calling the Win32 API directly won't update this variable.  The PowerShell native command processor relies on this variable value.

$OutputEncoding AFAIK just controls the encoding PowerShell uses when it's piping data into a native command like "test" | something.exe. It seems like the .NET code has changed quite a lot from the Framework days so it could explain the difference between Windows PowerShell and PowerShell. I'll have to look into it a bit more and see if it explains what is happening.

Trying to clear out any confusion there are 2 codepages associated with the console, the input and output codepage. For those 2 cp's I know of 3 "native" ways of setting them

Method
Input
Output
Notes

PInvoke
SetConsoleCP
SetConsoleOutputCP
I would consider this the absolute truth, it sources the data from the console itself and is how the other methods ultimately get/set the cp

chcp
chcp
chcp
This sets both the input/output in 1 command. Running chcp by itself only reports the input value though

.NET
[Console]::InputEncoding
[Console]::OutputEncoding
The value of these are cached, once it is retrieved once it will stay that way unless they were set in .NET itself

I could be wrong here but from observation the output code page is important here. If something is writing bytes directly to the conhost STDOUT pipe (like [Console]::WriteLine does) then it's the value of the output cp that decodes the bytes back to text. If the bytes sent don't match up with what encoded it then we will get back garbage text.
To illustrate this a bite more let's say we run `[Console]::WriteLine('café'). here is what happens in the .NET land

Console.WriteLine calls Console.Out.WriteLine
Console.Out is a SyncTextWriter stream
This stream will encode any strings using the Console.OutputEncoding value set

Console.OutputEncoding is cached so on the first call it will source the cp from GetConsoleOutputCP() which happens when pwsh first starts up

The bytes are then written to a ConsoleStream which is based on a file handle for the STDOUT pipe
This "pipe" sends the bytes to conhost
Conhost will use the value of GetConsoleOutputCP() to decode those bytes back to text
Conhost will then print that text to the screen buffer

In Windows PowerShell, all 3 methods work to change the console input/output codepage. I can use all 3 to set the console output cp to UTF-8 so that my example in https://github.com/PowerShell/PowerShell/issues/14941#issuecomment-790225777 outputs the proper character.
In PowerShell the 3 methods work for getting/setting the input console codepage but the output one is complex. Setting the output cp does not work with the PInvoke or chcp way. I can do [Console]::OutputEncoding = [Text.Encoding]::UTF8 and it does actually work. Looking at the code setting [Console]::OutputEncoding is just flushing the existing out/err pipes and then calling SetConsoleCP() so I'm confused as to why I cannot just do that in PowerShell itself.
Taking out the Console class althogether and I can show another example that fails on PowerShell but works on Windows PowerShell.
Add-Type -TypeDefinition @'
using Microsoft.Win32.SafeHandles;
using System;
using System.Runtime.InteropServices;

namespace RawConsole
{
    public class NativeMethods
    {
        [DllImport("Kernel32.dll")]
        public static extern SafeFileHandle GetStdHandle(
            int nStdHandle);

        [DllImport("Kernel32.dll")]
        public static extern bool SetConsoleOutputCP(
            uint wCodePageID);
    }
}
'@

$stdoutHandle = [RawConsole.NativeMethods]::GetStdHandle(-11)
$fs = [IO.FileStream]::New($stdoutHandle, 'Write')

[RawConsole.NativeMethods]::SetConsoleOutputCP(65001)
$data = [Text.Encoding]::UTF8.GetBytes('café')
$fs.Write($data, 0, $data.Length)
$fs.Flush()

# This outputs 'caf├⌐' which is the UTF-8 bytes of my input decoded using cp 437

Doing the exact same with with PowerShell out of the picture and it works just fine
using Microsoft.Win32.SafeHandles;
using System;
using System.IO;
using System.Runtime.InteropServices;
using System.Text;

namespace myApp
{
    public class NativeMethods
    {
        [DllImport("Kernel32.dll")]
        public static extern uint GetConsoleOutputCP();

        [DllImport("Kernel32.dll")]
        public static extern SafeFileHandle GetStdHandle(
            int nStdHandle);

        [DllImport("Kernel32.dll")]
        public static extern bool SetConsoleOutputCP(
            uint wCodePageID);
    }

    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine(NativeMethods.GetConsoleOutputCP());  // Just verifies the original was 437 in my case

            using (SafeFileHandle stdoutHandle = NativeMethods.GetStdHandle(-11))
            using (FileStream fs = new FileStream(stdoutHandle, FileAccess.Write))
            {
                NativeMethods.SetConsoleOutputCP(65001);

                byte[] data = Encoding.UTF8.GetBytes("café");
                fs.Write(data, 0, data.Length);
                fs.Flush();
            }
        }
    }
}

So ultimately I have 2 questions:

Why does PInvoke SetConsoleOutputCP() not work when calling from PowerShell (Windows PowerShell is fine)
Why does [Console]::OutputEncoding = ... work in PowerShell but the PInvoke method does not

It essentially does the same thing

Granted these are edge cases and it looks like doing [Console]::OutputEncoding = ... works for the majority of scenarios. What I really want to understand is why that's the case.

Final thing to add it seems like this is only a problem if I'm calling SetConsoleOutputCP() directly from PowerShell.
The following does work and replicates the behaviour I'm seeing when running my pure .NET binary.
Add-Type -TypeDefinition @'
using Microsoft.Win32.SafeHandles;
using System;
using System.IO;
using System.Runtime.InteropServices;
using System.Text;

namespace RawConsole
{
    public class NativeMethods
    {
        [DllImport("Kernel32.dll")]
        public static extern int GetConsoleOutputCP();

        [DllImport("Kernel32.dll")]
        public static extern SafeFileHandle GetStdHandle(
            int nStdHandle);

        [DllImport("Kernel32.dll")]
        public static extern bool SetConsoleOutputCP(
            int wCodePageID);
    }

    public class Program
    {
        public static void Main(string value, Encoding encoding)
        {
            Console.WriteLine(NativeMethods.GetConsoleOutputCP().ToString());
            NativeMethods.SetConsoleOutputCP(encoding.CodePage);
            Console.WriteLine(NativeMethods.GetConsoleOutputCP().ToString());  // It actually updates here!

            using (SafeFileHandle stdoutHandle = NativeMethods.GetStdHandle(-11))
            using (FileStream fs = new FileStream(stdoutHandle, FileAccess.Write))
            {
                byte[] data = encoding.GetBytes(value);
                fs.Write(data, 0, data.Length);
                fs.Flush();
            }
        }
    }
}
'@

[RawConsole.Program]::Main('café', [Text.Encoding]::UTF8)

Is my PInvoke definition wrong, am I calling it incorrectly in Powershell?

Thanks to @SeeminglyScience (once again!) the issue balls down to PSReadLine resetting the console codepage after every statement (maybe command I'm not 100% sure). So by setting it through chcp or PInvoke it is actually being sent to conhost PSReadLine is just setting it back afterwards before I go to check the value.
@SteveL-MSFT I can raise this issue in the PSReadLine repository but I assume it's expected behaviour and you are just expected to use [Console]::OutputEncoding to set this value going forward so will hold off unless you want me to open it.

PSReadLine resetting the console codepage after every statement (maybe command I'm not 100% sure).

It would be between interactive evals, if it's happening between statements then I can't explain why disabling PSRL fixes it tbh. Make sure your paste isn't being translated into multiple ReadLine loops (e.g. right clicking or ctrl v in some terminal apps).  Easiest way is to try it all on one line with semi colons or wrap in & { }.

Yea I am just copy and pasting my examples so that would explain it even more.

Great sleuthing, but do we know why PSReadLine does that? Just to guard against external programs changing the code page?

It saves the initial output encoding on first ReadLine call, and then resets it back to that after every ReadLine finishes.
It can probably be updated to allow for mid-process encoding customizations.

Looking at git blame, it seems https://github.com/PowerShell/PSReadLine/commit/c3868e125069d87adbc5ee8f0274762a99510957 to address an issue with CJK chars coming from native commands


time-selected doesn't trigger in "month" view

I setup the time selected and it's only working in "week" and "day" views. Nothing triggers when I click on days in "month" view.

@ricardo-arango It should be triggered, you can verify that behavior in the demo page


Can this reproduce the results of instant-ngp? What parameter should I use?

I tried to reproduce NeRF results of instant-ngp with this repo, but I the PSNR I got is quite a bit lower than the original one. So what should I do to reproduce the results? Can this repo reproduce the results in the original paper?

Unfortunately, there is still a performancee gap (e.g., 33 compared to 35 for Lego).

@GaiZhenbiao which flags are you using for training?
@ashawkey do you have a feeling where the problem might be coming from?

There are lots of details and tricks in instant-ngp unimplemented here, I doubt about the density grid updation most now. You can check those TODOs (e.g., https://github.com/ashawkey/torch-ngp/blob/main/nerf/renderer.py#L415) if you are interested.

closed for now, any improvement will be noted in readme.


VSCode ESP-IDF lvgl 例程编译报错 (IDFGH-8675)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
V4.4.2
Operating System used.
Windows
How did you build your project?
VS Code IDE
If you are using Windows, please specify command line type.
PowerShell
What is the expected behavior?
在VSCode上编译 lvgl 例程
What is the actual behavior?
在VSCode上编译 lvgl 例程，编译失败
Steps to reproduce.
1、打开VSCode插件 ESP-IDF ，
2、复制并创建例程 "lvgl"，
3、选择target 为 "esp32"，点击"build"，
4、开始编译。。。
5、结果编译不通过！log如下
Build or installation Logs.
*  正在执行任务: cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DESP_PLATFORM=1 ..

-- Found Git: D:/ESP-IDF/espressif/tools/idf-git/2.30.1/cmd/git.exe (found version "2.30.1.windows.1")
-- The C compiler identification is GNU 8.4.0
-- The CXX compiler identification is GNU 8.4.0
-- The ASM compiler identification is GNU
-- Found assembler: D:/ESP-IDF/espressif/tools/xtensa-esp32-elf/esp-2021r2-patch3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc.exe
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: D:/ESP-IDF/espressif/tools/xtensa-esp32-elf/esp-2021r2-patch3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: D:/ESP-IDF/espressif/tools/xtensa-esp32-elf/esp-2021r2-patch3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Project is not inside a git repository, or git repository has no commits; will not use 'git describe' to determine PROJECT_VER.
-- Building ESP-IDF components for target esp32
Processing 2 dependencies:
[1/2] idf (4.4.2)
[2/2] lvgl/lvgl (8.0.2)
CMake Error at D:/ESP-IDF/esp-idf/tools/cmake/build.cmake:454 (message):
  Traceback (most recent call last):

    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\urllib3\connection.py", line 174, in _new_conn
      conn = connection.create_connection(
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\urllib3\util\connection.py", line 72, in create_connection
      for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
    File "socket.py", line 918, in getaddrinfo

  socket.gaierror: [Errno 11001] getaddrinfo failed

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):

    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
      httplib_response = self._make_request(
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\urllib3\connectionpool.py", line 386, in _make_request
      self._validate_conn(conn)
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\urllib3\connectionpool.py", line 1042, in _validate_conn
      conn.connect()
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\urllib3\connection.py", line 358, in connect
      self.sock = conn = self._new_conn()
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\urllib3\connection.py", line 186, in _new_conn
      raise NewConnectionError(

  urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection
  object at 0x0000017EEF2350D0>: Failed to establish a new connection: [Errno
  11001] getaddrinfo failed

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):

    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\requests\adapters.py", line 489, in send
      resp = conn.urlopen(
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
      retries = retries.increment(
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\urllib3\util\retry.py", line 592, in increment
      raise MaxRetryError(_pool, url, error or ResponseError(cause))

  urllib3.exceptions.MaxRetryError:
  HTTPSConnectionPool(host='components-file.espressif.com', port=443): Max
  retries exceeded with url: /6fdc1d19-9f86-4bb2-b0f2-09850aaa7ad5.tgz
  (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object
  at 0x0000017EEF2350D0>: Failed to establish a new connection: [Errno 11001]
  getaddrinfo failed'))

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):

    File "runpy.py", line 194, in _run_module_as_main
    File "runpy.py", line 87, in _run_code
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\idf_component_manager\prepare_components\__main__.py", line 3, in <module>
      main()
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\idf_component_manager\prepare_components\prepare.py", line 126, in main
      args.func(args)
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\idf_component_manager\prepare_components\prepare.py", line 39, in prepare_dep_dirs
      ComponentManager(args.project_dir).prepare_dep_dirs(
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\idf_component_manager\core.py", line 404, in prepare_dep_dirs
      downloaded_component_paths = download_project_dependencies(
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\idf_component_manager\dependencies.py", line 128, in download_project_dependencies
      download_paths = fetcher.download()
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\idf_component_tools\sources\fetcher.py", line 51, in download
      return self.source.download(self.component, self.managed_path)
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\idf_component_tools\sources\web_service.py", line 139, in download
      with requests.get(url, stream=True, allow_redirects=True) as r:
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\requests\api.py", line 73, in get
      return request("get", url, params=params, **kwargs)
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\requests\api.py", line 59, in request
      return session.request(method=method, url=url, **kwargs)
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\requests\sessions.py", line 587, in request
      resp = self.send(prep, **send_kwargs)
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\requests\sessions.py", line 701, in send
      r = adapter.send(request, **kwargs)
    File "d:\ESP-IDF\espressif\python_env\idf4.4_py3.8_env\lib\site-packages\requests\adapters.py", line 565, in send
      raise ConnectionError(e, request=request)

  requests.exceptions.ConnectionError:
  HTTPSConnectionPool(host='components-file.espressif.com', port=443): Max
  retries exceeded with url: /6fdc1d19-9f86-4bb2-b0f2-09850aaa7ad5.tgz
  (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object
  at 0x0000017EEF2350D0>: Failed to establish a new connection: [Errno 11001]
  getaddrinfo failed'))

Call Stack (most recent call first):
  D:/ESP-IDF/esp-idf/tools/cmake/project.cmake:384 (idf_build_process)
  CMakeLists.txt:4 (project)

-- Configuring incomplete, errors occurred!
See also "D:/VSCode_Pro/ESP-IDF/lvgl/build/CMakeFiles/CMakeOutput.log".

 *  终端进程“C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -Command cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DESP_PLATFORM=1 ..”已终止，退出代码: 1。

More Information.
No response

seems a network issue happened when downloading the LvLG library from the component registry. Can retry help?

seems a network issue happened when downloading the LvLG library from the component registry. Can retry help?
我已经重试过，从打开ESP-IDF插件的实例例程，再复制创建项目，再重新编译，编译结果log显示是一样的。


support python 2.7 or 3.x ?

Could you  write which python supported in README.md please.
Thanks 😄

It's seems can't run code from .config import [somthing] in Python 2.7

Python 2.7. Relative import in non-package is not allowed.


Prefixing of elements of sub-highlighters

Currently all highlighters share the same element names, this is a request that a File inclusion be able to inject a prefix for the element names.
So if you use CSS.json directly f.i. its keywords would use the Attribute element (as currently), but if you include it with
{
    "File": "CSS.json",
    "AttributePrefix": "css"
}

then the importer would inject "css" as prefix on elements, and it would use "cssAttribute" element (and "cssEditor" and "cssReservedWords", etc.) this way it becomes possible to apply different colors/style to css attributes and html attributes.

Added ElementPrefix. Usage as you wished except AttributePrefix is obviously ElementPrefix.


v2.3 rc

Alaska Airlines Pull Request
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Summary:
Please summarize the scope of the changes you have submitted, what the intent of the work is and anything that describes the before/after state of the project.
Release PR for v2.3-rc
Type of change:
Please delete options that are not relevant.

[x] New capability
[x] Revision of an existing capability

Checklist:

[x] My update follows the CONTRIBUTING guidelines of this project
[x] I have performed a self-review of my own update

By submitting this Pull Request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Pull Requests will be evaluated by their quality of update and whether it is consistent with the goals and values of this project. Any submission is to be considered a conversation between the submitter and the maintainers of this project and may require changes to your submission.
Thank you for your submission!
-- Auro Design System Team

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.5 out of 6 committers have signed the CLA.:white_check_mark: jason-capsule42:white_check_mark: geoffrich:white_check_mark: leeejune:white_check_mark: blackfalcon:white_check_mark: DZwell:x: semantic-release-botYou have signed the CLA already but the status is still pending? Let us recheck it.

:tada: This PR is included in version 2.3.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


⚠ Test Homepage has degraded performance

In 4cf53ec, Test Homepage (https://test.h-edu.cz) experienced degraded performance:

HTTP code: 500
Response time: 760 ms

Resolved: Test Homepage performance has improved in c8c7cb0.


Hacktoberfest-2022

Added new problem Unique path sum

[ ] 🚀 Added Name
[ ] ✨ Feature
[ ] 🌟 stared the repo
[ ] 🐛 Grammatical Error
[ ] 📝 Documentation Update
[ ] 🚩 Other

Description
Add Link of GitHub Profile
https://github.com/swatantra-coder

please add my file to in your repository with label hacktoberfest


Making the site more tablet friendly

8649e6a
The next/previous buttons just kept shrinking as the screen did, making them much too small on anything but a computer.
a3a7888
The steps are too large for smaller screens, regardless of how they are oriented. And as they are not strictly necessary, I simply hid them on tablet and mobile. There is still a range close to the cutoff where the steps don't look great, but it is an improvement. (The cutoff for tablet could be changed to address this issue.)
Some alternatives to simply hiding the steps might be replacing them with a dropdown or hiding all but the current step's text.
The alerts sidebar also needs to be updated, but there were a couple of things I wanted to ask first. Is the <Reveal> in <BenefitCard> necessary? The information the <Reveal> displays now doesn't seem to be super helpful and hover effects are pretty useless on touch devices. Would it be alright to move the <AlertSidebar> above the <FormSection> on smaller screens? Otherwise I would suggest shortening the header 'Warnings and Alerts' to just 'Alerts', as the current header tends to overflow on smaller screens.

Forgot to update the snapshots...

The alerts are being/have been? removed. There may be other alternatives to the steps, but maybe we can talk about those tonight. Feel free to update the snapshots and push again.

The alerts are being/have been? removed.

Well shoot. I do kind of remember that.
Snapshots updated.


mitmdump memory usage is always constantly growing

(orignally mentioned in #4451 , but i don't think it's related to the issue discussed there and should get a separate ticket)
Problem Description
i use mitmdump for interactive browsing a lot.
that is, i run an mitmdump instance continuously, and use it from multiple browsers on multiple computers.
i find that mitmdump's memory usage constantly grows,
it appears as if it allocates memory for any request/response data it processes and then never frees it.
note that i would expect this behaviour from mitmproxy running interactively, as it displays all the data in the UI,
but mitmdump should have no reason to keep flows in memory after writing them out.
here's a typical mitmdump memory usage graph out of my monitoring stack:

mitmdump memory usage appears as a sawtooth pattern, rising until it's terminated.
the growth rate is currently typically around 500mb/hour, it was less in the past, when it got through a whole day with under 2gb,
and is higher than the rate of data actually dumped (under 1gb/day).
in this graph mitmdump is manually terminated a few times, because of debugging #4451, and by a daily restart at midnight.
(i also run a script that terminates it if it exceeds 3gb, to avoid it taking down my system.)
some very rare phases of non-growth (as seen in the graph above) are very unlikely related to zero traffic - thanks to websites constantly loading stuff in the background nowadays, and many browsers and tabs, it's unlikely that traffic is ever zero.
i see some extremely rare occurences (at most once a week) of larger amounts of memory being allocated and freed again, in the expected pattern of allocating it to handle a request, and freeing it again after the request is finished.
Steps to reproduce the behavior:

run mitmdump for a longer period and use it, possibly with multiple browsers with a large number of tabs open
observe memory usage

System Information
i think it has been the case ever since i started using mitmproxy, the above graph is of current git master. (running with --set proxy_debug -vvv, but it doesn't make a difference to the behaviour.)

 I have written a small mitmproxy addon that overwrites SIGUSR1 and outputs which other objects still hold a reference:
import collections
import gc
import os
import signal

from mitmproxy import flow

def debug(*_):
    print()
    print("Memory")
    print("=======")
    gc.collect()

    d = collections.Counter()
    for o in gc.get_objects():
        t = str(type(o))
        if "mitmproxy" in t:
            d[t] += 1

    for t, count in d.most_common(10):
        print(count, t)

    print()
    print("Flow References")
    print("===============")
    objs = gc.get_objects()
    for i in range(len(objs)):
        if isinstance(objs[i], flow.Flow):
            print(f"{objs[i]}")
            holder = {"val": objs[i]}

            # We hold references here ourselves. Ignore those, and make sure that we don't run into cycles.
            _seen = {id(objs), id(holder)}

            def seen(x):
                if x in _seen:
                    return True
                _seen.add(x)

            for _ in range(20):
                referrers = [
                    val for val in gc.get_referrers(holder["val"])
                    if not seen(id(val))
                ]
                if referrers:
                    print(f"  {len(referrers)=} {type(referrers[0])=}")
                    holder["val"] = referrers[0]
                    del referrers
                else:
                    break
            break
    else:
        print("No flows!")

def load(loader):
    print(f"Registering debug signal... {os.getpid()}")
    signal.signal(signal.SIGUSR1, debug)

If you run this as a regular mitmproxy addon, sending SIGUSR1 should show you 1) object statistics and 2) the reference chain for the first flow object found. Does that yield anything useful?

cool, i was wondering if the object stats in that that dump are expected or not.
i'll try this as i get to it, but as mentioned, #4451 is the more important issue for me.
(i have also not used addons before, time to learn something new ;) )

cool, i was wondering if the object stats in that that dump are expected or not.

If there are no active connections, I have zero flow.Flow instances in mitmdump. Anything else is unintended behavior. :)

curiously, since the three fixes from #4451 (and #5532 ) are in, memory usage has never exceeded 200mb.
is it possible that objects were simply not being cleaned up after those crashes?
did you expect the memory usage and crashes to be related? it was not obvious to me.

the three marks are deploys of the last three commits,
https://github.com/mitmproxy/mitmproxy/commit/c8eca9a396adb63f50e00506c53cf24ec723291d
https://github.com/mitmproxy/mitmproxy/commit/ff4120d4f3dd1d786377a541ec790945ae321b08
https://github.com/mitmproxy/mitmproxy/commit/70e08c880cadc5b87eb5187787a65c7857e0750c
but i also added --no-http2 about an hour after the last one.

is it possible that objects were simply not being cleaned up after those crashes?

There really isn't anything that should keep them around... some cyclic references in the traceback objects, maybe? I am not convinced, but I also don't mind them going away without much effort. 😅 To put a positive spin on this: if they reappear the script from https://github.com/mitmproxy/mitmproxy/issues/4456#issuecomment-780710897 should help us out. :)

you have got me to the point that I'm tunneling all my own traffic through mitmproxy
already losing my newly gained tester positon ;(

the run starting at ~1:50 is with --no-http, at 16:30 i turned http2 back on and loaded the script you provided above.
so:
Memory
=======
4515 <class 'mitmproxy.websocket.WebSocketMessage'>
573 <class 'mitmproxy.certs.Cert'>
563 <class 'mitmproxy.http.Headers'>
378 <class 'mitmproxy.proxy.context.Context'>
342 <class 'mitmproxy.http.HTTPFlow'>
342 <class 'mitmproxy.http.Request'>
342 <class 'mitmproxy.http.RequestData'>
262 <class 'mitmproxy.connection.Server'>
221 <class 'mitmproxy.http.Response'>
221 <class 'mitmproxy.http.ResponseData'>

Flow References
===============
<WebSocketFlow (1109 messages)>
  len(referrers)=2 type(referrers[0])=<class 'set'>
  len(referrers)=1 type(referrers[0])=<class 'dict'>
  len(referrers)=1 type(referrers[0])=<class 'mitmproxy.addons.save.Save'>
  len(referrers)=3 type(referrers[0])=<class 'list'>
  len(referrers)=2 type(referrers[0])=<class 'dict'>
  len(referrers)=1 type(referrers[0])=<class 'mitmproxy.addonmanager.AddonManager'>
  len(referrers)=49 type(referrers[0])=<class 'dict'>
  len(referrers)=1 type(referrers[0])=<class 'mitmproxy.tools.dump.DumpMaster'>
  len(referrers)=36 type(referrers[0])=<class 'dict'>
  len(referrers)=1 type(referrers[0])=<class 'module'>
  len(referrers)=34 type(referrers[0])=<class 'dict'>
  len(referrers)=2 type(referrers[0])=<class 'dict'>
  len(referrers)=1 type(referrers[0])=<class 'module'>
  len(referrers)=198 type(referrers[0])=<class 'builtin_function_or_method'>

Imitation is the sincerest form of flattery. 😋 Jokes aside, really appreciate your super useful bug reports. :)
The log explains a lot. You are probably running mitmdump -w something and the Save addon is simply not discarding errored flows right away. This should be easily fixable! :)

#4461 has the fix for the Save addon. There might be other memleaks, but I don't see any on my system at the moment. If there are, the script should point out the next one. :)

is it plausible that all of that memory is from messages on websocket connections that are still open?
if so, would it be a valid feature for mitmproxy (or probably just a possible plugin) to close close all active websockets and flush their data?
(as it would probably also mean that that data is lost on exit.)
i updated my daily restart script to send SIGUSR1 before sigterm, so this is the memory graph for yesterday, with associated memory stats from just before shutdown.
(graph with gigabyte scale this time)

Memory
=======
16500 <class 'mitmproxy.websocket.WebSocketMessage'>
1347 <class 'mitmproxy.certs.Cert'>
818 <class 'mitmproxy.proxy.context.Context'>
816 <class 'mitmproxy.http.Headers'>
416 <class 'mitmproxy.http.HTTPFlow'>
416 <class 'mitmproxy.http.Request'>
416 <class 'mitmproxy.http.RequestData'>
416 <class 'mitmproxy.proxy.layers.http.HttpStream'>
400 <class 'mitmproxy.http.Response'>
400 <class 'mitmproxy.http.ResponseData'>

Flow References
===============
<WebSocketFlow (2778 messages)>
  len(referrers)=2 type(referrers[0])=<class 'set'>
  len(referrers)=1 type(referrers[0])=<class 'dict'>
  len(referrers)=1 type(referrers[0])=<class 'mitmproxy.addons.save.Save'>
  len(referrers)=3 type(referrers[0])=<class 'list'>
  len(referrers)=1 type(referrers[0])=<class 'dict'>
  len(referrers)=1 type(referrers[0])=<class 'mitmproxy.addonmanager.AddonManager'>
  len(referrers)=1 type(referrers[0])=<class 'dict'>
  len(referrers)=1 type(referrers[0])=<class 'mitmproxy.tools.dump.DumpMaster'>
  len(referrers)=33 type(referrers[0])=<class 'dict'>
  len(referrers)=1 type(referrers[0])=<class 'module'>
  len(referrers)=34 type(referrers[0])=<class 'dict'>
  len(referrers)=2 type(referrers[0])=<class 'dict'>
  len(referrers)=1 type(referrers[0])=<class 'module'>
  len(referrers)=198 type(referrers[0])=<class 'builtin_function_or_method'>

That is at least somewhat plausible, but did you have ~400 active HTTP connections when creating this screenshot? Probably not, so it looks like we still leak somewhere.

if so, would it be a valid feature for mitmproxy (or probably just a possible addon) to have a way to close all active (websocket) connections and flush their data?
(as it would probably also mean that that data is otherwise lost on exit.)

You can probably write a custom addon that iterates through asyncio.all_tasks() and then cancels all connections... but that's a terrible workaround, not a solution. 😅  FWIW we do have a watchdog that closes connections after ten minutes of absolutely no activity.

(as it would probably also mean that that data is otherwise lost on exit.)

No, at least all active flows are written to disk on exit:
https://github.com/mitmproxy/mitmproxy/blob/4bd7b6c4eadeaca712f63e0e73f20bcf6aadbffb/mitmproxy/addons/save.py#L115-L118
Do you have any particular website that you suspect could be leaking here? Maybe you can modify your script to check for  if isinstance(objs[i], http.HTTPFlow): and then print(objs[i].request.url). I'll also see if I can reproduce this locally. This week is super busy for me, so probably not before Sunday :)

thanks for the explanations on handling of flows!

but did you have ~400 active HTTP connections when creating this screenshot?

actually i think i saw 150 client connections when i checked after reading your post, so 400 is not completely unthinkable.
whatever my browsers were doing there - sadly i did not check in detail. ofcourse an excessive number of clients could explain the usage, but i can't imagine what those would be doing, i will re-check.
i added this to the script you provided:
    objs = gc.get_objects()
    for i in range(len(objs)):
        if isinstance(objs[i], http.HTTPFlow):
            print(f"HTTPFlow: {objs[i].request.url}")

at a first look with very little uptime, i find that the top of the list is always https://venus.web.telegram.org/apiw1 (i see that URL all over my logs too, i guess that is from http://web.telegram.org/ ), and the count seems to be slowly but steadily growing. but i don't think those are possibly large enough to account for the usage observed.

it got killed by my script due to exceeding 2.5gb, 10 hours later.
in this case it's kind of obvious that some tab with twitch.tv open decided to start playing a video.
so as for that, this is possibly related to #4470 ... i don't have stream_large_bodies set, so mitmproxy buffers the whole stream in memory? for what i remember, it's actually possible to watch twitch streams through mitmproxy, as it loads seperate video segments in separate http requests, only latency is a bit higher than ususal. (but in this case all http request point to the exact same url, i will find out what exactly was playing there.)
in any case, telegram and wordpress look weird.
{code}
Memory
36545 <class 'mitmproxy.websocket.WebSocketMessage'>
9801 <class 'mitmproxy.certs.Cert'>
7506 <class 'mitmproxy.proxy.context.Context'>
7456 <class 'mitmproxy.http.Headers'>
3729 <class 'mitmproxy.http.HTTPFlow'>
3729 <class 'mitmproxy.http.Request'>
3729 <class 'mitmproxy.http.RequestData'>
3729 <class 'mitmproxy.proxy.layers.http.HttpStream'>
3728 <class 'mitmproxy.connection.Server'>
3727 <class 'mitmproxy.http.Response'>
{code}
{code}
$ tail -n 10024 2021-03-02c.log | tac | sed -n '1,/Flow References/p' | grep ^HTTPFlow: | cut -d? -f1 | sort|uniq -c|sort -nr
2156 HTTPFlow: https://venus.web.telegram.org/apiw1
754 HTTPFlow: http://private-wordpress-site/wp-admin/admin-ajax.php
79 HTTPFlow: https://gql.twitch.tv/gql
68 HTTPFlow: https://embeds.driftcdn.com/embeds/***.json
63 HTTPFlow: https://c.go-mpulse.net/api/config.json
59 HTTPFlow: https://video-edge-b0b5d3.pdx01.abs.hls.ttvnw.net/v1/segment/removed.ts
59 HTTPFlow: https://static.twitchcdn.net/config/manifest.json
6 HTTPFlow: https://connectivitycheck.gstatic.com/generate_204
{code}
(the references list seem to be missing because the script did not give mitmproxy enough time to log them before killing it.)

@r00t- I have stream_large_bodies set with stream_large_bodies: "1k" to enhance my comparable setup.
Any obvious use cases I can try to reproduce?

@hazcod:
setting stream_large_bodies is not an option for me, as i do want all data logged.
so if it's that, i rather live with the memory usage.
i could never pinpoint a specific site that causes or reproduces this.
the last stats dump i posted, showing even wordpress stuff sticking around made it seem to be a more general issue,
but i have no idea about the details of this.

I am able to reproduce with :
% curl -x localhost:3128 -v -o ~/Downloads/1gb.bin -k https://speed.hetzner.de/1GB.bin

mitmproxy will get killed in my case when memory usage raises above 1GB, which happens every time I try download this file.

that is a different issue, or rather expected behaviour.
mitmproxy holds any stream in memory until exit,
mitmdump until it writes it our to disk.
this issue is about mitmdump's memory usage growing over time,
while the number of active streams should not be growing.

@mhils there is any pointers you can give me to how to solve or debug this problem?
I do have a constant running mitmproxy instance handling hundreds of connections, and I had to restart it daily because the memory do not stop growling

Thanks @solovievv! The problem is that we keep all streams for a HTTP/2 connection around until the client disconnects, and some clients just keep using a single connection for a long time... That's not to blame the clients though, we "just" need to implement some logic to delete completed streams while the connection is still active.

The problem is that we keep all streams for a HTTP/2 connection around until the client disconnects, and some clients just keep using a single connection for a long time...

@mhils Could you explain why keepalive_expiry=5 mins doesn't help? As I understand it should close the connection every 5 mins
Is it possible to delete flows older than 5 mins(or any defined value) by a scheduler for example? It can be an additional option flow_lifetime.
mitmproxy is a great tool and solving this issue will allow using it in high loads. Please let me know what you think about it. Possibly I can help somehow.

Could you explain why keepalive_expiry=5 mins doesn't help?

My guess would be that this five minute timer is reset after every request, the connection would only be expired after 5 mins of inactivity (which never happens).
Deleting old/completed flows is on my to-do list, it's just a somewhat nontrivial change that requires some thought. https://github.com/mitmproxy/mitmproxy/issues/4786#issuecomment-928002790 has another workaround. If not using keepalive causes performance issues, you can maybe do it randomly for every Nth flow.

I have the same (or similar) issue.
I start the proxy with mitmproxy --set stream_large_bodies=1 --set  command_history=false --view-filter '!~all' (tried to turn everything off which looked to keep state/history inside the process).
After this, RSS size of the process is 59888 bytes.
After running this:
import requests

proxies = {
   'http': 'http://localhost:8080',
   'https': 'http://localhost:8080',
}

for i in range(1000):
    requests.get("http://ftp.fsn.hu/README")

the RSS grows to 73516 bytes and grows constantly with each HTTP requests.
Is it possible to use mitmproxy as a constantly running service without storing (meta)data from each flows getting through it?

@bra-fsn: view_filter only controls which flows are currently visible. mitmproxy always keeps all flows in memory. If you want to avoid that, use mitmdump (which discards after completion) or manually clear the flow list within mitmproxy.

@bra-fsn: view_filter only controls which flows are currently visible. mitmproxy always keeps all flows in memory. If you want to avoid that, use mitmdump (which discards after completion) or manually clear the flow list within mitmproxy.

I can see the same with mitmdump. RSS grows with each completed requests.

@bra-fsn: Can I ask you to try the memoryleak branch (https://github.com/mitmproxy/mitmproxy/tree/memoryleak, https://mitmproxy.org/downloads/#branches/memoryleak/)?
rss stabilizes after a few minutes for me. I'm using your example and mitmdump -q -s script.py:
import gc
import os
import psutil

def response(flow):
    gc.collect()
    print("rss", psutil.Process(os.getpid()).memory_info().rss)

@bra-fsn: Can I ask you to try the memoryleak branch (https://github.com/mitmproxy/mitmproxy/tree/memoryleak, https://mitmproxy.org/downloads/#branches/memoryleak/)?
rss stabilizes after a few minutes for me. I'm using your example and mitmdump -q -s script.py:
import gc
import os
import psutil

def response(flow):
    gc.collect()
    print("rss", psutil.Process(os.getpid()).memory_info().rss)

Sure. Starting the proxy with PYTHONPATH=. python release/specs/mitmdump --set stream_large_bodies=1 from branch memoryleak gives an initial size of 54640.
After 10.000 queries its RSS shows 59120.
If I add -s /tmp/script.py, initial: 55568, after 10k: 56544.
If I use the main branch without the script:
initial: 54600, 10k: 118152
with the script:
initial: 55484, 10k: 117708
And the response times deteoriate pretty much (the last request is more than the double of the first one) with the number of completed requests.
Both branches are actual ATM.

If I add -s /tmp/script.py, initial: 55568, after 10k: 56544.

This pretty much seems like it fixes it, or would you disagree?

And the response times deteoriate pretty much (the last request is more than the double of the first one) with the number of completed requests.

This is fixed on the memoryleak branch as well, correct?

@mhils I'm super excited by this. Thank you!
(BTW, I previously spent a couple of hours debugging to discover the leak. I was planning to do another round using the https://github.com/bloomberg/memray released a few days ago. If you don´t know about it, may be helpful).

If I add -s /tmp/script.py, initial: 55568, after 10k: 56544.

This pretty much seems like it fixes it, or would you disagree?

And the response times deteoriate pretty much (the last request is more than the double of the first one) with the number of completed requests.

This is fixed on the memoryleak branch as well, correct?

Yes to both!

Nice one, we also just had a bunch of headache to debug this! Thanks for all the notes and work on this.

@mhils I think this alone deserves a new release. :)

mitmproxy 8.1 is out now. :)


listener: Use v4-mapped addressing consistently.

Currently the local address of a new connection is mapped to an IPv4
address instance if the listening socket is not in V6ONLY mode and the
local address of the socket is an IPv4-mapped IPv6 address. This leads
to connections having an (IPv4-mapped) IPv6 instance for the remote
address and an IPv4 instance for the local address, which is
confusing. Fix this by allowing the remote address to be v4-mapped if the
local address is an IPv4 address. Or conversely, set the 'v6only'
parameter to 'true' if the local address is an IPv6 address instance.
Fixes: 3b3c0094d098 ("address_impl: Return an Ipv4Instance for v4-mapped v6 addresses. (#2309)")
Signed-off-by: Jarno Rajahalme<EMAIL_ADDRESS>Risk Level: Low

Added a test case that fails without the fix.

@zuercher can you take a first pass on this? Thanks.

Sorry, I read the wrong list of names. @htuch I think this is ready for you to look at.

Format fixed, sorry for the churn.

No idea why it shows I've dismissed your (stale) review. I just pushed the fix for the line you commented on.

Yeah, that's normal GitHub :) it shows dismissed stale review, because you updated a line that had a review comment so it needs to be reviewed again  ¯_(ツ)_/¯


Find new term for report

Jaishri finds "report" as a catch all term for powerpoints, pdfs, certificate of analysis, etc, confusing because it's confused with a milestone report. We can: be more granular with the resourceType, or come up with a different catch-all term. Maybe it would be good to be slightly more granular.

Going to close this as we currently have result and split last comment into separate enhancement issue.


Moar team meta

Adds the following properties to message.meta, populated by the corresponding headers.

bot_user_name => bb-slackbotusername
team_name => bb-slackteamname
team_domain => bb-slackteamdomain

Coverage remained the same at 100.0% when pulling ec636e802560ecf1159deb8c9ebaf620270f6e3f on moar-team-meta into 6270d32ed909af2c224ef40d811e72e7fef71f25 on master.

👍


IAC-1268: fix docs

Describe your changes
Comment Triggers

PR Check triggers

Build: trigger build
Sub Category Field Check: trigger subcategoryfieldcheck
gitleaks: trigger gitleaks

This pull request has been automatically closed due to 30 days of inactivity. If you believe this was done in error, please feel free to reopen the PR or reach out to the BT team for further assistance. Thank you for your contributions!


Lossless JSON

Implements Lossless JSON encoding/decoding.
For now, used only in RaidenState/persistency, but we need to keep consistency with messages encoding, to avoid needing multiple codecs for each type to support different serialization methods.
Based on top of #193

TODO: remove specific types after https://github.com/josdejong/lossless-json/pull/10 is merged and released


rambox: 0.4.5 -> 0.5.2

Motivation for this change
version bump, new checksums.
also, the binary was renamed in the new download (from Rambox to rambox).
Things done

[ ] Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] macOS
[X] Linux

[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[X] Tested execution of all binary files (usually in ./result/bin/)
[ ] Fits CONTRIBUTING.md.

@gnidorah: I won't be near a nixos machine until monday, so I'll close this to allow you to send a new PR.

@phryneas Done, thanks! I've preserved you as commit author ;-)

Very cool, thanks ^^


Create a test workflow

A new PR should create a new heroku app with a unique endpoint
The new app with the proposed code should be tested against a repository
The new app should be destroyed by the PR being merged.

Related - https://github.com/OrkoHunter/pep8speaks/issues/46

Step 1: Log in to Heroku and deploy the latest Pull Request.

Created a test app where the payload URL can be updated for testing each PR. The test app has been installed on the https://github.com/OrkoHunter/test-pep8speaks repository.

Using a Probot app called auto-comment, to comment the details of testing when a new PR is created.

Check out Probot apps! They are awesome - https://probot.github.io/apps/

Hello @OrkoHunter! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file tests/pep8speaks/test_workflow.py:

Line 15:1: E302 expected 2 blank lines, found 1

Finally! Our CI will test the core features on incoming Pull Requests.


Feature Request: Proxy requests through a specified server

Is your feature request related to a problem? Please describe.
My Okta org requires MFA when connecting from outside our network (eg: on an EC2 instance), making okta-awscli depend on answering a push notification on my phone. This works but  when CICD jobs are triggered periodically I have to answer notifications that I didn't originate (bad security practice).
Describe the solution you'd like
Allow users to proxy requests to Okta for secure credentials through a specified server in their organization's network to avoid MFA requirements when possible (after an appropriate warning). This would reduce friction in automated awscli uses such as CICD systems.
Add argument: okta-awscli --okta-profile $profile_name --proxy-server <IP_ADDRESS>:1234
Describe alternatives you've considered
Currently I use Okta app based MFA, and it has the limitations described above.
Additional context
Any suggestions are appreciated. Thanks for considering!

Dunno if it helps, but was skimming through the issues and saw this one.
If you play your cards right, maybe the technique outlined in the links below in unison could help you?
https://github.com/boto/boto3/issues/338 and https://2.python-requests.org/en/master/user/advanced/#proxies
TL;DR - set proxies env var and see if that helps

I'll have to give that a try once my org sets up an on-prem proxy. Thanks!

Sorry for the late reply here. @arahayrabedian's suggestion makes sense to me. Since okta-awscli is just using boto3 and requests under the hood, setting HTTPS_PROXY should be sufficient to cover everything.
You could even do it as a one-liner like HTTPS_PROXY=https://<IP_ADDRESS>:5678; okta-awscli ...
Going to close this for now, but if people find that the proxy envvar solution doesn't work, we can revisit building that into the project itself.


Perhaps xcodebuild clean should run before xcodebuild

Hey guys,
I've been seeing a weird issue when installing XCActionBar via Alcatraz where the build process will fail to copy over the required nib files.
I doesn't seem like you run xcodebuild clean before re-building the project, and doing so seems to solve the issues I'm seeing - wouldn't it make sense to add that step? Any particular reason why you're not going it?
Thanks.

Hi,
have the same issue with my plugin BBUncrustifyPlugin.
Steps to reproduce the issue:

Build the plugin from the sources.
The xib should be copied as expected: ~/Library/Application Support/Developer/Shared/Xcode/Plug-ins/UncrustifyPlugin.xcplugin/Contents/Resources/XCFPreferencesWindowController.nib
Remove the plugin in Alcatraz.
Reinstall the plugin in Alcatraz.
The xib file is now missing.
If I build again the plugin from the sources, the xib is no more copied. I need to peform a "Clean Build Folder" in Xcode in order to see the xib copied correctly.

As @pdcgomes said, I think that cleaning is needed because Xcode fails to do the incremental copy.

Fixed in #261


Add windows host enhancement

Rebased PR of https://github.com/rancher/cattle/pull/3117

https://github.com/rancher/rancher/issues/10442
Add ins go-agent and agent image: https://github.com/rancher/rancher/issues/10448
Add ins per-host-subnet: https://github.com/rancher/rancher/issues/10447


CLI command "dcos  connection" is broken

It seems like the /v1/connection endpoint used by the CLI client is no longer available.
At least I see the following error when trying to use the command:
$ dcos mongodb-replicaset connection
2017/01/11 00:58:36 HTTP GET Query for http://<IP_ADDRESS>/service/mongodb-replicaset/v1/connection failed: 404 Not Found

As the /v1/connection endpoint is also not listed in the Swagger API docs, and I can't find a API resource class under com/mesosphere/sdk/api, I suspect that this was replaced by the (yet undocumented) /v1/endpoints endpoint.

Sorry, I just saw that this is already implemented, but not yet activated:
https://github.com/mesosphere/dcos-commons/blob/master/cli/commands.go#L57
The ConnectionResource seems to have been removed though, and is apparently no longer usable.

This was in an odd state for a while where the connections endpoint was implemented in the common CLI code while only derived services (Cassandra/Kafka/HDFS) were implementing it as a custom resource. This became a problem when the dcos-commons repo started having frameworks of its own which didn't have that endpoint implemented yet.
The longer term solution is to have everything use the new Endpoints resource that's meant to replace all the current custom Connection implementations with a single universal version, but for now the Connection CLI command should just be removed from the common CLI in favor of having it manually added to CLIs of services that still implement the endpoint.

@nickbp Thanks for the fast feedback. Sorry, but I didn't really understand why you can't activate the /endpoints resource in the CLI... Currently it's, as you said, in an odd state, because the ConnectionResource class was already removed.
I was trying to do this on my branch, and use it in my framework, but somehow failed. I guess during the go compile step, it still uses the sources from the dcos-commons GitHub repo.

Oh I see. You should avoid calling HandleCommonArgs as it's due to be removed, and instead add the sections you need directly, see TODO for that function. This is because of how golang works when you aren't manually using vendoring: Changes made in golang code in master IMMEDIATELY take effect in any future downstream builds, so you can end up with new sections in your golang CLI that your Java code doesn't yet implement.

For an example see: https://github.com/mesosphere/dcos-kafka-service/commit/204983ced63c5f1e985af93b14ddb57decb92f5e

FWIW this isn't as much of a problem for services that are in the dcos-commons repo as the Java code is kept up to date in lockstep with the rest of the repo, but it can cause problems when the Java code is in a separate repo and the Java code is referencing a specific dcos-commons Java release.
In theory the real solution would be versioning of the golang CLI alongside the Java library versioning but that hasn't been done yet. I'll file an issue about finding a solution as it's been a problem for a while.

@nickbp Thanks again, and for keeping track of this! So, I'll try to modify my code at https://github.com/tobilg/dcos-commons/blob/mongodb-replicaset/frameworks/mongodb-replicaset/cli/dcos-mongodb-replicaset/main.go#L15 as in the example you linked 👍

@nickbp I changed my CLI implementation now as you hinted. If you have time, please have a look at https://github.com/tobilg/dcos-commons/blob/mongodb-replicaset/frameworks/mongodb-replicaset/cli/dcos-mongodb-replicaset/main.go
It didn't change anything though, the endpoints command is still not available. I left everything in there as-is, compared to dcos-commons master.
I'm really puzzled though how I can fix this behavior. This is the CLI output I'm seeing:
$ dcos package install --cli mongodb-replicaset
<IP_ADDRESS> - - [12/Jan/2017 01:35:41] "GET /stub-universe-mongodb-replicaset.zip HTTP/1.1" 200 -
Installing CLI subcommand for package [mongodb-replicaset] version [stub-universe]
New command available: dcos mongodb-replicaset
[vagrant@dcos-docker-sdk mongodb-replicaset]$ dcos mongodb-replicaset --help
usage: mongodb-replicaset [<flags>] <command> [<args> ...]

Provides CLI integration for the mongodb-replicaset package

Flags:
  -h, --help            Show context-sensitive help (also try --help-long and
                        --help-man).
      --version         Show application version.
  -v, --verbose         Enable extra logging of requests/responses
      --info            Show short description.
      --force-insecure  Allow unverified TLS certificates when querying service
      --custom-auth-token=DCOS_AUTH_TOKEN
                        Custom auth token to use when querying service
      --custom-dcos-url=DCOS_URI/DCOS_URL
                        Custom cluster URL to use when querying service
      --custom-cert-path=DCOS_CA_PATH/DCOS_CERT_PATH
                        Custom TLS CA certificate file to use when querying
                        service
      --name="mongodb-replicaset"
                        Name of the service instance to query

Commands:
  help [<command>...]
    Show help.

  config list
    List IDs of all available configurations

  config show <config_id>
    Display a specified configuration

  config target
    Display the target configuration

  config target_id
    List ID of the target configuration

  connection [<type>]
    View connection information (custom types: foo, bar)

  plan active
    Display the active operation chain, if any

  plan show
    Display the full plan

  plan continue
    Continue a currently Waiting operation

  plan interrupt
    Interrupt the current InProgress operation

  plan force <phase> <step>
    Force the current operation to complete

  plan restart <phase> <step>
    Restart the current operation

  state framework_id
    Display the mesos framework ID

  state status <name>
    Display the TaskStatus for a task name

  state task <name>
    Display the TaskInfo for a task name

  state tasks
    List names of all persisted tasks

Nevermind, this was due to an outdated version of the subcommand in home/vagrant/.dcos/subcommands... What I think is strange is that the actual subcommand binary doesn't seem to be replace by running
dcos package install --cli mongodb-replicaset
Is this the desired behavior?

I'd guess the CLI is refusing to re-install (new builds of the) module if it's already installed. In theory they should have the SHA256 available to check for changes. I'd file a bug against github.com/dcos/dcos-cli/