Can't download videos when self hosting.

Self hosting on a raspberry pi 4 running ubuntu 20.04, both transcoding and downloading from youtube do not work. I've verified that the paths to ffmpeg and youtube-dl are correct, and I've installed all requirements. The error messages from the terminal are below.
Downloading youtube video

[2020-07-26 16:07:41,451] ERROR in app: Exception on /yt [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/ubuntu/AudioAndVideoConverter/yt.py", line 122, in yt_downloader
user_ip = request.environ['HTTP_X_FORWARDED_FOR']
KeyError: 'HTTP_X_FORWARDED_FOR'
<IP_ADDRESS> - - [26/Jul/2020 16:07:41] "POST /yt HTTP/1.1" 500 -

Converting to MKV

<IP_ADDRESS> - - [26/Jul/2020 16:05:37] "POST / HTTP/1.1" 200 -
<IP_ADDRESS> - - [26/Jul/2020 16:05:37] "POST / HTTP/1.1" 200 -
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'uploads/YMCA_Spider.mp4':
Metadata:
major_brand     : isom
minor_version   : 512
compatible_brands: isomiso2avc1mp41
encoder         : Lavf58.29.100
Duration: 00:01:11.03, start: 0.000000, bitrate: 410 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 460x258 [SAR 1:1 DAR 230:129], 186 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
Metadata:
handler_name    : ISO Media file produced by Google Inc.
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 217 kb/s (default)
Metadata:
handler_name    : ISO Media file produced by Google Inc.
Output #0, matroska, to 'conversions/YMCA Spider.mkv':
Metadata:
major_brand     : isom
minor_version   : 512
compatible_brands: isomiso2avc1mp41
encoded_by      : "free-av-tools.com"
comment         : "free-av-tools.com"
encoder         : Lavf58.29.100
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 460x258 [SAR 1:1 DAR 230:129], q=2-31, 186 kb/s, 25 fps, 25 tbr, 1k tbn, 12800 tbc (default)
Metadata:
handler_name    : ISO Media file produced by Google Inc.
Stream #0:1(und): Audio: aac (LC) ([255][0][0][0] / 0x00FF), 48000 Hz, stereo, fltp, 217 kb/s (default)
Metadata:
handler_name    : ISO Media file produced by Google Inc.
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame= 1775 fps=0.0 q=-1.0 Lsize=    3538kB time=00:01:11.00 bitrate= 408.2kbits/s speed= 974x
video:1617kB audio:1885kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.042236%
<IP_ADDRESS> - - [26/Jul/2020 16:05:37] "POST / HTTP/1.1" 200 -
[2020-07-26 16:05:37,831] ERROR in app: Exception on /conversions/YMCA Spider.mkv [GET]
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functionsrule.endpoint
File "main.py", line 222, in send_file
user_ip = request.environ['HTTP_X_FORWARDED_FOR']
KeyError: 'HTTP_X_FORWARDED_FOR'
<IP_ADDRESS> - - [26/Jul/2020 16:05:37] "GET /conversions/YMCA%20Spider.mkv HTTP/1.1" 500 -

I know what the issue is. I'm already in the process of making some changes to the code, so I will also make the changes that will prevent this issue from occurring. This should be done within an hour, I will let you know when I have pushed the changes and then I would like you to re-clone this repository and try again.

Done. Re-clone this repository and follow the instructions here. Let me know if you run into any issyes.

Working fine now, I've managed to download several mp3's just fine. I did try downloading an hour long mp3 and it has been adding a thumbnail for 20 minutes now, checking htop tells me its at 100% cpu (out of 400), no throttling, just ffmpeg at 100% usage. The file is an mp3 and around 62 megabytes. Terminal outputs many of this message;

<IP_ADDRESS> - - [27/Jul/2020 18:57:15] "GET /yt-progress/1595875106.txt HTTP/1.0" 304 -

Killing and restarting the process hasn't done anything.
I've just removed the 'embed--thumbnail' option in yt.py for mp3's and waited about five minutes. The ffmpeg process then died, but the webpage still says "Initializing" and the terminal pushes the same message as above.

@Rick1029 I would like you to re-clone and try again as I have just pushed some changes. It's best to be running the same piece of code when diagnosing an issue. If you still have an issue with the hour-long video, let me know and share the link if you don't mind.

Also, uninstall your current installation of youtube-dl using pip/pip3 uninstall youtube-dl, then install the latest version by following the instructions here. The version of youtube-dl that was installed with pip/pip3 install -r requirements.txt currently has an issue with downloading videos, so I have removed it from requirements.txt


Profile images

Hey Sara,
I've given it a touch to the profile image and added two new ones. I still have the SVG files if you want to change the colors or do something similar.

Hello @uzluisf
thank you very much for your help. I really do like these two:

Would it be possible to make the tie instead of checked, completely colored? Then it would be more neutral. A PR would be really great!

That sounds great, @salomonelli!
One thing I noticed is that with the original image, the actual resolution (719x719) doesn't seem to be the most optimal and this is more noticeable with templates (e.g. material-blue) that seems to enlarge the image.  Do you want to use a higher resolution?  Or just keep it as before?

No you can enlarge them of course. That would be great!


[Ansible] Error to fetch Linux binaries to the ansible host

Could you help me, please?
I am trying to execute the playbook and I am receiving an error message.
I tried to increase the memory in the hosts, I tried to execute the playbook using --become and --become-user=root, but the problem continues.
This is the message:
TASK [linux/kubernetes : Kubernetes bins | fetch Linux binaries to the ansible host] ***********************************************************************************************************
ok: [node1] => (item=kubectl)
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 12] Cannot allocate memory
fatal: [node1]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}
The full traceback:
<<IP_ADDRESS>> ESTABLISH SSH CONNECTION FOR USER: ubuntu
<<IP_ADDRESS>> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ae5b4174cc <IP_ADDRESS> '/bin/sh -c '"'"'rm -f -r /home/ubuntu/.ansible/tmp/ansible-tmp-1544312243.77-250979832628827/ > /dev/null 2>&1 && sleep 0'"'"''
The full traceback is:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 104, in run
item_results = self._run_loop(items)
File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 326, in _run_loop
res = self._execute(variables=task_vars)
File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 558, in _execute
result = self._handler.run(task_vars=variables)
File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/action/fetch.py", line 216, in run
self._remove_tmp_path(self._connection._shell.tmpdir)
File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/action/init.py", line 312, in _remove_tmp_path
tmp_rm_res = self._low_level_execute_command(cmd, sudoable=False)
File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/action/init.py", line 902, in _low_level_execute_command
rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/connection/ssh.py", line 984, in exec_command
(returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/connection/ssh.py", line 258, in wrapped
return_tuple = func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/connection/ssh.py", line 878, in _run
return self._bare_run(cmd, in_data, sudoable=sudoable, checkrc=checkrc)
File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/connection/ssh.py", line 633, in _bare_run
p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "/usr/lib/python2.7/subprocess.py", line 711, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1235, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
fatal: [node1]: FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}

I think this is due to an out-of-memory error. Possible duplicate #494 which is being addressed by pull request #506

I think this can be closed.


include pcre into the next release

It seems that roughly half of the systems don't have pcre installed:
http://matrix.cpantesters.org/?dist=Nginx-Perl+<IP_ADDRESS>
So I think it's better to just ship pcre with nginx-perl for cpan and build it for those systems.

So, I guess travis-ci doesn't have PCRE either.

Including pcre was a bad idea. So, I'm simply disabling pcre and zlib if not found. This helps to success on travis and probably will succeed on many cpantesters' nodes as well.


Research and Share information about making renderered charts reactive, based on the container that they're in.

de-priortize.  On hold.

Nice to have, this has to be rendered on a larger screen,  we don't support small screens.


Process Territory Battles Settings from Spreadsheet

Set up data structures necessary for processing territory battles settings from spreadsheet

#18 pull request


ConfigMap as module source

What problem are you facing?
My kubernetes cluster does not have network access to my git server.
Concatenating all of the tf files into a single string and using it with Inline is my only option today that doesn't try to pull the module source from a system outside kubernetes (unless I host tar.gz-d modules in-cluster).
How could Official Terraform Provider help solve your problem?
A more kubernetes-native way would be to support terraform modules stored in configmaps.
If one could define module source to come from a configmap, then it would be very easy to package up an existing terraform module into kubernetes manifests and deploy them like any other kubernetes resource.
This would be a welcome improvement over Inline module sources.
Example:
kustomization.yaml:
configMapGenerator:
- name: my-terrfaorm-module
  files:
  - main.tf
  - variables.tf
  - outputs.tf
  - versions.tf

kustomize build | kubectl apply -n crossplane-system -

workspace.yaml:
apiVersion: tf.upbound.io/v1beta1
kind: Workspace
metadata:
  name: my-terraform-module
spec:
  forProvider:
    source: ConfigMap
    module:
      name: my-terraform-module
      namespace: crossplane-system

The feature makes sense! Thanks for documenting it.

I just use Helm. File.Get loop to include all files inside the inline. Therefore we just copied our current TF structure into folder "terraform" inside a chart. This enables normal TF work if you want to (assuming you extend provider.tf with configuration for the real cloud, provider.tf to be include has to be ONLY for aliases, not for the provider configured in crossplane ProviderConfig).
{{/*
Helper to setup Terraform Files

Include like this to ensure proper formatting:

{{ include "compositionTerraform.include" . | trim | indent 14 }}

*/}}
{{- define "compositionTerraform.include" -}}
{{- $currentScope := .}}
{{- with $currentScope}}
{{- range (tuple "provider" "variables" "locals" "data" "main" "outputs") }}
{{- $file := $currentScope.Files.Glob (printf "terraform/%s.tf" .) }}
{{- if $file }}
{{ $currentScope.Files.Get (printf "terraform/%s.tf" .) | trim }}
{{ end -}}
{{- end }}
{{- end }}
{{- end }}

Thanks @alkuzad, that sounds like a reasonable alternative.

this would be extremely beneficial for us as well

Yes, I would like this to ...
or atleast the possibility to have a way of including the TF into the configuration once I build/bundle crossplane into a OCI image (sry I am not sure of the correct terminology here )

Since this request started with a kustomize example, another workaround is to use replacements to substitute a field from a ConfigMap into the module field.
If you have a single file, you'd do something like:
# kustomization.yaml

configMapGenerator:
  - name: my-terraform-module
    files:
      - main.tf

replacements:
  - source:
      fieldPath: data.[main.tf]
      kind: ConfigMap
      name: my-terraform-module
      version: v1
    targets:
      - fieldPaths:
          - spec.forProvider.module
        select:
          group: tf.upbound.io
          kind: Workspace
          name: my-terraform-module
          version: v1beta1

resources:
  - workspace.yaml

# workspace.yaml

apiVersion: tf.upbound.io/v1beta1
kind: Workspace
metadata:
  name: my-terraform-module
spec:
  forProvider:
    source: ConfigMap
    module: <Kustomize will replace this with the contents of `main.tf`>

I'm not entirely sure how you'd do this with multiple files, but maybe there's some way to concatenate those files into a single string with some other generator/transformer? The vars field seems like it would make that possible, but it's deprecated.


[QUERY] how to use criteria builder  in cosmos

Hi,
I am trying to find how to use criteria builder in cosmos but i didn't find anywhere.
I am working to create dynamic query if any other options there please tell.
my requirement is i need to call 4 parameters but need to be work on any a combinations like with one param or two params or all 4 params
Thank you

@s998899 - which SDK are you using ? and what version ?


Add YamlLoader::load_from_str_with_markers to provide AST with source markers

fixes #103, replaces #105
The new function produces an alternative representation for YAML
documents where each YAML node is paired with a Marker to indicate the
corresponding line and column in the source markup. The new
representation takes the form of two new types, Node and YamlMarked.
Node is a pair of YamlMarked and Marker.
YamlMarked mimics the existing Yaml enum; the difference is that array
elements and hash keys and values are Node values instead of Yaml or
YamlMarked values. I created a new enum because I did not know of a way
to switch child nodes in Yaml between Yaml and Node types without
backward-incompatible changes to the Yaml enum.
The the behavior of the existing load_from_str function and Yaml
enum are unchanged, so pattern matching on results from load_from_str
will work as before.
To ensure consistent behavior for the Node and Yaml I moved methods
from the impl Yaml block to a new trait called YamlNode which is
implemented by Yaml, Node, and YamlMarked. This is a breaking
change since it means that consumers will have to import YamlNode to
use methods like .as_str() and .is_array().
I want to present this pull request as one proposal. I think there is also
an argument for changing the existing Yaml type to incorporate source
location markers instead of maintaining two parallel enums.
While making changes I split up yaml.rs into three nested modules.
I can put it back the way it was if that is preferable.

So I used the derivative crate to define Hash and PartialEq implementations for Node. Derivative uses procedural macros, which breaks compatibility with Rust v1.24. The reason I did that is that derivative makes it easy to ignore certain struct fields in the hash and equality implementations. Because Node values are used as hash keys, and YAML's hash keys do not differ based on source location, the source location marker in the Node struct should be ignored when comparing Node values.
If you want to maintain compatibility with earlier Rust versions I can make changes to remove derivative, and implement Hash and PartialEq by hand.


fix: fix build

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


add and Fix eror with Format (Data endpoint )

What does this PR do?

Before submitting / marking as 'ready to review'

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

Who can review?
Feel free to tag members/contributors who may be interested in your PR.

Add Data_endpoint


refactor(java11) - convert most modules to java 11

This PR is one of two Java 11 options (other) - only merge one. This version converts modules to Java 11 where they are not tied to spark-lineage which requires Java 8. Unlike the other PR, there are no duplicate modules for Java 11 and Java 8, only one java version is chosen based on its usage by spark-lineage
All modules except these are converted to java 11:
li-utils
metadata-models
test-models
metadata-integration:java:datahub-client
metadata-integration:java:spark-lineage
Checklist

[x] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[x] Links to related issues (if applicable)
[x] Tests for the changes have been added/updated (if applicable)
[x] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[x] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Hello @leifker
Thank you for tackling this large effort! It seems you have some failing checks in CI. The spark smoke tests seem to be real, the rest are flakes.
More generally...
What is the intended objective with this PR (or it's alternative)?
We have another smaller, more contained PR that is intended to upgrade the java runtime for all DataHub's components to Java 11 with the objective of making our images more cloud native and use more recent optimisations.
Is this your intended goal? If not, could you please elaborate?

Hello @leifker
Thank you for tackling this large effort! It seems you have some failing checks in CI. The spark smoke tests seem to be real, the rest are flakes.
More generally... What is the intended objective with this PR (or it's alternative)?
We have another smaller, more contained PR that is intended to upgrade the java runtime for all DataHub's components to Java 11 with the objective of making our images more cloud native and use more recent optimisations. Is this your intended goal? If not, could you please elaborate?

@pedro93 - The intent is to build any, or all, modules with jdk11. The docker images would need to updated to jdk11 as a requirement to run the jars generated with jdk11. Only certain modules would require jdk8, specifically those for supporting spark-lineage. For this later case, those specific modules can either be compile for jdk8 or for both jdk8 and jdk11 (this is the difference between the two PRs).
The failing tests seem related to other changes to the browse path that are outside of these changes. I will make a few updates to get the tests to pass and continue to update with the main branch to incorporate on-going changes. There is a thread here which outlines what I see with respect to changing browse paths (which occurs in the main branch).

@pedro93 - The intent is to build any, or all, modules with jdk11. The docker images would need to updated to jdk11 as a requirement to run the jars generated with jdk11. Only certain modules would require jdk8, specifically those for supporting spark-lineage. For this later case, those specific modules can either be compile for jdk8 or for both jdk8 and jdk11 (this is the difference between the two PRs).

My question is more, what is the benefit of compiling datahub code with java 11 jdk? byte code improvements? Java 11 language features?
Following https://github.com/datahub-project/datahub/pull/5875 all datahub images will run in Java 11 runtime with java 8 byte code.

There are a couple benefits of compiling with jdk11 where possible.
One of them are language features for sure. For example, :metadata-integration:java:datahub-protobuf, in the project is using jdk11 and is not compatible with jdk8. I wrote that module before realizing the jdk8 dependence of the project.
I think that the most important reason is ultimately that many libraries are deprecating jdk8 support and without updating to jdk11 this locks DataHub into older versions of libraries. In turn, these older libraries may not contain important security fixes and newer functionality.


Namespace is not removed when parse xml with lxml library

Original discussion in https://groups.google.com/forum/#!topic/robotframework-users/pNdU51HXnQc
When using lxml library and parse xml, namespaces are included when you get element from specific xpath. This does not occur when use standard ElementTree module (aka. use_lxml=False)
Can be reproduce by following data
<feed xmlns:verr="http://www.xyz.com/error" xmlns:vaext="http://www.xyz.com/atom-ext" xmlns="http://www.w3.org/2005/Atom">
<example>
  <first id="1">text</first>
</example>
</feed>

This is test case
*** Setting ***
Library  XML  use_lxml=True

*** Variables ***
${xml}  test.xml

*** Test Cases ***
test 1
   ${root}=  parse xml  ${xml}
   ${elem}=  get element  ${root}  example/first
   log element  ${elem}

When log element is executed, result is
<first xmlns:verr="http://www.xyz.com/error" xmlns:vaext="http://www.xyz.com/atom-ext" xmlns="http://www.w3.org/2005/Atom" id="1">text</first>

instead of
<first id="1">text</first>

Output of parse xml on lxml and ElementTree library should be the same.
Version information

Robot Framework 3.0
Python 2.7.11
Windows 7

First of all, sorry for ignoring this issue for so long. It was buried under a pile of other issues during summer time when I didn't have much time to follow this tracker.
Including namespace information in child elements seems to be a feature in lxml and avoiding that doesn't seem to be easy. Instead of that, I think it is better to add new strip_namespaces argument to Parse XML to allow stripping namespaces altogether and I already submitted #2534 about that.
With #2534 implemented, the problem in this issue can be avoided easily and I think this issue can be closed. Comment here if you disagreee and have use cases where the new functionality wouldn't work.


Plan for 1.0.0 release

[ ] Add strict mode option to isValid to consider dashes and spaces when validating or not.
[ ] Add node 5 to the test matrix (https://github.com/seegno/tin/blob/master/.travis.yml#L4) or consider using docker-compose run --rm sut.
[ ] Fix README title (Numbers -> Number)
[ ] Invert order of "See" with the introduction (https://github.com/seegno/tin/blob/master/src/index.js#L3).
[ ] Remove default importing in favor of importing the named function isValid* (https://github.com/seegno/tin/blob/master/src/index.js#L12).
[ ] Remove isValid(tin) in favor of using the type only to mask the value (https://github.com/seegno/tin/blob/master/src/index.js#L51).
[ ] Remove the "NOTE:" prefix (https://github.com/seegno/tin/blob/master/src/index.js#L27).
[ ] Remove unnecessary return (https://github.com/seegno/tin/blob/master/src/index.js#L43).
[ ] Rename tinType to type (https://github.com/seegno/tin/blob/master/src/index.js#L55).
[ ] Rename to tin-validator.
[ ] Update package.json (e.g. based on this one), with special focus on scripts.
[ ] Update README to include lodash-like arguments (based on this).
[ ] Update eslint-config-seegno to 2.0.0 and subsequently the rc file, including renaming to yml.

Bump.


Address micromatch cspell lock update

Resolved the setuptools CG alerts for azure-sdk-for-python, this PR addresses the remaining moderate CG issue in the sdk-for-python. If it's hitting us I'm certain the other repos are seeing it too.
Just a package lock update.

The following pipelines have been queued for testing:java - templatejava - template - testsjs - templatenet - templatenet - template - testspython - templatepython - template - tests You can sign off on the approval gate to test the release stage of each pipeline. See eng/common workflow


3.0: only expect non-filtered Async tests to be reported as completed

We have noticed the SuiteCompleted event is not fired to the reporters when there is an ignored test or a filtered test in an ASync test suite.
After investigation, it seems that the TestSortingReporter has a completedTestCount which is tracked off against the tests executed, however tests ignored don't count towards this total.  The ignored/excluded tests are added to the suite as tests with an ignore tag, so we need to also use the filtering that we use during execution when we create the total.
I was also considering just making the TestIgnored handler in the TestSortingReporter add one to the completed count, however that seemed like the wrong fix.  It did work though.
The overall aim is to fix the problem we are seeing with our test run where lots of test status output appears after the test run finished, meaning it's hard to tell whether the run passed or failed.
https://github.com/guardian/support-frontend/pull/2555 is the original issue workaround.
I have targeted this against 3.0.x but I notice the same bug appears in the main dev branch. Happy to raise a fix there as well if it helps?
Also, I haven't added any tests, I am not sure if that is required (being scalatest you'd expect so!!) and if so where they would go.

@cla-bot[bot] check


Upgrade to pyproject.toml and pypa/gh-action-pypi-publish

https://til.simonwillison.net/python/pyproject
https://til.simonwillison.net/pypi/pypi-releases-from-github

Tried the in-development version of this to build datasette-test:
./LICENSE
./pyproject.toml
./tests
./tests/test_datasette_test.py
./README.md
./setup.py
./.gitignore
./datasette_test
./datasette_test/__init__.py
./.github
./.github/workflows
./.github/workflows/publish.yml
./.github/workflows/test.yml

Annoying: https://github.com/simonw/python-lib/actions/runs/7546633197/job/20544794655

I think I can fix that by manually editing that workflow myself.

Yeah I need to run this manually on my own laptop to help solve this:

git clone git@github.com:simonw/python-lib-template-demo
git clone git@github.com:simonw/python-lib
mv python-lib-template-demo python-lib-template-demo-old
cat python-lib/input-for-demo.txt | cookiecutter python-lib
# Now python-lib-template-demo exists as well
mv python-lib-template-demo-old/.git python-lib-template-demo
cd python-lib-template-demo
git diff
# Looks good
git commit -a -m 'Updates'
git push

That seemed to work - here's the new example output, which passes its tests: https://github.com/simonw/python-lib-template-demo/tree/8ca44689

Next test: use https://github.com/new?template_name=python-lib-template-repository&template_owner=simonw to create a datasette-test package and then ship it!

The test.yml workflow in that is completely wrong - full of setup.py references: https://github.com/datasette/datasette-test/blob/8d5f8262dc3a88f3c6d97f0cef3b55264cabc695/.github/workflows/test.yml

I'm really confused. https://github.com/simonw/python-lib/blob/main/{{cookiecutter.hyphenated}}/.github/workflows/test.yml has the correct stuff in it. Why did datasette-test get the old version?

Oh! That's because I needed to update these specific files:

https://github.com/simonw/python-lib-template-repository/blob/main/.github/workflows/test.yml
https://github.com/simonw/python-lib-template-repository/blob/main/.github/workflows/publish.yml

That's my workaround for the problem where you can't push an update to a workflow - I include copies of them in the template repository itself.

Fixing that like so, after generating a fresh output in /tmp:
cd python-lib-template-repository
cp /tmp/datasette-test/.github/workflows/test.yml .github/workflows/test.yml
cp /tmp/datasette-test/.github/workflows/publish.yml .github/workflows/publish.yml

I'm going to try generating a repo from that manually: https://github.com/new?template_name=python-lib-template-repository&template_owner=simonw

Created: https://github.com/simonw/python-lib-issue-6
I'll copy the workflows from that into datasette-test.

I'm going to ship 0.1 of datasette/datasette-test to see if the publish workflow works.

Trying that with a Trusted Publisher:

https://github.com/datasette/datasette-test/actions/runs/7547266249/job/20546831786 it worked!
https://pypi.org/project/datasette-test/0.1/ was released by the new template.

There's one sharp edge left: the tests fail the first time the repository is cloned from the template: https://github.com/simonw/python-lib-issue-6/commit/54887b223d7e4fa17f3b424ef6703e4c5713ee1b
https://github.com/simonw/python-lib-issue-6/actions/runs/7547200692/job/20546630262 failed:

It doesn't like that there are no tests.

One more test: https://github.com/simonw/python-lib-issues-6-demo-2
Tests now pass on initial commit: https://github.com/simonw/python-lib-issues-6-demo-2/commit/73d216e017ecc645edc0e02a3773ce47a9ba3100

The setup.py did show some warnings: https://github.com/simonw/python-lib-issues-6-demo-2/actions/runs/7547373090

I'll use that to create a new test repo, check for no errors, then delete that repo.

Yes, that worked - I added whitespace to the tests/*.py file too to make sure the tests passed.

This is done!
One last thing: I'm going to update the documentation on https://github.com/simonw/python-lib to remind me how to update the template in the future.

That documentation is here: https://github.com/simonw/python-lib/blob/main/README.md#notes-on-updating-this-cookiecutter-template


After terraform import charm config isn't correctly recognised

Terraform Version
$ terraform -v
Terraform v1.3.0
on linux_amd64
+ provider registry.terraform.io/juju/juju v0.4.1

Affected Resource(s)
juju_application
Terraform Configuration Files
resource "juju_application" "indico" {
  name  = "indico"
  model = juju_model.staging.name

  charm {
    name = "indico"
  }
  config = {
   indico_no_reply_email =<EMAIL_ADDRESS>  }
  units = 1
}

Debug Output
Showing juju config is already set to<EMAIL_ADDRESS>juju config indico indico_no_reply_email
<EMAIL_ADDRESS>
Terraform plan output
  # juju_application.indico will be updated in-place
  ~ resource "juju_application" "indico" {
      ~ config = {
          + "indico_no_reply_email"       =<EMAIL_ADDRESS>        }
        id     = "stg-events:indico"
        name   = "indico"
        # (3 unchanged attributes hidden)
     }

Panic Output
N/A
Expected Behavior
Terraform plan should recognise that the config value is already set to the one it says it's going to change it to.
Actual Behavior
Terraform plan shows it's going to change the config.
Steps to Reproduce
Please list the steps required to reproduce the issue, for example:

juju deploy indico --config<EMAIL_ADDRESS>terraform import juju_application.indico stg-events:indico
terraform plan

Important Factoids
N/A
References
N/A

Can we have a 0.4.2 release so this change can be used?


Add Fragment.onViewCreated as a known initializer.

From the documentation:
"Called immediately after onCreateView(LayoutInflater, ViewGroup, Bundle) has returned, but before any saved state has been restored in to the view."
So, if one is an initializer, the other should be as well.

We currently don't have any support for forcing an order between initializers. Not sure if that will end up mattering :)

We currently don't have any support for forcing an order between initializers. Not sure if that will end up mattering :)

It could matter for checking for use-before-init bugs, if the later-running initializer reads a field that was assigned in the earlier-running initializer.  Right now I believe this case will lead to a false positive warning.  So if it matters we should see it 😄


Support version customization in manual folder

CLI developers can set version in manual folder to overwrite the version in generated folder so the CLI extension can be released with correct version.

should have supported, needs double verify.

Please use the latest private release and regenerate the code. Then follow the guideline here. https://github.com/Azure/autorest.az/blob/master/doc/03-manual-customizations.md#overriding-version

Merged and Closed


Preallocate JTC variables to avoid resizing in realtime loops

Preallocate several variables
last_commanded_state_ was previously used for open-loop control. I think it's redundant with state_desired_ now, so delete it

I don't understand why quite a few unit tests are failing. Marking as draft until that's fixed.

I think #296 broke the JTC tests which is why many of them fail.
I think #334 is an attempt to fix the issue but it hasn't yet been resolved.

OK, it does look like this PR does not make the test failures worse. Re-marking it as ready for review.

The failing CI things are OK. Merging this.


Definitely needs a test. Fixed in https://github.com/Mic92/sops-nix/pull/669


Don't increment counter in constructor

This same change was done in TelepathyTransport.  The counter is incremented when the first client connects so it doesn't need to be done here.  With this change, first client gets ID 1 instead of 2.

:tada: This PR is included in version 3.14.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


[REQUEST] Change "Playing iTunes" in Discord to "Playing Apple Music"

I would recommend using the apple emoji for a better look
example: "Music"

@kevinmussi now I'm curious. I changed the description in the ITUNES enum under Application.java, and it still shows up as "iTunes". Am I looking at the wrong place? (I did repackage it in maven)

The name that shows is the name of the Discord application that the Java program connects to, in this case it's an ad-hoc application that I called "iTunes", so that it shows that name. I guess you could create your own Discord application and put its ID in DiscordHelper.java (variable APP_ID), but that would also make the images not show up anymore (since those are loaded from the Discord application you link).
Anyway, I think for now I'll change it to "Music", since "Playing Music" makes the most sense to me and it's valid both for iTunes and Music app.


P2P file system

http://ori.scs.stanford.edu/#

Looks cool, but also dead.  Moving to discuss in #72.


Old parameter in example

The parameter "-TrackClicks" is deprecated, but still part of the mentioned example. Should be updated.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 06a1f9fa-7dd6-ebf1-3e64-ef47bd3962b2
Version Independent ID: 5ea9bde3-9fe9-8ae1-0992-b12223c4d7c4
Content: New-SafeLinksPolicy (ExchangePowerShell)
Content Source: exchange/exchange-ps/exchange/New-SafeLinksPolicy.md
Service: exchange-powershell
GitHub Login: @chrisda
Microsoft Alias: chrisda

@SaschaSeipp, fixed. I also fixed set-SafeLinksPolicy for the same issue.
Thanks for the feedback.
Closing


A concrete example for using data generator for large datasets such as ImageNet

I am already aware of some discussions on how to use Keras for very large datasets (>1,000,000 images) such as this and this. However, for my scenario, I can't figure out the appropriate way to use the ImageDataGenerator or write my own dataGenerator.
Specifically, I have the following four questions:

From this link: when we do datagen.fit(X_sample), do we assume that X_sample is a big enough chunk of data to calculate mean, perform feature centering/normalization and whitening on?
From the same previous link, Another thing is, X_sample cannot obviously be the entire data, so will the augmentation (i.e. flipping, width/height shift) happen on partial data? For example, X_sample = 10000 out of total 1,000,000 pictures. After augmentation, suppose we get 2 * 10,000 more pictures. Note that we are not running datagen.fit() again, so will our augmented data contain only 1,000,000 + 2 * 10,000 samples? How do we augment entire data (i.e. 1,000,000 + 2 * 1,000,000 samples)?
This is about fetching data from the manually written generator. Since the data is so large, it won't fit into one big HDF5 file, so we split it into 8 files. Now, the data generator has to run in an infinite loop. This and this answer mentions data generators, but a concrete example of it will be more helpful.

My approach for building a data generator (for a very large data) which loops indefinitely is as follows (which fails):
def myGenerator() #this will give chunk of 10K pictures, 100 such chunks form entire dataset:
    fileIndex=0
    while 1:
            # following loads data from HDF5 file numbered with fileIndex
            (X_train, y_train) = LOAD_HDF5_OF_10K_SAMPLES(fileIndex)
            fileIndex=fileIndex+1
            if fileIndex == numOfHDF_files
                fileIndex=0 #so that fileIndex wraps back and loop goes on indefinitely

The above code doesn't work in the sense that once it enters into the above function from fit_generator(), it just stays in the while 1 loop forever. A detailed example will help a lot.

Also, if we are to use ImageDataGenerator as in this link (which is preferable instead of writing our own), should we put (X_train, y_train), (X_test, y_test) = LOAD_10K_SAMPLES_OF_BIG_DATA() in a for loop and write datagen.fit(X_train) and model.fit_generator(datagen.flow(...)) in that loop?

Hi there, my answers to your questions below:

Yes that assumption is implicit.
datagen.fit() doesn't generate a batch of examples continuously like a generator; it actually does some once-off data transformations given the X matrix which you supplied. To continuously generate data with on-the-fly augmentations, you need to pass the .flow() function of the ImageDataGenerator class to the fit_generator function.  HDF5 has no size limits afaik so you don't need to worry about storing your data into one huge HDF5 file. Your only issue would be loading it all into memory at a go, which I would advise against since that is what the batch generators are for: avoiding a huge memory hog. Your example looks alright (looping forever is what you want), though you might want to add in some lines for augmenting the data you just loaded.

I have some follow up questions @wongjingping :
Response to 3. I should have been clearer. I am not concerned about one "big" HDF5 file, the question is entire data can't be loaded as you say. You say that my example looks fine, but I think its wrong. I illustrate that with the following snippet of data generator:
def myGenerator():
    #loading data
    (X_train, y_train), (X_test, y_test) = mnist.load_data()

    #some preprocessing
    y_train = np_utils.to_categorical(y_train,10)
    X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
    X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
    X_train = X_train.astype('float32')
    X_test = X_test.astype('float32')
    X_train /= 255
    X_test /= 255
    while 1:
        for i in range(1875):
            if i%125==0:
                print "i = " + str(i)
            yield X_train[i*32:(i+1)*32], y_train[i*32:(i+1)*32]

The above function is called by fit_generator(). Now, it should print till i=1750 and then train the model. However, it just keeps printing i=0 to i=1750 and then starts again from i=0 without training the model.
If I comment the line while 1, it runs perfectly, but then it violates the assumption of infinite loop, doesn't it? Can you clear up my confusion by providing a concrete example or talking with respect to this example?
If you want a self-contained code snippet, it is as follows. You can just run it.
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, Flatten, Reshape
from keras.layers.convolutional import Convolution1D, Convolution2D, MaxPooling2D
from keras.utils import np_utils

def myGenerator():
    (X_train, y_train), (X_test, y_test) = mnist.load_data()
    y_train = np_utils.to_categorical(y_train,10)
    X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
    X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
    X_train = X_train.astype('float32')
    X_test = X_test.astype('float32')
    X_train /= 255
    X_test /= 255
    while 1:
    for i in range(1875): # 1875 * 32 = 60000 -> # of training samples
        if i%125==0:
            print "i = " + str(i)
        yield X_train[i*32:(i+1)*32], y_train[i*32:(i+1)*32]

batch_size = 128
nb_classes = 10
nb_epoch = 12

# input image dimensions
img_rows, img_cols = 28, 28
# number of convolutional filters to use
nb_filters = 32
# size of pooling area for max pooling
nb_pool = 2
# convolution kernel size
nb_conv = 3
model = Sequential()

model.add(Convolution2D(nb_filters, nb_conv, nb_conv,
                        border_mode='valid',
                        input_shape=(1, img_rows, img_cols)))
model.add(Activation('relu'))
model.add(Convolution2D(nb_filters, nb_conv, nb_conv))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adadelta')

model.fit_generator(myGenerator(), samples_per_epoch = 60000, nb_epoch = 2, verbose=2, show_accuracy=True, callbacks=[], validation_data=None, class_weight=None, nb_worker=1)

Hi there,
I'm afraid I'm having some problems running your code with the debugger, but from what I can see I think you need to assign the generator instance to a new variable before passing it to the fit_generator() method as below:
my_generator = myGenerator() model.fit_generator(my_generator, samples_per_epoch = 60000, nb_epoch = 2, verbose=2, show_accuracy=True, callbacks=[], validation_data=None, class_weight=None, nb_worker=1)
Let me know if it doesn't work! (I'm not too sure myself)

I can run it in pycharm debugger after putting breakpoint inside my_generator() function. However, I don't think running with a debugger is a good idea, in case of generators, that's why I am printing the value of i. With the code I had, you should see i going from 0 to 1750, then immediately warping back and printing 0 to 1750, repeat this indefinitely. Ideally, it should print till 1750, train, the again print till 1750, repeat this for NUMBER_OF_EPOCHS times.
Anyway, with your suggestion, same thing happens, it just keeps repeating indefinitely. However, if I remove while 1, it goes till 1750, trains and again goes back to 0 till 1750, trains, then terminates (as desired).

Your model is training actually. You can add this snippet of code to verify (print out) the progress of your model:
`class printbatch(Callback):
def on_batch_end(self, epoch, logs={}):
print(logs)
...
pb = printbatch()
model.fit_generator(myGenerator(), samples_per_epoch = 60000, nb_epoch = 2,
verbose=2, show_accuracy=True, callbacks=[pb], validation_data=None, class_weight=None, nb_worker=1)
`

@wongjingping Thanks. It works. Just one small doubt (rather observation) is that when the logs are printed, the numbers being printed in front of "Batch: " are actually sample numbers. So if there are 60000 samples / epoch, then in the logs being printed through callbacks, we saw "Batch: 59999".
Anyway, I am closing this issue now. I still haven't had success in running the data generators with >1 workers, but I have asked another question for that. You may take a look at it if time permits. That would be great. That is Issue #1638

Hi @parag2489 ,
I'm having an issue with the batch sizes showing up correctly; and my suspect of the offending line is
                batch_logs = {}                 batch_size = len(X[0])                 batch_logs['batch'] = batch_index

@wongjingping @parag2489 Hi~ May I ask you guys how to specify batch size if I wrote my own data generator since fit_generator doesn't have the batch_size parameter and in the fit_generator, we only yield one sample at a time.

@sunshineatnoon You can pass the batch_size as an argument to the generator:
    i = 0
    while i < epoch_size:
        # add in image reading/augmenting code here
        yield X[i:i+batch_size,...],y[i:i+batch_size,...]
        if i + batch_size > epoch_size:
            i = 0
        else:
            i += batch_size

You might want to check out this link that introduces generators

@wongjingping Thanks! I will look into this. BTW, what does samples_per_epoch mean in fit_generator exactly? Say if I use a batch size of 64. Does this mean a total of 64*samples_per_epoch is seen every epoch?

@sunshineatnoon sorry for the poor formatting; the samples_per_epoch is the number of examples you expect to see in an epoch, not batch_size * samples_per_epoch :)

@wongjingping So it means that if I use a batch size of 64, I will have samples_per_epoch / 64  batches per epoch? But when I specify batch_size and generate a batch, my network training time slows down, it seems like it trains on more samples each epoch if I increase the batch_size. Here is my generator:
def generate_batch_data(vocPath,imageNameFile,batch_size):
    sample_number = 5000
    class_num = 20

    while 1:

        for i in range(0,sample_number,batch_size):
            #Read a batch of images from files
            imageList = prepareBatch(i,i+batch_size,imageNameFile,vocPath)
            #process imageList to np arrays images and boxes
            yield np.asarray(images),np.asarray(boxes)

@sunshineatnoon  samples_per_epoch means fit_generator() will stop asking samples from data generator. In another words, samples_per_epoch = batch_size * number_of_batches.
Regarding why your training slows down, its best to profile your code. There is a feature in Theano for that (I think mode=Profile). You can increase speed if you call prepareBatch() for a large number of samples (large means that they can fit in your cpu RAM but not in GPU). Also, convert images and boxes in numpy array only once. Then just yield in batches of 32. In short, prepareBatch and two calls to np.asarray will go outside the for loop.

@parag2489 Thanks! It's very nice of you to give such a detailed explanation, I will try to change my code.

this page helps,thanks! btw,it seems that new version comes quickly

Just to be clear can someone confirm:

In model.fit() you specify the batch_size so it knows how to break a finite data set (the corresponding  x and y numpy arrays) into chunks for gradient calculation.  100% of the data set gets consumed each epoch.
With model.fit_generator() the generator you provide should loop infinitely, the as samples_per_epoch is basically giving a bound to total data samples to run through.  The batch_size isn't specified as each tuple returned from the generator is a single batch.  You control the size of the batch via the generator, so if you return one sample per yields, it's like setting a batch size of 1.

  If the generator is handling the batching, why do you need another queue?

@raymondjplante
Q1. Your understanding of model.fit() is correct.
Q2. Correct.
Even I am not sure what the max_q_size. I think this answer has a mention of queue. So the queue is used to ensure that the generator is thread-safe.
You can also look at #1638 to see how to make the data generator thread-safe.

@parag2489  Someone on SO provides a good explanation of the purpose of the generator queue: http://stackoverflow.com/questions/36986815/in-keras-model-fit-generator-method-what-is-the-generator-queue-controlled-pa

For the case of multi-inputs, such as we have two pathways in the network, each corresponding to different input, Can we still use "data-generator" to generate image regions parallel with training process?

The problem I'm facing is keras fit_generator is good for processing images with collective size more than RAM size,but what if those files are actually not in image format.For example I've taken huge number of images(500k) and have used them against a pre-trained inception v3 model to get the feature out of them.Now each of those files are nothing but (1,384,8,8) array or npy files.Any idea how I can use fit generator to read them in batch as collectively they won't fit in my RAM.

@tanayz It would be the exact same as if they were image instead of pickled/numpy data files:

Get a list of all of the files, and pass this list into the generator
In the generator:

Infinite loop
Shuffle the list of files
For each slice of the shuffled files, where len(slice) == batch_size

Open files, read to a single array with first shape[0] == batch_size; yield data
Have an edge case to handle the case where batch_size is not a multiple of the number of files, such that the generator will always yield batch_size number of examples

@seasonwang I'm afraid I haven't tried that out before - sorry for the late reply!

Is there a way to use train_on_batch with a generator?

you means?
for batch in generator:
    model.train_on_batch(batch)

Hi,
I have a question while using "predict_generator". How to ensure that the prediction is done on all test samples once.
For example-
predictions = model.predict_generator(
test_generator,
steps=int(test_generator.samples/float(batch_size)), # all samples once
verbose = 1,
workers = 2,
max_q_size=10,
pickle_safe=True
)
predicted_classes = np.argmax(predictions, axis=1)
true_classes = test_generator.classes
So the dimensions of predicted_classes and true_classes is different since total samples is not divisible by batch size.
The size of my test_set is not consistent, so the no. of steps in predict_generator would change each time depending upon the batch size. I am using flow_from_directory and cannot use predict_on_batch since my data is organized in a directory structure.
One solution is running with batch size of 1, but makes it very slow.
I hope my question is clear. Thanks in advance.

The comments and suggestions in this issue and its cousin #1638 were very helpful for me to efficiently process large numbers of images.  I wrote it all up in a tutorial fashion that I hope can help others.
https://techblog.appnexus.com/a-keras-multithreaded-dataframe-generator-for-millions-of-image-files-84d3027f6f43

Hello,
I am trying to use model.fit_generator with a custom Callback that tries to access Validation data. However, whatever I do, when accessing validation data from within the Callback, it always equates to None.
class RecallMetrics(Callback):
    def on_train_begin(self, logs=None):
        print('RecallMetrics ... validating')
        self.val_f1s = []
        self.val_recalls = []
        self.val_precisions = []

    def on_epoch_end(self, epoch, logs=None):
        x=(self.validation_data[0])
        if x is None :
            print ('Error: validation_data is None')
            return
        else:
            val_predict = (np.asarray(self.model.predict(self.validation_data[0]))).round()
            val_targ = self.validation_data[1]
            _val_f1 = f1_score(val_targ, val_predict)
            _val_recall = recall_score(val_targ, val_predict)
            _val_precision = precision_score(val_targ, val_predict)
            self.val_f1s.append(_val_f1)
            self.val_recalls.append(_val_recall)
            self.val_precisions.append(_val_precision)
            print (" — val_f1: % f — val_precision: % f — val_recall % f" % (_val_f1, _val_precision, _val_recall))
            return

history = model.fit_generator(generator=train_gen,
                                  validation_data=validate_gen,
                                  # validation_data=None,
                                  steps_per_epoch=len(train_file_list),
                                  validation_steps=len(val_file_list) * 3,
                                  verbose=2,
                                  epochs=int(tc.config["LUNA16"]["epochs"]),
                                  callbacks=callbacks,
                                  workers=multiprocessing.cpu_count(),
                                  use_multiprocessing=True)

How can I access validation data from a custom Callback when using fit_generator?
Best,

you means?
for batch in generator:
    model.train_on_batch(batch)

Hi,
Tried using this but got the following error:
   dloss_real = disc.train_on_batch(dataBatch, valid)   File "/home/macman/miniconda3/lib/python3.7/site-packages/keras/engine/training.py", line 1211, in train_on_batch     class_weight=class_weight)   File "/home/macman/miniconda3/lib/python3.7/site-packages/keras/engine/training.py", line 751, in _standardize_user_data     exception_prefix='input')   File "/home/macman/miniconda3/lib/python3.7/site-packages/keras/engine/training_utils.py", line 92, in standardize_input_data     data = [standardize_single_array(x) for x in data]   File "/home/macman/miniconda3/lib/python3.7/site-packages/keras/engine/training_utils.py", line 92, in <listcomp>     data = [standardize_single_array(x) for x in data]   File "/home/macman/miniconda3/lib/python3.7/site-packages/keras/engine/training_utils.py", line 27, in standardize_single_array     elif x.ndim == 1: AttributeError: 'tuple' object has no attribute 'ndim'


Unable to create a new version of a Unlocked Package with dependencies

I have an unlocked package Pack1, which is the dependency of an unlocked package Pack2, which is the dependency of an unlocked package Pack3.
I can create new versions of Pack1 and Pack2, but if I want to create a new version of Pack3, then I get the error message:
An error occurred while trying to install a package dependency, ID PACK2: The package you're installing depends on package 'PACK1', version '0.1'. Install package 'PACK1' in the target org before you install package 'PACK2'.
I do not think I am able to manually install dependencies in this process and it should be done automatically.
This worked perfectly until two week ago, where I used it last.

I have an unlocked package Pack1, which is the dependency of an unlocked package Pack2, which is the dependency of an unlocked package Pack3.
I can create new versions of Pack1 and Pack2, but if I want to create a new version of Pack3, then I get the error message:
An error occurred while trying to install a package dependency, ID PACK2: The package you're installing depends on package 'PACK1', version '0.1'. Install package 'PACK1' in the target org before you install package 'PACK2'.
I do not think I am able to manually install dependencies in this process and it should be done automatically.
This worked perfectly until two week ago, where I used it last.

If you pass a config file for the package version creation command, it will create an "invisible" scratch org, deploy the dependencies of that package and then start the package version creation of your package. But to ensure it works, your package 3 must have the "dependencies" defined inside the "sfdx-project.json".
Could you post here the "sfdx-project.json" of the three packages?

@AllanOricil @sfvb I will share this with the packaging team, but if you need immediate assistance please file a customer case

@clairebianchi Im ok. I was just trying to help @sfvb but did not get a response. Thank you anyway!


Please add the description of the bug for unaffected_affected_no_reg.py

Fixed:
https://github.com/mozilla/relman-auto-nag/pull/197


Tweak scripts

Execute bacchus scripts from any path, no need to cd to bacchus/scripts directory

thanks !


Hot reload HTML

It would be awesome if this worked for html files as well.
In my app, I am importing html files with the !text plugin. I have an idea for implementing it, but I need to be able to find the parent module that imported the html file. The hot reloader seems to be using System._loader.moduleRecords to find the module hierarchy for js files.
Does anyone with more internal SystemJs knowledge know where I can find the module that imported an html file? @capaj @guybedford
I can see the full module name in System._loader.modules, but can't seem to find the hierarchy.

Editing HTML did work in my limited testing (after basically following the simple tutorial to set this up).

@JordanBelford please see https://github.com/capaj/systemjs-hot-reloader/issues/74#issuecomment-221845425
Closing for now, if it still doesn't work, let me know.


Add routable project categories

i.e. /projects/category/web should show all the projects that are web projects.

No longer needed / now a frontend app concern


maxiops

Max IOPs is 18K and not 20K.

Learn Build status updates of commit ece4786:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

articles/postgresql/flexible-server/concepts-compare-single-server-flexible-server.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

thank you @Court72 - checking

#sign- off

#sign-off (per contributor comment above)

@AwdotiaRomanowna, is this OK to merge?

#hold-off


Error installing on ubuntu 20.04.3 (org-yt -> wrong-type-argument arrayp nil)

What did you expect to happen?
Install to be successful.
On Ubuntu 20.04.3
Using the commands for cloning and "doom install"
What actually happened?

Describe your attempts to resolve the issue
At first I was trying to upgrade doom but got an error with a package, so I ended up eliminating the straight repos and doing a doom upgrade but the problem persisted. At the end I deleted .emacs.d and the doom config folder to make a clean install and I'm still getting errors when attempting to install Doom Emacs.
Steps to reproduce

On Ubuntu 20.04.3
Run the installation commands given in the instructions:
git clone --depth 1 https://github.com/hlissner/doom-emacs ~/.emacs.d
~/.emacs.d/bin/doom install

System Information
https://pastebin.com/JL6an4NR

I had the same issue, except it triggered on org-contrib instead. However, I don't think this is a duplicate of #5207  -- it's a different error message, and the repository is never actually downloaded, so the fix doesn't work (although the workaround in https://github.com/hlissner/doom-emacs/issues/5207#issuecomment-932053166 allows installation to continue).
But I don't think disabling SSL verification is a reasonable long-term fix.

I had the same issue, except it triggered on org-contrib instead. However, I don't think this is a duplicate of #5207 -- it's a different error message, and the repository is never actually downloaded, so the fix doesn't work (although the workaround in #5207 (comment) allows installation to continue).
But I don't think disabling SSL verification is a reasonable long-term fix.

same thing happened to me , using this git config --global http.sslVerify false fixed it for me .


Fix the bug with the wrong format.

This is a little problem, it will be better to fix it.

Good catch. Thanks!


Thread error

i have a problem with commitig data to db.
models.py
class Task(db.Model):
id = db.Column(db.Integer, primary_key=True)
task = db.Column(db.String(64), index=True, unique=True)
description = db.Column(db.String(120), index=True, unique=True)
start = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
password_hash = db.Column(db.String(128))
def __repr__(self):
    return '<Task {}>'.format(self.task)

config.py
class Config(object):
SECRET_KEY = os.environ.get('SECRET_KEY') or 'you-will-never-guess'
SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL') or
'sqlite:///' + os.path.join(basedir, 'app.db')
SQLALCHEMY_TRACK_MODIFICATIONS = False
inti.py
from flask import Flask
from config import Config
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
app = Flask(name)
app.config.from_object(Config)
db = SQLAlchemy(app)
migrate = Migrate(app, db)
from app import routes, models
i have a problem committing data or starting any new queries in python interpreter. Because each time i do a get an thread error:
Traceback (most recent call last):
File "/Users/janzaplatil/Desktop/taskmanager/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1127, in _execute_context
context = constructor(dialect, self, conn, *args)
File "/Users/janzaplatil/Desktop/taskmanager/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 639, in _init_compiled
self.cursor = self.create_cursor()
File "/Users/janzaplatil/Desktop/taskmanager/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 954, in create_cursor
return self._dbapi_connection.cursor()
File "/Users/janzaplatil/Desktop/taskmanager/venv/lib/python3.7/site-packages/sqlalchemy/pool.py", line 977, in cursor
return self.connection.cursor(*args, **kwargs)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id<PHONE_NUMBER>50336 and this is thread id<PHONE_NUMBER>14048.
The above exception was the direct cause of the following exception:
sometimes it works if i run a db.session.rollback() command but if i try to run tasks = Task.query.all() than it does not.
sory for bothering you i have no idea what to do

I need to see a complete session in the Python shell, until you get the error.

PyDev console: starting.
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28)
[Clang 6.0 (clang-600.0.57)] on darwin
from app.models import Task
u = Task(task='zlozi', description='zlozi hlace')
u

db.session.add(u)
Traceback (most recent call last):
File "", line 1, in
NameError: name 'db' is not defined
from app.models import db
db.session.add(u)
db.session.commit()
tasks = Task.query.all()
than the error listed above appears

Can you try running the same test but this time using my code? This error is odd in this situation, because you are not using multiple threads. Makes me think something in your version of the code is different.

ok so i did it like this i started a new project in new venv so it should be isolated if i understand it right.
2. i downloaded your https://github.com/miguelgrinberg/microblog/tree/v0.4 part 4 database zip
3. i did all the imports in terminal...flask , wtf,
4. now the commit part:
(venv) Jans-MBP:microblog2 janzaplatil$ flask db init
Error: Directory migrations already exists  **dont understand why if i started from scratch **
(venv) Jans-MBP:microblog2 janzaplatil$ flask db migrate -m "users table"
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
ERROR [root] Error: Target database is not up to date. again how if it is new
(venv) Jans-MBP:microblog2 janzaplatil$ flask db upgrade
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> e517276bb1c2, users table
INFO  [alembic.runtime.migration] Running upgrade e517276bb1c2 -> 780739b227a7, posts table
(venv) Jans-MBP:microblog2 janzaplatil$ flask db migrate -m "posts table"
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.env] No changes in schema detected.
(venv) Jans-MBP:microblog2 janzaplatil$ flask db upgrade
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
(venv) Jans-MBP:microblog2 janzaplatil$
now in python console
from app.models import User
u = User(username='susan'<EMAIL_ADDRESS>u

from app import db
from app.models import User, Post
u = User(username='john'<EMAIL_ADDRESS>db.session.add(u)
db.session.commit()
u = User(username='susan'<EMAIL_ADDRESS>db.session.add(u)
db.session.commit()
Traceback (most recent call last):
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1127, in _execute_context
context = constructor(dialect, self, conn, *args)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 639, in _init_compiled
self.cursor = self.create_cursor()
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 954, in create_cursor
return self._dbapi_connection.cursor()
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/pool.py", line 977, in cursor
return self.connection.cursor(*args, **kwargs)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id<PHONE_NUMBER>43616 and this is thread id<PHONE_NUMBER>14048.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2345, in _flush
flush_context.execute()
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 395, in execute
rec.execute(self)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 560, in execute
uow
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
mapper, table, insert)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 866, in _emit_insert_statements
execute(statement, params)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
return meth(self, multiparams, params)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
compiled_sql, distilled_params
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1132, in _execute_context
None, None)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
raise value.with_traceback(tb)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1127, in _execute_context
context = constructor(dialect, self, conn, *args)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 639, in _init_compiled
self.cursor = self.create_cursor()
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 954, in create_cursor
return self._dbapi_connection.cursor()
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/pool.py", line 977, in cursor
return self.connection.cursor(*args, **kwargs)
sqlalchemy.exc.ProgrammingError: (sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread. The object was created in thread id<PHONE_NUMBER>43616 and this is thread id<PHONE_NUMBER>14048. [SQL: 'INSERT INTO user (username, email, password_hash) VALUES (?, ?, ?)'] [parameters: [{'username': 'susan', 'email'<EMAIL_ADDRESS>'password_hash': None}]] (Background on this error at: http://sqlalche.me/e/f405)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 704, in _rollback_impl
self.engine.dialect.do_rollback(self.connection)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 459, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id<PHONE_NUMBER>43616 and this is thread id<PHONE_NUMBER>14048.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "", line 1, in
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/orm/scoping.py", line 153, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 943, in commit
self.transaction.commit()
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 467, in commit
self._prepare_impl()
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 447, in _prepare_impl
self.session.flush()
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2254, in flush
self._flush(objects)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2381, in _flush
transaction.rollback(capture_exception=True)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 76, in exit
compat.reraise(type, value, traceback)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 249, in reraise
raise value
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2381, in _flush
transaction.rollback(_capture_exception=True)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 532, in rollback
util.reraise(*rollback_err)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 249, in reraise
raise value
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 497, in rollback
t[1].rollback()
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1632, in rollback
self._do_rollback()
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1670, in _do_rollback
self.connection._rollback_impl()
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 706, in _rollback_impl
self._handle_dbapi_exception(e, None, None, None, None)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
raise value.with_traceback(tb)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 704, in _rollback_impl
self.engine.dialect.do_rollback(self.connection)
File "/Users/janzaplatil/Desktop/microblog2/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 459, in do_rollback
dbapi_connection.rollback()
sqlalchemy.exc.ProgrammingError: (sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread. The object was created in thread id<PHONE_NUMBER>43616 and this is thread id<PHONE_NUMBER>14048. (Background on this error at: http://sqlalche.me/e/f405)

Regarding your questions:

(venv) Jans-MBP:microblog2 janzaplatil$ flask db init
Error: Directory migrations already exists **dont understand why if i started from scratch **

You really didn't start from scratch. My code includes a migration repository. The flask db init command is used to create the repository, it is not used when you get someone else's code that comes with migrations included.

(venv) Jans-MBP:microblog2 janzaplatil$ flask db migrate -m "users table"
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
ERROR [root] Error: Target database is not up to date. again how if it is new

Same as above. This repository includes the database migrations, you are trying to generate them yourself and that fails.

now in python console

Is this really a Python console, or are you using a different shell? In your previous post you appeared to be using PyDev. Do you get the error if you use the normal Python shell instead?

i am using a pycharm and to write these commands now in python console
from app.models import User
u = User(username='susan'<EMAIL_ADDRESS>u
from app import db
from app.models import User, Post
u = User(username='john'<EMAIL_ADDRESS>db.session.add(u)
db.session.commit()
u = User(username='susan'<EMAIL_ADDRESS>db.session.add(u)
db.session.commit()
i open python console
should i be doing this directly in terminal ?

It could be that the way the PyCharm terminal works conflicts with the way SQLite works, so maybe that is the problem and you have to use the normal Python shell.
The UNIQUE constraint failed: user.username error means that you are adding a duplicate user to your database, which is not allowed.

Yes, I got this as well in PyCharm.  Opening a terminal, activating the environment manually and issuing the commands, works perfectly.


Fix typos under New Zealand

Really minor - just spelling too correctly for the context.

Coverage remained the same at 99.399% when pulling 70d2a09afd5bdc3a12ecf3a911abb66d59599c91 on dwightwatson:patch-1 into 7a4722c67fdd98cd0973e7b673cc540ee6dd9cd8 on floere:master.

Coverage remained the same at 99.399% when pulling 70d2a09afd5bdc3a12ecf3a911abb66d59599c91 on dwightwatson:patch-1 into 7a4722c67fdd98cd0973e7b673cc540ee6dd9cd8 on floere:master.

Coverage remained the same at 99.399% when pulling 70d2a09afd5bdc3a12ecf3a911abb66d59599c91 on dwightwatson:patch-1 into 7a4722c67fdd98cd0973e7b673cc540ee6dd9cd8 on floere:master.

Coverage remained the same at 99.399% when pulling 70d2a09afd5bdc3a12ecf3a911abb66d59599c91 on dwightwatson:patch-1 into 7a4722c67fdd98cd0973e7b673cc540ee6dd9cd8 on floere:master.

@dwightwatson Thanks Dwight! Minor maybe but you took the time to fix it anyway 😊 Thanks so much! Released in 2.16.4 🚀


fix(object): Takes into account the range of bytes starting with 0

Fixes errors when trying to re-create directories that already exist.
Uses a helper function to verify for the directory existence first.
Note this PR also includes the fix for options.start !== undefined as I've built on top of my previous branch.

These look pretty good to me, do you think you could add a quick test case with the range that wasn't working before? There should be some existing tests that you can borrow from.

Hey @kherock , the ranges changes were done in a separate PR: https://github.com/jamhall/s3rver/pull/753
I'll revisit this PR again to refactor a few more things and get back to you. Thanks!

This PR is related to this issue: https://github.com/jamhall/s3rver/issues/754


I can not run configure after install.

After install and run gitrob configure, I get this :
./gitrob configure
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in require': cannot load such file -- gitrob (LoadError) 	from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in require'
from ./gitrob:3:in `'
uname -a
Linux kali 4.9.0-kali3-amd64 #1 SMP Debian 4.9.16-1kali1 (2017-03-24) x86_64 GNU/Linux

Did you install with gem install gitrob or did you clone the repo?


Fixed #1502: Proxying of unannounced trailers

As per #1512 this PR only contains the necessary changes to fix #1502 and nothing more.
Sorry for closing the other PR and sorry for the delays - I'm having a kinda hard time in life right now and things like that are the result I guess. 😕
P.S.: I have no idea why one of the commit tests failed... Maybe a race condition?  With Rust that wouldn't have happened.

Thanks @lhecker. It's not a problem - take care of things you need to. I hope things look up for you soon!
I'm not sure about the failing test either, but it's something worth looking into! Even if this change doesn't go into this next release, it'll get in one of the next ones.
Know that I and many others (perhaps unknowingly) appreciate your work. I'll review this soon, but might also wait until you're ready to circle back around to it.

@mholt My life is slowly getting back to normal and starting next month everything should finally be a bit more relaxed again. 😂
I just rebased this PR on the current master. So how about we take another look at this PR now? The new & shiny support for some glorious gRPC proxying has been waiting far too long already. 🙂
P.S.: When the tests on this PR originally failed after submitting this PR it I bet a race condition was at fault. Since the underlying issue might not have been fixed yet I think we should carefully look out for that in caddy in the future. I mean: You do run the tests with -race enabled but who knows if that catches the rather spurious bugs (e.g. those who only happen if caddy is preempted or interrupted at a critical moment etc.).

@mholt I fixed the 2 issues you mentioned. The PR indeed was quite a bit messy, but I cleaned it up now (no TODOs anymore or removed code).
The issue with the test was - or rather "is" - that it's only a fake HTTP request test. Since we never actually serialize and submit r (the http.Request), but instead submit the fake request/response directly to the proxy's ServeHTTP we will never know if proxying of unannounced trailers actually works. That will only ever be the case if we do a proper E2E test over localhost.
Code wise on my side everything's ready for a proper E2E though, after I added the new trailerTestStringReader struct just now. 🙂

Feel free to submit small PRs with the other changes you had in #1512 if you still want to do those. :+1:

Yeah I'll do that. 🙂 And if I have the opportunity to contribute something again afterwards it'd probably be something to integrate caddy with kubernetes or so... I don't really want to use linkerd for that. 😂 And maybe something to make the config file more generic - I honestly fear that the current format might become a mayor painpoint as soon as we accumulate more and more plugins which aren't necessarely strictly centered around HTTP or domain names at all.

Let's discuss it more, either in an issue, or I've also sent you an email with some information. Thanks again! :+1:


"PTRACE_GETREGS returns ffffffffffffffff"

When getting the threads and their stack traces for a .NET Core 2.1 process on Linux x64 I get this in the output:
PTRACE_GETREGS returns ffffffffffffffff for 42009
PTRACE_GETREGS returns ffffffffffffffff for 42029
PTRACE_GETREGS returns ffffffffffffffff for 42034
PTRACE_GETREGS returns ffffffffffffffff for 42038
PTRACE_GETREGS returns ffffffffffffffff for 42041
PTRACE_GETREGS returns ffffffffffffffff for 42042
PTRACE_GETREGS returns ffffffffffffffff for 42043
PTRACE_GETREGS returns ffffffffffffffff for 42046

I don't know what this means. I guess the register values could not be read?

@mikem8361 Any ideas?

This is part of live linux reader that @xinyanmsft added a few months ago.

Hi, @loop-evgeny does the caller has ptrace permission on the .net core process?
Can you explain a bit in more detail what you are trying to accomplish? That would help me understanding what's going on. Right now this indicates the ptrace call with GETREGS failed. Thanks!

Yes, this happens even when running with sudo. I'm printing the stack trace of every managed thread. The code does something like this:
                var dataTarget = DataTarget.AttachToProcess(pid, 5000, AttachFlag.Passive);
                var runtime = dataTarget.ClrVersions[0].CreateRuntime();

                foreach (var thread in runtime.Threads)
                {
                    var st = thread.StackTrace; // This is what prints the PTRACE_GETREGS message...
                    // ... print it ...
                }

fix merged. thanks!


Decryption failed

On 2024-06-11 I could successfully run the program and decrypt all my tokens with the backup password. After that i removed the new device from the account.
However I tried this again today (registering as a new device) and after entering the same backup password as last time it failed.
2024/06/18 15:38:13 Failed to decrypt token XXX:xxx: decryption failed
I have tried to disable the backup password and enable it again, but to no avail.

Authy now dynamically increases the amount of rounds of password-based key derivation function for encrypting seeds.
Mine is set 100000 currently.
Run AUTHY_DEBUG=1 authy-export
Check what "key_derivation_iterations": is set at
Then modify crypto.go kdfRounds to match yours and rebuild
for example I set: kdfRounds    = 100000

Thank you! That worked perfectly. Maybe this could be mentioned somewhere in the Readme or automatically be set when the api response comes back.

tried to modify kdfRounds to match key_derivation_iterations but still got  'Decryption failed'

The master branch still couldn't decrypt keys, can you check and reopen this issue?


fix "Edit this page" links everywhere

Closes #287
"Edit this page" links do not work because the editUrl in docusaurus.config.js is clearly templated but is never expanded for REPOSITORY_NAME
My quick look at docusaurus docs doesn't show any information on templating the config.
I also didn't find anywhere else REPOSITORY_NAME is used, so I ruled out it was injected at build time (like in github workflows or build scripts).
This change hardcodes the editUrl to what would be the expanded template.
A simple test of rewriting the URL I was directed to with what's involved in this change confirms that it will place the user on the associated markdown file that they can then edit.

Looks good, thanks!


[Inquiry]: Is it possible to create a list of allowed URLs

Description
I'd like to isolate my application such that it can only access URLs that I allowed in the manifest. Is this possible? The granularity appears to be rather limited: https://learn.microsoft.com/en-us/windows/win32/secauthz/appcontainer-isolation#network-isolation
This would be useful to mitigate the scenario when my app is compromised (e.g. via an input file) and the compromised app wants to exfiltrate stolen data.

Hi, @szilvaa-adsk.
Thank you so much for your input! This is currently not supported, and we don't have immediate plans for it. We will try to take a look in the future.
Thank you so much for your valuable participation!

I don't think it's worth implementing such a mechanism for requests made by the Windows API. Most applications use their own HTTP library.
Allowing the App Container to specify a virtual network device (driver) or other L2/L3 network interception mechanism could be useful for more advanced cases.
However, with the arrival of ECH (Encrypted Client Hello) over TLS, it would no longer be possible to determine an origin via Deep Packet Inspection (DPI). DNS detection will also not work when using Secure DNS over HTTPS.


Move to Microsoft identity platform for Azure AD authentication

I want to use Microsoft identity platform.
When we use below it works:
services.AddAuthentication().AddMicrosoftAccount(microsoftOptions =>
{
microsoftOptions.ClientId = Configuration["Authentication:Microsoft:ClientId"];
microsoftOptions.ClientSecret = Configuration["Authentication:Microsoft:ClientSecret"];
});

But when I wan to use below it is showing error.
services.AddSignIn(Configuration);
 services.AddAuthentication(AzureADDefaults.AuthenticationScheme)
            .AddAzureAD(options => Configuration.Bind("AzureAd", options));

Error:
OptionsValidationException: The 'Instance' option must be provided.
Microsoft.Extensions.Options.OptionsFactory<TOptions>.Create(string name)
Can anyone please suggest how to move to this new v2 option.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a4c264c0-0c5c-298e-bb45-9b4891c8d162
Version Independent ID: ce69b990-0b4c-abda-cd2d-68f85cd8031e
Content: Microsoft Account external login setup with ASP.NET Core
Content Source: aspnetcore/security/authentication/social/microsoft-logins.md
Product: aspnet-core
Technology: aspnetcore-security
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

See https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-v2-aspnet-core-webapp

@Rick, thank you so much.


Shift the FFT data

Title

Function to Shifts the FFT data into a useful position

Linting

Passes pylint with 10/10

Labels and Milestones

ALL pull requests must have an attached label and Milestone before a reviewer can be assigned.

@zbpetersbuf I've added you as reviewer for this request. Please do this as soon as possible.

so for conflicts all you need to do is press resolve conflicts and delete 3 lines, the lines are 11 (<<<<<<< main), 32 (=======), 75 (>>>>>>> main) and your good to go

@Priyanka-43 You should not merge your own PR. Always the maintainer @kylemasc917 merges it

I havent seen any major effects yet so it should be good for now but I will revert it if that changes.


Face Shape Classification/Detector

Pull Request for ML-Crate 💡
Issue Title:  Face Shape Detector #417

Info about the related issue (Aim of the project) :  Detect Face Shape
Name: Harsh
GitHub ID: Parker-X
Idenitfy yourself:  I'm a bca student participating in kWoC22

Closes: #issue number that will be closed through this PR  #417
Describe the add-ons or changes you've made 📃
folders added: Model, Images, Dataset
files added: readme.md, requirement.txt
Type of change ☑
What sort of change have you made:

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Code style update (formatting, local variables)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested? ⚙
Describe how it has been tested: Tested on vscode and Jupyter Lab and kaggle
Describe how have you verified the changes made: Code executes without errors
Checklist: ☑

[X] My code follows the guidelines of this project.
[X] I have performed a self-review of my own code.
[X] I have commented my code, particularly wherever it was hard to understand.
[X] I have made corresponding changes to the documentation.
[ ] My changes generate no new warnings.
[X] I have added things that prove my fix is effective or that my feature works.
[ ] Any dependent changes have been merged and published in downstream modules.

Your README.md file should be bit more elaborative, add more information about your project there. Also at the last put the conclusion for the model by providing the accuracy scores.
Project folder name should be, Face Shape Classification.

@PARKER-X

Updated plz check
Thanks!


A mod thief.

I believe someone has stolen your work. The animations are identical to yours. When you open the file of their mod, everything is identical to yours.
https://www.curseforge.com/minecraft/mc-mods/pandas-animations

I also left a comment on his mod saying that it looked remarkably similar to yours. However, the comment section no longer exists.

Thanks for letting me know


Add setup_dataloader2_creator

Codecov Report
Patch coverage: 100.00% and no project coverage change.

Comparison is base (627e41d) 100.00% compared to head (00ee134) 100.00%.

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #63   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           30        30
  Lines          521       532   +11
  Branches        24        25    +1
=========================================
+ Hits           521       532   +11

Impacted Files
Coverage Δ

src/gtvision/creators/dataloader2/__init__.py
100.00% <100.00%> (ø)

src/gtvision/creators/dataloader2/base.py
100.00% <100.00%> (ø)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Local/Remote IOM connection - Add Library panel support

Describe the solution you'd like
When using a local/remote IOM connection, I would like to see a Libraries panel similar to what is already support for SAS Viya 4 (REST) connections.
The libraries panel should support:

Listing all libraries defined on the SAS server
Expanding a library should list all tables in that library
Dragging a table into a SAS Code editor tab should result in the string . being added to the editor
An option to download a table that was selected in the Libraries panel, which downloads the table content as a CSV and asks the user for a location and name to store the CSV file

Note: Clicking on a table should open a table viewer, but that is a separate (but related) feature request, see #595.
Environment
Local/Remote IOM connection to SAS 9

As part of library panel support, look into using eventing for managing ods html output. See comment here: https://github.com/sassoftware/vscode-sas-extension/pull/592#discussion_r1447785917

As part of library panel support, look into using eventing for managing ods html output. See comment here: #592 (comment)

This will not be a part of implementing library support.


[RHCLOUD-28745] Add a new disable flag for deployments

I want the ability to disable reconciliation for a single deployment but not disable the entire clowdapp.

I am going to defer this. We're going to try it at the clowdapp level.

Can we closed this out?


unable to edit a downloded file.

Type: Feature Request
I am unable to edit an html file which i have permission for after opening the file in vs studio. could you please help?
here is the file source:  https://www.dropbox.com/s/elkh6vcct1r84ev/my-first-web-page.zip?dl=1 https://www.dropbox.com/s/elkh6vcct1r84ev/my-first-web-page.zip?dl=1
VS Code version: Code 1.85.1 (0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2, 2023-12-13T09:49:37.021Z)
OS version: Windows_NT x64 10.0.19043
Modes:

We closed this issue because it is a question about using VS Code rather than an issue or feature request. Please search for help on StackOverflow, where the community has already answered thousands of similar questions. You may find their guide on asking a new question helpful if your question has not already been asked. See also our issue reporting guidelines.
Happy Coding!


Remote AWS Environment Variables code example in README makes no sense

Context
The code shown in this section appears to be checking that the code isn't running locally?
if 'SERVERTYPE' in os.environ and os.environ['SERVERTYPE'] == 'AWS Lambda':
appears to be a check that the code is running in Lambda, when it should fully inverted as far as I can tell? I'm happy to fix this myself, I just want to ensure I am correctly interpreting what's happening here.

Expected Behavior

N/A
Actual Behavior

N/A
Possible Fix

not the if
Steps to Reproduce

N/A
Your Environment

N/A

Hey @texonidas, thanks for the report.  I agree with you - this code example in the README doesn't make sense to me.  In addition to the issue you pointed out, in the very next line it calls import os, but the os module would clearly have to been already imported since the line before is already using os.environ... 🙄
Unfortunately, this portion of the README is 6 years old, so I have no idea what the original author was thinking.  But, I'm convinced it should be changed, so I will be altering that line to provide an example of the modern canonical check for whether code is running in the Lambda runtime environment or not.  Hope that will clear things up for everyone!

PR with the fix is here: #1281 - we should be able to get that merged and included in the next release.  Thanks again for pointing it out!

Thank you! Muuuuuch clearer now


Wrong redirect for link Use PowerShell to register and scan Power BI

[Enter feedback here]
The link Use PowerShell to register and scan Power BI redirects to the same page (not powershell instructions)

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f2058d99-ebe7-6d5d-1337-43de20df8116
Version Independent ID: 643c4310-00e2-3dac-c0db-7e4f9c588d22
Content: Register and scan a Power BI tenant (preview) - Azure Purview
Content Source: articles/purview/register-scan-power-bi-tenant.md
Service: purview
Sub-service: purview-data-catalog
GitHub Login: @viseshag
Microsoft Alias: viseshag

@jovalen Thanks for the feedback. We shall evaluate and update the document as appropriate.

@jovalen This seems to have been a recent change to the docs. I'm checking with the author and will get back to you as I hear more.

@jovalen,  ADC PowerShell module is not yet available, that's the reason the link is redirecting to the same page.
Once the ADC PowerShell module is available, we will update the link to point to the right document.
Hope this helps.


To implement /v1/embed-yourself

[x] old-release-protocol,old-release-address,old-release-port - request body attributes are made as optional
[x] registry-office-application-release-number - description changed from update to "find or update"

Issue merged to branch v2.1.1_impl as a part of the PR https://github.com/openBackhaul/ApplicationPattern/pull/967.
Hence closing this issue.