Invoke-RestMethod 7.4 not working with charset, multipart/form-data for .pfx certificate and target device doesn't seem to accept charset. PS 7.3 works.

Prerequisites

[X] Write a descriptive title.
[X] Make sure you are able to repro it on the latest released version
[X] Search the existing issues.
[X] Refer to the FAQ.
[X] Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce
I have tried several attempts over countless hours, and nothing seems to work.  I've also read dozens of Issues (open and closed).
I'm uploading a multipart/form-data certificate (.pfx) from a decoded Base64 Latin1/ISO-8895-1 string. Setting charset just returns an error code. I've had the same issue with another device but setting -ContentType="application/x-pkcs12;boundary=someBoundary;charset=iso-8895-1" worked. but for this device nothing works. Can't upload it as UTF-8 and can't pass charset. I guess e.g. Chrome detects the form-data/file's encoding and submits it accordingly or it uses ISO-8859-1/ASCII.
I don't have access to modify the device and I don't see any way to force the old pwsh behavior.

Could it be that the device doesn't accept a charset?
Is there a way to force the old behavior, like something of the sort

pwsh::SetRequestEncoding [System.Text.Encoding]::Latin1
Invoke-RestMethod ...
pwsh::SetRequestEncoding [pwsh]::DefaultRequestEncoding

I can't imagine I'm the only person having an issue now that it defaults to UTF-8
We have dozens of these devices and they need to be updated automatically

Expected behavior
Any way to force previous behavior or detect charset/encoding automatically.

Actual behavior
No way for to use PS 7.4 when device doesn't accept charset.

Error details
No response
Environment data
Name                           Value
----                           -----
PSVersion                      7.4.2
PSEdition                      Core
GitCommitId                    7.4.2
OS                             Microsoft Windows 10.0.22000
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

Visuals
No response

A PFX certificates is PKCS12 format data which ASN.1 which in turn is binary. There should be no character set involved because there is nothing to translate. The Base64 data was text, when you decode Base64 you have an octet-stream or byte array.
Your mime type looks correct, what is the charset for?

A PFX certificates is PKCS12 format data which is ASN.1 which in turn is binary. There should be no character set involved because there is nothing to translate. The Base64 data was text, when you decode Base64 you have an octet-stream or byte array. Your mime type looks correct, what is the charset for?

I pass the file data like:
$boundary = 'someBoundary'
 $body = (
            "--$boundary",
            "Content-Disposition: form-data; name=`"PWD1`"$LF",
            $certPw,
            "--$boundary",
            "Content-Disposition: form-data; name=`"FILE`"; filename=`"certificate.pfx`"",
            "Content-Type: application/x-pkcs12$LF",
            [System.Text.Encoding]::Latin1.GetString($bytes),
            "--$boundary--$LF"
        ) -join $LF

        $response = Invoke-RestMethod -ContentType "multipart/form-data; boundary=$boundary" -Uri $certupload_uri -Method Post -Body $body -WebSession $session -ErrorAction Stop

...of course I also tried the same with $mulitpart = [System.Net.Http.MultipartFormDataContent]::new()
I mean, when I see the sent Form-Data it's in now in UTF-8 data, which doesn't work, so setting it to Latin1 should correctly send the data with ISO-8895-1 encoding. As mentioned, that's also what happens when I manually upload it. For the other devices setting charset=iso-8895-1 correctly POSTed the data with that encoding. In Fiddler I see the content is correctly set to ANSI (when viewing in Notepad) which is obviously not the case with 7.4, as it defaults to UTF-8.

I must admit I am not familiar with multipart/form-data
This looks wrong
The bytes are bytes and should not be converted, they should stay as bytes. I don't know how the length of the mime data within each part is encoded.
My example
$ od -t x1  random.bin
0000000 0f cc eb 1c bc b7 30 8f aa aa bc 40 77 d4 a7 65
0000020 9c 5a fb 35 ee b2 1e bf af 89 ea 58 94 8d 5b 2b
0000040
$ curl --trace - --user guest:changeit --form<EMAIL_ADDRESS>--fail http://localhost:8080
== Info:   Trying <IP_ADDRESS>:8080...
== Info: Connected to localhost (<IP_ADDRESS>) port 8080 (#0)
== Info: Server auth using Basic with user 'guest'
=> Send header, 229 bytes (0xe5)
0000: 50 4f 53 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d POST / HTTP/1.1.
0010: 0a 48 6f 73 74 3a 20 6c 6f 63 61 6c 68 6f 73 74 .Host: localhost
0020: 3a 38 30 38 30 0d 0a 41 75 74 68 6f 72 69 7a 61 :8080..Authoriza
0030: 74 69 6f 6e 3a 20 42 61 73 69 63 20 5a 33 56 6c tion: Basic Z3Vl
0040: 63 33 51 36 59 32 68 68 62 6d 64 6c 61 58 51 3d c3Q6Y2hhbmdlaXQ=
0050: 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 63 75 ..User-Agent: cu
0060: 72 6c 2f 37 2e 38 31 2e 30 0d 0a 41 63 63 65 70 rl/7.81.0..Accep
0070: 74 3a 20 2a 2f 2a 0d 0a 43 6f 6e 74 65 6e 74 2d t: */*..Content-
0080: 4c 65 6e 67 74 68 3a 20 32 33 34 0d 0a 43 6f 6e Length: 234..Con
0090: 74 65 6e 74 2d 54 79 70 65 3a 20 6d 75 6c 74 69 tent-Type: multi
00a0: 70 61 72 74 2f 66 6f 72 6d 2d 64 61 74 61 3b 20 part/form-data;
00b0: 62 6f 75 6e 64 61 72 79 3d 2d 2d 2d 2d 2d 2d 2d boundary=-------
00c0: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
00d0: 2d 62 39 38 64 33 38 39 63 64 62 34 62 35 32 64 -b98d389cdb4b52d
00e0: 32 0d 0a 0d 0a                                  2....
=> Send data, 234 bytes (0xea)
0000: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
0010: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 62 39 38 64 33 38 ----------b98d38
0020: 39 63 64 62 34 62 35 32 64 32 0d 0a 43 6f 6e 74 9cdb4b52d2..Cont
0030: 65 6e 74 2d 44 69 73 70 6f 73 69 74 69 6f 6e 3a ent-Disposition:
0040: 20 66 6f 72 6d 2d 64 61 74 61 3b 20 6e 61 6d 65  form-data; name
0050: 3d 22 66 69 6c 65 22 3b 20 66 69 6c 65 6e 61 6d ="file"; filenam
0060: 65 3d 22 72 61 6e 64 6f 6d 2e 62 69 6e 22 0d 0a e="random.bin"..
0070: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 70 Content-Type: ap
0080: 70 6c 69 63 61 74 69 6f 6e 2f 6f 63 74 65 74 2d plication/octet-
0090: 73 74 72 65 61 6d 0d 0a 0d 0a 0f cc eb 1c bc b7 stream..........
00a0: 30 8f aa aa bc 40 77 d4 a7 65 9c 5a fb 35 ee b2<EMAIL_ADDRESS>00b0: 1e bf af 89 ea 58 94 8d 5b 2b 0d 0a 2d 2d 2d 2d .....X..[+..----
00c0: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
00d0: 2d 2d 2d 2d 2d 2d 62 39 38 64 33 38 39 63 64 62 ------b98d389cdb
00e0: 34 62 35 32 64 32 2d 2d 0d 0a                   4b52d2--..

I can see the 32 bytes of my file random.bin in the content from 009A to 00B9 but I don't know how it encodes that there are 32 bytes there. I can see that where is a boundary immediately following.
Given you are trying to pass binary data I don't think you can create the body with string appends. I think you may need something like MultipartFormDataContent Class

https://github.com/PowerShell/PowerShell/issues/21386#issuecomment-2023863985 Has an example of building form data for Invoke-WebRequest, specifically with raw file bytes.

Thank you both for the responses and examples!

I posted a multiform example in the comment above too (shortly after answering, so not after you responded), that version has the exact same problem.
The text version worked when I was able to give the charset (for other devices) or using pwsh 7.3, and am 99.99% sure the multiform version will work with 7.3 too, but for this issue without being able to tell it to encode the body as Latin1/ISO-8859-1 it won't work.
The problem is that the Invoke-... will still post it as UTF-8 encoding, since 7.4. The other person also mentions it still didn't work.
Ideally I can force powershell to Invoke-... with using the previous encoding without having to pass "charset" as the device doesn't seem to accept it.

it expects the bytes to be Latin1.

The end system doesn't expect the bytes to be Latin1, it expects them to just be bytes. Raw bytes have no character encoding, they are just eight bits. Character-set encoding only applies to text. What has changes is that PowerShell's default character set it UTF8, and you may have previously been lucky with the encoding and it was a one to one mapping, but really there should be no mapping at all.

Thanks for the clarification, that clears that part up for me! What that really means is that the form-data submitted needs to be Latin1 (and the file bytes within the form-data has nothing to do with the issue). The issue still stands but doesn't specifically relate to the .pfx bytes.
Note that when base64 decoding the file and saving it as UTF-8, uploading the .pfx manually doesn't work, this is probably because then Chrome will see the file was saved with UTF-8 encoding and post the form-data with UTF-8 encoding.
What I still don't understand what to do with the link, when I wrote in my first answer that it's the exact same issue, the device returns a JSON with the exact same error code. Here my code again, as in the first answer, using mulitpart-form with just bytes, exact same issue. Maybe I should delete the Latin1 part from my answers and just post the multipart-form version, as the issue is I can't send charset but the body may not be UTF-8 encoded:
$bytes = [System.Convert]::FromBase64String('MII....=')

$FileHeader = [System.Net.Http.Headers.ContentDispositionHeaderValue]::new('form-data')
$FileHeader.Name = 'FILE'
$FileHeader.FileName = 'certificate.pfx'
$FileContent = [System.Net.Http.StreamContent]::new([System.IO.MemoryStream]::new($bytes))
$FileContent.Headers.ContentDisposition = $FileHeader
$FileContent.Headers.ContentType = [System.Net.Http.Headers.MediaTypeHeaderValue]::Parse('application/x-pkcs12')

$MultipartContent = [System.Net.Http.MultipartFormDataContent]::new()
$MultipartContent.Add([System.Net.Http.StringContent]::new($certPw), 'PWD')
$MultipartContent.Add($FileContent)

Invoke-RestMethod -Body $MultipartContent ....

If I was going to try and solve this, I would get to the basics.
Can you do the equivalent with curl ( I mean real curl, not curl the PowerShell 5.1 alias)?
Something like
Set-Content -LiteralPath 'certificate.pfx' -Value $bytes -AsByteStream
curl --trace - --user XXX:YYYY --form<EMAIL_ADDRESS>--fail http://yoururl

Can you use Invoke-WebRequest to do the post to make sure the Cmdlet is not doing any extra translation of the data body?

Thanks a lot for all your help so far!
I will definitely try your suggestions, but it'll be on Monday, and comment. I'll try the MultipartForm version too, but am certain that'll work with 7.3. I'll definitely try the Invoke-WebRequest and see about curl, although I don't know curl very well and I will need to pass session data after logging in to the device.

I notice that the MultipartFormDataContent has a mechanism to convert the whole encoding to a byte stream, I think you need to be sending those bytes exactly as is through Invoke-WebRequest.

I haven't followed this thread in detail, so there may be a better solution, but note that you can pass a [byte[]] array directly to the -Body argument, so to emulate the 7.3- behavior you can try to ISO-8859-1-encode the entire $body string:
$body = [System.Text.Encoding]::Latin1.GetBytes((
    "--$boundary",
    "Content-Disposition: form-data; name=`"PWD`"`n",
    $certPw,
    "--$boundary",
    "Content-Disposition: form-data; name=`"FILE`"; filename=`"certificate.pfx`"",
    "Content-Type: application/x-pkcs12`n",
    [System.Text.Encoding]::Latin1.GetString($bytes),
    "--$boundary--`n"
  ) -join "`n")

Note that any single-byte, 8-bit, fixed-width text encoding (all of whose code points must be valid) can be used to losslessly encode arbitrary binary data.
ISO-8859-1 encoding (which is closely related to, but not the same as Windows-1252) is special in that it is a true subset of Unicode, covering the 8-bit range of Unicode code points (� (NULL, U+0000) through ÿ (LATIN SMALL LETTER Y WITH DIAERESIS, U+00FF)
As such, you can cast arbitrary binary data to a .NET string, and the Unicode code points of the resulting characters will reflect the original byte values; e.g.:
[byte[]] $bytes = 0..255

$str = [string]::new([char[]] $bytes)

# -> 0..255
[int[]] [char[]] $str

This is my solution, you need $path and $uri set up, My credentials are just as an example. This has no text or character sets. Everything is done using System.Net.Http objects to do the correct encoding.
Set up credentials
[string]$userName = 'guest'
[string]$userPassword = 'changeit'
[securestring]$secStringPassword = ConvertTo-SecureString $userPassword -AsPlainText -Force
[pscredential]$credObject = New-Object System.Management.Automation.PSCredential -ArgumentList $userName, $secStringPassword

Read file into memory and get name and content-type
[byte[]]$bytes=Get-Content -LiteralPath $Path -AsByteStream -Raw
$name=(Get-Item -LiteralPath $Path).Name
[string]$contentType = 'application/octet-stream'

Create the multippart body
$multipartFormDataContent = New-Object -TypeName 'System.Net.Http.MultipartFormDataContent'
$byteArrayContent = New-Object -TypeName 'System.Net.Http.ByteArrayContent' -ArgumentList @(,$bytes)
$multipartFormDataContent.Add($byteArrayContent,'FILE',$name)

Get bytes and contentType from the multipart content
$memory = New-Object -TypeName 'System.IO.MemoryStream'
$multipartFormDataContent.ReadAsStream().CopyTo($memory)
$bytes = $memory.ToArray()
$contentType = $multipartFormDataContent.Headers.ContentType

Finally do the post with Invoke-WebRequest
Invoke-WebRequest -Credential $credObject -AllowUnencryptedAuthentication -Uri $Uri -Method 'POST' -Body $bytes -ContentType $contentType

In my case I get a 201 created or a  204 no content as a successful response.

You guys are the best! It worked!
@mklement0 It worked as you mentioned. I had to save the bytes to a variable first. For some reason I couldn't get doing the GetBytes within the Invoke-RestMethod to work, and actually tried that before too.
@rhubarb-geek-nz I tried quite a while but couldn't fully get it to work, but super helpful for using the MultipartFormDataContent. I tried to use your example with the solution of getting the bytes from Latin1. I think the issue is that to pass the body I just need the MultipartFormDataContent Content and not the headers etc., so not sure getting the bytes returns other stuff too? I tried removing the headers too, but that didn't fully work. I guess I could use the enumerator and then parse the bytes, but for now I'm just happy I have a solution.
Thank you both very very much!

Glad to hear it helped, @ray-wn (I've updated my previous comment to amend the background information about encodings).
You don't strictly a need an intermediate variable, though, so I suspect you merely had a syntax problem, which is presumably easily remedied by enclosing the [System.Text.Encoding]::Latin1.GetBytes() call in (...); a simplified example:
# Note the (...) around the [System.Text.Encoding]::Latin1.GetBytes() call.
Write-Host ([System.Text.Encoding]::Latin1.GetBytes('abc'))

In argument-parsing mode, [ is not a metacharacter; to force recognition of an expression such as a method call starting with [System.Text.Encoding] as such, enclosure in (...) is needed.

Thanks again @mklement0!!! You're absolutely right, I just retried it as I was certain that's what I did, but it worked. I have no idea what I must have done, as adding parentheses would be the first logical thing to do. Guess I couldn't see the forest for the trees.

@rhubarb-geek-nz You're totally right and I agree that this is the cleanest way. Right now I have other issues to resolve, but will definitely revisit it when I have a bit more time to look into it, but thanks a lot for the code and all your help!


Fix notebook invalid JSON

This is one of the issues preventing the site from updating.

thanks @markmcd cc @MarkDaoust @shilpakancharla


Can't find ZeroMQ from CMake project

Hi,
I installed ZeroMQ using "vcpkg install zeromq:x64-windows" command. But when I try to use ZeroMQ in my project, I got the following CMake error:
CMake Error at C:/vcpkg/installed/x64-windows/share/zeromq/ZeroMQConfig.cmake:48 (get_target_property):
  get_target_property() called with non-existent target "libzmq-static".
Call Stack (most recent call first):
  C:/vcpkg/scripts/buildsystems/vcpkg.cmake:222 (_find_package)
  CMakeLists.txt:48 (find_package)

It seems ZeroMQConfig.cmake wants to find both static and dynamic versions of ZeroMQ. This is wrong behavior because "zeromq:x64-windows" was supposed to be a dynamic library.

I have a similar issue.  CMake can no longer find libzmq.dll using find_library.  It worked a few months ago.  It also seems that since then the verbosity of the build was reduced so I can no longer see which files vcpkg actually installs.  I reproduced my problem in a test repo and set up a test build on AppVeyor.
https://github.com/hmenke/vcpkg_zeromq/tree/master
https://ci.appveyor.com/project/hmenke/vcpkg-zeromq/build/1.0.1


Allow custom tool installation

In addition to the tools available on the CVMFS, we need to enable the option to install additional tools. There are probably 2 parts to this: (1) data persistence via CloudMan, and (2) Galaxy configuration. In this issue, we'll focus on the Galaxy configuration.

[x] Add a local shed_tool_conf.xml
[x] Define an SQLite install database
[ ] Test and document the procedure

First part done with init container here: https://github.com/CloudVE/galaxy-kubernetes/pull/34

See this commit message in the above commit but it turns out we do not need a dedicated install database because we are not packaging a deployment to include tool installations.

How would we go about building a container with preinstalled tools though?

We would not without an install database.


jinja2 in Python 3 import error

I installed Jupyterhub on Ubuntu 14.04, then run it as sudo. When logging in, the following error comes up:

[I 2016-02-01 13:28:07.586 JupyterHub spawner:409] Spawning jupyterhub-singleuser --user=michael --port=51590 --cookie-name=jupyter-hub-token-michael --base-url=/user/michael --hub-prefix=/hub/ --hub-api-url=http://localhost:8081/hub/api --ip=localhost
Traceback (most recent call last):

File "/usr/local/bin/jupyterhub-singleuser", line 15, in <module>

from jinja2 import ChoiceLoader, FunctionLoader

ImportError: cannot import name 'ChoiceLoader'

I tried reinstalling Jupyterhub and jinja2.
When I run python3, I can do import jinja2; dir(jinja2) and it looks like something is wrong there:

>>> import jinja2; dir(jinja2)
['doc', 'loader', 'name', 'package', 'path', 'spec']

Here is my system information:

$ python3 -c "import IPython; print(IPython.sys_info())"
{'codename': 'An Afternoon Hack',
'commit_hash': '9c4f59d',
'commit_source': 'installation',
'default_encoding': 'UTF-8',
'ipython_path': '/usr/lib/python3/dist-packages/IPython',
'ipython_version': '1.2.1',
'os_name': 'posix',
'platform': 'Linux-3.16.0-30-generic-x86_64-with-Ubuntu-14.04-trusty',
'sys_executable': '/usr/bin/python3',
'sys_platform': 'linux',
'sys_version': '3.4.3 (default, Oct 14 2015, 20:28:29) \n[GCC 4.8.4]'}

What's jinja2.__version__ and/or jinja2.__path__?

>>> jinja2.__path__

_NamespacePath(['/usr/local/lib/python3.4/dist-packages/jinja2'])

there's no __version__

Then it looks like you have a partial install of jinja2 (i.e. missing everything). Try:
pip install --upgrade jinja2

If that doesn't work, you might try uninstalling it first.

I'm really at a loss here. Is it a requirement of python 2.7, or 3.4? Those are the two I have installed locally, but I believe Jupyterhub uses 3.4.
sudo pip uninstall jinja2

Not uninstalling Jinja2 at /usr/lib/python2.7/dist-packages, owned by OS

sudo pip3 uninstall jinja2

...
Successfully uninstalled Jinja2

Want to verify that it was removed:
sudo ls /usr/local/lib/python3.4/dist-packages/jinja*

ls: cannot access /usr/local/lib/python3.4/dist-packages/jinja*: No such file or directory

Looks like it's gone, so reinstall:
sudo pip3 install --upgrade jinja2

Downloading/unpacking jinja2
Downloading Jinja2-2.8-py2.py3-none-any.whl (263kB): 263kB downloaded
Requirement already up-to-date: MarkupSafe in /usr/local/lib/python3.4/dist-packages (from jinja2)
Installing collected packages: jinja2
Successfully installed jinja2
Cleaning up...

Now check if it was completely installed:
python3

Python 3.4.3 (default, Oct 14 2015, 20:28:29)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import jinja2
>>> dir(jinja2)
['doc', 'loader', 'name', 'package', 'path', 'spec']

And what's jinja2 after that, and what files do you see when you ls the directory that you find?

>>> jinja2.__path__

_NamespacePath(['/usr/local/lib/python3.4/dist-packages/jinja2'])

$ sudo ls /usr/local/lib/python3.4/dist-packages/jinja2

bccache.py constants.py environment.py filters.py loaders.py optimizer.py runtime.py tests.py
_compat.py   debug.py exceptions.py init.py  meta.py parser.py sandbox.py utils.py
compiler.py defaults.py ext.py lexer.py nodes.py pycache   _stringdefs.py  visitor.py

Something's really fishy. What do you get from from jinja2 import loaders?
When you uninstall jinja2, does that directory still exist?

>>> from jinja2 import loaders

Traceback (most recent call last):
File "", line 1, in
ImportError: cannot import name 'loaders'

When I'd uninstalled jinja2, the directory was gone:
sudo ls /usr/local/lib/python3.4/dist-packages/jinja*

ls: cannot access /usr/local/lib/python3.4/dist-packages/jinja*: No such file or directory

It looks like there is something wrong with my pip3. I worked around the above errors by using apt-get to install the packages jinja2,tornado under Python 3.4. Now jinja2 and tornado are installed to /usr/lib/python3/dist-packages/ and seem to be installed correctly.
Unfortunately, now there is an error under notebook:

[I 2016-02-02 17:32:58.097 JupyterHub spawner:409] Spawning jupyterhub-singleuser --user=michael --port=42835 --cookie-name=jupyter-hub-token-michael --base-url=/user/michael --hub-prefix=/hub/ --hub-api-url=http://localhost:8081/hub/api --ip=localhost
Traceback (most recent call last):
File "/usr/local/bin/jupyterhub-singleuser", line 43, in
from notebook.notebookapp import NotebookApp, aliases as notebook_aliases
ImportError: No module named 'notebook.notebookapp'

I tried also installing ipython-notebook for python3 with apt-get, but that wasn't found in the class path for jupyterhub. It appears that my pip3-installed packages are just not installing completely.

Try this:
pip3 install --upgrade setuptools pip
pip3 uninstall jinja2 tornado
pip3 install --upgrade notebook jupyterhub

pip3 install --upgrade notebook jupyterhub ended with this:
Installing collected packages: jinja2, tornado, notebook, jupyterhub, ipython
  Found existing installation: ipython 4.0.3
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.4/dist-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/usr/local/lib/python3.4/dist-packages/pip/req/req_set.py", line 725, in install
    requirement.uninstall(auto_confirm=True)
  File "/usr/local/lib/python3.4/dist-packages/pip/req/req_install.py", line 709, in uninstall
    for path in pip.wheel.uninstallation_paths(dist):
  File "/usr/local/lib/python3.4/dist-packages/pip/wheel.py", line 532, in unique
    for item in fn(*args, **kw):
  File "/usr/local/lib/python3.4/dist-packages/pip/wheel.py", line 551, in uninstallation_paths
    r = csv.reader(FakeFile(dist.get_metadata_lines('RECORD')))
  File "/usr/local/lib/python3.4/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 1624, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/usr/local/lib/python3.4/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 1621, in get_metadata
    return self._get(self._fn(self.egg_info, name)).decode("utf-8")
  File "/usr/local/lib/python3.4/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 1727, in _get
    with open(path, 'rb') as stream:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.4/dist-packages/ipython-4.0.3.dist-info/RECORD'

Arg, that seems like setuptools and/or pip has really messed itself up. When it gets to that degree, I'm tempted to trash the entirety of /usr/local/lib/python3.4/dist-packages and start over, but be careful when doing something that destructive.

@statueofmike If you installed pip or pip3 using apt-get it is possible that you may have an old version of pip or pip3. Before trashing the entirety of /usr/local/lib/python3.4/dist-packages, you may wish to just remove pip and pip3 (and possibly setuptools). http://stackoverflow.com/questions/28917534/pip-broken-on-ubuntu-14-4-after-package-upgrade

You could also use a virtualenv to see if the problem goes away there, instead of trashing your /usr/local/lib/

SOLVED
I tried reinstalling pip and pip3 altogether first. This didn't seem to help.
Then I trashed my entire /usr/local/lib/python3.4/dist-packages (backed up).
Here I realized the problem: for some reason, the file system permissions were restricting user Python sessions from loading these packages properly.
$ python3
Python 3.4.3 (default, Oct 14 2015, 20:28:29)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import jinja2
>>> dir(jinja2)
['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']

sudo chmod -R 775 /usr/local/lib/python3.4/dist-packages/
$ python3
>>> import jinja2
>>> dir(jinja2)
['BaseLoader', 'BytecodeCache', 'ChoiceLoader', 'DebugUndefined', 'DictLoader', 'Environment', 'FileSystemBytecodeCache', 'FileSystemLoader', 'FunctionLoader', 'Markup', 'MemcachedBytecodeCache', 'ModuleLoader', 'PackageLoader', 'PrefixLoader', 'StrictUndefined', 'Template', 'TemplateAssertionError', 'TemplateError', 'TemplateNotFound', 'TemplateSyntaxError', 'TemplatesNotFound', 'Undefined', 'UndefinedError', '__all__', '__builtins__', '__cached__', '__doc__', '__docformat__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_compat', '_stringdefs', 'bccache', 'clear_caches', 'compiler', 'contextfilter', 'contextfunction', 'defaults', 'environment', 'environmentfilter', 'environmentfunction', 'escape', 'evalcontextfilter', 'evalcontextfunction', 'exceptions', 'filters', 'is_undefined', 'lexer', 'loaders', 'make_logging_undefined', 'nodes', 'optimizer', 'parser', 'runtime', 'tests', 'utils', 'visitor']

Now I seem to have created a NEW problem. The Jupyterhub launcher also had too restrictive permissions. I'd installed with sudo -H pip3 install jupyterhub; sudo -H pip3 install notebook
$ ls -lah /usr/local/bin
-rw-r--r--  1 root root   59 Feb 15 11:47 jupyterhub
-rw-r--r--  1 root root 7.9K Feb 15 11:47 jupyterhub-singleuser

Open Question
Why did I have this file permission issue? Should I have installed differently?

That's quite strange. Do you see similar permission issues on any other packages?

I'm seeing other strange errors now.
$ pip3 install matplotlib
Collecting matplotlib
  Downloading matplotlib-1.5.1.tar.gz (54.0MB)
    100% |████████████████████████████████| 54.0MB 8.9kB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "/tmp/pip-build-y9xlz1k7/matplotlib/distribute_setup.py", line 152, in use_setuptools
        pkg_resources.require("setuptools>=0.7b")
    AttributeError: 'module' object has no attribute 'require'

Jupyterhub throws the following error:
[I 2016-02-15 12:53:31.558 JupyterHub spawner:409] Spawning jupyterhub-singleuser --user=michael --port=59855 --cookie-name=jupyter-hub-token-michael --base-url=/user/michael --hub-prefix=/hub/ --hub-api-url=http://localhost:8081/hub/api --ip=localhost
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/path.py", line 123, in <module>
    __version__ = pkg_resources.require('path.py')[0].version
AttributeError: 'module' object has no attribute 'require'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/jupyterhub-singleuser", line 21, in <module>
    from IPython.utils.traitlets import (
  File "/usr/local/lib/python3.4/dist-packages/IPython/__init__.py", line 49, in <module>
    from .terminal.embed import embed
  File "/usr/local/lib/python3.4/dist-packages/IPython/terminal/embed.py", line 16, in <module>
    from IPython.core.interactiveshell import DummyMod
  File "/usr/local/lib/python3.4/dist-packages/IPython/core/interactiveshell.py", line 31, in <module>
    from pickleshare import PickleShareDB
  File "/usr/local/lib/python3.4/dist-packages/pickleshare.py", line 41, in <module>
    from path import Path
  File "/usr/local/lib/python3.4/dist-packages/path.py", line 126, in <module>
    except pkg_resources.DistributionNotFound:
AttributeError: 'module' object has no attribute 'DistributionNotFound'```

That means that the pkg_resources module, the underpinning of setuptools itself, is completely broken or missing. When you do
import pkg_resources
print(pkg_resources.__file__)

what do you see?

I see this:
>>> import pkg_resources
>>> print(pkg_resources.__file__)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute '__file__'
>>> dir(pkg_resources)
['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']

And ls -la /usr/local/lib/python3.4/dist-packages/pkg_resources?

$ sudo ls -la /usr/local/lib/python3.4/dist-packages/pkg_resources
total 114
drwxr-Sr--  5 root staff   1024 Feb 15 12:46 .
drwxrwsr-x 64 root staff   3072 Feb 15 12:46 ..
drwxr-Sr--  3 root staff   1024 Feb 15 12:46 extern
-rw-r--r--  1 root staff 108616 Feb 15 12:46 __init__.py
drwxr-Sr--  2 root staff   1024 Feb 15 12:46 __pycache__
drwxr-Sr--  4 root staff   1024 Feb 15 12:46 _vendor

All users should have x permissions on directories in there, otherwise they can't read the files. The fact that you had to sudo ls is the same problem. In order for these packages to be imported, you have to be able to cd/ls/read all files in there.
If this is the only issue, you can fix it by ensuring that all directories are world-readable:
sudo find /usr/local/lib/python3.4/dist-packages -type d -exec chmod a+x "{}" \;

That's: find all directories in dist-packages and give everyone execute (aka listing) permissions on those directories

I ran the command to enable world-readable directories. pip3 has the same problem trying to install e.g. matplotlib.
Is there a safe way to just remove all python altogether and reinstall it? Maybe I used easy_install early on and messed things up that way?

You can remove all of the pip-installed packages easily enough, by removing /usr/local/lib/python3.4/, and start again with:
sudo python3 -m pip install --upgrade setuptools pip

That will leave some broken entrypoints in /usr/local/bin, but that shouldn't be too problematic as they will be replaced by subsequent installations.
It would be unwise to remove the system-installed packages without going through apt.

is it supposed to be sudo python3 -m pip ... or ... -m pip3 ... ?

@statueofmike It's as written by @minrk. :sunny:

Closing this issue as it's been inactive for several months. Please feel free to open a new issue at the jupyter/help repo or leave a comment requesting to reopen this issue. Thanks! :sunflower:


Loosen gemspec dependency on rest-client

There is a strick dependency in the gemspec so if a new version of rest_client is released (ie.1.6.8) things will break.  Keeping the dependency constraints loose on this (i.e. '>= 1.6.7') should avoid publishing a new version of right_api_client whenever an new rest-client is released.

"~> 1.6" is probably called for: which means anything >= 1.6.0 and <= 2.0.0


Fixed broken Link for Docker Reference Architecture

Proposed changes
Fixed broken link in networks/index.md file that was raised by @GMartinez-Sisti in issue #6713.
Fixes #6713

Thanks for addressing this! Let's ship it


Bumping to snapshot version for development

Codecov Report

Merging #44 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #44   +/-   ##
=========================================
  Coverage     59.73%   59.73%
  Complexity      140      140
=========================================
  Files            15       15
  Lines           611      611
  Branches         91       91
=========================================
  Hits            365      365
  Misses          224      224
  Partials         22       22

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a6d3925...8362e18. Read the comment docs.


Default pdftotext to UTF-8: Issue 1582

GitHub Issue: https://github.com/Islandora/documentation/issues/1582
What does this Pull Request do?
Changes the default output of pdftotext form Latin-1 to UTF-8.
What's new?

Added -enc UTF-8 as a parameter to pdftotext.

How should this be tested?

Take an existing Islandora 8 site and add a Digital Document node/media with a PDF original file including non-latin characters. I used https://fsi-languages.yojik.eu/languages/PeaceCorps/Arabic-Moroccan/MO_Arabic_Language_Lessons.pdf for my test case.
Ensure the text was extracted by looking at the node's extracted text media. You won't see any of the non-latin characters.
Apply the PR
Run the Extract Text action on the Digital Document node again to extract the text again.
Look at the extracted text media again and you should now see non-latin characters in your output.

Interested parties
@Islandora/8-x-committers

@mjordan when did you build the box? I did the PR based on @dannylamb's pre-built release box. As I mentioned to @jordandukart, this is likely dependent on the pdftotext version on the box where you have Crayfish installed.

Dev Playbook. Let me try to get it to fail on a release box.

I just built an 8.x-1.1 vagrant, and PDF text extraction is not working on the Arabic file above - Extracted Text for it never shows up. (This happens with and without the PR applied.) In fact, no other derivatives are created for that file, and when I view its parent node, even though I have the PDFjs display hint selected, it's not rendering in the node. Watchdog is empty. Derivatives are being created for other PDFs and the display hint is working as expected for them as well, so if I had to guess I'd say that PDF file is invalid or corrupted.
OTOH, text extraction on the other PDFs containing non-latin characters I was testing with earlier is successful in both the dev and 8.x-1.1 vms without the PR applied. In the 8.x-1.1 box without the PR, here's the extracted text from a PDF created from a LibreOffice word processing file encoded as Windows 1255 (Hebrew):

The text I extracted from the LibreOffice Writer file is in Windows 1255/ISO 8859:
mark@mark-ThinkPad-X1-Carbon-6th:~/Downloads$ file עוֹלָם.txt
עוֹלָם.txt: ISO-8859 text, with no line terminators

yet the text from the LibreOffice-derived PDF extracted by Islandora is in UTF-8:
mark@mark-ThinkPad-X1-Carbon-6th:~/Downloads$ file encodingtest.txt
encodingtest.txt: UTF-8 Unicode text

As long as adding the -enc UTF-8 parameter won't have any side effects, I don't see any harm in adding it, but I have been unable to reproduce the problem that adding it would solve while running pdftotext version 0.62.0, which is the version on both the dev and 8.x-1.1 vms.
Maybe someone else should test?

Huh, I just re-tested this by rolling back my VM to before the fix and it still worked... 🤷♂
I guess this PR isn't really necessary. No sense changing something doesn't need to be changed.


HTML5 D&D broken with 0.5.0

Resolve why D&D has broken, this prevents 0.5.0 release.

Might have to drop this feature, can't get it work well enough. Packery would offer functionality required, so maybe I should implement GPLv3 licensed add-on on top of that.

To be resolved in Packery add-on.

Resolved in gridstack add-on: https://github.com/alump/GridStack


New devserver

It seems that devserver has been deprecated. This is an ahead of time patch that should fix potential CI problems.
Let me know if you have other devserver preferences. Default port on sfz is 5000. I did not test the Makefile.

So there's a built-in one in firebase tools, which provides stuff like emulating the backend DB and things like that. My intention was that we would eventually use that, but it's maybe a bit a ways off still.

The makefile is malformed btw:
[cargo-make] ERROR - Unable to parse external file: "./Makefile.toml", expected a right bracket, found a string at line 68 column 3

The makefile is malformed btw:

Thanks for reporting this, fixed.

So there's a built-in one in firebase tools, which provides stuff like emulating the backend DB and things like that. My intention was that we would eventually use that, but it's maybe a bit a ways off still.

Oh cool I didn't know this. The main thing is that right now devserver is broken on my machine. I think the best course of action would be for me to merge this PR unless there is strong opposition, and then you can open a new one in the future with that changes. I expect it will mostly look like the diff in this PR, adapted to the firebase webserver ofc.

The makefile doesn't work for me :(

Do you have sfz installed?

Do you have sfz installed?

Yes!

Works on my machine, I only have a cargo watch error but the webserver spawns fine. Can you send me the error you're seeing?

I'm not getting any errors. It says "build done" but when I go to localhost:5000, I see nothing.

Oh yeah sometimes that happens, try to do a force refresh.

I've tried that numerous times, still doesn't work.


PHP Warning:  PHP Startup: redis: Unable to initialize module

After I installed redis extention, this happened.
How can I fixed it? THANKS.
docker image: mattrayner/lamp:latest-1404-php5
installation steps：
$ cd redis
$ phpize
$ ./configure
$ make
$ make install

It seems that there is no way to solve this problem except to reinstall PHP.


GET  /taxonomy/{uuid}/term API route does not work

Describe the bug
Requests to GET /taxonomy/{uuid}/term?path=TERM return a 200 response with no body, indicating that a term exists but not giving access to its UUID so no further operations can occur. For instance, this makes it difficult to nest child terms under existing terms when programmatically constructing a taxonomy.
To Reproduce
Steps to reproduce the behavior:

Create a taxonomy with a single term, e.g. "TERM"
Send a request to GET /taxonomy/{taxonomy uuid}/term?path=TERM
The response is an empty array []
The response headers also don't contain the term's UUID anywhere

Expected behavior
A single result should be returned like:
[
  {
    "term": "TERM",
    "fullTerm": "TERM",
    "uuid": "2ae3f425-b4aa-459d-933f-22f7a4b6683c",
    "index": 0,
    "readonly": false
  }
]

This is consistent with what happens when you send a request without a path parameter and receive a list of all the root terms complete with their UUIDs.
Screenshots
Here's the problem demonstrated in the apidocs.do tool:
No path parameters specified returns all the root-level terms, note the first one:

Specifying the first term's path causes nothing useful to be returned:

Platform:

OpenEquella Version: 2019.2
OS: Ubuntu on the server

Further details
This gap in the APIs is problematic for me because I often try to add child terms before checking that they already exist. I'd like to write a method that tries to add a term and returns the newly created term so that children can be nested under it; if there's a pre-existing a term, I need to catch the 406 "duplicate sibling" error, then send the existing term. This bug means the existing term cannot be found because /api/taxonomy/{uuid}/term doesn't give access to UUIDs and /api/taxonomy/{uuid}/search doesn't return any UUIDs in its results list (which seems like another bug? It mars the utility of the search results).

@cathfitz Hi Cath, it's working as intended.  No functionality was removed as a result of 2019.2 changes.  Possibly UUIDs not being returned was broken, but haven't had a chance to look into that.  (However, one of the tests inspects the UUID of a term, so it mustn't be broken on that endpoint at least)
@phette23
Hi Eric, the term?path=X endpoint returns child terms of the given path, hence the empty array returned.  There doesn't seem to be any way of retrieving an individual term other than by its UUID.  I assume if you supply a path that doesn't exist you will get a 404 response instead of a 200?  Can you use that to determine if the term exists or not?
I agree, the term UUIDs should be returned, and I can't see anything in the code preventing them being returned.  Will investigate tomorrow when I have an Equella instance to test against.

Thanks for the quick responses. I wasn't aware that path=X is meant to return child terms, that's not obvious from the way the API route is described.

There doesn't seem to be any way of retrieving an individual term other than by its UUID. I assume if you supply a path that doesn't exist you will get a 404 response instead of a 200? Can you use that to determine if the term exists or not?

This is the crux of the matter; I want to find a term's UUID given its path, but that's not possible, I can only determine if it exists or not, which isn't good enough. For instance, unless I'm missing something, I can't nest a child term underneath it since I need to know it's UUID to supply in the parentUuid field.
Perhaps I've just misunderstood the purpose of this route though and the real solution is to have the /taxonomy/{uuid}/search route return UUIDs in its results.


Build 3617+ crashes with

Standard Mach-O file. Crash seems related to the new string-related stuff:
* thread #27, stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
  * frame #0: 0x000000010c8d4450 libbinaryninjacore.1.dylib`BinaryNinjaCore::TypeData::GetOffset(this=0x0000000000000000) const at type.cpp:499:6 [opt]
    frame #1: 0x000000010c975fec libbinaryninjacore.1.dylib`BinaryNinjaCore::Unicode::CheckForUTF32Annotation(view=0x0000000104f98600, addr=4296120896, value=0x0000000175ea5c48, allowShortStrings=true, maxLength=32, unicodeBlocks=size=3) at unicode.cpp:836:31 [opt]
    frame #2: 0x000000010c0b7784 libbinaryninjacore.1.dylib`BinaryNinjaCore::Analysis::CheckForStringAnnotation(this=0x0000000144431a00, addr=4296120896, value=0x0000000175ea5c48, allowShortStrings=true) at analysis.cpp:1159:24 [opt]
    frame #3: 0x000000010c6d3550 libbinaryninjacore.1.dylib`BinaryNinjaCore::MediumLevelILTranslator::TranslateExpr(this=0x0000000175ea6800, src=0x00000002ac485a00, dest=0x000000017b270200, ...)

Paletro.zip

I'm also seeing a reliable crash when loading a certain dylib on 3.1.3617; let me know if more details are needed.

Resolved in builds >= 3.1.3622


docs: alias LATEST_VERSION as TS_LATEST_VERSION in example snippets

Fix https://github.com/aspect-build/rules_js/issues/817
WDYT of changing this in 2.0+? Could add it as an alias for now...

We could export both and then its not breaking

Only if we plan to remove it in 2.0. Sound good?

We have DEFAULT_NODE_VERSION in rules_nodejs and I saw this in one of our bazel-examples
load("@aspect_rules_esbuild//esbuild:repositories.bzl", "esbuild_register_toolchains", LATEST_ESBUILD_VERSION = "LATEST_VERSION")

esbuild_register_toolchains(
    name = "esbuild",
    esbuild_version = LATEST_ESBUILD_VERSION,
)

so perhaps be explicit and follow the same convention with LATEST_TYPESCRIPT_VERSION as the exported symbol with LATEST_VERSION around for backward compat and to be removed in 2.0?

Should probably make the same change in rules_swc, rules_esbuild & rules_cypress

Ok I've switched to the LATEST_*_VERSION style and marked the LATEST_VERSION to be removed in 2.0. If that looks right I'll do the same in other rulesets as well.


istio-proxy CPU overload

Bug description
Hi,
We recently had an issue caused by istio-proxy cpu going rather crazy as can be seen by the green line on this graph.  This resulted in no requests being handled by istio-proxy at all (just connect timing out).

The interesting thing here is that the proxy was maxed out at 1 core, we have no limits on our proxies so I think this is telling me that it's a single thread.
Unfortunately, due to a process breakdown, we didn't capture proxy logs at the time.
We have never seen this in the past, so feels it's potentially something related to 1.4.0 upgrade.  Based on nothing more than my gut, it feels like https://github.com/istio/istio/issues/19112 is some how related.
Expected behavior
Proxy not to go haywire
Steps to reproduce the bug
Unfortunately, we haven't been able to replicate it
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
1.4.0
How was Istio installed?
Helm
Environment where bug was observed (cloud vendor, OS, etc)
GKE v1.14.8-gke.17

From the graph it's more like https://istio.io/news/security/istio-security-2019-006/, but it is supposed to be fixed in 1.4
I feel there is no way to fix it if we only have this graph. However, more details are welcomed.
@Stono When you say no requests being handled by istio-proxy at all (just connect timing out), how did you obtain this assertion? This is the major divergence vs the above istio-security-2019-006

Yeah I didn't anticipate that we'd be able to root cause it from the graph alone, I was more trying to trigger "oh I've seen that before" style comments such as that security bulletin.
We actually already mitigated against that issue (and upgraded to 1.4 too) so I don't think that's the problem :-(
We obtained that by exec'ing onto the pod and trying to curl downstream services, which just connect timed out, the same call from other unaffected pods in the same workload was fine, and restarting the trouble pod resolved it.

Hello
We've just had this happen again on 1.4.5.
We've cordoned the pod which has a proxy stuck at 100%.
I've dumped all the config (can provide privately).
Proxies readiness probe passes but all upstream requests fail.

Is there anyone around that can help debug?

I believe the readiness probe is handled by a different thread than normal traffic, so it would make sense those are succeeding. That likely means the issue was on the data path side not from xds config or anything

@lambdai @yxue fyi

it's cordoned so it's taking no traffic right now, so it should be at 0 load but it's stuck at 1 cpu, right?

Correct.
As a side note a readiness probe should really reflect its ability to proxy traffic too!

That's a good point, we should probably proxy the readiness probe through
envoy. I'll open an issue to track that
On Fri, Feb 21, 2020, 7:18 AM Karl Stoney<EMAIL_ADDRESS>wrote:

Correct.
As a side note a readiness probe should really reflect its ability to
proxy traffic too!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/19634?email_source=notifications&email_token=AAEYGXLCDGH5J3TH75PZMITRD7WFLA5CNFSM4J33ZWQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTBEVQ#issuecomment-589697622,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEYGXMOYZH4QHFRWPX55TTRD7WFLANCNFSM4J33ZWQA
.

@Stono Is the cpu usage at userspace or sys?

Sorry still not much progress made this time. Looks like it's data plane worker threads.
If it happens in the future, we should try to obtain the cpu profile, or some quick info

worker threads(data path) or master thread(xds). The clue shows it's worker threads this time. Let's confirm next time.
Single worker threads or more?
cpu util sys or us?
^^ Those two question could be answered by top -H
logs around the moment when cpu util was sky rocketing.


Consider performing font fallback analysis only if the chosen font is not adequate

Description of the new feature/enhancement
Right now the _AnalyzeFontFallback is always called if the text is "complex":
https://github.com/microsoft/terminal/blob/e7108332f72a3c14c72a65484d091c4e79ab28b2/src/renderer/dx/CustomTextLayout.cpp#L246-L247
We already know that even for pure ASCII characters, some fonts will treat them as "complex". But the real need for font fallback emerges only when the font don't actually contains the needed glyph. For example when dealing with "ABC中文". You have to find "Microsoft YaHei" as fallback, otherwise "中文" is nowhere to be find in "Cascadia Code“.
Proposed technical implementation details (optional)
To avoid the unnecessary overhead of finding fallbacks (MapCharacters, which is relatively expensive), my proposed solution is to delay the fallback analysis and put it in _ShapeGlyphRun. Inside _ShapeGlyphRun, the method GetGlyphs will tell you precisely if there are available glyphs inside the desired font (_glyphIndices):
https://github.com/microsoft/terminal/blob/e7108332f72a3c14c72a65484d091c4e79ab28b2/src/renderer/dx/CustomTextLayout.cpp#L414

If no glyphs can be found (which I assume is a rather rare case), then the fallback analysis is necessary.

/cc @miniksa


Fixed bug of DiscreteMIP lax/tight precedence constraints

Please refer to the comment on commit.

Just checked, precedence constraints between groups of different size dont seem to work. This is the problem, right? In the example above replacing the size of group2 to 5 gives a valid solution.
However, this should also work for groups of different size. The root cause is that the constraint implementation
affine = [(x[P.task_left, t_],1) for t_ in range(t,self.horizon)] + \
       [(x[P.task_right, t_],-1) for t_ in range(min(t+P.task_left.length+P.offset,self.horizon),self.horizon)]
cons.append(mip.con(affine, sense=-1, rhs=0))

does not take different groups sizes into account, but I think it is possible to fix this by simply adding the difference in groups sizes to one of the sides of the constraint. I will check and test this on the weekend. This would not require additional variables which increase the solving time.
Thanks for pointing out this problem! If I will not be able to fix it this way I would merge the pull request.

This is the problem, right? In the example above replacing the size of group2 to 5 gives a valid solution.

Hmm... I'm not sure. In my opinion, even replace the size of group2 to 5, the answer seems not right either.
For example,
# output of "S += S['T2_0'] < S['T1_0']"
[(T2_0, R2, 0, 3), (T1_0, R1, 3, 5), (T2_1, R2, 3, 6), (T1_1, R1, 6, 8), (T2_2, R2, 6, 9), (T1_2, R1, 9, 11), (T2_3, R2, 9, 12), (T1_3, R1, 12, 14), (T2_4, R2, 12, 15), (T1_4, R1, 15, 17)]

Except the head tasks (T1_0, T2_0), the others remain in random order (My expected order is T2_0, T2_1,.... T2_4, T1_0, ... T1_4).
And because this statement:
if P.task_left not in self.task_groups or P.task_right not in self.task_groups:
	continue

I can't control the orders of them...
Perhaps it's my misunderstanding of the usage. Hope you can clarify, that's also why I posted this request. Thank you!

In addition, I just changed the action of up bound, tight up/low bound a little.
Mainly two points:

At least for my case, "T < x" means T must finish before x, so T.length should be considered.
For tight bound,
mip.con([(x[P.task, P.bound],1)], sense=0, rhs=task_group_size)
makes all tasks start exactly at P.bound... not very fit for my case.

First, regarding the ordering in
[(T2_0, R2, 0, 3), (T1_0, R1, 3, 5), (T2_1, R2, 3, 6), (T1_1, R1, 6, 8), (T2_2, R2, 6, 9), (T1_2, R1, 9, 11), (T2_3, R2, 9, 12), (T1_3, R1, 12, 14), (T2_4, R2, 12, 15), (T1_4, R1, 15, 17)]
This looks correct. To clarify the usage of precedence relations in groups, T2_0 < T1_0 implies that group2 should be before group1. This means that the first task in group2 should be before the first task in group1, the second task in group2 should be before the second task in group1, etc.
A scenario for this would be the production of some product which requires two tasks, first on one machine, and then on a second. So the xth task on the first machine should be before the xth task on the second machine...
The output above does that T1_0 starts at time step 3 after T2_0 finishes,T1_1 starts after time step 6 after T2_1 finishes, and so on. It is much easier to show this on a picture with plotters.matplotlib.plot, but it is not possible to include pictures here.
Does this make sense?
For the two machine use case described above, both groups need to have tha same size, in which case everything runs nicely. I spend some thinking what to do for different sized groups. Then it is not clear what is the best default, there are different options that make sense.

Second, regarding the bounds:

Yes, you are correct, there is a bug in the upper bound (T<0), the T.length is missing. Thanks. In your commit, are you sure about the +1 in affine = [(x[P.task, t],1) for t in range(P.bound-P.task.length+1,self.horizon)]? Also, it should be range(max(P.bound-P.task.length,0),self.horizon) to avoid going to negative periods.

Regarding the tight bounds (T<=x and T>=x). For groups, a tight bound T>=x would imply that all tasks in the group of task T should start in exactly period x. Your change

affine = [(x[P.task, t],1) for t in range(P.bound)]
cons.append(mip.con(affine, sense=0, rhs=0))
cons.append(mip.con([(x[P.task, P.bound],1)], sense=1, rhs=1))

would imply that at least one task in the group of task T should start in exactly period x. Now comes the question which makes more sense? I would probably go for your option, because the current "all" option can be implemented by also adding an upper bound, but the "one" option cannot.
After checking the +1 and the range problem, could you please create a pull request just for the bounds, i would then merge this in right away.

Thanks for your valuable explanations.

This looks correct. To clarify the usage of precedence relations in groups, T2_0 < T1_0 implies that group2 should be before group1. This means that the first task in group2 should be before the first task in group1, the second task in group2 should be before the second task in group1, etc.

OK, I see. Hmm, in my case perhaps different size is more common... anyway, I'll consider how to make use of this "group", or create another one.
And also about tight bounds, perhaps the "one" option makes more sense for me ...
Maybe, supplying options to users will be a good idea.
Very interesting discussion, thank you again.

After checking, I'm sure +1 should be necessary for up bounds...


go get error: could not determine kind of name for C.RBD_FEATURES_DEFAULT

root@9f755bc45121:/go# go get github.com/ceph/go-ceph
# github.com/ceph/go-ceph/rbd
src/github.com/ceph/go-ceph/rbd/rbd.go:32:23: could not determine kind of name for C.RBD_FEATURES_DEFAULT
src/github.com/ceph/go-ceph/rbd/rbd.go:30:28: could not determine kind of name for C.RBD_FEATURE_DATA_POOL
src/github.com/ceph/go-ceph/rbd/rbd.go:952:22: could not determine kind of name for C.rbd_trash_image_info_t
src/github.com/ceph/go-ceph/rbd/rbd.go:948:5: could not determine kind of name for C.rbd_trash_list
src/github.com/ceph/go-ceph/rbd/rbd.go:969:2: could not determine kind of name for C.rbd_trash_list_cleanup
src/github.com/ceph/go-ceph/rbd/rbd.go:264:18: could not determine kind of name for C.rbd_trash_move
src/github.com/ceph/go-ceph/rbd/rbd.go:979:18: could not determine kind of name for C.rbd_trash_remove
src/github.com/ceph/go-ceph/rbd/rbd.go:990:18: could not determine kind of name for C.rbd_trash_restore

OS: Debian stretch
golang: 1.10.3
libcephfs-dev, librbd-dev, librados-dev version: 10.2.5-7.2
I'm not sure if this is a version issue.

after upgrade libcephfs-dev to 12.2.8, this issue fixed.

how to upgrade?


Hide polls in Jitsi Chat in Flutter with jitsi_meet_flutter_sdk: ^0.1.7 ??

I want to know how to hide polls in Jitsi Chat in Flutter with jitsi_meet_flutter_sdk: ^0.1.7
We are getting Polls tab under Chat but we couldnt find out the way to rid of Polls tab from chat window.
Is anyone has any idea how we can hide Polls from chat window

Hey! Sorry for the late reply. Please try setting disablePolls: true in the configOverwrites.


Setting up Adhoc (wifi-less communication)

https://www.raspberrypi.org/forums/viewtopic.php?f=41&t=50070

Solved in #265


update tutorials

companion to https://github.com/stratum-mining/stratum/pull/726

I yet have to dig deep into reviewing the docs, but at the first glance it seems too detailed. Should we optimize for what we have in Google Docs already?

Run hosted setup
Run all roles

And just mention options for configuration, I was under impression we'll just port over entire Google Doc here and nuke anything else we have as currently this wall of text does seem a bit overwhelming.

I yet have to dig deep into reviewing the docs, but at the first glance it seems too detailed. Should we optimize for what we have in Google Docs already? As our goal for quick start is to allow anyone to test SV2 quickly.

Run hosted setup
Run all roles

And just mention options for configuration, I was under impression we'll just port over entire Google Doc here and nuke anything else we have as currently this wall of text does seem a bit overwhelming.

indeed, I went through what we currently have (mostly focused on Config C and D).
but if the team agrees I can also just port over the Google Docs instructions over here.
@GitGab19 @Fi3 wdyt?

or also @pavlenex can open a new PR going in this different direction, whatever you guys think is best

I yet have to dig deep into reviewing the docs, but at the first glance it seems too detailed. Should we optimize for what we have in Google Docs already? As our goal for quick start is to allow anyone to test SV2 quickly.

Run hosted setup
Run all roles

And just mention options for configuration, I was under impression we'll just port over entire Google Doc here and nuke anything else we have as currently this wall of text does seem a bit overwhelming.

indeed, I went through what we currently have (mostly focused on Config C and D) and updated it. but if the team agrees I can also just port over the Google Docs instructions over here.
@GitGab19 @Fi3 wdyt?

I agree with @pavlenex, now I would optimize what we have on the google doc as well.
It puts the focus on what we need to test more


update 1.43.0 and deps

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

@specter119 Thanks for pushing this !


Connection error: Ensure server is up and running

I am running this on replit and receiving the above error.
Could you please help me on this?
Thanks

This is what I am receiving.
@victordibia  Could you please have a look at this. I will be very thankful to you.

Hi,
I have pushed an update which should solve this.
Essentially, the UI expects the server to be at url localhost:8081.
Now when you install by default, it should look for the server on the same url.
pip install autogenui

to run ui
autogenui


Dek rename sample in vcf test

Completed integration test.  Thanks to @akiezun.

@dekling I'm closing this pull request because it looks like the right one is #857


add discourse shortcode

This adds support for embedding discourse topics within a page via shortcode.

/lgtm

/lgtm


Contributing [getting started]

Hi, I enjoyed reading the codebase, and I'd like to contribute. I've looked into the issues, but I am not sure where to get started. I'd appreciate if you can point me to something.

Hey, what sort of thing are you interested in working helping with? Features? Bugs? Documentation? Examples?

Right now I am occupied with final exams, so I like to work on something with low priority. I prefer working on features, examples, documentation, and bugs and in that order.


Add context to output

Description of changes:
Add the context to output.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

I really think we need this but I understand it's contentious. Especially with middleware as private, we need a way to inspect request context not just for testing but for debugging. I think context can be also response context so we want to make sure it round trips. It also I think shouldn't be documented on output, we should make it API private maybe. I'm not certain.

Probably should close this for now if we aren't doing config context.


Difference between transfered coins using transferCoins and transferredCoins

Describe the bug
If you transfer 1,0000,0000 using transferCoins function in a smart contract and check the transferred coins using transferredCoins, you get the transferred coins being 1000 times bigger than the amount actually transfered.
To Reproduce
Steps to reproduce the behavior:

Create a simple smart contract including
Transfer a certain amount of coins to a given address using transferCoins function
Immediately check the amount transfered using transferredCoins function
Compare the amount transfered using transferCoins and amount obtained usingtransferredCoins.

Expected behavior
Both transferCoins and transferredCoins should give the same result.
Version
massa-as-sdk - v2.5.2 and buildnet DEVN.27.6

Extract of the smart contract is shown below:
`
let transferAmount = 1000000;
transferCoins(colleeAddress,transferAmount);
let transferredAmount = transferredCoins();
if (transferAmount != transferredAmount){
generateEvent("Bug detected");
}
`

After further experimentation, I learned that transferredCons function is getting the amount of coins transfered when calling the smart contract but not the one transfered within the smart contract using transferCoins or transferCoinsOf. So this is no longer an issue.


fix(modules): entrypoint and env

set entrypoint and cmd such that

podman run -it runs activation and drops into bash
podman run cmd runs activation and then the command

container.config.env should always be overriden with environmentVariables
when flox activation is occurring, don't set env vars in container.config.env, since environmentVariables may need to be evaluated by a shell

This discards aliases...may need to rework

@tomberek added another bit of hackery, thoughts?


Move images to media folders

.PNG renamed to .png
Items moved from article-images\ folder to media\article\ folder.
Articles updated to image location

Closing after discussing with david


Pass "all" the attributes defined on the block to the Image node

Currently, we create a new JSON object and we selectively add attributes: https://github.com/Mogztter/asciidoctor-kroki/blob/51a3c2e9d66b6886093aa1c9a43d7b03683a6c2e/src/asciidoctor-kroki.js#L63-L67
As a result, some important attributes are missing:

align
width
height
format
fallback
float
...

The user could also define custom attributes that could be used in a custom template/converter.
We should try to get closer to the original implementation: https://github.com/asciidoctor/asciidoctor-diagram/blob/66876e9c11c401eec943ff6968cd0c5f6c443906/lib/asciidoctor-diagram/diagram_processor.rb#L61
A few things we should do:

Use the original attributes but

Extract (ie. remove from the original attributes) title and caption.
Update target attribute accordingly
Set alt attribute

Use $assign_caption to assign the caption on the image block (currently this method is not mapped in Asciidoctor.js)

Related issue: https://github.com/Mogztter/asciidoctor-kroki/issues/35


Completion without space at the end

Hi Remko,
I was able to write a completer that makes a coma separated list of values, at first all values are shown, then only the values not used already are added to the list. Like this:
values are x,y,z
no input: shows x y z as candidates
input = x: shows x,y and x,z as candidates
This is similar to the behavior for paths, the problem is, that after completing a value (longer than 1 char) with , a space is appended.
For jline3 I found that it is possible to add candidates with the field complete=false, is there a way to tell picocli to complete without adding a space?
Thank you in advance and best regards,
Mayra

fyi: I'm using 'picocli-shell-jline3', version: '4.6.2' and 'jline-terminal-jansi', version: '3.21.0'


Load remote

Data from a remote file.
The files currently selected are the potential files in this repo, but you can easily change them to somewhere else.

Do you think you can resolve these conflicts easily? if not, I'll try to resolve them and resubmit...


Allow parallel Replays

Problem Description
From the docs:
"Client-side replay
Client-side replay does what it says on the tin: you provide a previously saved HTTP conversation, and mitmproxy replays the client requests one by one. Note that mitmproxy serialises the requests, waiting for a response from the server before starting the next request."
I didnt find any way to change this behaviour and simply send out all replays concurrently (or with a limit like 10 concurrent replays). Im building a universal pentest proxy script for mitmproxy and would like to have a fuzzer addon, which replays requests with a number of payloads from a wordlist. This works great but is significantly slower than other tools because there is only one concurrent replayed request at a time. Here is a snippet of my code:
  for word in fuzz_list:
    counter += 1
    modified_flow = flow.copy()
    # path
    modified_flow.request.path = modified_flow.request.path.replace("FUZZ", word)
    # headers
    for header_name in modified_flow.request.headers:
      modified_flow.request.headers[header_name] = modified_flow.request.headers[header_name].replace("FUZZ", word)
    # request body
    if modified_flow.request.content:
      modified_flow.request.content = modified_flow.request.content.decode().replace("FUZZ", word).encode()
    ctx.master.commands.call("replay.client", [modified_flow])

Proposal
I would like to have an option to send all replayed flows in parallel, preferably with a number of concurrent requests/replays allowed (0 being unlimited).
Alternatives
I tried using requests/httpx/urllib3 and some other frameworks but they all have their own problems, especially when you need full control over payload encoding, special characters in uncommon places etc. mitmproxy does this much better, as it simply creates an exact copy of my request send with burp (mitmproxy as upstream proxy).

Are you aware of client_replay_concurrency?

Wow, how did i manage to miss that one. Thanks!
But yeah, being able to set a specific value would be nice.


[firebase_messaging] On resume and on launch not trigger for IOS

Hi everyone.
Works in all situations on android but On launch and on resume do not trigger when app is in background or terminated.
Flutter doctor version
ertugrul@ertugruls-Mac ~ % flutter doctor -v
[✓] Flutter (Channel stable, v1.12.13+hotfix.8, on Mac OS X 10.15 19A602, locale
en-TR)
• Flutter version 1.12.13+hotfix.8 at /Users/ertugrul/flutter
• Framework revision 0b8abb4724 (5 months ago), 2020-02-11 11:44:36 -0800
• Engine revision e1e6ced81d
• Dart version 2.7.0

[!] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
• Android SDK at /Users/ertugrul/Library/Android/sdk
• Android NDK location not configured (optional; useful for native profiling
support)
• Platform android-28, build-tools 29.0.3
• Java binary at: /Applications/Android
Studio.app/Contents/jre/jdk/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build
1.8.0_212-release-1586-b4-5784211)
✗ Android license status unknown.
Try re-installing or updating your Android SDK Manager.
See https://developer.android.com/studio/#downloads or visit
https://flutter.dev/setup/#android-setup for detailed instructions.

App Delegate.swift
import UIKit
import Flutter
import FirebaseMessaging

@UIApplicationMain
@objc class AppDelegate: FlutterAppDelegate {
override func application(
_ application: UIApplication,
didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?
) -> Bool {
GeneratedPluginRegistrant.register(with: self)
if #available(iOS 10.0, *) {
UNUserNotificationCenter.current().delegate = self as? UNUserNotificationCenterDelegate
}
return super.application(application, didFinishLaunchingWithOptions: launchOptions)
}
override func application(_ application: UIApplication, didRegisterForRemoteNotificationsWithDeviceToken deviceToken: Data) {
let deviceTokenString = deviceToken.reduce("", {$0 + String(format: "%02X", $1)})
print("==== didRegisterForRemoteNotificationsWithDeviceToken ====")
print(deviceTokenString)
Messaging.messaging().apnsToken = deviceToken
}
}

Fcm code
_fcm.configure(onMessage: (Map<String, dynamic> message) async {
print("onMessage tetkilendi: $message");
showNotification(message);
},
onBackgroundMessage: Platform.isIOS?null:myBackgroundMessageHandler,
onLaunch: (Map<String, dynamic> message) async {
print("onLaunch tetiklendi: $message");
showNotification(message);
},
onResume: (Map<String, dynamic> message) async {
print("onResume tetiklendi: $message");
showNotification(message);
},
);

Info.plist
<key>FirebaseAppDelegateProxyEnabled</key> <string>NO</string>

Firebase messagin version is 6.0.16. The code in the above on launch and on resume doesn't work print line when the above code is in the background or closed. Is there an error in the code I wrote or is it in the fcm package?

Hi @curlyeto
Looks like there is an issue open for this
Closing duplicate of https://github.com/FirebaseExtended/flutterfire/issues/2284


Feature: support registry resource schemas

cfn-lint version: (cfn-lint --version)
any/all
Description of issue.
My company uses this tool to ensure correctness of our CloudFormation templates and would like to explore CloudFormation's new registry for third party resources.  In particular, we'd are trialing those provided by datadog.
Understandably cf-lint is not currently aware of the schema's these declare for custom resources but I was wondering if adding support for them would possible given the current schema definition formation cf-lint relies on? The problem for us is that cf-lint will always fail for templates containing these resources. The current work around is to put them into separate files and use a naming convention that cf-lint avoids but I don't think that's an ideal approach for the way CloudFormation registry resources are intended to be used.
Strawman idea. Add a new multi value parameter the s3 schema uris these resources are required to publish to cf-lint and have it incorporate those in its validation. This allows for organizations to change resource schema versions of these resources and to update s3 uris provided to cfn lint in sync

@softprops how would you feel about a cfn-lint command that would update a patch file from that registry?   I'm not sure I would like to do it with --update-specs at this point because we have tried to make that command to not require authentication to AWS.  @PatMyron any thoughts here?

I don't know enough implementation details to know what that does. Rather than focus on implementation I'll try to make clearer the goal. I would like to have a way to continue to use cfn-lint in ci and not fail when it encounters a cfn template for a custom resource.
A common property of all third party resources is that they all require hosting of a schema file describing that resource. What would be great if there was a way for cfn to leverage that type to schema mapping to adapt aws third party resource schema model to what ever schema model cfn lint uses internally.
How does the patch file system work?

There are now publicly accessible CloudFormation Registry resource provider schemas for all of the PUBLIC resource types in Amazon's namespaces
The Linter has a lot of Amazon resource type information that is not available in these schemas that we don't want to lose:

public resource types not yet fully migrated to the CloudFormation Registry are lacking most schema information. primaryIdentifier may even be incorrect for these types
constantly evolving enums like instance types, IAM actions, lambda runtimes, availability zones, etc. don't make sense to maintain in CloudFormation resource schemas as blocking validations, especially with the significant time and effort to deploy changes to AWS' most regulated regions
some schema information doesn't match up exactly as previously expected due to technical limitations: writeOnlyProperties, readOnlyProperties, createOnlyProperties
etc.

Resource schemas definitely also contain information the Linter does not currently check, so it's worth figuring out how to start utilizing them as well. In order to not lose any existing information checked not available in these schemas, it probably makes sense to start utilizing resource schemas writing additional new rule(s)
The trickiest part will be handling CloudFormation template syntax like dynamic references that cannot be resolved. Some things like combined JSON schemas will make it particularly tricky to properly ignore not yet resolvable template syntax because one property utilizing unresolvable template syntax may also hinder our ability to easily validate other properties of that resource as well. In order to reduce false positive noise rates, we should err on the side of caution and stop evaluating in circumstances like those where schema validation may fail simply due to unresolved template syntax
Once the initial framework utilizing these resource schemas is in place, expanding support for this rule beyond Amazon's PUBLIC resource type namespaces should be trivial: we should able to provide the ability for customers to provide their own directory of resource schemas or something similar easily. Even including some of the most popular non-PUBLIC resource types by default would probably make sense as well

As far as current errors from non-PUBLIC CloudFormation Registry type usage, I believe it should be limited to this existing rule, E3001, that we need to re-evaluate:
https://github.com/aws-cloudformation/cfn-python-lint/blob/93659fba5cf86ee34177c55b28df0aa9140b7dc9/src/cfnlint/rules/resources/Configuration.py#L105
Resource-level ignores should mitigate these errors until then:
  Resource:
    Type: Datadog::IAM::User
    Metadata:
      cfn-lint:
        config:
          ignore_checks:
          - E3001

Here are a few options we have as we continue to think about this issue.

One solution would be to create a way to convert Resource Provider Schema into CloudFormation Resource Spec format.  Our current rules will continue to work as needed.  I don't feel like this is a sustainable long term answer as the Schemas continue to evolve we will continually run into issues as things that won't convert correctly to the CloudFormation Resource Spec format.  The Registry Schema has additional data that we could leverage for checks that we would be missing.

Rewrite the existing rules to leverage the the new Resource Provider Schema. This is going to take some work as some of the rules need to be rewritten. Ideally we could use a JSON Schema validation package to do most of that heavy lifting and providing additional logic on top of that package based on our needs. This is probably the most ideal but will take the most time. Additionally we may have to think about how people write custom rules and the impact that will result from PropertyTypes not existing any more.  Those names may not match the property names so there is a chance things will break as we lose those names.

After next release, one or more directories of CloudFormation Registry Resource Schemas can be passed in with:
-s / --registry-schemas


no added DOIs in TEI from consolidation (docker or local) and (crossref or glutton)

This might be a really basic one...
I've installed grobid on a linux box and configured it to consolodate using bibli-gluton (on the same computer). I installed glutton because when I ran consolodation with crossref I didn't see DOIs added to the TEI file references section. Same for biblioglutton.
Am I missing something? I thought that consolidation would enrich the TEI...and it does not seem to do so
what I have for logs is
WARN  [2024-07-07 12:41:47,702] com.scienceminer.glutton.web.resource.LookupController: DOI did not matched or did not pass post validation
from glutton...there is only 1 DOI in the article as ingested
and I see a bucketload of these
INFO  [2024-07-07 12:41:48,430] org.grobid.core.utilities.Consolidation: Consolidation service returns error (500) : Server Error
but bibliogutton seems to be properly installed
curl localhost:8075/service/data
{"ISTEX size":"{istex_doi2ids=0, istex_istex2ids=0, istex_pii2ids=0}","Crossref Metadata stored size":"{crossref_Jsondoc=0}","Total metadata indexed size":"{crossref_Jsondoc=0}","HAL Metadata stored size":"{hal_Jsondoc=0}","PMID size":"{pmid_doi2ids=0, pmid_pmc2ids=0, pmid_pmid2ids=0}","DOI OA size":"{unpayWall_doiOAUrl=
and the config.yaml for grobid does say that it is on localhost:8075
not running anything in docker as the docker install (full one) didn't consolidate using crossref either...well..I didn't see the DOIs in the reference section of the TEI.

Hi @bozo32, if you look at the counter from /service/data, you will notice that every counter is 0.
Biblio glutton needs to be "loaded" after installing it. Please check here: https://biblio-glutton.readthedocs.io/en/latest/Build-Databases/


opentelemetry-collector: 0.101.0 -> 0.103.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/open-telemetry/opentelemetry-collector/releases.
meta.description for opentelemetry-collector is: A vendor-agnostic implementation on how to receive, process and export telemetry data
meta.homepage for opentelemetry-collector is: https://github.com/open-telemetry/opentelemetry-collector
meta.changelog for opentelemetry-collector is: https://github.com/open-telemetry/opentelemetry-collector/blob/v0.102.0/CHANGELOG.md
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.102.0 with grep in /nix/store/6q2b71hhyc889wjr84fqp8acxn9k31dz-opentelemetry-collector-0.102.0
found 0.102.0 in filename of file in /nix/store/6q2b71hhyc889wjr84fqp8acxn9k31dz-opentelemetry-collector-0.102.0

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

opentelemetry-collector

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/6q2b71hhyc889wjr84fqp8acxn9k31dz-opentelemetry-collector-0.102.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A opentelemetry-collector https://github.com/r-ryantm/nixpkgs/archive/6d9466d9e8b1fa6cafd62dbab529d7f7931a51a4.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/6d9466d9e8b1fa6cafd62dbab529d7f7931a51a4#opentelemetry-collector

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/6q2b71hhyc889wjr84fqp8acxn9k31dz-opentelemetry-collector-0.102.0
ls -la /nix/store/6q2b71hhyc889wjr84fqp8acxn9k31dz-opentelemetry-collector-0.102.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

opentelemetry-collector

Maintainer pings
cc @uri-canva @06kellyjac for testing.

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/open-telemetry/opentelemetry-collector/releases.
meta.description for opentelemetry-collector is: Vendor-agnostic implementation on how to receive, process and export telemetry data
meta.homepage for opentelemetry-collector is: https://github.com/open-telemetry/opentelemetry-collector
meta.changelog for opentelemetry-collector is: https://github.com/open-telemetry/opentelemetry-collector/blob/v0.102.1/CHANGELOG.md
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.102.1 with grep in /nix/store/97lc7w80i7w5ar3ljwvb2xyzcjx4sn4w-opentelemetry-collector-0.102.1
found 0.102.1 in filename of file in /nix/store/97lc7w80i7w5ar3ljwvb2xyzcjx4sn4w-opentelemetry-collector-0.102.1

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

opentelemetry-collector

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/97lc7w80i7w5ar3ljwvb2xyzcjx4sn4w-opentelemetry-collector-0.102.1 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A opentelemetry-collector https://github.com/r-ryantm/nixpkgs/archive/d42d63408a2bffe5d3b501e7574fb0d6df616a98.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/d42d63408a2bffe5d3b501e7574fb0d6df616a98#opentelemetry-collector

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/97lc7w80i7w5ar3ljwvb2xyzcjx4sn4w-opentelemetry-collector-0.102.1
ls -la /nix/store/97lc7w80i7w5ar3ljwvb2xyzcjx4sn4w-opentelemetry-collector-0.102.1/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

opentelemetry-collector

Maintainer pings
cc @uri-canva @06kellyjac for testing.

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/open-telemetry/opentelemetry-collector/releases.
meta.description for opentelemetry-collector is: Vendor-agnostic implementation on how to receive, process and export telemetry data
meta.homepage for opentelemetry-collector is: https://github.com/open-telemetry/opentelemetry-collector
meta.changelog for opentelemetry-collector is: https://github.com/open-telemetry/opentelemetry-collector/blob/v0.103.0/CHANGELOG.md
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.103.0 with grep in /nix/store/1i9p31xdkda0jiq5ia7hmbip13pdakjl-opentelemetry-collector-0.103.0
found 0.103.0 in filename of file in /nix/store/1i9p31xdkda0jiq5ia7hmbip13pdakjl-opentelemetry-collector-0.103.0

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

opentelemetry-collector

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/1i9p31xdkda0jiq5ia7hmbip13pdakjl-opentelemetry-collector-0.103.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A opentelemetry-collector https://github.com/r-ryantm/nixpkgs/archive/abc9236ff3b11c86abdf112f8cdd7e37ce84a4b7.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/abc9236ff3b11c86abdf112f8cdd7e37ce84a4b7#opentelemetry-collector

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/1i9p31xdkda0jiq5ia7hmbip13pdakjl-opentelemetry-collector-0.103.0
ls -la /nix/store/1i9p31xdkda0jiq5ia7hmbip13pdakjl-opentelemetry-collector-0.103.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

opentelemetry-collector

Maintainer pings
cc @uri-canva @06kellyjac for testing.

Flagging as security related, it includes the fix for CVE-2024-36129
https://opentelemetry.io/blog/2024/cve-2024-36129/


no element is specified to initialize PerfectScrollbar

Hello,
We're using Jest and Enzyme to do unit tests that includes components with react-perfect-scrollbar. When doing a snapshot test, we got this error: "no element is specified to initialize PerfectScrollbar". The code works fine on the website, just only in the unit test snapshot, we get an error. Any ideas on how to fix this? Thanks!
Here's a code snippet:
`import React from 'react';
import renderer from 'react-test-renderer';
import AbstractHistory from '../src/components/Abstract/AbstractHistory';
import configureStore from 'redux-mock-store';
import expect from 'expect';
import { mount } from 'enzyme';
import { initialState } from '../src/redux/initial-state'
import { Modal } from 'reactstrap';
import PerfectScrollbar from 'react-perfect-scrollbar'
import '../../css/perfect-scrollbar.css'
describe('AbstractHistory (Snapshot)', () => {
const mockStore = configureStore()
let store

beforeEach(() => {
    store = mockStore(initialState)
})

it('AbstractHistory snapshot renders', () => {
    const component = renderer.create(<AbstractHistory store={store} />);
    const json = component.toJSON();
    expect(json).toMatchSnapshot();
});

});`

The react-perfect-scrollbar uses refs, so you may need to reference https://reactjs.org/docs/test-renderer.html#ideas to create a mocked ref.

Thanks for the reply, @goldenyz. I see from the example that createNodeMock is mocking a focus function. Which function should I mock for react-perfect-scrollbar?

@Trancedified This is a good question. I have glanced at the source code of perfect-scrollbar, a lot of DOM related method has been called. jsdom may give some help with jest

@Trancedified did you solved this problem?

@Trancedified did you solved this problem?

No, we just disabled the unit test.


[BanyanDB-Server]Backup and Restore Data

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
Description
We propose to add a backup and restore feature to BanyanDB. This feature will allow users to create and restore data backups when necessary.
Requirements

Backup Functionality:

Implement a method to create a consistent snapshot of the BanyanDB data directory.
Ensure minimal performance impact during the snapshot process.
Allow users to configure the backup frequency and retention period.

Restore Functionality:

Provide a way to restore BanyanDB data from a previously created snapshot.
Ensure the restoration process is robust and handles potential issues such as incomplete or corrupted snapshots.
Support restoring to the same instance or a new instance of BanyanDB.

Use case
No response
Related issues
No response
Are you willing to submit a pull request to implement this on your own?

[X] Yes I am willing to submit a pull request on my own!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

We should consider restoring from N-1 version snapshot as well. This would be critical for the upgrade.


Client Call Controls on Escalated Calls are Inconsistent

Describe the issue

User A calls Bot
Bot Accepts
Bot adds User B

We have a call now with three participants.
Here's how the call controls look like for the Users
User A:
Hold Button
Transfer Buttons
User B:
Transfer Buttons
Expected behavior
I expected that these buttons would not be there as it is essentially a meeting call now.
If you have a p2p call in teams and then add a third user now Hold or Transfer buttons are present for any User.
However the buttons actually do work... but it's a bit unpredictable from my point of view.
What is the offical story here? Is this a bug? How should these buttons work exactly?

From what I've seen with these escalations, it's also not really a meeting.  You have multiparty audio, but no IM capabilities.  Would be nice if the behaviour was consistent and it created a read ad-hoc meeting.


fixed workflow issues

Fixed redundant testing and set executable permissions on scripts.

changes didn't trigger test workflow, so had to force skip them


refactor: rewrite the project

This is a complete rewrite of the project, using a more efficient and flexible design based around the concept of a forward-only "view" into the source document. Some capabilities are regressed in this initial implementation, including built-in non-UTF-8 support and some of the writer functionality.

Benchmark results compared to the old Reader implementation:
Benchmark 1 (43 runs): ../../zig-xml-main/bench/zig-out/bin/reader zig-out/Gtk-4.0.gir
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           118ms ± 2.11ms     115ms …  125ms          1 ( 2%)        0%
  peak_rss           7.49MB ± 77.4KB    7.32MB … 7.58MB          0 ( 0%)        0%
  cpu_cycles          465M  ± 6.90M      456M  …  477M           0 ( 0%)        0%
  instructions       1.17G  ± 21.0      1.17G  … 1.17G           0 ( 0%)        0%
  cache_references   1.07M  ±  312K      728K  … 2.70M           1 ( 2%)        0%
  cache_misses       26.3K  ± 9.07K     13.3K  … 55.4K           1 ( 2%)        0%
  branch_misses      1.38M  ± 4.20K     1.38M  … 1.40M           2 ( 5%)        0%
Benchmark 2 (73 runs): zig-out/bin/reader zig-out/Gtk-4.0.gir
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          68.9ms ±  626us    68.1ms … 71.4ms          4 ( 5%)        ⚡- 41.3% ±  0.4%
  peak_rss           7.53MB ± 58.3KB    7.45MB … 7.60MB          0 ( 0%)          +  0.5% ±  0.3%
  cpu_cycles          265M  ± 1.41M      262M  …  269M           0 ( 0%)        ⚡- 43.0% ±  0.4%
  instructions        604M  ± 16.8       604M  …  604M          12 (16%)        ⚡- 48.3% ±  0.0%
  cache_references    481K  ±  265K      268K  … 2.42M           5 ( 7%)        ⚡- 54.9% ± 10.1%
  cache_misses       19.8K  ± 3.77K     11.0K  … 27.5K           5 ( 7%)        ⚡- 24.6% ±  9.1%
  branch_misses       899K  ± 3.71K      893K  …  910K           2 ( 3%)        ⚡- 35.0% ±  0.1%

Follow-up work will be to implement new features and optimizations. Aside from the core design, very little attempt at optimizing the code has been made so far.


[api] add rel filter on emerald/transactions

tested e2e locally.

Yep tested locally and confirmed that the returned transactions are associated with the rel address. It should be noted that for evm.Call transactions, the rel address often only appears in the events associated with that transaction and is not displayed within the transaction itself (eg as sender or to). This can be checked by querying emerald/events?tx_hash={some_tx_hash} and inspecting the returned events for the rel address


Crash when exchanging Spawner

In All The Mods 9 version 0.0.53, when using the exchanging gadget, enchanted with silk touch, on a spawner, the game crashes. Attached is the crash report generated.
I had dirt selected to replace the spawner with, Affect Block entities was the default value as I had just crafted the gadget. I had gone into my inventory while the exchange was happening.
crash-2023-08-19_15.23.26-server.txt

Thanks for the report I’ll investigate :)

I couldn't reproduce this crash, however I added some code that should protect against it.


Test Flowzone PR 485

Auto-generated by https://github.com/product-os/flowzone/pull/485
Will be closed automatically once the PR is merged.

Auto-closing pull request


Refine pre-defined type of andalso and orelse

The shortcut type can be more specific to allow more programs to type check, e.g.:
-spec a() -> false.
a() -> false andalso no.

The current type specification does not allow this.

This example still fails, though:
-spec double_andalso(term(), term()) -> false | {number(), atom()}.
double_andalso(N, A) ->
    is_number(N) andalso is_atom(A) andalso {N, A}.


Fixed typo in FAQ-journey.md

Closes https://github.com/MicrosoftDocs/OfficeDocs-SkypeForBusiness/issues/5987.

@JohanFreelancer9 Copy editing is needed for this PR. Thanks.

@LanaChin PR has been copyedited and is ready for final review, could you please check and merge? Thanks!


Add docker setup

This PR adds docker setup for the server access
This should work on both GPU and CPU
To test the docker enviroment, commands in Makefile can be used. For example, make up sets the container up

Glad to see that Docker is now supported!
Your commit on auto_explanation.py also looks good for me.