Github action for ping pong test in Staging Deployment

Enable GitHub action that for every commit to master, it deploys changes to staging infrastructure, launches the ping pong test, and verifies it passes.
if test passes, commit tag should be updated in deployment repo as the last valid version.
if test fails, it shouldn’t prevent from pushing to master, but at least we know there is something that needs fixing.

We no longer have a staging deploy from master so closing this PR


Split 1.17 module into 1.17-common, 1.17, and 1.17.1

This PR takes the old commandapi-1.17 module and splits it into three modules: commandapi-1.17-common, commandapi-1.17, and commandapi-1.17.1.
This is necessary because the field MinecraftServer.resources (accessed in reloadDataPacks()) is mapped to two different names (aB and aC) for the two versions 1.17 and 1.17.1. This caused the following exception on a 1.17 Spigot Server:
[15:14:47] [Server thread/INFO]: [CommandAPI] Reloading datapacks...
[15:14:47] [Server thread/WARN]: [CommandAPI] Task #2 for CommandAPI v8.4.1 generated an exception
java.lang.NoSuchFieldError: aB
    at dev.jorel.commandapi.nms.NMS_1_17_R1.reloadDataPacks(NMS_1_17_R1.java:647) ~[?:?]
    at dev.jorel.commandapi.CommandAPI.lambda$onEnable$0(CommandAPI.java:186) ~[?:?]
    at org.bukkit.craftbukkit.v1_17_R1.scheduler.CraftTask.run(CraftTask.java:81) ~[spigot-1.17.jar:3170-Spigot-a483d2c-ec116f6]
    at org.bukkit.craftbukkit.v1_17_R1.scheduler.CraftScheduler.mainThreadHeartbeat(CraftScheduler.java:400) ~[spigot-1.17.jar:3170-Spigot-a483d2c-ec116f6]
    at net.minecraft.server.MinecraftServer.b(MinecraftServer.java:1252) ~[spigot-1.17.jar:3170-Spigot-a483d2c-ec116f6]
    at net.minecraft.server.dedicated.DedicatedServer.b(DedicatedServer.java:436) ~[spigot-1.17.jar:3170-Spigot-a483d2c-ec116f6]
    at net.minecraft.server.MinecraftServer.a(MinecraftServer.java:1200) ~[spigot-1.17.jar:3170-Spigot-a483d2c-ec116f6]
    at net.minecraft.server.MinecraftServer.x(MinecraftServer.java:1027) ~[spigot-1.17.jar:3170-Spigot-a483d2c-ec116f6]
    at net.minecraft.server.MinecraftServer.lambda$0(MinecraftServer.java:307) ~[spigot-1.17.jar:3170-Spigot-a483d2c-ec116f6]
    at java.lang.Thread.run(Thread.java:831) [?:?]

It seems that this is the only difference relevant to the CommandAPI between 1.17 and 1.17.1, though if other differences are found, these classes can definitely be expanded. To cut down redundant code, commandapi-1.17-common contains almost all the NMS methods except for reloadDataPacks(), which is implemented identically in commandapi-1.17 and commandapi-1.17.1, but mapped differently by specialsource.
To allow commandapi-1.17 and commandapi-1.17.1 access commandapi-1.17-common both before and after it is remapped, they depend on both the mojang mapped and unmapped versions of Spigot 1.17.1-R0.1-SNAPSHOT.
I tried to match the code style and project layout, but I'm not sure if I got the annotations correct on the NMS classes. Feel free to change anything that's not quite right!

Thanks for this - I was not aware Maven would let you compile against two versions of the same dependency.
The NMS class requires a method compatibleVersions() to be implemented, this will need to be implemented in each submodule (i.e. in 1.17 and 1.17.1):
@Override
public String[] compatibleVersions() {
    return new String[] { "1.17", "1.17.1" };
}

Each NMS module should have a test folder with a SafeReflect.java file. This is used for compil-time reflection checks for code that has reflection. The reloadDatapacks() method uses reflection to modify CustomFunctionManager's command dispatcher.
Although we can already assert this won't fail, I'd still like the 1.17 module and 1.17.1 module to independently have this reflection check (as opposed to just commandapi-1.17-common), because the compile-time reflection checks use the spigot libraries used for compile time (i.e. it'll perform reflection checks against spigot 1.17 for commandapi-1.17 and perform reflection checks against spigot 1.17.1 for commandapi-1.17.1).


Error message click handling

When an error message is shown, it pops up in a box with a red background that you have to click to get past.
Unfortunately, it's not obvious that you have to click the message to close it. Either:
a) Allow clicking anywhere in the browser window to close that message
or
b) Replace that error message with a modal window that has an actual "Close" button so it's obvious. If using b, just have one window class that allows you to set different messages (rather than a separate Window class for each error message). The window would only need a "Close" button that closes itself.

Need to see how people like this before worrying about it more - just changed to a "warning" message, which will disappear automatically in a few seconds (or also if you click it), and it doesn't block the UI.
Based on the API docs and this question, there's not a super easy way to make it so you can click anywhere to dismiss the message. (I'm sure it could be done - but this was easier)


Logout returns 204, not 200.

Logout typically returns 200 when successful, not 204. Not the worst thing, but should be poked at.
https://github.com/terjekv/hubuum/blob/6e388c8457616f066ab13a2ef4f8dd234423c931/hubuum/api/v1/tests/test_01_auth.py#L19

Knox default, see https://github.com/James1345/django-rest-knox/blob/develop/knox/views.py#L86. There are PRs to allow for modifying the return, but not really urgent. Closing until further notice.


import os before using os

Fixes #3730.

I think we have tests for notebooks, so there should be a way. Not sure the same is true for checking test coverage...

I think we have tests for notebooks, so there should be a way. Not sure the same is true for checking test coverage...

Yeah, this notebook is excluded as it uses the engine API :/
There could be a way to make notebooks somehow mockable + even do test coverage possibly but currently it's not possible to test this.


Mouse events not firing after keypres on Mac

I am testing iohook in node.js with the standard example both on Windows and Mac.
On Mac after running the code mousemove events logs the event, but after pressing any key it stops firing. This issue is not present on Windows, it works as expected.
I have tested other computers too, but all of them were running macOS Mojave version 10.14.4

Similar problem. The events work only on the first action. The second event doesn't trigger any action

Hi there,
I'm sorry I can't reproduce that. If anyone can reproduce it and is able to help, it would be great

Same issue. I noticed that "mousemove" event works as intended.

Same issue here, OS is Mojave. Seems to be the same issue as https://github.com/wilix-team/iohook/issues/124. Would love a fix for this.

It's only issue with node. Electron works fine

Currently using it also with electron and works without issues with Mojave 10.14.13. Events I am using are: mousemove, mousewheel and keyup

Also experiencing this in a generic node script, first keydown triggers but not subsequent ones.

Seeing this too on 10.14.5

I'm experienced this too. Node 10.16.0 and MacOS Mojave 10.14.1

Hello, check this comment for a workaround for this problem https://github.com/wilix-team/iohook/issues/124#issuecomment-513026388.

Also experiencing this in a generic node script on mac, first keydown triggers but not subsequent ones.
For example, when I press the character 'e' (or any key...), this is the output:
 process_key_pressed [367]: Key 0X20 pressed. (0X2)
 dispatch_event [108]: Dispatching event type 4.
 process_key_pressed [379]: Using dispatch_sync_f for key typed events.
 { shiftKey: false,
   altKey: false,
   ctrlKey: false,
   metaKey: false,
   keycode: 32,
   rawcode: 2,
   type: 'keydown' }

Keycode 32 seems to be the space bar which I'm not pressing.

Same with you

Is this still an issue?

yes, same issue

yes, the same issue
"iohook": "^0.9.3",

What's your environment please? Node/electron version, os, etc.

Thanks for response,
Don't worry, I solved my issue by using other library

Thanks for response,
Don't worry, I solved my issue by using other library

Could you share it with us so others running into your use case can use it as well?

Here is the code that I used
https://github.com/djkskqyr3/samples.git


Table with stickyHeader drags content of headers while scrolling horizontally

[x] The issue is present in the latest release.
[x] I have searched the issues of this repository and believe that this is not a duplicate.

Current Behavior 😯
Content of header cell is dragged with horizontal scroll.

Expected Behavior 🤔
As this is stickyHeader not stickyColumn, header should be sticky only on vertical scroll.
I believe this can be fixed by removing left: 0 from .MuiTableCell-stickyHeader class. I am not sure if this would affect something else tho.
Steps to Reproduce 🕹

Reduce windows size and try horizontal scrolling.
sandbox: stickyHeader example from documentation

Check the DataGrid component: https://material-ui.com/components/data-grid/, it's free from this issue.

As an alternative https://github.com/archfz/vh-sticky-table-header


Cannot access shared dataset

What steps will reproduce the problem?

Create a dataset in one project in account (a)
Share this dataset with a project in another account (b)
Create a service account in account b and give it view or editor permissions
Attempt to list the tables in the shared dataset using the python API

What is the expected output? What do you see instead?
I expect to see the list of tables for the shared dataset. Instead I get the following error:
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.6/site-packages/oauth2client/util.py", line 137, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/googleapiclient/http.py", line 760, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/bigquery/v2/projects//datasets//tables?alt=json returned "Access Denied: Dataset :: The user  does not have permission to list tables in dataset :">
What version of the product are you using? On what operating system?
OS:
CentOS release 6.7 (Final)
Python packages:
google-api-python-client (1.5.0)
Please provide any additional information below.
Using the user account (project owner) I can list the tables in the dataset using the UI or the bq command line tool

It might be better to ask this question on stackoverflow, as this team does not maintain the APIs themselves, just the client library. Authorization issues are unfortunately out of our area of focus.


feat(xo/xo): scaffold xo/xo

xo/xo: xo is a command-line tool to generate idiomatic code for different languages code based on a database schema or a custom query.
$ aqua i -g xo/xo

Check List

[x] Read CONTRIBUTING.md

[x] Read OSS Contribution Guide

[x] All commits are signed

This repository enables Require signed commits, so all commits must be signed

[x] Avoid force push
[x] Do only one thing in one Pull Request
[x] Execute cmdx s to scaffold code
[x] Install and execute the package and confirm if the package works well

Thank you for your contribution!

You checked this, but actually your commits aren't signed.
Could you sign your commits according to the guide?

All commits are signed
This repository enables Require signed commits, so all commits must be signed

If you can't, I'll create a pull request to add xo/xo instead of you.

@suzuki-shunsuke Thank you! I'm sorry I forgot to specify the option on committing.
I'll recreate the PR with the signed commit!

@suzuki-shunsuke
I recreated the PR here. Thanks!
https://github.com/aquaproj/aqua-registry/pull/30314


Translation issue with the "pick" button in the ColorPicker

Microsoft PowerToys version
0.79.0
Utility with translation issue
ColorPicker
🌐 Language affected
Brazilian Portuguese
❌ Actual phrase(s)
Picareta (means pickaxe in portuguese)

✔ Expected phrase(s)
The word should be "Conta-gotas" (dropper) or "Escolher" (pick).
ℹ Why is the current translation wrong
Pick, in this context, is translated to "Escolher", and not "Picareta".

/dup #31699


Issue exporting xpub via QR

I'm trying to export an xpub from Jade, for the purpose of setting up a Watch Only Wallet using Green.
When I scan Jade's xpub QR [which is on high-density], the following data is read by Green.

This data is in some format I [and Green] do not recognise.
Any advice on where things have gone wrong?

Gotcha. Thanks for the speedy reply.
Feel free to delete this issue.


Remove peer/server structure

Description
Remove the requirement for the endpoint (peer) metadata is added by Fishjam. Therefore, if we add such type definitions they should be in the FishjamClient, not WebRTCEndpoint.
Why? So far the WebRTCEndpoint has been used as an SDK for Membrane RTC Engine. If we add such changes, this effectively means that we no longer support SDK's for the RTC Engine.
Motivation and Context
Why is this change required? What problem does it solve? If it fixes an open
issue, please link to the issue here.
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to
not work as expected)

this apparently safe change broke the tests :C


CSS Customization for Pagination Failed

I'm trying to customize pagination:
`#app

div.app
div.send-tokens
div.track-tokens
div
div.transaction-table.box
div
nav
ul {
background-color: #666769;
}

#app

div.app
div.send-tokens
div.track-tokens
div
div.transaction-table.box
div
nav
ul
li {
background-color: #666769;
list-style-type: none;
}`
The result is:

Version info:
"vuejs-smart-table": "0.0.5"
"vue": "^2.6.6"

Hello, please refer to the Pagination CSS Customization Section to see how it is done:
https://tochoromero.github.io/vuejs-smart-table/pagination/#css-customization

I'm still having issues. I'm using CSS selectors (described in the example provided above) to customize the element but no luck. Here is an example:
/* this one is not working /
ul.pagination {
background-color: aqua;
list-style: none;
display: flex;
}
/ this one is also not working */
.pagination {
background-color: aqua;
list-style: none;
display: flex;
}
It would be great if you can provide a minimum example. Thanks


Showing also my code here for Legendre polynomials following the Chebyshev example.

Good work! I was wondering if we both have different implementation of Legendre polynomials. This is inspired from the Chebyshev KAN layer.
class LegendreKANLayer(nn.Module):
    """ Inspired by Kolmogorov-Arnold Networks but using Legendre polynomials instead of splines coefficients. """
    def __init__(self, input_dim: int, output_dim: int, degree: int):
        super(LegendreKANLayer, self).__init__()
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.degree = degree

        self.legendre_coefficients = nn.Parameter(torch.empty(input_dim, output_dim, degree + 1))
        nn.init.normal_(self.legendre_coefficients, mean=0.0, std=1 / (input_dim * (degree + 1)))

    def forward(self, x):
        x = torch.reshape(x, (-1, self.input_dim)) # shape = (batch_size, input_dim)
        # Since the Legendre polynomial is defined in [-1, 1]
        # We need to normalize x to [-1, 1] using tanh
        x = torch.tanh(x)
        # Initialize the Legendre polynomial tensors
        leg = torch.ones(x.shape[0], self.input_dim, self.degree + 1, device=x.device)
        if self.degree > 0:
            leg[:, :, 1] = x # Set the second Legendre polynomial to x
        for i in range(2, self.degree + 1):
            leg[:, :, i] = ((2 * i - 1) * x * leg[:, :, i-1].clone() - (i - 1) * leg[:, :, i - 2].clone()) / i
        # Compute the Legendre interpolation
        y = torch.einsum('bid,iod->bo', leg, self.legendre_coefficients) # shape = (batch_size, output_dim)
        y = y.view(-1, self.output_dim)
        return y

Would appreciate any feedback/comment. Thank you!

Well, while the idea is the same, I think our structures, representations and computations are going to be significantly different. As you can see you are using a single layer; the differences are in the computations of the polynomials, the activation, etc. You have invited an inductive bias of normalising as well as initialisation weights of selecting non-uniform weights. As you can realise I have changed the structure of the KAN itself, you probably have quite a bit of a wiggle room depending on the cascading under the outer class.

Right! These insights are helpful. I'm still new to this topic and would really appreciate your suggestions. As for your points (2) and (3):

Normalization and Activation: I'm planning on using a linear normalization instead. I do agree with the bias brought by using .tanh(). As for the final output, may I politely ask why an explicit activation on the final output could be helpful, and how is it different in terms of performance (if ever you have tried on using one without activation on the final output)?
Initialization: Thank you for pointing this out as well. Will also try Kaiming initialization.

Once again, thank you!

Here's a reworded version of your message:

I haven't thoroughly examined the entire paper, so I don't profess to be an expert. My approach primarily stems from my basic knowledge of MLPs. The first point is straightforward: for a classification task like MNIST, using normalizations along with an explicit activation function is essential. Without these, you're essentially requiring the network to learn the output data manifold directly. Given the transition from high-dimensional input to the low-dimensional classification task, this would create a significant bottleneck in terms of information flow within the model.

Hi @egmaminta, I haven't thoroughly examined the entire paper, so I don't profess to be an expert. My approach primarily stems from my basic knowledge of MLPs. The first point is straightforward: for a classification task like MNIST, using normalizations along with an explicit activation function is essential. Without these, you're essentially requiring the network to learn the output data manifold directly. Given the transition from high-dimensional input to the low-dimensional classification task, this would create a significant bottleneck in terms of information flow within the model.

Right. Thank you!


cannot generate models for cpp sample.

When run optimum-cli export openvino --trust-remote-code --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 TinyLlama-1.1B-Chat-v1.0 locally, it reports like:
Traceback (most recent call last):
  File "//workspace/llm_dev/venv/llm/bin/optimum-cli", line 8, in <module>
    sys.exit(main())
  File "//workspace/llm_dev/venv/llm/lib/python3.10/site-packages/optimum/commands/optimum_cli.py", line 163, in main
    service.run()
  File "//workspace/llm_dev/venv/llm/lib/python3.10/site-packages/optimum/commands/export/openvino.py", line 310, in run
    task = infer_task(self.args.task, self.args.model)
  File "//workspace/llm_dev/venv/llm/lib/python3.10/site-packages/optimum/exporters/openvino/__main__.py", line 51, in infer_task
    task = TasksManager.infer_task_from_model(model_name_or_path)
  File "//workspace/llm_dev/venv/llm/lib/python3.10/site-packages/optimum/exporters/tasks.py", line 1668, in infer_task_from_model
    task = cls._infer_task_from_model_name_or_path(model, subfolder=subfolder, revision=revision)
  File "//workspace/llm_dev/venv/llm/lib/python3.10/site-packages/optimum/exporters/tasks.py", line 1581, in _infer_task_from_model_name_or_path
    raise RuntimeError(
RuntimeError: Cannot infer the task from a local directory yet, please specify the task manually (masked-im, question-answering, image-to-image, zero-shot-object-detection, feature-extraction, object-detection, automatic-speech-recognition, text-generation, token-classification, audio-xvector, stable-diffusion-xl, text-classification, depth-estimation, audio-frame-classification, image-classification, mask-generation, image-to-text, text-to-audio, stable-diffusion, image-segmentation, fill-mask, multiple-choice, sentence-similarity, audio-classification, text2text-generation, zero-shot-image-classification, conversational, semantic-segmentation).

You have TinyLlama folder which contains incomplete model. Remove or rename it

I checked the SHA256. They are same.

sha256sum filename

--task text-generation-with-past --weight-format fp16 can resolve the issue.
I am not sure that this issue occurs on other models. If so, the doc readme may need a refresh.


fatal error: 'string' file not found

I'm on Linux.
When running make in the top-level, I get:
[ ISOBMFF ]> Compiling C++ file [ Release - x86_64 ]: HVCC.cpp
In file included from ISOBMFF/source/HVCC.cpp:31:
In file included from ISOBMFF/include/ISOBMFF/HVCC.hpp:36:
In file included from ISOBMFF/include/ISOBMFF/FullBox.hpp:36:
In file included from ISOBMFF/include/ISOBMFF/Box.hpp:36:
ISOBMFF/include/ISOBMFF/BinaryStream.hpp:34:10: fatal error: 'string' file not found
#include <string>
         ^
1 error generated.
make[1]: *** [Submodules/makelib/Targets.mk:220: Build/Release/Intermediates/x86_64/HVCC.cpp.o] Error 1
make[1]: *** Waiting for unfinished jobs....
[ ISOBMFF ]> Compiling C++ file [ Release - x86_64 ]: STSD.cpp
In file included from ISOBMFF/source/STSD.cpp:31:
In file included from ISOBMFF/include/ISOBMFF/STSD.hpp:36:
In file included from ISOBMFF/include/ISOBMFF/FullBox.hpp:36:
In file included from ISOBMFF/include/ISOBMFF/Box.hpp:36:
ISOBMFF/include/ISOBMFF/BinaryStream.hpp:34:10: fatal error: 'string' file not found
#include <string>
         ^

It looks like it is compiling C instead of C++? With make SHELL='sh -x', I see:
+ echo -e [\x1b[32;01m ISOBMFF \x1b[0m]> Compiling C++ file [ \x1b[36;01mRelease - x86_64\x1b[0m ]: \x1b[33;01m\x1b[30;01mILOC-Item-Extent.cpp\x1b[0m\x1b[0m
-e [\x1b[32;01m ISOBMFF \x1b[0m]> Compiling C++ file [ \x1b[36;01mRelease - x86_64\x1b[0m ]: \x1b[33;01m\x1b[30;01mILOC-Item-Extent.cpp\x1b[0m\x1b[0m
+ clang -Werror -Wall -fPIC -Os -stdlib=libc++ -fno-strict-aliasing -ISubmodules/PIMPL/PIMPL/include -IISOBMFF/include/ -std=c++11 -o Build/Release/Intermediates/x86_64/ILOC-Item-Extent.cpp.o -c ISOBMFF/source/ILOC-Item-Extent.cpp
In file included from ISOBMFF/source/ILOC-Item-Extent.cpp:31:
In file included from ISOBMFF/include/ISOBMFF/ILOC.hpp:36:
In file included from ISOBMFF/include/ISOBMFF/FullBox.hpp:36:
In file included from ISOBMFF/include/ISOBMFF/Box.hpp:36:
ISOBMFF/include/ISOBMFF/BinaryStream.hpp:34:10: fatal error: 'string' file not found
#include <string>
         ^
1 error generated.
make[1]: *** [Submodules/makelib/Targets.mk:220: Build/Release/Intermediates/x86_64/ILOC-Item-Extent.cpp.o] Error 1
make[1]: *** Waiting for unfinished jobs....

It looks like it is using clang instead of clang++ for some reason, but CC=clang++ make, make CC=clang++ or changing clang to clang++ in the Makefile don't seem to work either.
Compiling a minimal example with clang test.cpp -std=c++11 && ./a.out works however, so it must be a build system issue.
#include <iostream>
#include <string>

int main()
{
    std::cout << std::string( 5, 'A' ) << std::endl;
    return 0;
}

Which Linux distribution are you using?

You probably don't have libc++ installed. Try your example with -stdlib=libc++.

-std=libc++ does indeed lead to the same problem for the example.

I'm using Debian.
I had libc++ installed:
ii  libc++1:amd64                        3.7.0-1                 amd64                   LLVM C++ Standard library

Would have surprised me if not. I tried to install libc++-dev but it didn't help and now I have these package versions (I hope my system doesn't break...)
+++-====================================-=======================-=======================-==============================================================================
ii  libc++-7-dev:amd64                   1:7.0.1-8               amd64                   LLVM C++ Standard library (development files)
ii  libc++-dev:amd64                     1:7.0-47.1              amd64                   LLVM C++ Standard library (development files)
un  libc++-x.y                           <none>                  <none>                  (no description available)
un  libc++-x.y-dev                       <none>                  <none>                  (no description available)
ii  libc++1:amd64                        1:7.0-47.1              amd64                   LLVM C++ Standard library
ii  libc++1-7:amd64                      1:7.0.1-8               amd64                   LLVM C++ Standard library
un  libc++abi-x.y                        <none>                  <none>                  (no description available)
un  libc++abi1                           <none>                  <none>                  (no description available)
ii  libc++abi1-7:amd64                   1:7.0.1-8               amd64                   LLVM low level support for a standard C++ library

This is a known package issue on Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=855222
You may try to change the Makefile to use libstdc++ and GCC instead, but I haven’t tried it.

Ok, changing the -stdlib=libc++ part in FLAGS_OTHER to -stdlib=libstdc++ make the .cpp compile, but then linking fails with:
/usr/bin/ld: cannot find -lc++abi
clang: error: linker command failed with exit code 1 (use -v to see invocation)

After removing -lc++ from LIBS := in the Makefile, it links correctly.
Unfortunately, it only builds the static and shared libraries and not the ISOBMFF-Dump command line tool?
I got that to work somehow by manually compiling it:
clang -fno-strict-aliasing -ISubmodules/PIMPL/PIMPL/include -IISOBMFF/include/ \
-std=c++o isobmff-dump ISOBMFF-Dump/main.cpp \
Build/Release/Products/x86_64/libISOBMFF.a -lstdc++ -lm

and running it with ./isobmff-dump IMG_1234.heic works.
Btw, the dump prints in one line - Handler name: [NUL-ASCII-CHAR]. I don't think printing non-printable ascii characters is intended? It's basically the only non-printable char in the whole output.

I’m currently on holiday. I’ll take a closer look when I’m back.

Ok, changing the -stdlib=libc++ part in FLAGS_OTHER to -stdlib=libstdc++ makes the .cpp compile, but then linking fails with:
/usr/bin/ld: cannot find -lc++abi
clang: error: linker command failed with exit code 1 (use -v to see invocation)

After removing -lc++ from LIBS := in the Makefile, it links correctly.
Unfortunately, it only builds the static and shared libraries and not the ISOBMFF-Dump command line tool?
I got that to work somehow by manually compiling it:
clang -fno-strict-aliasing -ISubmodules/PIMPL/PIMPL/include -IISOBMFF/include/ \
-std=c++o isobmff-dump ISOBMFF-Dump/main.cpp \
Build/Release/Products/x86_64/libISOBMFF.a -lstdc++ -lm

and running it with ./isobmff-dump IMG_1234.heic works.
Btw, the dump prints in one line - Handler name: [NUL-ASCII-CHAR]. I don't think printing non-printable ascii characters is intended? It's basically the only non-printable char in the whole output.

Also close the line #LIBS                := -lc++
if linker gives error.

Maybe you should use g++ instead of clang.
update Makefile like this:
CC                  := g++
FLAGS_CPP           :=
LIBS                :=


Fix StringIO import in Python3

In environment with Python 3.5.2 an attempt to render template with ini as input data fails with the following traceback:
$ jinja2 cloud-config.jinja2 node.ini > result.yaml

Traceback (most recent call last):
  File "/usr/local/var/pyenv/versions/jinja2-tools/bin/jinja2", line 11, in <module>
    sys.exit(main())
  File "/usr/local/var/pyenv/versions/3.5.2/envs/jinja2-tools/lib/python3.5/site-packages/jinja2cli/cli.py", line 335, in main
    sys.exit(cli(opts, args))
  File "/usr/local/var/pyenv/versions/3.5.2/envs/jinja2-tools/lib/python3.5/site-packages/jinja2cli/cli.py", line 232, in cli
    data = fn(data) or {}
  File "/usr/local/var/pyenv/versions/3.5.2/envs/jinja2-tools/lib/python3.5/site-packages/jinja2cli/cli.py", line 85, in _parse_ini
    import StringIO
ImportError: No module named 'StringIO'

This PR fixes this issue.

Thanks. 🍕

Great fix. Any chance this would be released?


oc-mirror does not account for umask restrictions on host machine

Version
$ oc-mirror version
4.12 - 4.15-rc.4

What happened?
Hi team!
We are using oc-mirror in a fairly restricted environment, with a default umask of 0077 on both our internet connected, and disconnected environment hosts. This has resulted in some confusing, unexpected issues - specifically with the redhat-operator-index container.
I don't have full context on the exact process of how oc-mirror handles the export, tar/un-tar and finally import of these imagest into an offline quay instance, but I do see the impacts of system default umask settings.
Looking at some areas of the oc-mirror codebase, I do see where explicit chmod is ran on several directories at creation time, so my assumption is that there is an expectation of correct permissions being set
However: os.Mkdir("./example/directory", 0775) is not respected by the system default umask, and in our case with a 0077 umask, that directory would have 0700 permissions.
I believe it would make sense to take advantage of  golang.org/x/sys/unix to handle setting the umask in the session oc-mirror is ran, then setting it back.
Here's an example borrowed from this blog post:
package main

import (
	"fmt"
	"strconv"
	"strings"

	"golang.org/x/sys/unix"
)

func main() {
	fmt.Println("Checking umask of current session:")

	checkUmask()

	var forcedUmask int = 0022

	fmt.Println("Setting umask to more permissive 0022")
	savedUmask := unix.Umask(forcedUmask)
	checkUmask()
	_ = unix.Umask(savedUmask) // Return the umask to the original
}

func checkUmask() {
	currentUmask := unix.Umask(0)

	umaskOctal := strconv.FormatInt(int64(currentUmask), 8) // Convert the uint32 to int64 and format as octal
	desiredLength := len("0022")                            // Set desired length of output (we want 0022, not just '22')
	leadingZeros := strings.Repeat("0", desiredLength-len(umaskOctal))
	fmt.Printf("The current session umask is: %s\n", leadingZeros+umaskOctal)
}

Then, we run the program:
❯ umask
077
❯ go run .
Checking umask of current session:
The current session umask is: 0077
Setting umask to more permissive 0022
The current session umask is: 0022

I believe an addition of session umask handling, with appropriate logging would greatly reduce some of the issues that impact Red Hat customers, and support personnel who are impacted by this problem.
I also believe it is fair to assume that a large majority of oc-mirror users are in some form of restricted environment, especially with the inroads Open Shift is making in the gvt/dod space, so these impacts are going to occur more frequently.
What did you expect to happen?
container and operator images to be migrated from the internet to a disconnected quay instance reliably, and correctly. (as in, correct permissions retained)
How to reproduce it (as minimally and precisely as possible)?
use oc-mirror to move redhat/redhat-operator-index from an internet connected system to an offline system, that are both set to a system umask of 0077
deploy that new redhat-operator-index container in an OCP cluster
observe the container go into crashLoopBackoff as it cannot read/write to /configs since that dir is created with 0700 vs 0755 permissions.
Anything else we need to know?
There are quite a few OCPBUG tickets, and Red Hat support cases covering this issue
References
https://issues.redhat.com/browse/OCPBUGS-27125 ( the exact issue we're impacted by)
https://issues.redhat.com/browse/OCPBUGS-26078 (parent ticket)
https://access.redhat.com/solutions/7006771
I am happy to work with the team on testing some strategies for implementing a umask-aware fix to help alleviate the issues we're running into when using oc-mirror to move images and operators into our disconnected environment.
Many thanks,
-BadgerOps

Any thoughts on this as a potential path forward? As we see an increase in disconnected OCP deployments, in hardened environments, I believe it makes sense to ensure the reliability and ease-of-use of the surrounding tooling is addressed.
I've spent a little time reading through the codebase, but unfortunately do not have a good enough grasp on the flow to submit an example PR at this point. I'm hoping someone on the team with better context can help out here!
Thanks!

golang mkdir documentation shows that it respects the system umask. We would likely need to use something like os.Chmod after running os.Mkdir.
https://pkg.go.dev/os#Mkdir

I'm not actively using oc-mirror at the moment, but I do still think this request is still helpful for many folks who are using the tool.

I recently discovered that setting a temporary session umask is unreliable when using automation like Ansible, because Ansible makes multiple connections / logins which use the system's defined (unusable) umask. It would be better if oc-mirror set the required/expected permissions itself via os.chmod or similar...


Form don't like a schema with refine on it, and I can't understand why

I created a schema with a .refine (following your example on https://remix-forms.seasoned.cc/examples/schemas/zod-effects) and it has a great UX on front-end by displaying the error & focusing the creditCardInvoiceSlug field when it is empty.
But TS is not happy with it and I needed to put a // @ts-expect-error to move forward but that disabled all Zod key checking in my form.
My schema (simplified with the fields involved on refine):
export const transactionFormSchema = z
  .object({
    paymentSource: z.string().min(1, {
      message: "Escolha Conta ou Cartão",
    }),
    creditCardInvoiceSlug: z.string().nullish(),
  })
  .refine(
    ({ creditCardInvoiceSlug, paymentSource }) =>
      paymentSource.startsWith("creditCard") && !creditCardInvoiceSlug,
    {
      message: "É necessário informar a fatura do cartão",
      path: ["creditCardInvoiceSlug"],
    }
  );

The error:
Type 'ZodEffects<ZodObject<{ paymentSource: ZodString; creditCardInvoiceSlug: ZodOptional<ZodNullable<ZodString>>; }, "strip", ZodTypeAny, { ...; }, { ...; }>, { ...; }, { ...; }>' is missing the following properties from type 'ZodObject<ZodRawShape, UnknownKeysParam, ZodTypeAny, { [x: string]: any; }, { [x: string]: any; }>': _cached, _getCached, shape, strict, and 14 more.

I tried moving the refine to the mutation but when I do it I lose the front-end auto focus feature on the missing field and that's bad.

It was a problem in my form wrapper type, it worked fine with the refine now 🥳


Automatic fullscreen on android

On iOS the player goes fullscreen once the video is loaded. I cannot replicate this behavior on android and i need to manually click on the fullscreen button. This is my config
<Vimeo videoId={this.props.route.params.videoId} params={"api=1&autoplay=1&fullscreen=1"} handlers={videoCallbacks} style={{ backgroundColor: "black" }} androidLayerType='hardware' />
Any idea to replicate this automatic fullscreen behavior on android ?
Nice lib guys, thanks.

Any update on this? still facing this in android.


HIVE-28209: Allow the use of a proxy user

What changes were proposed in this pull request?
Allow delegation for HiveMetaStore connections
Why are the changes needed?
The log message indicates that HADOOP_PROXY_USER is set, but then doesn't allow its use by calling getRealUser() which will be null.  This change allows delegation for HiveMetaStore connections by using either the real user or self.  The code for this is coming from Hadoop 3.3.6.
Does this PR introduce any user-facing change?
Allows proxying instead of throwing an exception.
Is the change a dependency upgrade?
No
How was this patch tested?
Local builds which have removed getRealUser() to allow the use of delegation work as expected.  Running the existing code always fail because getRealUser() will always return null.

+1

@snmvaughan Thanks for the contrib! Merged to master.


Error in compiling custom_ops

I wonder what is the reason for this problem? Usually, this problem is caused by a failure to bind successfully. But since the code has been around for a while, that shouldn't be the reason.
I use

Windows 11
MSVC x64 19.38.33130
python3.10
torch 2.1

same error

I just updated the get_patches; it should run without compiling. For more information, refer to this code.

same error, how to run without compiling?


Replace is_ajax with accepts

django.http.request.HttpRequest.is_ajax is deprecated. The documentation for version 3 indicated that instead of calling is_ajax, the code should check if the text/html MIME type is acceptable based on the request's Accept header.

Django 2.2 is in LTE for a few more years. Can you write in a compatibility check for 2.2 to fix the tests?

Maybe the JS code should explicitly set an Accept: application/json header to discern between AJAX and other requests.
I know that the docs suggest replacing request.is_ajax() with not request.accepts("text/html"), however I'm not sure it's the best way to do it. After all, until recently we also did fetch HTML, too.
I'd like a more explicit solution for this, not just in django-debug-toolbar but also in my own projects.

I can check if the request has an accepts method and fall back to is_ajax if it doesn't, like this:
accepts = getattr(request, "accepts", None)
if accepts is None:
    is_ajax = request.is_ajax()
else:
    is_ajax = not accepts("text/html")

Another option is to add a custom header to all fetch calls, such as X-Toolbar-AJAX: 1.
That can be checked easily in all supporter versions of Django:
is_ajax = request.headers.get("X-Toolbar-AJAX") == "1"

Maybe the JS code should explicitly set an Accept: application/json header to discern between AJAX and other requests.

I understand this to mean we should have the toolbar's JS manipulate the header on AJAX requests for the developer's application. I don't think that's what we want.
This part of the code isn't related to the history panel's internal API requests. It's checking for AJAX or JSON requests that we should capture the toolbar's measurements for so the history panel can show them.

Checking for JSON requests is easy, but an AJAX request is just a normal HTTP request; there is no built-in way to determine if a request is AJAX or not.
The code currently checks for a jQuery-specific header to determine if a request is an AJAX request or not. As more developers move from jQuery to fetch, that check will become less useful.
In my view (which could be totally wrong), the best way forward is to introduce a new custom header that developers can choose to add to their AJAX requests if those requests return anything other than JSON. If all of the developer’s AJAX APIs respond with JSON, they won’t have to change anything, since JSON is already properly handled, but if the response is HTML or XML, adding the header indicates that the request was made via AJAX.

I'm not sure we need this check. If someone has an app that runs some
business logic before returning a file, we should be trying to help them
debug that process. We shouldn't exclude that use case based on the
response type.
Granted, moving in this direction means we have to verify the existing
panels don't break on non html/json responses.

I'm going to close this PR and open a new issue to discuss the best path forward.


Feature / Widget redesign

This PR redesigns this plugin with the new "Insert" menu in mind.
To make sense of this, "Insert article" and "Insert decision title" are static buttons that are now displayed in the new insert menu (unless there already is a decision title in the document). Other buttons about articles are more context driven and are still in their original card.

No idea what the best approach is for this plugin, because it seems that half of it is semi-static and the other half is more context driven.
Also, package upgrades to new AppUniversum are included, which makes this PR https://github.com/lblod/ember-rdfa-editor-besluit-plugin/pull/12 redundant (?).

Thanks to @Dietr, lay-out has been updated to a more concise plugin-card, like so:

As per new instructions, always show all buttons, but disabled when it's functionality is irrelevant due to the current context of the editor. This will be done in the next commit(s).

adding articles below/above seem to trigger errors im on latest dev gn, editor, and this branch


Implement delete page command

We should be able to delete a single page from the index. I.e. A page has many modifications, so it should be deleted and then re-indexed.

Continued with #260


Section 7.2. (mis)Optimization : Can not play the debug session as described in the tutorial

Hi everyone,
I have some troubles at running this section correctly.
First, here are my version of rustup and rustc :
$ rustup --version
rustup 1.23.1 (3df2264a9 2020-11-30)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.53.0-nightly (07e0e2ec2 2021-03-24)`

The problem is that I can not execute step by step (using the gdb next instruction) the binary launched with cargo run --release. Here after is my session :
$ cargo run --release
    Finished release [optimized + debuginfo] target(s) in 0.02s
     Running `arm-none-eabi-gdb -q -x openocd.gdb '/home/XXX/Dev/Rust/Embedded_Discovery/discovery-master/target/thumbv7em-none-eabihf/release/registers'`
Reading symbols from /home/XXX/Dev/Rust/Embedded_Discovery/discovery-master/target/thumbv7em-none-eabihf/release/registers...
0x08000252 in registers::__cortex_m_rt_main () at /rustc/07e0e2ec268c140e607e1ac7f49f145612d0f597/library/core/src/ptr/mod.rs:1150
1150	/rustc/07e0e2ec268c140e607e1ac7f49f145612d0f597/library/core/src/ptr/mod.rs: Aucun fichier ou dossier de ce type.
Loading section .vector_table, size 0x194 lma 0x8000000
Loading section .text, size 0x1220 lma 0x8000194
Loading section .rodata, size 0x1b4 lma 0x80013b4
Start address 0x8000194, load size 5480
Transfer rate: 13 KB/sec, 1826 bytes/write.
Breakpoint 1 at 0x8000224: file src/07-registers/src/main.rs, line 9.
Breakpoint 2 at 0x8001078
Breakpoint 3 at 0x80013b2
Note: automatically using hardware breakpoints for read-only addresses.

Breakpoint 1, main () at src/07-registers/src/main.rs:9
9	#[entry]
(gdb) s
halted: PC: 0x0800022a
registers::__cortex_m_rt_main () at src/07-registers/src/main.rs:11
11	    aux7::init();
(gdb) n
halted: PC: 0x08000254
halted: PC: 0x08000236
halted: PC: 0x0800023a
halted: PC: 0x0800023e
18	        ptr::write_volatile(GPIOE_BSRR as *mut u32, 1 << 9);
(gdb) n
halted: PC: 0x08000240
halted: PC: 0x08000244
halted: PC: 0x08000246
halted: PC: 0x0800024a
halted: PC: 0x0800024c
halted: PC: 0x08000250
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
halted: PC: 0x08000252
^Chalted: PC: 0x08000252
halted: PC: 0x08000252

Program received signal SIGINT, Interrupt.
0x08000252 in registers::__cortex_m_rt_main () at /rustc/07e0e2ec268c140e607e1ac7f49f145612d0f597/library/core/src/ptr/mod.rs:1150
1150	/rustc/07e0e2ec268c140e607e1ac7f49f145612d0f597/library/core/src/ptr/mod.rs: Aucun fichier ou dossier de ce type.

after the second nextinstruction, I enter in the loop rather to go to the next write_volatile ...
If I disassemble, I get :
Loading section .vector_table, size 0x194 lma 0x8000000
Loading section .text, size 0x1220 lma 0x8000194
Loading section .rodata, size 0x1b4 lma 0x80013b4
Start address 0x8000194, load size 5480
Transfer rate: 13 KB/sec, 1826 bytes/write.
Breakpoint 1 at 0x8000224: file src/07-registers/src/main.rs, line 9.
Breakpoint 2 at 0x8001078
Breakpoint 3 at 0x80013b2
Note: automatically using hardware breakpoints for read-only addresses.

Breakpoint 1, main () at src/07-registers/src/main.rs:9
9	#[entry]
(gdb) s
halted: PC: 0x0800022a
registers::__cortex_m_rt_main () at src/07-registers/src/main.rs:11
11	    aux7::init();
(gdb) n
halted: PC: 0x08000254
halted: PC: 0x08000236
halted: PC: 0x0800023a
halted: PC: 0x0800023e
18	        ptr::write_volatile(GPIOE_BSRR as *mut u32, 1 << 9);
(gdb) disassemble /m
Dump of assembler code for function registers::__cortex_m_rt_main:
10	fn main() -> ! {
   0x0800022a <+0>:	push	{r7, lr}
   0x0800022c <+2>:	mov	r7, sp

11	    aux7::init();
   0x0800022e <+4>:	bl	0x8000254 <aux7::init>
   0x08000232 <+8>:	movw	r0, #4120	; 0x1018
   0x08000236 <+12>:	mov.w	r1, #512	; 0x200
   0x0800023a <+16>:	movt	r0, #18432	; 0x4800
=> 0x0800023e <+20>:	str	r1, [r0, #0]
   0x08000240 <+22>:	mov.w	r1, #2048	; 0x800
   0x08000244 <+26>:	str	r1, [r0, #0]
   0x08000246 <+28>:	mov.w	r1, #33554432	; 0x2000000
   0x0800024a <+32>:	str	r1, [r0, #0]
   0x0800024c <+34>:	mov.w	r1, #134217728	; 0x8000000
   0x08000250 <+38>:	str	r1, [r0, #0]
   0x08000252 <+40>:	b.n	0x8000252 <registers::__cortex_m_rt_main+40>

End of assembler dump.

I have noticed that the file mod.rs is not found. Could it be the problem ?
I have searched also in the directory that we can see in the tutorial (~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs) but it is not. May anyone explain me what I have missed ?
Many thanks for your responses :-)

I see the individual string movements are there, try using stepi (or si) to step by machine instruction rather than line. Lights should then individually switch on and off.

Hi there @pcardune @benschneider @the-cat-that-codes I am also trying to debug in release mode I am using cargo run --release command in discovery/src/07-registers directory.
The error i am getting is-
knoldus@knoldus-Vostro-3590:~/Documents/discovery/src/07-registers$ cargo run --release     Finished release [optimized + debuginfo] target(s) in 0.04s      Running arm-none-eabi-gdb -q -x openocd.gdb /home/knoldus/Documents/discovery/target/thumbv7em-none-eabihf/release/registerserror: could not execute processarm-none-eabi-gdb -q -x openocd.gdb /home/knoldus/Documents/discovery/target/thumbv7em-none-eabihf/release/registers` (never executed)
Caused by:
No such file or directory (os error 2)
`
I am using rustup 1.24.2 (755e2b07e 2021-05-12)

I am also working on the same chapter and facing the same issue in mis(optimization). Nothing comes this
Breakpoint 1, main () at src/07-registers/src/main.rs:7 7       #[entry] (gdb) s registers::__cortex_m_rt_main () at src/07-registers/src/main.rs:9 9           aux7::init(); (gdb) n 16              ptr::write_volatile((GPIOE_BSRR as *mut u32), (1<<9));

I think this is an artefact of optimisation, which can create a bit of a discontinuity between your source and the assembly.  The debugger attempts to step through your source, but the source no longer sufficiently corresponds with the assembly and it gets a bit lost.  It still executes the program.  The repeated halted: PC: 0x08000252 is the loop after it turns the LEDs off.
The simple solution, as @benschneider points out, is to just step through the assembly one instruction at a time with stepi.  You can view the assembly in the debugger as it's executed with layout asm.
@japaric: I'm happy to submit a PR, but there appear to be several stale issues and PRs associated with this section already (#238, #245, and #237 are the obvious ones).  Also, it looks like the most recent commit to discovery/src/07-registers/optimization.md (cdf6be) actually changed from stepping through the assembly to stepping through the source, so maybe we in this thread are simply doing something wrong...


consistency with codelab

Tweaks so the code more closely matches the codelab which was updated to support the product name change and other minor fixes.

Updated files in PR to include Node.js files as well as moving Python code to its own subfolder.


Emergency withdraw is no longer "emergency"

With the current state of the emergency withdraw function, its only difference from the normal withdraw function is that it does not call the safeTransferReward of the reward controller. This is not really an "emergency" withdraw, it just withdraw without claiming the reward. This could be useful in case the reward token is paused, or if there is not enough of the reward token in the reward controller to pay the reward.
However, in both these cases, the user will likely not want to lose their reward, but would be happier with being able to just claim it at a later time. Also, in both these situations, the user will currently be forced to withdraw everything just to be able to withdraw from the vault, but they may want to just withdraw part of their deposit in the vault.
I would propose to remove the emergency withdraw function, and just completely separate the sending of rewards from updating the vault balance, so that on each action the "unclaimed" rewards of the user are registered and accumulated, and then at any time, the user can call claim reward to claim these rewards.

I sort of think this is a good idea. Also others (balancer say) do that. We can also add a convenience function "withdrawAndClaim" like curve has.
@shayzluf wdyt

I sort of think this is a good idea. Also others (balancer say) do that. We can also add a convenience function "withdrawAndClaim" like curve has.
@shayzluf wdyt

Let's do it and add withdrawAndClaim for backwards compatibility with the ui

OK added as part of #281


Question about compiling.

If I already have a tensorflow program, how can I get the .a file, which contains the openTPU instruction sequence of the tf program. You have a compiler to compile the tf program or you just do it manually?

We do not have a compiler --- you must manually generate the .a file.


yesterday issue

{{ Date::parse($me->created_at)->ago() }}

I want it will show as like yesterday then  2days ago . there any way to solve this .

Now than jenssegers/date supports Carbon 2, you can use it to get "yesterday"/"tomorrow":
Date::parse('yesterday')->diffForHumans(['options' => CarbonInterface::ONE_DAY_WORDS])


Go through API docs, make sure using proper Typescript syntax

About time to do this

Calling this done.


Add docstring to revision param for downloadFile

Copies the docstring in the Python counterpart

you can run pnpm -r format to format

Thanks


HTML SAX parsing works when using a string but fails when using a file

I noticed that the HTML SAX parser fails when reading the same html from a file vs a string. It chokes on any badly formed tags such as <br > from an IO source
I've added a test to highlight the issue in  #1576
If anyone could point me in a general direction I would be more than happy to try fix this.

Thanks for reporting this, and for PRing the failing test. This is a weird one, looking into it now.

Yeeeeaaaaaah, this is a weird one. The TL;DR is that HTML::SAX::Parser is using XML::ParserContext instead of HTML::ParserContext for parsing IO. This bug has been present since ... the dawn of time.
SOOOOoooo ... how does it feel to be the first person in the universe to SAX-parse an HTML file through an IO object? Must feel good, right? Right?
Sigh.
I'll have a fix as soon as I work through the inconsistencies in API between the XML and HTML SAX parsers.


Fix parse problem in EwsUtilities

fixes a parsing problem in EwsUtilities. Also includes some tests for
the parsing process as well as some additional parsing behaviors.
ref.: #296, #300, #301

Hi @serious6, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
We will now validate the agreement and then real humans will evaluate your PR.
TTYL, AZPRBOT;

thanks @tomravn for the given input on the defect.

maybe to make the api work again we should avoid a long delay in merging this. So any feedback is welcome.

:+1:


No module named 'win32_setctime'

win 10  执行 python startup.py "写一个俄罗斯方块"
报错信息如下：
Traceback (most recent call last):
File "E:\code\mategpt\metagpt\startup.py", line 5, in
from metagpt.software_company import SoftwareCompany
File "E:\code\mategpt\metagpt\metagpt\software_company.py", line 9, in
from metagpt.config import Config
File "E:\code\mategpt\metagpt\metagpt\config.py", line 10, in
from metagpt.logs import logger
File "E:\code\mategpt\metagpt\metagpt\logs.py", line 10, in
from loguru import logger as logger
File "E:\Programs\Python\Python310\lib\site-packages\loguru-0.7.0-py3.10.egg\loguru_init.py", line 10, in
from ._logger import Core as _Core
File "E:\Programs\Python\Python310\lib\site-packages\loguru-0.7.0-py3.10.egg\loguru_logger.py", line 95, in
from ._file_sink import FileSink
File "E:\Programs\Python\Python310\lib\site-packages\loguru-0.7.0-py3.10.egg\loguru_file_sink.py", line 12, in
from ._ctime_functions import get_ctime, set_ctime
File "E:\Programs\Python\Python310\lib\site-packages\loguru-0.7.0-py3.10.egg\loguru_ctime_functions.py", line 57, in
get_ctime, set_ctime = load_ctime_functions()
File "E:\Programs\Python\Python310\lib\site-packages\loguru-0.7.0-py3.10.egg\loguru_ctime_functions.py", line 6, in load_ctime_functions
import win32_setctime
ModuleNotFoundError: No module named 'win32_setctime'

解决方案：pip install win32-setctime


Error fetching access token. Not connecting to Firebase Auth Emulator if application_default_credentials.json not present

[READ] Step 1: Are you in the right place?

For issues related to the code in this repository file a Github issue.
If the issue pertains to Cloud Firestore, read the instructions in the "Firestore issue"
template.
For general technical questions, post a question on StackOverflow
with the firebase tag.
For general Firebase discussion, use the firebase-talk
google group.
For help troubleshooting your application that does not fall under one
of the above categories, reach out to the personalized
Firebase support channel.

[REQUIRED] Step 2: Describe your environment

Operating System version: Debian 10 (but I think is OS independent). Runing in the container produced by this Dockerfile

ARG VARIANT="14-buster"
FROM mcr.microsoft.com/vscode/devcontainers/typescript-node:0-${VARIANT}
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
   && apt-get -y install --no-install-recommends openjdk-11-jre-headless
RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg  add - && apt-get update -y && apt-get install google-cloud-sdk -y
RUN sudo -u node npm install -g<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

Firebase SDK version: v8.14.0
Firebase Product: auth
Node.js version: v10.22.0
NPM version: 6.14.6

[REQUIRED] Step 3: Describe the problem
If the 'application_default_credentials.json' is not present in the gcloud config path I get the error from below when I try to call any auth method even though the emulator is started, the FIREBASE_AUTH_EMULATOR_HOST env var is correctly set and the initializeApp is correctly called.
Based on my research this problem comes from the getApplicationDefault function inside the src/credential/credential-internal.ts file. I solved this by running gcloud auth application-default login (found on internet, don't know exactly what it does) which generated a valid application_default_credentials.json file
I don't completely understand the underlying mechanisms but I think this behaviour is not great for my use case which is probably very similar with a CI/CD pipeline use case. Also, there's nothing mentioned in the Firebase Auth emulator about this and I really like to be able to run all emulators without a real project but, doing this gcloud config makes me worried not to interact by mistake with the real gcloud project.
 Error: Credential implementation provided to initializeApp() via the "credential" property failed to fetch a valid Google OAuth2 access token with the following error: "Error fetching access token: Error while making request: getaddrinfo ENOTFOUND metadata.google.internal metadata.google.internal:80. Error code: ENOTFOUND".
    at FirebaseAppError.FirebaseError [as constructor] (C:\Users\Marian\Desktop\test\tp\node_modules\firebase-admin\lib\utils\error.js:43:28)
    at FirebaseAppError.PrefixedFirebaseError [as constructor] (C:\Users\Marian\Desktop\test\tp\node_modules\firebase-admin\lib\utils\error.js:89:28)
    at new FirebaseAppError (C:\Users\Marian\Desktop\test\tp\node_modules\firebase-admin\lib\utils\error.js:124:28)
    at C:\Users\Marian\Desktop\test\tp\node_modules\firebase-admin\lib\firebase-app.js:124:23
    at process._tickCallback (internal/process/next_tick.js:68:7)

Steps to reproduce:

Run the firebase emulators (especially the auth emulator)
Set the FIREBASE_AUTH_EMULATOR_HOST env var as described in the docs
Create a simple npm project that uses the firebase-admin
Write a simple script (see detailed code below) that makes any call on the auth using the firebase-admin

Relevant Code:
I'm running a dev environment using the VSCode .devcontainer feature using the following files:
main.Dockerfile
ARG VARIANT="14-buster"
FROM mcr.microsoft.com/vscode/devcontainers/typescript-node:0-${VARIANT}
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends openjdk-11-jre-headless
RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg  add - && apt-get update -y && apt-get install google-cloud-sdk -y
RUN sudo -u node npm install -g<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
firebase_emulator.Dockerfile
ARG VARIANT="14-buster"
FROM mcr.microsoft.com/vscode/devcontainers/typescript-node:0-${VARIANT}

RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends openjdk-11-jre-headless

RUN sudo -u node npm install -g<EMAIL_ADDRESS>
COPY ./firebase.json .
COPY ./.firebaserc .

RUN firebase setup:emulators:database
RUN firebase setup:emulators:firestore

ENTRYPOINT [ "firebase",  "emulators:start"]

docker-compose.yml
version: '3'
services:
  main:
    build:
      context: .
      dockerfile: main.Dockerfile
      args:
        VARIANT: 10
    volumes:
      - ..:/workspace
    command: sleep infinity
    environment:
      FIRESTORE_EMULATOR_HOST: "firebase_emulator:8080"
      FIREBASE_DATABASE_EMULATOR_HOST: "firebase_emulator:9000"
      FIREBASE_AUTH_EMULATOR_HOST: "firebase_emulator:9099"
      GCLOUD_PROJECT: "local"
    links:
      - firebase_emulator
  firebase_emulator:
    build:
      context: .
      dockerfile: firebase_emulator.Dockerfile
      args:
        VARIANT: 10
    ports:
      - "4000:4000"
      - "5001:5001"
      - "8080:8080"
      - "9000:9000"
      - "8085:8085"
      - "4500:4500"
      - "4400:4400"
      - "9005:9005"
      - "9099:9099"

devcontainer.json
{
	"name": "Node.js & TypeScript & GCP & Firebase & Nest",
	"dockerComposeFile":"docker-compose.yml",
	"service": "main",
	"workspaceFolder": "/workspace",
	"settings": {
		"terminal.integrated.shell.linux": "/bin/zsh",
		"files.autoSave": "afterDelay",
		"git.autofetch": true,
		"workbench.iconTheme": "material-icon-theme"
	},
	"extensions": [
		"dbaeumer.vscode-eslint",
		"ms-vscode.vscode-typescript-tslint-plugin",
		"eg2.vscode-npm-script",
		"redhat.vscode-yaml",
		"abhijoybasak.nestjs-files",
		"nrwl.angular-console",
		"esbenp.prettier-vscode",
		"firsttris.vscode-jest-runner",
		"github.vscode-pull-request-github",
		"pkief.material-icon-theme",
		"humao.rest-client"
	],
	"remoteEnv": {
		"GITHUB_TOKEN": "${localEnv:GITHUB_TOKEN}",
		"FIREBASE_TOKEN": "${localEnv:FIREBASE_TOKEN}"
	},
	"postCreateCommand": "npm install && npm install --only=dev",
	"remoteUser": "node"
}

firebase.json
{
    "emulators": {
      "firestore": {
        "port": 8080,
        "host": "<IP_ADDRESS>"
      },
      "database": {
        "port": 9000,
        "host": "<IP_ADDRESS>"
      },
      "auth": {
        "port": 9099,
        "host": "<IP_ADDRESS>"
      },
      "ui": {
        "enabled": true,
        "port": 4000,
        "host": "<IP_ADDRESS>"
      }
    }
 }

.firebaserc
{
  "projects": {
    "default": "local"
  }
}

The minimal code required is to call some auth function using the firebase-admin package. For example:
// The test script make sure the FIREBASE_AUTH_EMULATOR_HOST environment var is set
const process=require("process")
process.env.FIREBASE_AUTH_EMULATOR_HOST = "localhost:9099";

var admin = require('firebase-admin');

admin.initializeApp({
    projectId: 'local'
})

admin.auth().createUser({});

Also running into this issue after trying to use the admin SDK as mentioned by @yuchenshi here: https://github.com/firebase/firebase-tools/issues/2749#issuecomment-718141997
One thing I did notice as well (likely separate ticket but maybe could be an easier workaround?). In @firebase/rules-unit-testing you can call initializeAdminApp and get what is suppose to be an admin app, but it's Auth doesn't have all the same methods as firebase-admin's Auth, namely createUser() which is what I specifically was trying to use to create users with specific uid's for testing

@imarian97 you mentioned: Firebase SDK version: v8.14.0. Note that the emulator support is only available on latest version (v9.3.0) of the SDK.

@imarian97 you mentioned: Firebase SDK version: v8.14.0. Note that the emulator support is only available on latest version (v9.3.0) of the SDK.

Sorry! I have confused firebase-tools with firebase-admin-node. So, to clarify, I'm using the latest versions at the moment of writing and these versions are:

Firebase SDK (firebase-admin-node): v9.3.0
Firebase CLI (firebase-tools): v8.14.1

@IsaiahByDayah On your side comment on rules testing and initializeAdminApp: I think it's a completely different abstraction from the admin client, only meant for bypassing rules restrictions. Anyways, out of scope of this issue.

Haha, that would explain things wouldn't it 😅. My bad. I guess I just saw "admin" and made assumptions

@IsaiahByDayah ok there's a lot going on here so I have some questions.

Where do you get this error  Error: Credential implementation provided to initializeApp(), is this from code running within the Functions emulator?  A script running elsewhere?  Do you have to do anything besides 'emulators:start` to trigger this error?
It seems like you have the FIREBASE_TOKEN environment variable set, does the behavior change with/without that?
Is there a simpler way I could reproduce this behavior? Are you able to get this to happen outside of docker / docker compose?

@IsaiahByDayah ok there's a lot going on here so I have some questions.

Where do you get this error  Error: Credential implementation provided to initializeApp(), is this from code running within the Functions emulator?  A script running elsewhere?  Do you have to do anything besides 'emulators:start` to trigger this error?
It seems like you have the FIREBASE_TOKEN environment variable set, does the behavior change with/without that?
Is there a simpler way I could reproduce this behavior? Are you able to get this to happen outside of docker / docker compose?

Hi, Considering the questions I suppose they are also addressed to me so:

The script is not running in the emulator. In my real use case, I get the error inside a Jest test and in a Nest.js app but, I also tested with a simple script ( the last script from the initial comment) in a clean npm project and I still get the error.
I have removed the environment variable and the behaviour is the same, I still get the same error.
I was able to reproduce the same issue in my host machine (running windows) so, it is not docker related. To reproduce the issue, follow the steps in the initial comment. I will rewrite them here in a simpler form:

Assuming you have the specified versions of node, npm and firebase-tools:

Create a folder
Run npm init -y in the folder
Run npm i firebase-admin in the folder
Create the firebase.json file in the folder with the content from the initial comment
Create the .firebaserc file in the folder with the content from the initial comment
Create the index.js file with the following content:

// The test script make sure the FIREBASE_AUTH_EMULATOR_HOST environment var is set
const process=require("process")
process.env.FIREBASE_AUTH_EMULATOR_HOST = "localhost:9099";

var admin = require('firebase-admin');

admin.initializeApp({
   projectId: 'local'
})

admin.auth().createUser({});

Run node index.js

So I think this is working as intended, although I'd like to do some experiments myself to try.  Setting the FIREBASE_AUTH_EMULATOR_HOST doesn't really change how firebase.initializeApp() works, since that initializes things for much more than Auth.  And initializeApp() needs to be able to find a credential from somewhere.  Some ways it can find a credential:

You could manually pass a credential param with projectId
You can point the GOOGLE_APPLICATION_CREDENTIALS env var at a service account JSON file, thus authorizing as a service account.
You can use gcloud auth application-default login to put user credentials on your machine, thus authorizing as yourself.

Now we try to hide this from you in a few cases:

When you run initializeApp() inside Google Cloud, like in a Cloud Function or on App Engine or on Compute Engine, we can automatically discover the necessary credentials due to the trusted environment.
When you run initializeApp() inside the Cloud Functions Emulator, we simulate (1) by morphing your firebase login credentials into something the Admin SDK recognizes.

So when running a script in a completely unauthorized docker container, I'd expect this error.

So I think this is working as intended, although I'd like to do some experiments myself to try. Setting the FIREBASE_AUTH_EMULATOR_HOST doesn't really change how firebase.initializeApp() works, since that initializes things for much more than Auth. And initializeApp() needs to be able to find a credential from somewhere. Some ways it can find a credential:

You could manually pass a credential param with projectId
You can point the GOOGLE_APPLICATION_CREDENTIALS env var at a service account JSON file, thus authorizing as a service account.
You can use gcloud auth application-default login to put user credentials on your machine, thus authorizing as yourself.

Now we try to hide this from you in a few cases:

When you run initializeApp() inside Google Cloud, like in a Cloud Function or on App Engine or on Compute Engine, we can automatically discover the necessary credentials due to the trusted environment.
When you run initializeApp() inside the Cloud Functions Emulator, we simulate (1) by morphing your firebase login credentials into something the Admin SDK recognizes.

So when running a script in a completely unauthorized docker container, I'd expect this error.

I forgot to mention an important step but I suppose you already noticed it. The step was to remove or rename the application_default_credentials.json file which on Windows is located in %appdata%\gcloud
I understand that the initializeApp() needs the credentials but I don't really like to have to provide as a requirement a service account or some other form of credential to my CI/CD pipeline or, in my case, (at least in this stage of the development) to the local development environment, as long as I'm using only the emulators. For Firestore and RTDB emulators, based on my experience, the credentials are not required and I feel like the firebase-admin code can be updated (I have not spent enough time on it to understand how) to also not require the credentials for Auth Emulator.
I don't really have a problem with the credentials but I can't "fit the moment where you generate/obtain the credentials". For the CI/CD I know there are some ways to store a secret (the service account) and then retrieve it but that's extra work and configs considering that only emulators are used. But, for the local development scenario, it is even worse.
The ideal scenario for local development which I managed to achieve using the method described in the first comment is to have any developer of my team clone the repo and immediately start working on code (because I've containerized both the dev env and the Firebase Emulators so,  everything is configured and starts automatically and the developer does not have to do anything). In this scenario, obtaining the credentials can be a bit annoying considering that including the service account in the repo is not a good practice and the developer might not even have access to a GCP/Firebase project (extremely possible for public projects)
I hope the scenarios and my issue are a bit clear now and I hope to find together (or, maybe, it already exists but I'm missing it) a good solution for completely local development and easy to use CI/CD.

Facing a similar issues here. Admin SDK cannot be used with the Auth emulator without providing valid application credentials.
The particular use case for me is a using the Admin SDK to seed/reset the emulators in a script. And while Admin SDK can successfully interact with the Firestore emulator without any application credentials, the Auth part of the Admin SDK requires application credentials with the Auth emulator.

I am also running into this issue. The documentation for the auth emulator recommends writing a script to create users, but I am not able to get admin-firebase to work this way unless I provide it with credentials to a real environment.

@dbanisimov @nVitius thanks for your feedback!  According to @yuchenshi we may be able to fix this by lazily requiring credentials in firebase-admin-node when using Auth with the emulator.

This is fixed and released in version 9.4.0 of firebase-admin


Update to Swift 5.5 release in CI

urgh. jazzy.

Nightly failing for known reasons. Merging over it.


⚠ Bofrost.be has degraded performance

In c35c975, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1888 ms

Resolved: Bofrost.be performance has improved in 536e4e7.


Expand API to predict against a model with a database document

partisan-discourse contains a REST api using django-rest-framework that currently exposes data against the corpus app's Document and Label models, which is used by the frontend UI to trigger document annotations.
To further expose the use of a REST api in a machine learning context, this issue seeks to provide an API endpoint against models in the arbiter application that returns predictions against a GET request off a document in the database. This will need to consider performance; the pycon poster suggested to initialize the 'best' Estimator into memory before request/on application load(?) so the GET itself is fast. While we may in the future want to support on-demand/model-specific predictions, this issue is to implement that paradigm first.
This issue is closed when an authenticated user can GET against a 'best Estimator' endpoint with a document ID in the database and return the results of predict().

Very interesting discussion here; because "best" is going to come up against context. E.g. best user model, best global model, etc.
I would suggest the following as a start to this project/slight redefinition of the RPC implementation:

POST to /estimators a model type (svm, maxent, nbayes) in order to trigger a user-specific model build (we'll build the global model routinely every night) on their entire corpus. Optionally POST an ID to a corpus as well with specific documents.
GET /estimators to return a list of estimators, including a ?best query flag that returns a listing of the best models for every model type above.
GET /estimators/:id to retrieve model specific information
POST /estimators/:id with a document id or url to return a prediction.
PUT /estimators/:id to update model meta information
DELETE /estimators/:id to delete the model.

Then let's do some benchmarking to see how long it takes to load the model from the database and employ it to estimate. Maybe we can use caching to speed it up?

Ok I agree with that. I originally thought this was going to be your 2 and 3, though I was trying to reconcile what I read in the pycon poster. I do want to expose prediction endpoints for all models if possible and as you mention then we can see if/in what way we hit a performance bottleneck.
Also I haven't used it but I know yhat does something similar, so I may creep on them just a little to see if it is related to what we are proposing here.

Yeh the poster was our initial thoughts; and it was definitely one of the places we struggled - an RPC model is much better for this than an REST model ... But I think that slight definition should get us going with this app anyway and allow us to explore it in more detail.
Would love to know about yhat!

Hey @lauralorenz should this one be moved over into In Progress?


mis-spelled in bib file corrected

@AnjuJoon I'll merge this but please be sure to also change any rst files that use this reference.

Thanks, @jstac I am taking care of that!


FT: putObjectTagging

ARSENAL: https://github.com/scality/Arsenal/pull/248
UTAPI: https://github.com/scality/utapi/pull/127
INTEGRATION: https://github.com/scality/Integration/tree/ft/putobjecttagging
==============
BEFORE MERGING:
remove Arsenal branch dependency in package.json

We are now storing an object "tags" instead of an array of tag's object.
@LaurenSpiegel

you might want to do a quick review (just to be sure)
we can merge as soon as http://ci.ironmann.io/gh/scality/Integration/12750 passes.


Should @types/lodash be a non-dev dependency?

I noticed on Snyk (https://snyk.io) that this project pulls in @types/lodash as a non-dev dependency.
https://github.com/2muchcoffeecom/ngx-restangular/blob/master/projects/ngx-restangular/package.json#L49
This should be moved to the devDependencies section right?

Hi @jaydiablo, Thanks for your question. It shouldn't be moved to devDependecies as in this way it won't be installed when ngx-restangular is installed.


Vulnerabilities found in dbatools.library 2024.4.12

Hey, there! 👋
On one of our security scans, some vulnerabilities were found in the module dbatools.library version 2024.4.12 (latest) regarding the package system.private.uri version 4.3.0. This package was found in both core/lib/sqlpackage.deps.json and core/lib/mac/sqlpackage.deps.json. Here are the vulnerabilities:

High severity: CVE-2019-0981
High severity: CVE-2019-0980
Moderate severity: CVE-2019-0657

Is it possible for these vulnerabilities to be addressed on a future release?
Thank you!

Thank you, we will address this in an upcoming release.

@potatoqualitee I was provided results of a security scan that found a couple others as well.
Alert Details:
The library Microsoft.Extensions.Caching.Memory version 8.0.0 was detected in Dotnet library manager located at /%HOME%/.PowerShellUniversal/Repository/Modules/dbatools.library/2024.4.12/core/lib/sqlpackage.deps.json and is vulnerable to CVE-2024-43483, which exists in versions >= 8.0.0-preview.1.23110.8, <= 8.0.0.
The vulnerability was found in the Github Security Advisory with vendor severity: High (NVD severity: High).
The vulnerability can be remediated by updating the library to version 8.0.1 or higher, using dotnet add package Microsoft.Extensions.Caching.Memory.

Alert Details:
The library Newtonsoft.Json version 12.0.2 was detected in Nuget library manager located at /%HOME%/.PowerShellUniversal/Repository/Modules/dbatools.library/2024.4.12/core/third-party/XESmartTarget/Newtonsoft.Json.dll and is vulnerable to CVE-2024-21907, which exists in versions < 13.0.1.
The vulnerability was found in the Github Security Advisory with vendor severity: High (NVD severity: High).
This vulnerability has a known exploit available. Source: Github.
The vulnerability can be remediated by updating the library to version 13.0.1 or higher, using dotnet add package Newtonsoft.Json.


feat: throw an error on seed command if there are migrations to apply

Fix: #572

@ivan-tymoshenko Great to see these seeding and migrations improvements and bug fixes landing! 🙌🏻


Rclone always finding new files to copy

So I have two gdrives that I am keeping mirrored. I have run it a handful of times now since I stopped uploading to the source gdrive and everytime I re-run the command it finds new files to copy even though there was absolutely no change to the files in the source.
It correctly finds existing files and skips those but finds files that have apparently been missed out before even though the outputs do no indicate any errors (-v)
Some examples of the outputs I am getting.

2017/07/30 18:20:56 DEBUG : 2016/02-14-16 full backup Sony -gopro/GoPro/117GOPRO/GOPR5074.THM: Unchanged skipping
2017/07/30 18:20:58 DEBUG : 2016/01-24-16 keikis aerial und gopro/gopro/GOPR5013.MP4: Sending chunk 67108864 length 8388608
2017/07/30 18:21:00 DEBUG : 2016/01-10-16 West side/aerial/DJI_0031.MOV: Sending chunk 92274688 length 8388608

Some more:

/07/30 18:23:21 DEBUG : 2017/01-29-17 Spitting cave whales and more x5s/x5s/A079_C025_20170129_P1/A079_C025_20170129_P0007.mov: Size and modification time the same (differ by 0s, within tolerance 1ms)
2017/07/30 18:23:21 DEBUG : 2017/01-29-17 Spitting cave whales and more x5s/x5s/A079_C025_20170129_P1/A079_C025_20170129_P0007.mov: Unchanged skipping
2017/07/30 18:23:21 DEBUG : 2017/01-29-17 Spitting cave whales and more x5s/x5s/A079_C025_20170129_P1/A079_C025_20170129_P0003.mov: Size and modification time the same (differ by 0s, within tolerance 1ms)
2017/07/30 18:23:21 DEBUG : 2017/01-29-17 Spitting cave whales and more x5s/x5s/A079_C025_20170129_P1/A079_C025_20170129_P0003.mov: Unchanged skipping
2017/07/30 18:23:22 DEBUG : pacer: Rate limited, sleeping for 16.336848143s (5 consecutive low level retries)
2017/07/30 18:23:22 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2017/07/30 18:23:22 DEBUG : 2016/01-10-16 West side/aerial/DJI_0031.MOV: Sending chunk 134217728 length 8388608
2017/07/30 18:23:22 DEBUG : 2016/03-11-16 waikiki plus firework sony/bilder/DSC05128.ARW: Sending chunk 0 length 8388608
2017/07/30 18:23:22 DEBUG : pacer: Resetting sleep to minimum 10ms on success
2017/07/30 18:23:22 DEBUG : 2017/03-28-17 koko arch makapuu lighthouse x5s/sd/DJI_0174.MP4: Sending chunk 58720256 length 8388608
2017/07/30 18:23:23 DEBUG : 2017/03-26-17 Westside x5s gopro/gopro/GOPR3039.MP4: Sending chunk 25165824 length 8388608
2017/07/30 18:23:23 DEBUG : 2017/03-28-17 koko arch makapuu lighthouse x5s/sd/DJI_0170.MP4: Sending chunk 8388608 length 8388608
2017/07/30 18:23:23 DEBUG : 2017/02-05-17 Spitting caves dan x5s/x5s/A081_C016_20170205_B1/A081_C016_20170205_B00043.dng: Sending chunk 16777216 length 4977420
2017/07/30 18:23:24 DEBUG : 2016/01-24-16 keikis aerial und gopro/gopro/GOPR5013.MP4: Sending chunk 109051904 length 8388608`

I tried lowering the transfer count, checking -vv for any errors and --ignore-existing flag with no change.

What is your rclone version (eg output from rclone -V)
1.37
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Ubuntu, VPS, x86
Which cloud storage system are you using? (eg Google Drive)
Gdrive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone -v --transfers=64 copy "source":/VideoBackup "dest":/VideoBackup

Check the source and destination for duplicates with rclone dedupe - that could be the cause of the problem.

Tried that in --dry-run but there seem to be only about 5 files on each side that are duplicates however there are far more files that rclone copies on a rerun of the copy command.

Can you send me a complete log with -vv please?  With --dry-run is fine.
Also can you do rclone ls2 -R source: > source-list.txt and rclone ls2 -R dest: > dest-list.txt and attach those.
If those have file names you don't want to be public then email them to me at<EMAIL_ADDRESS>otherwise attach them all to the issue.
Thanks

I will send you the logs you requested via email in a bit!

I will send you the logs you requested via email in a bit!

I've received those thanks.  I think I see what the problem might be - duplicate directories.
Can you send me a log of the sync with -vv too please?  With --dry-run is fine.

Here is an experimental version of dedupe which will merge duplicated directories which I think will fix your problem.
https://pub.rclone.org/v1.37-045-g3f98978c-dedupedirβ/
I can see you have duplicated directories here
grep '/$' < source-list.txt | sort  | uniq -c | sort -n | egrep -v '^ +1 '
      2 2016/Canada 2016/Day 4 DJI sunshine valley/
      2 2017/03-26-17 Westside x5s gopro/Neuer Ordner/
      2 2017/03-28-17 koko arch makapuu lighthouse x5s/Neuer Ordner/
      4 2017/Neuer Ordner/

If you want to give it a try, try rclone dedupe -v --dry-run remote:path to start with.  2016/Canada 2016 might be a good directory to start with. When you think the output looks sensible try without the --dry-run.  This will merge the contents of the duplicated directories and trash (not delete) the empty directory at the end.  It is brand new code so may have bugs.

Okay awesome! The folder is basically structured by Year - Day - Camera so the camera folders certainly exist several times however in different subfolders etc. Not sure if that is the issue.
I will run the experimental version in a bit, do you still want the sync -vv log of the copy command?

Just tested the experimental and yes it does indeed find a handful of duplicate files to remove in -dry-run. I will let it dedupe both the source and the dest folder after checking that those files were indeed duplicates. After that I will see if the copy command finds new files to copy again or not, right?

2017/08/03 14:20:06 INFO  : Transferred:      0 Bytes (0 Bytes/s) Errors:                 0 Checks:            115750 Transferred:            0 Elapsed time:    12m21.1s
Rclone doesn´t find any new files to copy so it seems to be all good with the experimental version!

Excellent - thanks for testing :-)
I'll merge that into master shortly

I'm going to close this now as I think the issue has been dealt with.


[Feature Request] Add RAG to Teams Chef Bot

Add Retrieval Augmentation Generation to the Teams Chef bot as done in the JS SDK sample: https://github.com/microsoft/teams-ai/pull/513

related to #571


wrong text content with column inside page

hi, I tested with this pdf file but it's return wrong text. It get the text on the row instead of the column side. Can u help?
document.pdf

to fix your trouble check this solution click
maybe this will solve your problem.


bibtex: don't escape spaces in journals

INSPIR-3416

Note that this requires reindexing (I don't remember where this should be mentioned exactly).


chore(main): release 1.4.3

:robot: I have created a release beep boop
1.4.3 (2024-03-26)
Bug Fixes

icon: Added snark (#18) (f84adfd)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/CoreySpohn/lod_unit/releases/tag/v1.4.3 :sunflower:


switch images to alpine

License Agreement for Contributions
By submitting this pull request, I acknowledge and agree that my contributions will be included in Stirling-PDF and that they can be relicensed in the future under MPL 2.0 (Mozilla Public License Version 2.0) license.
(This does not change the general open-source nature of Stirling-PDF, simply moving from one license to another license)

Wow this is awesome work I could never get alpine images working due to inexperienced docker skills
Have you been able to test these and the functionality of python/libre office etc?

Just ran a quick run of all 3 docker files
Ultra lite seems to be missing java

Cool with that it seems they all load to the homepage

I will do some tests on OCR and conversions just to see if anything else wrong in parallel with your testing etc

From my tests (of the full image) everything works, except one thing; If I want to convert a pdf to word, I see in the docker logs: Error: source file could not be loaded  and it downloads an empty zip file

doing docker compose i can expose the port
but when running in docker desktop i get

Without the Expose 8080
What is the norm for docker files

docker run --rm -it p 8080:8080  works for me on docker desktop

(Meaning i am unable to run it and access 8080)

yeah works via CMD for me as well, just find it odd that docker desktop UI doesnt work with it or let you define ports, so was wondering if this is something non standard

I've never used EXPOSE in a Dockerfile since it does nothing: https://docs.docker.com/engine/reference/builder/#expose

I've tried again, it downloads now the converted document, but it is unuseable (pdf => html:

Something is defo off with the outputs when compared to old version
(You can test here)
https://pdf.adminforge.de/pdf-to-word
pdf to work on your version gives me

vs

seems its just loaded the PDF file bytes into .doc format not actual converting

i see libreoffice-core is not avaible, i tried just 'libreoffice'
and get a slightly better conversion now.. all 3 pages are overlayed onto single page and rotation is not kept.. so something weird..
but clearly something different about the libre package

I've found the error

So right now the conversions are not happening in a way which i am happy with, it seems to throw everything onto single page, not sure why... Until then we wont be able to merge this.
However super appreciate your work, will try see whats up with libreoffice

it seems to be a bug in libreoffice, so it will also affect debian/ubuntu when they will update in the future

If it's the version and not general package can we pull the older version?

v<IP_ADDRESS>-r0 and earlier of libreoffice works, everything after v<IP_ADDRESS>-r0 not, so the bug seems to be added with v7.4 or v7.5

https://packages.ubuntu.com/jammy/libreoffice currently the dokcer image uses v7.3

using alpine:3.17 instead of :latest makes it work for lite and normal, ultra lite can stay at latest, but 3.17 is two versions old...

https://mirror1.hs-esslingen.de/pub/Mirrors/tdf/libreoffice/src/bugs-changelog-tag-libreoffice-<IP_ADDRESS>-release-<IP_ADDRESS>.log
"PDF: Conversion of pdf to docx or doc collapses all content onto one page (tdf#157589) [Kevin Suo]"

https://mirror1.hs-esslingen.de/pub/Mirrors/tdf/libreoffice/src/bugs-changelog-tag-libreoffice-<IP_ADDRESS>-release-<IP_ADDRESS>.log
"PDF: Conversion of pdf to docx or doc collapses all content onto one page (tdf#157589) [Kevin Suo]"

https://gitlab.alpinelinux.org/alpine/aports/-/issues/15628

https://gitlab.alpinelinux.org/alpine/aports/-/issues/15628

using alpine:3.17 instead of :latest makes it work for lite and normal, ultra lite can stay at latest, but 3.17 is two versions old...

Based on this and you using 3.19 is this PR now blocked on resolution of alpine updating its package manager?

using alpine:3.17 instead of :latest makes it work for lite and normal, ultra lite can stay at latest, but 3.17 is two versions old...

Based on this and you using 3.19 is this PR now blocked on resolution of alpine updating its package manager?

yes blocked until they fix libreoffice, but from what I know, they are fast in fixing this

should now work, I will create a PR when a fixed libreoffice verion from outside edge can be used

btw wondering if i can get  your advice,
I am trying to get calibre installed on alpine
I get
apk add --no-cache calibre --repository=http://dl-cdn.alpinelinux.org/alpine/edge/testing
fetch http://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
ERROR: unable to select packages:
  py3-pyqt6-webengine (no such package):
    required by: calibre-7.3.0-r0[py3-pyqt6-webengine]
  qt6-qtwebengine (no such package):
    required by: calibre-7.3.0-r0[qt6-qtwebengine]
  so:libicui18n.so.74 (no such package):
    required by: calibre-7.3.0-r0[so:libicui18n.so.74]
  so:libicuuc.so.74 (no such package):
    required by: calibre-7.3.0-r0[so:libicuuc.so.74]
  so:libpodofo.so.2 (no such package):
    required by: calibre-7.3.0-r0[so:libpodofo.so.2]

As a solution i added
http://dl-cdn.alpinelinux.org/alpine/edge/main
http://dl-cdn.alpinelinux.org/alpine/edge/community

to
/etc/apk/repositories
is this okay?

I would do this:
/etc/apk/repositories:
https://dl-cdn.alpinelinux.org/alpine/v3.19/main
https://dl-cdn.alpinelinux.org/alpine/v3.19/community

@testing https://dl-cdn.alpinelinux.org/alpine/edge/main
@testing https://dl-cdn.alpinelinux.org/alpine/edge/community
@testing https://dl-cdn.alpinelinux.org/alpine/edge/testing

apk add --no-cache calibre@testing

can I ask when this PR will be merged?-

can I ask when this PR will be merged?-

Once i figure out calibre and wkhtmltopdf
I added them as custom installs which users can do post docker image install
https://github.com/Stirling-Tools/Stirling-PDF/pull/682/files#diff-8e6e46e251822ef45509ec061b133c5788a5719a0670ac3158b49dbb58e3dab6R69

should I add them to this PR? calibre and wkhtmltopdf?

If you can make it conditional in docker file via some arg in the init scripts or changed to match in the java file i linked then sure!

should I add them to this PR? calibre and wkhtmltopdf?

If you can make it conditional in docker file via some arg in the init scripts or changed to match in the java file i linked then sure!

waht do you mean?

what does wkhtmltopdf do? Is it not the same as WeasyPrint? And I thought wkhtmltopdf is deprecated

Sorry missed comments
WeasyPrint does a pretty bad job, i have had requests to use wkhtmltopdf as it produces vastly better pdf files, i have it as an optional download due to its high security issues
so its not recommended other than for local users.. i guess a better solution would be chromium or puppeteer though....?
"waht do you mean? should I add calibre to the full image?"
So for this i mean that they are optional based on user parameters, i woudlnt want them as part of image but downloaded as a optional step post image via init script or the app itself

wkhtmltopdf is removed in newer alpine versions... - but I could add calibre to be installed if an env is set

wkhtmltopdf is removed in newer alpine versions... - but I calibre can be added by running apk add --no-cache calibre@testing
I've looked into the https://github.com/Stirling-Tools/Stirling-PDF/blob/main/src/main/java/stirling/software/SPDF/config/PostStartupProcesses.java - I think it would need to be changed completely to work with alpine - and I want to ask why you set the timezone in it?

I had issues with calibre install requiring user prompts without it

what do you mean? maybe try adding --no-interactive

what do you mean? maybe try adding --no-interactive

that didnt work for me, at least via ubuntu apt

I talk about apk

Okay lets remove wkhtmltopdf  completly thats fine, ill merge your changes today or tomorrow and remove wkhtmltopdf as well as add
apk add --no-cache calibre@testing

was just rebuilding the dockerbaseFile (I think we can remove this base file now doesnt serve much purpose)
I got
2.066 fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
2.242 ERROR: unable to select packages:
2.245   so:libpoppler.so.133 (no such package):
2.245     required by: libreoffice-draw-<IP_ADDRESS>-r0[so:libpoppler.so.133]

please retry

https://github.com/Stirling-Tools/Stirling-PDF/actions/runs/7950256132/job/21702556824
gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DFFI_BUILDING=1 -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python3.11 -c src/c/_cffi_backend.c -o build/temp.linux-aarch64-cpython-311/src/c/_cffi_backend.o
#22 98.48       error: command 'gcc' failed: No such file or directory
#22 98.48       [end of output]
#22 98.48
#22 98.48   note: This error originates from a subprocess, and is likely not a problem with pip.
#22 98.48   ERROR: Failed building wheel for cffi
#22 98.48 Failed to build cffi
#22 98.48 ERROR: Could not build wheels for cffi, which is required to install pyproject.toml-based projects
#22 ERROR: process "/bin/sh -c echo "@testing [https://dl-cdn.alpinelinux.org/alpine/edge/main](https://dl-cdn.alpinelinux.org/alpine/edge/main/)" | tee -a /etc/apk/repositories &&     echo "@testing [https://dl-cdn.alpinelinux.org/alpine/edge/community](https://dl-cdn.alpinelinux.org/alpine/edge/community/)" | tee -a /etc/apk/repositories &&     echo "@testing [https://dl-cdn.alpinelinux.org/alpine/edge/testing](https://dl-cdn.alpinelinux.org/alpine/edge/testing/)" | tee -a /etc/apk/repositories &&     apk add --no-cache         ca-certificates         tzdata         tini         bash         curl         openjdk17-jre    libreoffice@testing         python3 &&         wget https://bootstrap.pypa.io/get-pip.py -qO - | python3 - --break-system-packages --no-cache-dir --upgrade &&     pip install --break-system-packages --no-cache-dir --upgrade unoconv WeasyPrint &&     mkdir -p /configs /logs /customFiles /pipeline/watchedFolders /pipeline/finishedFolders &&     fc-cache -f -v &&     chmod +x /scripts/*.sh" did not complete successfully: exit code: 1
Any idea? works for other images

just not lite

try to add build-base at the beginning using apk and remove it at the end using apk del --no-cache build-base
and please do everything inside one RUN step to make the image smaller

@Zoey2936 for running as non root user how would be best with volume mounting etc?
I got stirling PDF running as none root user fine by using gosu/su-exec  to switch from root to stirlingpdf user via entrypoint script
But cant get it working by doing standalone user
My concern is that i volume mount folders such as /logs which the app (started by stirling user) doesnt have permission to write to as since its volume mounted it users the hosts permissions and often uses root.
I can fix this by changing the permissions on host side but that would mess everyones installs and i wouldnt want to force people to change permissions on host level for every install
So is running root then switch to new user the only way? (since i can use root user to set volume mounted folders to stirling user after startup)
Basically
docker file
ENV PUID=1000 \
    PGID=1000 \
    UMASK=022
RUN apk add --no-cache su-exec && \
    addgroup -g $PGID stirlingpdfgroup && \
    adduser -u $PUID -G stirlingpdfgroup -s /bin/sh -D stirlingpdfuser && \
    mkdir -p $HOME && chown stirlingpdfuser:stirlingpdfgroup $HOME
ENTRYPOINT ..... etc

entrypoint.sh
echo "Setting permissions and ownership for necessary directories..."
chown -R stirlingpdfuser:stirlingpdfgroup /logs /scripts /usr/share/fonts/opentype/noto /usr/share/tessdata /configs /customFiles
chmod -R 755 /logs /scripts /usr/share/fonts/opentype/noto /usr/share/tessdata /configs /customFiles

# Run the main command and switch to stirling user for rest of run
exec su-exec stirlingpdfuser "$@"

@Zoey2936  Seems 3.19.1 has the libreoffice bug again regardless of repository used,
If i move docker to tag 20240329 it fixes this but now this one has a issue with a python install as part of libreoffice..
0.648                py3-cffi-pyc-1.16.0-r1[python3~3.12]
0.648                py3-lxml-5.1.0-r0[python3~3.12]
0.648                py3-charset-normalizer-pyc-3.3.2-r1[python3]
0.648                py3-charset-normalizer-pyc-3.3.2-r1[python3~3.12]
0.649                py3-pygments-pyc-2.17.2-r1[python3]
0.649                py3-pygments-pyc-2.17.2-r1[python3~3.12]
0.650   so:libpython3.11.so.1.0 (no such package):
0.650     required by: libreoffice-common-<IP_ADDRESS>-r2[so:libpython3.11.so.1.0]
0.650   so:libmbedcrypto.so.7 (no such package):
0.650     required by: librist-0.2.10-r0[so:libmbedcrypto.so.7]

Any ideas?

just wait... they updated to python 3.12 and their builder currently rebuilds all packages against python 3.12

Yeah first part seems fixed now thanks for bringing that to attention
And now libreoffice isnt including soffice and other executables :(

What di you mean?

So it doesn't work?

I don't see any breaking chages in the last weeks https://gitlab.alpinelinux.org/alpine/aports/-/commits/master/community/libreoffice?ref_type=heads

How odd..
basically what i see is that after adding
openssl \
openssl-dev \
    	```

To the APK installs in current docker file as it seems to require that, when I check ``usr/lib/libreoffice/program/``
I would normally see soffice and several others and these are suddenly missing

https://pkgs.alpinelinux.org/contents?file=soffice&path=&name=&branch=edge but they exist

Then i don't understand why it doesn't show :(

Correct and the file itself isn't in the final docker build for some reason

there is also an issue opened: https://gitlab.alpinelinux.org/alpine/aports/-/issues/16005