JsonSyntaxException when creating API command response

ISSUE TYPE

Bug Report

COMPONENT NAME

API

CLOUDSTACK VERSION

master/4.15

CONFIGURATION

N/A

OS / ENVIRONMENT

N/A

SUMMARY

Issues when API command response is generated. Exception at the log and API command result. The command works fine, but to whom using the API it seems to have failed.
STEPS TO REPRODUCE

Call an API command, e.g. prepareHostMaintenance or cancelHostMaintenance

EXPECTED RESULTS

Proper API response and no log exceptions on the management node

ACTUAL RESULTS

When calling API command via CloudMonkey:

> prepare hostformaintenance id=host-id
Error: async API failed for job b43bbdbf-fe65-436c-9e7e-7c48e4fbfea1

log at the management node:

2020-09-29 16:55:15,853 DEBUG [c.c.a.t.Request] (API-Job-Executor-4:ctx-4d783c5f job-55326 ctx-7d98f674) (logid:b43bbdbf) Seq 24-3700551518814994444: Received:  { Ans: , MgmtId:<PHONE_NUMBER>957, via: 24(n06.2019.cldin.net), Ver: v1, Flags: 110, { MaintainAnswer } }
2020-09-29 16:55:15,874 DEBUG [c.c.r.ResourceState] (API-Job-Executor-4:ctx-4d783c5f job-55326 ctx-7d98f674) (logid:b43bbdbf) Resource state update: [id = 24; name = n06.2019.cldin.net; old state = Enabled; event = AdminAskMaintenance; new state = PrepareForMaintenance]
2020-09-29 16:55:15,917 DEBUG [c.c.r.ResourceManagerImpl] (API-Job-Executor-4:ctx-4d783c5f job-55326 ctx-7d98f674) (logid:b43bbdbf) Sent resource event EVENT_PREPARE_MAINTENANCE_AFTER to listener CapacityManagerImpl
2020-09-29 16:55:15,950 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-4:ctx-4d783c5f job-55326) (logid:b43bbdbf) Unexpected exception while executing org.apache.cloudstack.api.command.admin.host.PrepareForMaintenanceCmd
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unterminated object near assword":"c*****},"resourcestate":"Prepa
        at com.google.gson.Streams.parse(Streams.java:51)
        at com.google.gson.JsonParser.parse(JsonParser.java:83)
        at com.google.gson.JsonParser.parse(JsonParser.java:58)
        at com.google.gson.JsonParser.parse(JsonParser.java:44)
        at com.cloud.utils.HumanReadableJson.addElement(HumanReadableJson.java:52)
        at com.cloud.utils.HumanReadableJson.getHumanReadableBytesJson(HumanReadableJson.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.convertHumanReadableJson(AsyncJobManagerImpl.java:352)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.completeAsyncJob(AsyncJobManagerImpl.java:262)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:111)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:615)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:563)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.google.gson.stream.MalformedJsonException: Unterminated object near assword":"c*****},"resourcestate":"Prepa
        at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1111)
        at com.google.gson.stream.JsonReader.nextInObject(JsonReader.java:737)
        at com.google.gson.stream.JsonReader.quickPeek(JsonReader.java:379)
        ... 24 more
2020-09-29 16:55:15,955 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-4:ctx-4d783c5f job-55326) (logid:b43bbdbf) Complete async job-55326, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"com.google.gson.stream.MalformedJsonException: Unterminated object near assword":"c*****},"resourcestate":"Prepa"}

Note that the host got in the Maintenance state as expected and the issue is related only to the response.

@GabrielBrascher are you working on a fix? or can we triage that to someone else? thanks

@rhtyd just started to look at this. Do you have any idea of what is causing it? Have anyone seen this as well?

cc @borisstoyanov @andrijapanicsb @vladimirpetrov @DaanHoogland @shwstppr @harikrishna-patnala @nvazquez

@rhtyd @GabrielBrascher it looks like the double quote (") at the end of the password value is missing. probably a bug in the password filtering code. (just a quick assessment, sorry if i just kicked in an open door)

@rhtyd I discovered what caused this issue and how to reproduce.
It happens only with Hosts that have an OutOfBand driver (e.g. IPMI or Redfish) enabled.
The host password is passed on the response as first letter +  "****". For instance, a password "password" then it is set as "p****" on the response object. More details at OutOfBandManagementResponse.java#L98
Code:
this.setPassword(outOfBandManagementConfig.getPassword().substring(0, 1) + "****");
However, the String that represents the Host response JSON object is being processed as:
..."outofbandmanagement":{"powerstate":"On","enabled":true,"driver":"redfish","address":"oob-kvm-node.address.net","port":"80","username":"root","password":"p*****}....

Probably something related to special character handling that resulted on the lack of " on the String that is processed into a JSON object. I will be testing a few options to fix it.
Keep you updated.

@DaanHoogland you are right. I have a hunch of the cause but I still need to debug it.
For some odd reason, GitHub sometimes does not update issues/PR messages to me, after I answered and refreshed the page then your answer appeared to me.

Connected via remote debug on the JVM I was able to check the workflow. The password for Out of band response is obfuscated at OutOfBandManagementResponse.java#L98; then, the workflow will make that password as input to  obfuscatePassword .
constructor at OutOfBandManagementResponse.java#L98
https://github.com/apache/cloudstack/blob/1d05fead49f5c856257a741b07122f5633d2e359/api/src/main/java/org/apache/cloudstack/api/response/OutOfBandManagementResponse.java#L98
obfuscatePassword method at AsyncJobManagerImpl.java#L478
https://github.com/apache/cloudstack/blob/0f3f2a09370a18301db28ec3d28efe746b6437c9/framework/jobs/src/main/java/org/apache/cloudstack/framework/jobs/impl/AsyncJobManagerImpl.java#L478
The obfuscatePassword method removes " and causes the JSON issue.
@DaanHoogland I am curious, why do we log password? Even knowing that it is obfuscated I am not sure of why we shuld log it.
With that said, I see two solutions for this bug:

fix obfuscatePassword, o
remove password from Host responses and log messages.

:D or three: return '"password": ""' for any input :)
No real pref but one consideration: obfuscatePassword may be used in other places and all other results of the bug are not known yet. I think option 1 is your safest bet. maybe in combination with three 👯

@DaanHoogland @rhtyd I've created PR #4387. The PR has 2 commits, the first contains the fix; the second commit deletes commented lines on the test class and adds JUnit tests for obfuscatePassword.


getSVGMatrix not exposed

Previously you had stated this function could be imported as such
import SVGPathCommander from 'svg-path-commander';
const { getSVGMatrix } = SVGPathCommander;

but that didn't work and, looking at the code, it appears as though you have not exported it in any way.

That is correct, I didn't think people would actually need that.

v 2.0.6 will have the method exported to public static.


Release version 4 now?

Given that all pending https://github.com/mozilla/pdf.js/labels/release-blocker PRs have landed, should we perhaps attempt to release version 4 now?
The new version, being a major version bump, already contains a bunch of breaking changes as-is; some notable examples include outputting of JavaScript modules and removal of the SVG back-end.
(In this case, given the amount and kind of changes since the last release we might need to be prepared to do a follow-up release if any "serious" problem is found.)
/cc @timvandermeij

I agree; I was thinking the same yesterday. We have enough pending changes to do it now, and those changes had some time to settle now, so I think we can do it now and if need be take care of follow-ups (which will be easier now than if we stack more on top of the existing changes). I'll take care of the release later this afternoon because I'll need some time to compose the release highlights given that it's a major version release.

Closing since the release is done now.


Uploads Duplicated with Different Tags

I'm noticing that every once in a while, a user will send data twice for the same day, and the entire upload will be identical except for the tag. I'm not sure what could be causing this. The fact  that the total visits and ciphertexts are identical leads me to believe that the prepareAndSendDataBody function is only being called once, but this makes the differing tags all the more confusing.
The good news is that these duplicates are easy to deduplicate. However, it makes me think there might be something fundamentally going wrong every once in a while, and if it affects tags, it could affect payments.

This issue seems to be getting more frequent.


how to localize text on ios?

Hello guys,
when I click camera.getPicture() on ios I can see this action sheet. How I can change or localize this texts?

You can add languages with edit-config and config-file tags like
<config-file parent="CFBundleLocalizations" target="*-Info.plist">
  <array>
    <string>en</string>
    <string>es</string>
  </array>
</config-file>


HitEvent

Hi
Could it be possible to add an HitEvent to magic?

There is a CastEvent:
https://github.com/elBukkit/MagicPlugin/blob/master/MagicAPI/src/main/java/com/elmakers/mine/bukkit/api/event/CastEvent.java#L43
You can call getSpellResult, if the result is CAST it was a hit.

How can I get the target?

@NathanWolf

Hm- try getSpell().getCurrentCast().getTargetedEntities() (there may be more than one target, such as for AOE spells)


DUPLO-21234: [Develop]:TF:Airflow: Getting error for resource 'duplocloud_aws_mwaa_environment' on destroyfix tf regression, terraform assume delete message is null

Overview
DUPLO-21234: fix tf regression, terraform assume delete message is null
Summary of changes
This PR does the following:

parse delete response
...

Testing performed

[ ] Using unit tests
[X] Manually, on my local system
[X] Manually, on a remote test system

Describe any breaking changes

None

Task linked: DUPLO-21234 [Develop]:TF:Airflow: Getting error for resource 'duplocloud_aws_mwaa_environment' on destroy


MRG: Refactor field mapping LUT to use interp1d

We had implemented nearest-neighbor and linear interpolation ourselves in the field interpolation code. This uses the more-standard scipy.interpolate.interp1d instead. In theory we could get some speedups, but testing shows it to be about the same. In any case, should help keep things more maintainable.

ok so now fast and accurate are as fast?

At that stage, yeah. Sometimes fast and accurate are meant to describe how many coil integration points are used. Here it's whether linear or nearest interpolation are used. I think SciPy does that interpolation in compiled code, so it seems to be a negligible difference in the context of our calculations.

ok then we should just support the linear interp right? and remove the
nearest.

Okay with me. I'll change it now

Okay, now the difference between fast and accurate is just the number of coefficients used (50 vs 100). Should we get rid of mode altogether? That's another option.

... it would create deprecations in inst.interpolate_bads, evoked.as_type, and mne.make_field_maps

how much does it impact the results?
is there any memory usage change?

Okay, now the difference between fast and accurate is just the number of coefficients used (50 vs 100).

Actually it turns out linear is slower than nearest by a factor of ~2. 50 vs 100 coefficients is also a factor of 2. So we might as well have them both.
I don't see a difference by eye between fast and accurate in examples/preprocessing/plot_interpolate_bad_channels.py, but who knows.

...reverted so that it has the same behavior as before, 50/100 coefficients and nearest/linear interp for accurate/fast respectively. Now it's back to just changing the interpolation backend.

Codecov Report

Merging #3989 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3989      +/-   ##
==========================================
+ Coverage   85.98%   85.98%   +<.01%
==========================================
  Files         349      349
  Lines       62617    62591      -26
  Branches     9582     9579       -3
==========================================
- Hits        53840    53819      -21
+ Misses       6090     6087       -3
+ Partials     2687     2685       -2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0c31b1e...2c385a1. Read the comment docs.

merge when CIs are happy

Thanks for the reviews @agramfort @jaeilepp


Call to 'toArray()' with pre-sized array argument 'new String[map.key…

…Set().size()]'
Inspection info: There are two styles to convert a collection to an array: either using a pre-sized array (like c.toArray(new String[c.size()])) or using an empty array (like c.toArray(new String[0]).
In older Java versions using pre-sized array was recommended, as the reflection call which is necessary to create an array of proper size was quite slow. However since late updates of OpenJDK 6 this call was intrinsified, making the performance of the empty array version the same and sometimes even better, compared to the pre-sized version. Also passing pre-sized array is dangerous for a concurrent or synchronized collection as a data race is possible between the size and toArray call which may result in extra nulls at the end of the array, if the collection was concurrently shrunk during the operation.
This inspection allows to follow the uniform style: either using an empty array (which is recommended in modern Java) or using a pre-sized array (which might be faster in older Java versions or non-HotSpot based JVMs).

Coverage decreased (-0.009%) to 87.232% when pulling 56cc0e467f7463a5ad4cb00641c914731aa53be7 on DuoLaAMengDianHuanXiang:master into 7cb8acea16378632ca51ee9c0f0790a3742382dc on mybatis:master.

Thank you, @DuoLaAMengDianHuanXiang !

@harawata my honor


OCR Language can not be changed in settings

Option "dropdown" multiselect for OCR language to be performed is missing

Included with one of the latest changes


Add examples for all the supported tasks for huggingface runtime

"Fixes #issue-number" or "Add description of the problem this PR solves"
Proposed Changes

/lgtm
/approve


GH-Pages deploys are happening in forks

Is there a way to make gh-pages deploy only happen in the Hyperledger org repository?  I don't think they should be happening in (for example) my fork of the repo.
The deploy is failing in my local repo because of issue #4, but I don't think it should be even trying to be deployed.

I think we can restrict the GHA to a specific repo

I probably shouldn't have assigned you that -- others can figure that out.  The first one though -- #4 needs to come from you.


Timer events automation doesn't filter out timer events by specified timer entity id

Home Assistant release with the issue:

v0.102.3
Last working Home Assistant release (if known):
Operating environment (Hass.io/Docker/Windows/etc.):

Hass.io
Integration:

Description of problem:
As you see in the screenshots below event handler of "timer.vacuum_timer_to_start_on_return" timer executes automation action even though entity id is not matched (event was fired by  "timer.heating_bathroom_timer" timer)

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

Traceback (if applicable):

Additional information:

Temporary workaround by condition to fix the bug:

Same user error as described in #29796. I.e., you need to change entity in your event trigger to entity_id.


NavLink doesn't get active on hash links

Version
16.2
Steps to reproduce
<NavLink>
    to="/foo/#bar"
</NavLink>

Expected Behavior
Being on path"/foo/#bar" the NavLink should be active.
Actual Behavior
The NavLink isn't active.
Workaround
const to="/foo/#bar"
<NavLink>
    to={to}
    isActive={()=>{
        return window.location.pathname + window.location.hash === to;
    }}
</NavLink>

We don't support hash links. This is because of HashRouter and the need to support a hash-based history.

Ok, I understand this, but how come /foo#bar is working?

It's the trailing slash. We parse out /foo/#bar to {pathname: '/foo/', hash: 'bar'} and strip the trailing slash for the matching to provide normalization. It's an implementation quirk.


[logging-test]  logging-test elog exception

Feb 22 06:07:37 witherspoon logging-test[952]: This is a simple test error.
Feb 22 06:07:37 witherspoon logging-test[952]: Error in mapper call

02:04:36.783     INFO     Executing command 'logging-test -c AutoTestSimple'.
02:04:37.836     INFO     Command exited with return code 0.
02:04:37.837     INFO     ${stdout} = elog exception caught: example.xyz.openbmc_project.Example.Elog.AutoTestSimple
02:04:37.837     INFO     ${stderr}

logging-test itself doesn't call the mapper - that happens here https://github.com/openbmc/phosphor-logging/blob/master/elog.cpp
In the last few days, the mapper dbus path was changed from /xyz/openbmc_project/ObjectMapper to /xyz/openbmc_project/object_mapper. Are you sure you're running this on an updated system? If not, that would break your dbus calls.

Its the latest code with the object mapper with the lower case
Executing: GET  /xyz/openbmc_project/

/xyz/openbmc_project/
[
    "/xyz/openbmc_project/sensors",
    "/xyz/openbmc_project/software",
    "/xyz/openbmc_project/object_mapper",
    "/xyz/openbmc_project/logging"
]
-bash-4.1$

- [x] BUILD_ID   : https://openpower.xyz/job/openbmc-build/649/
- [x] VERSION_ID : v1.99.3-128-ga0040f7

I ran logging-test -c AutoTestSimple and watched it with dbus-monitor, resulting in
signal time=1487751939.183829 sender=org.freedesktop.DBus -> destination=:1.173 serial=2 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameAcquired
string ":1.173"
method call time=1487751939.189542 sender=:1.173 -> destination=xyz.openbmc_project.ObjectMapper serial=2 path=/xyz/openbmc_project/ObjectMapper; interface=xyz.openbmc_project.ObjectMapper; member=GetObject
string "/xyz/openbmc_project/logging/internal/manager"
array [
string "xyz.openbmc_project.Logging.Internal.Manager"
]
error time=1487751939.209330 sender=:1.19 -> destination=:1.173 error_name=org.freedesktop.DBus.Error.UnknownMethod reply_serial=2
string "Method "GetObject" with signature "sas" on interface "xyz.openbmc_project.ObjectMapper" doesn't exist
"
Note that path=/xyz/openbmc_project/ObjectMapper. You must not have the latest phosphor-logging.

It's also worth noting that that elog exception caught: example.xyz.openbmc_project.Example.Elog.AutoTestSimple is, in fact, intended behavior - the point of logging-test is to create that exception and then commit it to the error log. Looks like everything is working as intended.

    "/xyz/openbmc_project/logging/entry/3": {
        "AdditionalData": [
            "STRING=FOO"
        ],
        "Id": 3,
        "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
        "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
        "Timestamp":<PHONE_NUMBER>901,
        "associations": []
    },
    "/xyz/openbmc_project/logging/entry/4": {
        "AdditionalData": [
            "STRING=FOO"
        ],
        "Id": 4,
        "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
        "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
        "Timestamp":<PHONE_NUMBER>155,
        "associations": []
    },
    "/xyz/openbmc_project/logging/internal/manager": {}
}

Closing issue..


Otel / Open Telemetry for logs, metrics, traces and profiling

couldn't find an issue regarding Otel support in Pinots repository,
https://github.com/apache/pinot/issues?q=is%3Aissue+otel
-> so I just wanted to give a starting point and some input for discussion
With OpenTelemetry announces support for profiling with bidirectional linking to logs, metrics and traces
https://www.cncf.io/blog/2024/03/19/opentelemetry-announces-support-for-profiling/
and it is on the way to be suitable for low latency systems
which progress is tracked in the parent issues within Apache Pulsar (which is the motivating usecase)
https://github.com/apache/pulsar/issues/21121
and since it's somehow becoming the go to standard with an ever-growing number of integration
see https://opentelemetry.io/ecosystem/integrations/
=> it may be also very interesting for Pinot...

maybe also interesting:
Elastic donated their eBPF-based continuous profiling agent to OTel
"We're introducing native correlation between continuous profiling signals and OTel traces, starting with Java."
this is the background (and ad) article on this, to show the usecase:
Pinpointing performance culprits with continuous profiling and distributed tracing correlation
https://www.elastic.co/de/blog/continuous-profiling-distributed-tracing-correlation

cc @zhtaoxiang @xiangfu0


[Automation] Generate Fluent Lite from Swagger developerhub#package-preview-2022-10

[Automation] Generate Fluent Lite from Swagger developerhub#package-preview-2022-10

API change check
APIView has identified API level changes in this PR and created following API reviews.
com.azure.resourcemanager:azure-resourcemanager-devhub


Log full of errors when remote sleeps

Describe the bug
I was looking at my Home Assistant logs and noticed it was full of errors like below (2584 in about 4 days). The IP in the errors are from the UC remote, so I am guessing it is coming from this integration when the remote goes to sleep.
2024-03-20 21:15:54.008 INFO (MainThread) [websockets.client] ! connect failed again; retrying in 60 seconds
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/websockets/legacy/client.py", line 598, in __aiter__
    async with self as protocol:
  File "/usr/local/lib/python3.12/site-packages/websockets/legacy/client.py", line 629, in __aenter__
    return await self
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/websockets/legacy/client.py", line 647, in __await_impl_timeout__
    return await self.__await_impl__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/websockets/legacy/client.py", line 651, in __await_impl__
    _transport, _protocol = await self._create_connection()
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1120, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1102, in create_connection
    sock = await self._connect_sock(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1005, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.12/asyncio/selector_events.py", line 641, in sock_connect
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/selector_events.py", line 681, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
OSError: [Errno 113] Connect call failed ('<IP_ADDRESS>', 80)

To Reproduce
Steps to reproduce the behavior:

Put remote to sleep
Wait a while
Look at Home Assistant logs

Expected behavior
I understand an error when connection gets lost. Most integrations only give an error when it gets lost the first time and then not until connection gets restored.
Desktop (please complete the following information):

OS: Windows
Browser Chrome
Version 122.0.6261.112

Smartphone (please complete the following information):

Device: n.a.
OS: n.a.
Browser n.a.
Version n.a.

Additional context
Add any other context about the problem here.
Home Assistant OS: 12.1
Home Assistant Core: 2024.3.1
Integration version: v0.6.2

Yeah, that is the complete error. Which indeed is weird, so it is still only a guess it is this integration only based on the IP address in the message.
Ah, good point about the loglevel. I changed that to Info a long long time ago and totally forgot about that it was not at default anymore. That makes it easy to resolve.
So not really a problem then I guess.

Since the problem is on my side because I enabled the additional logging I am closing it


MINOR: Fix broken output layout of kafka-consumer-groups.sh

Output format of kafka-consumer-groups.sh with --delete-offsets or --reset-offsets has been broken since line separators are missing.
This PR will fix the problem.
--
Current:
$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete-offsets --topic topic1 --group TestGroup1
Request succeed for deleting offsets with topic topic1 group TestGroup1

TOPIC                          PARTITION       STATUS         topic1                         0               Successful     topic1                         1               Successful     topic1                         2               Successful     topic1                         3               Successful     topic1                         4               Successful     topic1                         5               Successful     topic1                         6               Successful     topic1                         7               Successful     topic1                         8               Successful     topic1                         9               Successful

Expect:
$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete-offsets --topic topic1 --group TestGroup1
Request succeed for deleting offsets with topic topic1 group TestGroup1

TOPIC                          PARTITION       STATUS
topic1                         0               Successful
topic1                         1               Successful
topic1                         2               Successful
topic1                         3               Successful
topic1                         4               Successful
topic1                         5               Successful
topic1                         6               Successful
topic1                         7               Successful
topic1                         8               Successful
topic1                         9               Successful

--
Current:
$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --from-file /tmp/offsets.csv --group TestGroup1 --execute

GROUP                          TOPIC                          PARTITION  NEW-OFFSET     TestGroup1                     topic1                         1          20             TestGroup1                     topic1                         0          10             TestGroup1                     topic1                         3          40             TestGroup1                     topic1                         2          30             TestGroup1                     topic1                         5          60             TestGroup1                     topic1                         4          50

Expect:
$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --from-file /tmp/offsets.csv --group TestGroup1 --execute

GROUP                          TOPIC                          PARTITION  NEW-OFFSET
TestGroup1                     topic1                         1          20
TestGroup1                     topic1                         0          10
TestGroup1                     topic1                         3          40
TestGroup1                     topic1                         2          30
TestGroup1                     topic1                         5          60
TestGroup1                     topic1                         4          50

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Hey @sasakitoa, could you please fix the spotbugs failures? you can re-run it with ./gradlew :tools:spotbugsMain to see the details. Thanks!

Thank you for reviewing and advising.
I fixed it and passed SpotBugs test locally.

Thanks for the fix @sasakitoa! (All builds completed with unrelated failures)


Project is listed as "HTML" project?

A minor thing, but the project is listed as "HTML" project.
Maybe you want to change this to a more sensible category, e.g. Python? :P

Are you referring to wengong-jin/chemprop or swansonk14/chemprop? I'm pretty sure wengong-jin/chemprop is listed as Python. Unfortunately it seems like swansonk14/chemprop is listed as HTML because we're using a chemical drawing library for our web interface which has a lot of HTML code. Do you know if there's any way to override github and tell it to list it as a Python project instead of an HTML project?


How to make v12 package for the browser smaller?

I need to upload to blob storage from client JS. Following the instructions here:
https://github.com/Azure/azure-sdk-for-js/blob/master/documentation/Bundling.md
I end up with a file that is about 1.6MB after following the webpack instructions here: https://github.com/Azure/azure-sdk-for-js/blob/master/documentation/Bundling.md. This makes me think I'm doing something wrong. I'm putting this in my index.js file for webpack as the docs suggest:
const { BlobServiceClient } = require("@azure/storage-blob");
But I end up with a distribution file that doesn't feel right. It's too big.
In contrast the minified version of the v10 client file is about 200KB. And the file to include in my build is clearly referenced in what I pull from npm.  It's right here<EMAIL_ADDRESS>That seems more conventional for a client side library: just give me the js or files that I can choose to integrate into my build process however I see fit.
It seems like v12 wants me to think about which modules to include. I'm happy to do that (I guess) but its not clear what modules I might need... Some subset of BlobServiceClient that would give me less code to distribute perhaps?

Thanks for reaching us.
For the bundling size, are there other project files involved into the bundling, or 1.6MB is purely for v12 @azure/storage-blob?
And would you have a try on RollUp, which is previoulsy used by v10 SDK for bundling.
Thanks,
Jiachen

@steveellis The bundling guide has webpack --mode=development (obviously for dev environment) so the output is not minified.  I just tried running webpack without the mode argument and got an result of ~350 KiB,  which is still a bit large compare to v10.
We had some ideas for improvement and will see how much we can shave off.

Thanks @jeremymeng  and @jiacfan. 350KB is fine for my purposes.
I do think including a pre-build in your v12 npm distribution (like v10) would make for a better experience for people integrating the browser who aren't using one of the bundlers you recommend. I'm using grunt for example.
Another feedback item for you: it's not in the docs (unless I missed it) what the main entry point is to use the library in the browser. In v10 it's azblob. I'd suggest perhaps adding that somewhere.

Thanks for your feedback!

I do think including a pre-build in your v12 npm distribution (like v10) would make for a better experience for people integrating the browser who aren't using one of the bundlers you recommend. I'm using grunt for example.

We looked at doing this before, though I think we were reluctant about bundling in ecosystem dependencies that would then be pinned to specific (and potentially insecure) versions.
There was also the consideration that bundlers would do a better job at producing a smaller bundle based on what a developer decides to import, though I admit that it's very easy at the moment to pull in many things from storage-blob since many clients have references to each other.
Ease of use and not being too heavy of a dependency are both very important to us, so we will continue to work on improving both!

Another feedback item for you: it's not in the docs (unless I missed it) what the main entry point is to use the library in the browser. In v10 it's azblob. I'd suggest perhaps adding that somewhere.

I believe this is dependent upon how one chooses to bundle it. It's entirely possible via webpack to not expose the package on the global object.


Tag folders and files like Mac

Description of the new feature / enhancement
Like any user can do with MacOS, it would be great to have a visual tagging system, where you can add a customized label; and then use it as a filter to show just the tagged folder or file. These tags should appear in the left nav bar. As an example, "Files" app let it do it.
Scenario when this would be used?
Whenever the user wants tp organise his/her files.
Supporting information
No response

/dup #150


Format tests

The Python source code is already formatted, this will re-format the tests and add the Github Action check.

Codecov Report

Merging #47 (4a14858) into main (4b179af) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #47   +/-   ##
=======================================
  Coverage   85.60%   85.60%
=======================================
  Files          10       10
  Lines        1778     1778
=======================================
  Hits         1522     1522
  Misses        256      256

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b179af...4a14858. Read the comment docs.


fix(remote): endpointReader deserialziation error handling #728

Description

Simple error handling of deserialization errors, with logging using configurable level.
Purpose
This pull request is a:

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation (if appropriate)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

👍🏻 Very good


Port to Newer Kernel? 3.18 and/or 4.4

Hey,
I watched your presentation at shmoocon on youtube. I found it quite interesting and was hoping to have a stab at trying out BinderFilter. Unfortunately, it looks like your code only works on very old kernels... Is there any change this code might be ported to msm-3.18 or higher??
I'm hoping to try it out on my Pixel... I'm more than capable of building AOSP, my kernel, android apps, etc... I was able to manually apply your changes and fix up a few errors/warning, But the changes needed in your code for new kernels/binder are a little beyond me...
thanks.

Hi 979,
I can look into porting the module code to a higher kernel, but it may take a week or so. I can also try to debug the kernel issues you're having if you send me specific errors.
Thanks,
David

Hey David,
First, thanks for the response! ... I'm not not in a rush and I am sure that you are busy, but that would be absolutely great if you can have a look at updating to a newer kernel.
When I'm in front of that machine, I can do a build again and show you specific errors.
Thanks again!

Update: So I did manage to get the kernel module to build and I can see /dev/binderfilter on my Pixel.
The errors during the compile went as follows;

fix the #include error in binder_filter.h; https://pastebin.com/qKaHmjRe
Next, I hit this;

drivers/staging/android/binder.c: In function ‘binder_transaction’:
drivers/staging/android/binder.c:2309:3: error: incompatible type for argument 4 of ‘filter_binder_message’
t->sender_euid, (void*)offp, tr->offsets_size);
^
drivers/staging/android/binder.c:46:12: note: expected ‘int’ but argument is of type ‘kuid_t {aka struct }’
extern int filter_binder_message(unsigned long, signed long, int, int, void*, size_t);
^~~~~~~~~~~~~~~~~~~~~
So I changed the 4th argument. compile continues.

I had to change fp->type in binder_filter.c to fp->handle. [line: 978] (but I'm not sure if that's correct)...But it continues to compile....

finally there were a few format warnings, all of which I corrected (enough to make the compiler happy, but not necessarily right);

printk(KERN_INFO "BINDERFILTER: type BINDER_TYPE_FD, handle %ld\n", fp->handle);
I did this on my own branch of 3.18.75 (Marlin/Google Pixel) so it was easy to create a diff. Which is right here; https://pastebin.com/2cNQiELa
my kernel sources are here: https://github.com/nine7nine/marlin/tree/EX-NoShadez-3.18.75-gcc6-plus
Unfortunately, I haven't gotten Picky / The Middleware to compile to test. I did get it stup in android studio, but I'm having a bit of a spat with it. LOL...Like this;
:app:compilePicky-jniMips64DebugSharedLibraryPicky-jniMainC
/tmp/picky-jni-69ae83.s: Assembler messages:
/tmp/picky-jni-69ae83.s:1391: Warning: unrecognized section type
/tmp/picky-jni-69ae83.s:1391: Warning: entity size for SHF_MERGE not specified
/tmp/picky-jni-69ae83.s:1391: Error: junk at end of line, first unrecognized character is `0'
never seen anything like that before, not sure how to fix that... I'm going to have to google around.
anyhoo. Again, not rush. but this is where I'm at so far...

UPDATE 2:
You can ignore the above regarding Picky and the middleware... I've have Picky built and I can build middleware.c with both the toolchain in your sources and the prebuilt toolchain in android-ndk, specifically; /opt/android-sdk/ndk-bundle/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin
(although, I would imagine your middleware code should be ported to aarch64 / arm64? .. like the kernel code?)
Anyway, I have managed to get BinderFilter to work somewhat... I can actually get some output using ./binderfilter.py ... there are a few issues. Selinux being one of them, as newer devices have much stricter sepolicy than old devices.... For now, whenever I am testing, I am setting selinux to permissive. (I can inject sepolicy fixes later)
Here is a document that sort of outlines what is working/not working for me; https://pastebin.com/hMXcDG7U
I also had to modify printIpcBuffersOnce() to use root (in binderfilter.py), otherwise it doesn't have enough permissions.
def printIpcBuffersOnce():
checkIpcBuffersAndFilterEnabled()
cmd='adb shell su -c dmesg | grep "BINDERFILTER"'
call(cmd, shell=True)
After that, it works. Sample output; https://pastebin.com/wUscm90w
Where I am stuck is applying rules, it doesn't work as expected. I get this sort of thing;
android.permission.RECORD_AUDIO:10070:1:0:
Write_len: 48
Fatal error: Policy could not be successfully applied!
Policy expected: android.permission.RECORD_AUDIO:10070:1:0:
but was:
Reading GPS also throws and error;
./binderfilter.py --get-gps-bytes
Traceback (most recent call last):
File "./binderfilter.py", line 556, in
main(sys.argv[1:])
File "./binderfilter.py", line 532, in main
getGpsStringForBinderFilter(results.latitude, results.longitude)
File "./binderfilter.py", line 266, in getGpsStringForBinderFilter
bLat = bytearray(struct.pack("d", float(latitude)))
TypeError: float() argument must be a string or a number
I assume that I messed up with my kernel patching and/or middleware.c needs to be adjusted to get things working properly. (and possibly updating it for arm64)...
I'm not sure how to fix these problems, but if/when you have the time, I do think it would be great to have BinderFilter find a new life on modern phones/tablets. I'm definitely into helping in any way that I can. Possibly improving Picky, small changes to the middleware (shell stuff) and testing/resolving selinux denials.

Hey David,
I just thought I would give you a friendly nudge and see if you've had any time to look at porting BinderFilter to a newer kernel?. .... not trying to bug you. Just curious 😉
I'm still very interested inuriluzing BinderFilter. I did actually fork Picky and started on some code fixes, just light stuff, like code that LInt points out, but I figured if/when you managed getting BinderFilter working, I would then look at improving Picky's interface/gui, more substantial updating to the app...
myself, I've been working on other Projects; a youtube player for android that I've committed a number of fixes, improvements and features to... and also getting Linaro's Energy Aware Scheduler up and running on the Pixel. It's actually pretty exciting, I've backported most of the code needed from MSM8996/linux-4.4(pixel 2/Op5 devices) to MSM8996/linux-3.18(Pixel 1)...
so there hasn't been any rush or anything, but still. I would love to be able to make use of BinderFilter.
I hope all is well. ttys

Hey,
I've looked at your documentation- which is very helpful. I need to install the new kernel on my testing device and try to reproduce the issues you've been seeing, which I'm hoping to get around to this weekend. By the way, Picky the android app may be a little lacking compared to the python command-line tool (the command line tool was written more recently and has less complexity when it comes to UI).
Also, if you want to, printk debugging the kernel module may shed some light on your errors
printk(KERN_INFO "BINDERFILTER: test\n");

Hey,
What is your testing device out of curiousity? Well, the nice thing about custom kernels is at least you don't need AOSP sources to do that - just an Arm/Aarch64 toolchain ;)
Regarding picky; I thought as much and I do prefer the commandline tool anyway. I just had thought having a gui app wouldn't hurt either. Maybe it can be extended in the future, but I would be happy even without it... Unreleated, but I had also considered making bilterfilter a proper service in my init.rc script that I generate in Anykernel2 which creating a flashable zip for the kernel. Pretty sure we can set sepolicy there and then inject any additional policies (as an aside)
I'll take a look at adding back binderfilter to my current kernel locally and take a closer look at my errors...
Let me know how things go! and good luck.
I really appreciate you getting back to me, Thanks a lot!

Hi,
I've updated the binderfilter.py file and that should fix some of your problems. Note that --get-gps-bytes needs some arguments passed to it (see below for proper usage). If your --print-buffers-once functionality works (see below for sample output), then that is most of the functionality and meat of the binderfilter project- this lets you see the buffer contents of all IPC messages, which contain very important information that app developers assume is secret ;). If you want to get the --print-system-context to work with GPS, then try opening google maps or something and asking an app for your location.
I will work more on the issues you're having with setting policy later this week.

Can you send me the output of the following commands?
adb shell "su -c ls -l /data/local/tmp/bf.policy"
adb shell "su -c cat /data/local/tmp/bf.policy"
adb shell "su -c ls -l /data/local/tmp/middleware"
adb shell "su -c '/data/local/tmp/middleware -a 1 -u 1000 -m binderfiltertest -c 0'"

Hey David,
The output from the above;
[ninez@nine7x cli]$ adb shell "su -c ls -l /data/local/tmp/bf.policy"
-rwxrwxrwx 1 root root 0 2017-11-05 21:07 /data/local/tmp/bf.policy
[ninez@nine7x cli]$ adb shell "su -c cat /data/local/tmp/bf.policy"
[ninez@nine7x cli]$ adb shell "su -c ls -l /data/local/tmp/middleware"
-rwxrwx--x 1 root root 11080 2017-11-05 20:48 /data/local/tmp/middleware
[ninez@nine7x cli]$ adb shell "su -c '/data/local/tmp/middleware -a 1 -u 1000 -m binderfiltertest -c 0'"
Write_len: 48
Appears to be ok, but when actually applying;
[ninez@nine7x cli]$ ./binderfilter.py -s -m "android.permission.RECORD_AUDIO" -u 10177 -a 1
android.permission.RECORD_AUDIO:10177:1:0:
Write_len: 48
Fatal error: Policy could not be successfully applied!
Policy expected: android.permission.RECORD_AUDIO:10177:1:0:
but was:
So it still fails... My binder filter is still a bit different from yours. You get this nice clean view, like;
buffer contents: {(0)@(28)(0)android.app.IProcessObserver(0)(0)D(
^^^^^^^^^^
Mine is like this;
buffer contents: {(4\xff)@(1\xff)2(0\xff)com.android.internal.telephony.IPhoneStateListener(0\xff)(0
^^^^^^^^^^^^^^^^^^
more output; https://pastebin.com/7UwtEdHQ
I think my patch on the kernel is wrong. (?)

If you recall, gcc complained when compiling about arguments being different (documented by me above)... IIRC;
your binder c;
#include "binder_filter.h"
extern int filter_binder_message(unsigned long, signed long, int, size_t, void*, size_t);
my binder.c
+#include "binder_filter.h"
+extern int filter_binder_message(unsigned long, signed long, int, kuid_t, void*, size_t);
(forth argument was size_t, but replaced with kuid_t)
the actual hook;
(yours)
I didn't actually change the 'hook' in binder_transaction....
Then in binder_filter.c;
--- binder_filter.c	2017-11-05 19:55:54.601030280 -0500
+++ binder_filter.c	2017-10-16 17:37:50.729788639 -0400
@@ -362,7 +362,7 @@
}
if (data_size > max_buffer_size) {

  printk(KERN_INFO "BINDERFILTER: data size %ld too large (max size %d)", data_size, max_buffer_size);

  printk(KERN_INFO "BINDERFILTER: data size %d too large (max size %d)", data_size, max_buffer_size);
  len = max_buffer_size;

}

@@ -975,8 +975,8 @@
if (offsets_size > 0) {
fp = (struct flat_binder_object *)offp;
if (fp != NULL) {

  	if (fp->handle == BINDER_TYPE_FD) {

  		printk(KERN_INFO "BINDERFILTER: type BINDER_TYPE_FD, handle %d\n", fp->handle);

  	if (fp->type == BINDER_TYPE_FD) {

  		printk(KERN_INFO "BINDERFILTER: type BINDER_TYPE_FD, handle %ld\n", fp->handle);
  	}
  }

}

and binder_filter.h
---binder_filter.h	2017-11-05 19:55:54.601030280 -0500
+++ binder_filter.h	2017-10-16 18:47:43.158513336 -0400
@@ -8,7 +8,11 @@
#ifndef _LINUX_BINDER_FILTER_H
#define _LINUX_BINDER_FILTER_H
-#include <linux/uidgid.h>
+static inline uid_t __kuid_val(kuid_t uid)
+{

   return uid.val;

+}
+
enum {
BLOCK_ACTION = 1,
The only other changes I made to any of your sources, was adding 'su' to use dmesg AND I pointed ccAndMove.sh to use a local toolchain from the NDK - My thought here was that carrying around an old toolchain is kind of a hassle - easier to just use a standard one... but I suppose I could try using the old one, if you think that actually makes a difference?
+export PATH=/opt/android-sdk/ndk-bundle/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin:$PATH
+export CC=arm-linux-androideabi-gcc
+$CC --sysroot=/opt/android-sdk/ndk-bundle/platforms/android-23/arch-arm -fPIE -pie -o middleware middleware.c

side note: (and part of the rason I asked about your testing device).
Is it possible that some of the work being done on hardening the kernel could be interfering with binderfilter.py failing to apply rules?? (assuming there is nothing wrong with my kernel patching to binder?
https://source.android.com/devices/architecture/kernel/hardening
https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project/Recommended_Settings
some of these are linux-4.4+, so don't apply (completely to my 3.18 marlin kernel)...
a lot of the Grsec/Pax security/hardening patches (or code that accomplishes the same) is making it's way into linux and and android. Not sure if anything so far would break binder filter or not.

Yeah that's interesting- I'll look into that param change this weekend. It is also possible that the kernel hardening affected the project, but I don't think it would reach as deep as the kernel module layer (compiling changes to the kernel layer should be trusted as an attacker can only recompile the kernel if they have root access and an unlocked bootloader).
My testing device is a 2013 nexus 7 (quite old at this point, but the pure android device I have).
The binder buffer content changes are probably just changing (char) to its hex representation. Remember that the python script is just a command line driver for the underlying middleware on the device- which seems to be working fine.
Can you try the following (applying a rule directly through middleware, then seeing if that rule was applied)
adb shell "su -c '/data/local/tmp/middleware -a 1 -u 1000 -m binderfiltertest -c 0'"
adb shell "su -c cat /data/local/tmp/bf.policy"

Hey,
slow / late response. Life gets a bit busy sometimes - but 100% still interested in Binder Filter... Later this afternoon ~ I'm going to patch (and commit) binder filter into one of my branches ~ so I don't need to keep doing it and can just cherry-pick the commits for Binder-filter....
I don't want to keep it in my main Pixel kernel branch. I also have been working on porting the newest binder code to my kernel - Have not yet tested Binder filter with it, but I expect it shouldn't be much different, as in where it hooks, arguments, etc....
I'll then try just using the middleware to apply rules and see if I have any success there. Because, as you say; the middleware does seem to be working.... I'll let you know later on! :-)
I hope all is well! ttys

Hope it goes well! Yeah it seems like the middleware is working (which would be great news since the python-middleware interaction is a lot easier to debug than the kernel layer). You could confirm with something like this:
adb shell "su -c '/data/local/tmp/middleware -a 1 -u 1000 -m binderfiltertest -c 0'"
adb shell "su -c cat /data/local/tmp/bf.policy"


Doesn't start web process automatically with multibuildpacks

I've been using this buildpack in conjunction with @ddollar's heroku-buildpack-multi, but the default_process_type listed in bin/release doesn't run. I manually created a Procfile that calls "bin/run" and then it starts, but it seems like this shouldn't be necessary. I'm not entirely sure if this is a problem within this buildpack, or with how I'm using it in conjunction with the multibuildpack, but I'd appreciate some feedback, thanks!

The multi buildpack uses the process defined by the last buildpack of the list. Make sure the PHP buildpack comes last if you want it to run the process automatically.

Ah, thanks!


this should get a warning tomorrow

#backlog

@lamebrain - This issue has been in the backlog for 80 days. In 10 days, it will be automatically closed.
For more information about the process by which we handle documentation feedback, see "How we respond to Azure technical documentation feedback" in our public contributor guide.

@bharney-msft nice ^

The feedback you logged for this should get a warning tomorrow has been in our backlog for 90 days. While we try to respond individually to as many feedback items as possible, we are not able to respond to all of them. Since this item has been in the backlog for 90 days, we’re closing it.
For more information about the process by which we handle documentation feedback, see "How we respond to Azure technical documentation feedback" in our public contributor guide.


Implement FCPath

Fixed Issues
Does this pull request fix any open issues? If so, list them here, one per line,
each in the format Fixes #<n>. Delete this comment.

Fixes #

Summary of Changes
Summarize the changes made by this pull request. Consider this a permanent record of the work you have done and make it sufficiently detailed and readable that a developer in the future would be able to get the complete picture without looking at the code changes. Skimming through the changed files will help you create the summary. Delete this comment.

Summary

Known Problems
Are there any relevant problems not fixed by, or problems caused by, this pull request? Delete this comment.

Problem

Summary by Bito
This PR implements FCPath as a replacement for FileCachePrefix, providing enhanced path handling across different operating systems with improved type safety and mypy integration. The implementation includes nthreads validation, strict type checking, and better cross-platform compatibility. The changes focus on API improvements, comprehensive path handling capabilities, and improved FileCache functionality with updated documentation.

Unit tests added: True

Estimated effort to review (1-5, lower is better): 5

AI Code Review is in progress (usually takes <3 minutes unless it's a very large PR).

Changelist by Bito
This pull request implements the following key changes.

Key Change
Files Impacted

Feature Improvement - Implement FCPath class to replace FileCachePrefix

- .coveragerc - Added filecache/my_glob.py to coverage

- .flake8 - Excluded filecache/my_glob.py from flake8 checks

- __init__.py - Updated import from FileCachePrefix to FCPath

- file_cache.py - Replaced FileCachePrefix with FCPath, added registered_scheme_prefixes method

- file_cache_path.py - Implemented new FCPath class to replace FileCachePrefix

Feature Improvement - Implement FCPath class to replace FileCachePrefix

- .coveragerc - Added filecache/my_glob.py to coverage

- .flake8 - Excluded filecache/my_glob.py from flake8 checks

- __init__.py - Updated import from FileCachePrefix to FCPath

- file_cache.py - Replaced FileCachePrefix with FCPath, added registered_scheme_prefixes method

- file_cache_path.py - Implemented new FCPath class to replace FileCachePrefix

Feature Improvement - Implement FCPath class to replace FileCachePrefix

- .coveragerc - Added filecache/my_glob.py to coverage

- .flake8 - Excluded filecache/my_glob.py from flake8 checks

- __init__.py - Updated import from FileCachePrefix to FCPath

- file_cache.py - Replaced FileCachePrefix with FCPath, added registered_scheme_prefixes method

- file_cache_path.py - Implemented new FCPath class to replace FileCachePrefix

- file_cache_prefix.py - Removed FileCachePrefix class

- file_cache_types.py - Added new file for type definitions

Feature Improvement - Implement FCPath class to replace FileCachePrefix

- .coveragerc - Added filecache/my_glob.py to coverage

- .flake8 - Excluded filecache/my_glob.py from flake8 checks

- __init__.py - Updated import from FileCachePrefix to FCPath

- file_cache.py - Replaced FileCachePrefix with FCPath, added registered_scheme_prefixes method

- file_cache_path.py - Implemented new FCPath class to replace FileCachePrefix

- file_cache_prefix.py - Removed FileCachePrefix class

- file_cache_types.py - Added new file for type definitions

- my_glob.py - Added new file with glob functionality

- test_file_cache.py - Updated tests to use FCPath instead of FileCachePrefix

Feature Improvement - Implement FCPath class to replace FileCachePrefix

- .coveragerc - Added filecache/my_glob.py to coverage

- .flake8 - Excluded filecache/my_glob.py from flake8 checks

- __init__.py - Updated import from FileCachePrefix to FCPath

- file_cache.py - Replaced FileCachePrefix with FCPath, added registered_scheme_prefixes method

- file_cache_path.py - Implemented new FCPath class to replace FileCachePrefix

- file_cache_prefix.py - Removed FileCachePrefix class

- file_cache_types.py - Added new file for type definitions

- my_glob.py - Added new file with glob functionality

- test_file_cache.py - Updated tests to use FCPath instead of FileCachePrefix

- test_file_cache_path.py - Added comprehensive tests for new FCPath class functionality

Feature Improvement - Implement FCPath class to replace FileCachePrefix

- .coveragerc - Added filecache/my_glob.py to coverage

- .flake8 - Excluded filecache/my_glob.py from flake8 checks

- __init__.py - Updated import from FileCachePrefix to FCPath

- file_cache.py - Replaced FileCachePrefix with FCPath, added registered_scheme_prefixes method

- file_cache_path.py - Implemented new FCPath class to replace FileCachePrefix

- file_cache_prefix.py - Removed FileCachePrefix class

- file_cache_types.py - Added new file for type definitions

- my_glob.py - Added new file with glob functionality

- test_file_cache.py - Updated tests to use FCPath instead of FileCachePrefix

- test_file_cache_path.py - Added comprehensive tests for new FCPath class functionality

- test_url_to_path.py - Updated tests to use new_path instead of new_prefix

AI Code Review is in progress (usually takes <3 minutes unless it's a very large PR).

AI Code Review is in progress (usually takes <3 minutes unless it's a very large PR).

AI Code Review is in progress (usually takes <3 minutes unless it's a very large PR).

AI Code Review is in progress (usually takes <3 minutes unless it's a very large PR).

AI Code Review is in progress (usually takes <3 minutes unless it's a very large PR).


ntr - ensure plugin load priority

1. Why is this change necessary?
Because of this DirectoryIterator (https://github.com/shopware/shopware/blob/5.5/engine/Shopware/Bundle/PluginInstallerBundle/Service/PluginInitializer.php#L83) it is possible that 2 App Servers have a different chain how the plugins are loaded. It can cause to very hard to debug issues if the priority of the plugin loading is necessary.
With this small change it is ensured that the plugins are loaded and processed with an alphabetical sorting.
2. What does this change do, exactly?
3. Describe each step to reproduce the issue or behaviour.
4. Please link to the relevant issues (if any).
5. Which documentation changes (if any) need to be made because of this PR?
6. Checklist

[ ] I have written tests and verified that they fail without my change
[ ] I have squashed any insignificant commits
[ ] This change has comments for package types, values, functions, and non-obvious lines of code
[ ] I have read the contribution requirements and fulfil them.

this is the same problem as #1785, isn't it?

Yes

It's also the same code 😉

And the issue is still there...

@t2oh4e that does not solve the entire problems. We want a loading which respects the dependencies and not just alphabetically.

@t2oh4e - before the comment of @uehler I didn't know that the pr #1785 exist. I'm not working in the core development so I know not every pull request for Shopware itself.
@shyim Your described fix is a much bigger solution and is not needed to fix the original problem. I have two different app server with the exact same sourcecode and they act different.
This one liner fix that.

Hi @teiling88,
thank you for your great idea of a config switch to enable the "predictable plugin load order". I think that is a great way to fix your (and @hhoechtel's) problem while providing a smooth upgrade path.
We'll mark this PR incomplete in the meantime.
Thanks!

I have no idea if I take good config names but the functionality is now there :-)

Hello,
thank you for creating this pull request.
I have opened an issue on our Issue Tracker for you. See the issue link: https://issues.shopware.com/issues/SW-22688
Please use this issue to track the state of your pull request.

HI @teiling88, your change got merged with 91dddf7066b1288f8e96d2bf406431ae8d0e683d. Thank you very much for your contribution! 💙


Rename String() method

DO NOT MERGE this is only used to track code calling non-deterministic .String() method, in the core.

Tests have been done. Closing it.


Publish Dapr operators on https://operatorhub.io/operator/

Ability to discover and deploy Dapr operators

Consider doing the same for helmhub

I've sketched something by using the operator-sdk's helm support and I got dapr almost running in an OpenShift cluster by using a custom catalog at this stage as nothing has been published to OperatorHub :
➜ k tree daprs.darp.dapr.io dapr-sample
NAMESPACE            NAME                                                     READY  REASON              AGE
openshift-operators  Dapr/dapr-sample                                         -                          115s
openshift-operators  ├─Configuration/daprsystem                               -                          108s
openshift-operators  ├─Deployment/dapr-dashboard                              -                          108s
openshift-operators  │ └─ReplicaSet/dapr-dashboard-74c8f46db4                 -                          108s
openshift-operators  │   └─Pod/dapr-dashboard-74c8f46db4-psfcg                True                       108s
openshift-operators  ├─Deployment/dapr-operator                               -                          108s
openshift-operators  │ └─ReplicaSet/dapr-operator-5dfc994cdd                  -                          108s
openshift-operators  │   └─Pod/dapr-operator-5dfc994cdd-q4kmg                 False  ContainersNotReady  108s
openshift-operators  ├─Deployment/dapr-sentry                                 -                          108s
openshift-operators  │ └─ReplicaSet/dapr-sentry-7f759d9bb                     -                          108s
openshift-operators  │   └─Pod/dapr-sentry-7f759d9bb-n5bwj                    False  ContainersNotReady  108s
openshift-operators  ├─Deployment/dapr-sidecar-injector                       -                          108s
openshift-operators  │ └─ReplicaSet/dapr-sidecar-injector-568756fbf8          -                          108s
openshift-operators  │   └─Pod/dapr-sidecar-injector-568756fbf8-v6zfb         True                       108s
openshift-operators  ├─Role/dapr-injector                                     -                          108s
openshift-operators  ├─Role/dapr-operator                                     -                          108s
openshift-operators  ├─Role/dapr-sentry                                       -                          108s
openshift-operators  ├─RoleBinding/dapr-injector                              -                          108s
openshift-operators  ├─RoleBinding/dapr-operator                              -                          108s
openshift-operators  ├─RoleBinding/dapr-sentry                                -                          108s
openshift-operators  ├─Secret/dapr-sidecar-injector-cert                      -                          108s
openshift-operators  ├─Secret/dapr-trust-bundle                               -                          108s
openshift-operators  ├─Secret/dapr-webhook-ca                                 -                          108s
openshift-operators  ├─Secret/dapr-webhook-cert                               -                          108s
openshift-operators  ├─Service/dapr-api                                       -                          108s
openshift-operators  │ └─EndpointSlice/dapr-api-h8vk5                         -                          108s
openshift-operators  ├─Service/dapr-dashboard                                 -                          108s
openshift-operators  │ └─EndpointSlice/dapr-dashboard-9h957                   -                          108s
openshift-operators  ├─Service/dapr-placement-server                          -                          108s
openshift-operators  │ └─EndpointSlice/dapr-placement-server-8jw96            -                          108s
openshift-operators  ├─Service/dapr-sentry                                    -                          108s
openshift-operators  │ └─EndpointSlice/dapr-sentry-c8n7g                      -                          108s
openshift-operators  ├─Service/dapr-sidecar-injector                          -                          108s
openshift-operators  │ └─EndpointSlice/dapr-sidecar-injector-cmhwf            -                          108s
openshift-operators  ├─Service/dapr-webhook                                   -                          108s
openshift-operators  │ └─EndpointSlice/dapr-webhook-sp2j2                     -                          108s
openshift-operators  ├─ServiceAccount/dapr-dashboard                          -                          108s
openshift-operators  ├─ServiceAccount/dapr-injector                           -                          108s
openshift-operators  ├─ServiceAccount/dapr-operator                           -                          108s
openshift-operators  ├─ServiceAccount/dapr-placement                          -                          108s
openshift-operators  ├─ServiceAccount/dapr-sentry                             -                          108s
openshift-operators  └─StatefulSet/dapr-placement-server                      -                          108s
openshift-operators    └─ControllerRevision/dapr-placement-server-548ccfd447  -                          108s

However, before going any further ad digg into the issues more, I'd like to understand if this is the right way to move forward as the resulting operator is a sort of a proxy that takes a CR and uses the values to deploy an helm chart so it seems quite a convoluted way. The operator now also requires a lot of privileges as it must install some roles, role bindings, CRDs, etc.
Link to sources and docker repo:

https://github.com/lburgazzoli/dapr-operator
https://github.com/lburgazzoli/dapr-operator-catalog
https://hub.docker.com/r/lburgazzoli/dapr-operator/tags
https://hub.docker.com/r/lburgazzoli/dapr-operator-bundle/tags
https://hub.docker.com/r/lburgazzoli/dapr-operator-catalog/tags

Some screenshots or it never happened:

Great work @lburgazzoli, as long as the operator adheres to the Operator Hub guidelines and can be used with OpenShift ALM then this approach is fine. Reusing our existing Helm Chart in an operator is better IMO than reinventing the wheel.

@lburgazzoli great work! what is missing for it to run on Openshift?

Great work @lburgazzoli, as long as the operator adheres to the Operator Hub guidelines and can be used with OpenShift ALM then this approach is fine. Reusing our existing Helm Chart in an operator is better IMO than reinventing the wheel.

thx for the confirmation, I'll work to refine the set-up in the coming days

@lburgazzoli great work! what is missing for it to run on Openshift?

I need to find out why some of the pods sometime are failing to start and to double check some errors I saw in the helm operator pod when it applies the chart plus of course some testing :)
Once that's done I can submit the operator to OperatorHub

@lburgazzoli I know that @cmoulliard is running Dapr on Openshift -> https://github.com/ch007m/my-dapr/blob/main/ocp/README.md are you setting those values when installing the chart?

@lburgazzoli I know that @cmoulliard is running Dapr on Openshift -> https://github.com/ch007m/my-dapr/blob/main/ocp/README.md are you setting those values when installing the chart?

Not yet, I got some error to solve before that point, most of the time rbac missing or mis-configured

I need to find out why some of the pods sometime are failing to start and to double check some errors I saw in the helm operator pod when it applies the chart plus of course some testing :)

Can you share what you did + instructions in order to test the OLM Operator on an existing OCP4 cluster where we can register it within the local operatorhub ?

I need to find out why some of the pods sometime are failing to start and to double check some errors I saw in the helm operator pod when it applies the chart plus of course some testing :)

Can you share what you did + instructions in order to test the OLM Operator on an existing OCP4 cluster where we can register it within the local operatorhub ?

Apply this manifest and then go to the operators installation page

I will be happy to try if someone share some instructions on the openshift side.. I am happy to test step by step tutorials.. if someone wrote that.. I can help review and improve and test

I will be happy to try if someone share some instructions on the openshift side.. I am happy to test step by step tutorials.. if someone wrote that.. I can help review and improve and test

Apply this manifest and then go to the operators installation page

How do you generate the content of the image:  docker.io/lburgazzoli/dapr-operator-catalog:latest ?

@cmoulliard it should be fixed with the latest release (0.0.16)

@salaboy if you have an openshift running, clone this repo then run the following commands:

provision the catalog (openshift-marketplace namespace):

kubectl apply -f config/samples/catalog.yaml

install the operator  (openshift-operators namespace):

kubectl apply -f config/samples/subscription.yaml

deploy dapr  (openshift-operators namespace):

kubectl apply -f config/samples/sample.yaml

You may need to wait a little between the commands so openshift reconciles the various resources.
However, you'll notice that the sentry pod won't start and that is because it looks like the related role, lacks the grant to delete secrets and the logs would show something like:
dapr-sentry-7f759d9bb-9fp85 dapr-sentry time="2023-05-25T12:17:31.157643483Z" level=info msg="starting watch on filesystem directory: /var/run/dapr/credentials" instance=dapr-sentry-7f759d9bb-9fp85 scope=dapr.sentry type=log ver=1.10.7
dapr-sentry-7f759d9bb-9fp85 dapr-sentry time="2023-05-25T12:17:31.157686144Z" level=info msg="certificate authority loaded" instance=dapr-sentry-7f759d9bb-9fp85 scope=dapr.sentry type=log ver=1.10.7
dapr-sentry-7f759d9bb-9fp85 dapr-sentry time="2023-05-25T12:17:31.158863513Z" level=info msg="Healthz server is listening on :8080" instance=dapr-sentry-7f759d9bb-9fp85 scope=dapr.sentry type=log ver=1.10.7
dapr-sentry-7f759d9bb-9fp85 dapr-sentry time="2023-05-25T12:17:31.16626055Z" level=info msg="root and issuer certs not found: generating self signed CA" instance=dapr-sentry-7f759d9bb-9fp85 scope=dapr.sentry.ca type=log ver=1.10.7
dapr-sentry-7f759d9bb-9fp85 dapr-sentry time="2023-05-25T12:17:31.179839599Z" level=fatal msg="error loading trust root bundle: error generating trust root bundle: failed saving secret to kubernetes: secrets \"dapr-trust-bundle\" is forbidden: cannot set an ownerRef on a resource you can't delete: , <nil>" instance=dapr-sentry-7f759d9bb-9fp85 scope=dapr.sentry type=log ver=1.10.7

To fix it, I've manually added delete  among the allowed verbs in the dapr-sentry role, I don't yet know if that's is something missing in the helm chart or something specific to openshift

@lburgazzoli I don't have Openshift running.. can you point me to how you are running it? Which version and can I run it locally? Or do I need to deploy it on a cloud provider?

To fix it, I've manually added delete among the allowed verbs in the dapr-sentry role, I don't yet know if that's is something missing in the helm chart or something specific to openshift

The verb delete is not defined part of the helm template - Role of sentry - https://github.com/dapr/dapr/blob/master/charts/dapr/charts/dapr_rbac/templates/sentry.yaml#L58

I can install dapr using the Dapr OLM operator and your instructions @lburgazzoli  ;-)

Note: I did some hacks to use the images pushed on quarky.io as I got the famous docker rate limit on our shared ocp4 cluster

@cmoulliard I've published the image on quay so it should solve the rate limiting issue.
The catalog can be installed with the following command:
kubectl apply -f https://raw.githubusercontent.com/lburgazzoli/dapr-operator/main/config/samples/catalog.yaml

@salaboy you can use CRC  but may be quicker and effective to get one cluster from a cloud provider

@salaboy you can use CRC but may be quicker and effective to get one cluster from a cloud provider

As I dont have a new account on azure/aws/gcp, I'm using our redhat openstack platform to test/run ocp4 ;-) but then I cannot showcase externally what I'm doing :-(

Quick update: the operator now install all the building blocks however there is an issue as the helm-operator goes in a sort of  infinite loop as for each reconciliation, some resources appears to be changing hence the   helm-operator tries to re-apply the chart

If needed I also have an ocp4 cluster running on IBMCloud that I could share privately with you

steps to install it in a cloud provider

This is documented here: https://docs.openshift.com/container-platform/4.13/installing/index.html

I foudn that the operator seems to behave as expected as in fact if you try to upgrade an dapr installation with the the same chart, helm effectively find some draft and generates patches:
upgrade.go:150: [debug] preparing upgrade for dapr
upgrade.go:158: [debug] performing update for dapr
upgrade.go:330: [debug] creating upgraded release for dapr
client.go:385: [debug] checking 40 resources for changes
client.go:678: [debug] Looks like there are no changes for ServiceAccount "dapr-dashboard"
client.go:678: [debug] Looks like there are no changes for ServiceAccount "dapr-injector"
client.go:678: [debug] Looks like there are no changes for ServiceAccount "dapr-operator"
client.go:678: [debug] Looks like there are no changes for ServiceAccount "dapr-placement"
client.go:678: [debug] Looks like there are no changes for ServiceAccount "dapr-sentry"
client.go:678: [debug] Looks like there are no changes for Secret "dapr-webhook-cert"
client.go:678: [debug] Looks like there are no changes for Secret "dapr-webhook-ca"
client.go:678: [debug] Looks like there are no changes for Secret "dapr-trust-bundle"
client.go:678: [debug] Looks like there are no changes for Secret "dapr-sidecar-injector-cert"
client.go:678: [debug] Looks like there are no changes for ClusterRole "dapr-dashboard"
client.go:678: [debug] Looks like there are no changes for ClusterRole "dapr-injector"
client.go:678: [debug] Looks like there are no changes for ClusterRole "dapr-operator-admin"
client.go:687: [debug] Patch ClusterRole "dapr-placement" in namespace
client.go:678: [debug] Looks like there are no changes for ClusterRole "dapr-sentry"
client.go:678: [debug] Looks like there are no changes for ClusterRoleBinding "dapr-dashboard"
client.go:678: [debug] Looks like there are no changes for ClusterRoleBinding "dapr-injector"
client.go:678: [debug] Looks like there are no changes for ClusterRoleBinding "dapr-operator-admin"
client.go:678: [debug] Looks like there are no changes for ClusterRoleBinding "dapr-placement"
client.go:678: [debug] Looks like there are no changes for ClusterRoleBinding "dapr-sentry"
client.go:678: [debug] Looks like there are no changes for Role "dapr-injector"
client.go:678: [debug] Looks like there are no changes for Role "dapr-operator"
client.go:678: [debug] Looks like there are no changes for Role "secret-reader"
client.go:678: [debug] Looks like there are no changes for Role "dapr-sentry"
client.go:678: [debug] Looks like there are no changes for RoleBinding "dapr-injector"
client.go:678: [debug] Looks like there are no changes for RoleBinding "dapr-operator"
client.go:678: [debug] Looks like there are no changes for RoleBinding "dapr-secret-reader"
client.go:678: [debug] Looks like there are no changes for RoleBinding "dapr-sentry"
client.go:678: [debug] Looks like there are no changes for Service "dapr-dashboard"
client.go:678: [debug] Looks like there are no changes for Service "dapr-api"
client.go:678: [debug] Looks like there are no changes for Service "dapr-webhook"
client.go:678: [debug] Looks like there are no changes for Service "dapr-placement-server"
client.go:678: [debug] Looks like there are no changes for Service "dapr-sentry"
client.go:678: [debug] Looks like there are no changes for Service "dapr-sidecar-injector"
client.go:678: [debug] Looks like there are no changes for Deployment "dapr-dashboard"
client.go:678: [debug] Looks like there are no changes for Deployment "dapr-operator"
client.go:678: [debug] Looks like there are no changes for Deployment "dapr-sentry"
client.go:678: [debug] Looks like there are no changes for Deployment "dapr-sidecar-injector"
client.go:678: [debug] Looks like there are no changes for StatefulSet "dapr-placement-server"
client.go:687: [debug] Patch Configuration "daprsystem" in namespace dapr-system
client.go:687: [debug] Patch MutatingWebhookConfiguration "dapr-sidecar-injector" in namespace
upgrade.go:165: [debug] updating status for upgraded release for dapr

Looking at what changes, it seems that only the clusterrole dapr-placement gets effectively patched but since such role has no rules, only the metadata gets updated so it could be an issue with helm itself but for better understanding, is there any reason to include an empty cluster role ?

is there any reason to include an empty cluster role ?

I think that this is a mistake as an empty rules array is only used with an aggregationRule
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monitoring
aggregationRule:
  clusterRoleSelectors:
  - matchLabels:
      rbac.example.com/aggregate-to-monitoring: "true"
rules: [] # The control plane automatically fills in the rules

I did some helm debugging and yes, so every time the helm chart is applied, then it generates a patch with {"rules":[]} so
it may be an helm misbehavior but in the meantime, I've opened an issue to discuss the empty rules thing https://github.com/dapr/dapr/issues/6429

@lburgazzoli can we book some time next week to talk about this? are you ok with doing a Zoom/Meet call to go over the state of this project?

I've reported my finding to the operator-sdk and helm projects:

https://github.com/operator-framework/operator-sdk/issues/6485
https://github.com/helm/helm/issues/12178

Operator repository created here: https://github.com/dapr-sandbox/dapr-kubernetes-operator


Notifications are not displayed if cartodb_com_hosted is true

Context
I think that this should affect OS and onpremises installs.
Steps to Reproduce

Set cartodb_com_hosted configuration parameter at app_config.yml to true. That means "this is self hosted, a local installation, not the CARTO SaaS".
Go to the dashboard

Current Result
Notification icon next to the avatar doesn't show up (it's hidden, as "guides" and "documentation" are).
Expected result
Notification icon appears.
Additional notes
Caused by this check.
cc @xavijam @danicarrion

This is pure frontend, taking care of it.

Deployed!


How to tell the qcluster to resume died jobs?

Is there a proper way to let died job rerun if the cluster died and is up again?
For example: The django project and the qcluster lives in a docker container. There is currently a long job executed by the qcluster and the docker container is rebuilt and deployed - the qcluster is shutted down and restarted.
I tested that case by having a long running oneshot task and killed the qcluster hart on my local windows machine. After the qcluster restarts, it is not restart the oneshot job.

@sparrow242 Did you find any solution for the same? Having same issue that you have described.


Support ASGIRequest instances in sync context.

Adds test case for authenticating against the app for #194
Fixes #194

@KundaPanda Django 3.1 is outside of its lifecycle. Do you feel strongly that this library should continue to support it? If not, then I'm inclined to create a new PR to drop the support to get these tests to pass rather than make them 3.1 compatible.

@KundaPanda Django 3.1 is outside of its lifecycle. Do you feel strongly that this library should continue to support it? If not, then I'm inclined to create a new PR to drop the support to get these tests to pass rather than make them 3.1 compatible.

Good idea, given the fact that the security support for 3.1 has been dropped as well and it will make the code easier to maintain.

Once #197 is merged, I'll rebase this.


Can we support rod?

Thanks for the project, can we support rod?
https://github.com/go-rod/rod

I'll look into it, thanks for the suggestion. Also feel free to PR if you have the time.

Rod supports the direct use of xvfb, so you can use it directly. Thank you again for your project.


Need help with wifi

Hi,
I'm in progress of collecting my first experiences with an Hackintosh. So far I'm already failing on getting it installed. I used this 1 how to to create a bootable USB stick until just before the 'Method 1" headline, choosing MacOS Big Sur.
From there on I tried to use this repo to move on. As far as I have understood it, I have to copy the BOOT and OC folder from this repo into the EFI folder of the boot stick? OpenCore already seems to be present, so I guess I don't have to install it manually? Right now I'm struggling, that no matter what I try, sometimes the touchpad doesn't work. And no matter what I do (using itlwm.kext from this repo or download it myself from 2), Wifi never works. And since the installation requires internet, I obviously cannot proceed. A heads up for what I am doing wrong would be greatly appreciated.

use the files in #release
& wifi should work fine with the latest macOS version

installation is here:
https://github.com/profzei/Matebook-X-Pro-2018/wiki/Upgrade-to-macOS-Big-Sur
use the files in #releases
& wifi should work fine with the latest macOS version

Thanks. But I don't have a working MacOSX yet. I am trying to create a working installer stick on Linux. But so far i am failing the get past the installation from the recovery, since I never have working wifi there.

@g0r3 My suggestion is creating (in some way depending on your starting OS...) a USB installer stick for a full offline installation without the need of a working Wi-Fi during installation process


Show current file in external file browser

Open the current file in external file browser. Useful for locating other files being worked on.

this is duplicated of #2628


Build errors with code downloaded

I'm getting build errors when I downloaded the sourcecode.
The tag 'ColorToBrushConverter' does not exist in XML namespace 'clr-namespace:PropertyTools.Wpf;assembly=PropertyTools.Wpf'. Line 9 Position 10.   C:\Users\rdoshi\Downloads\PropertyTools-master\PropertyTools-master\Source\Examples\Controls\ControlDemos\Pages\ColorPickerPage.xaml
It is giving similar errors for all the controls. What am I missing ?

The CI build is green, so I think something is wrong on your side. What version of VS are you using?
https://ci.appveyor.com/project/objorke/propertytools

Can't be reproduced. Please provide more information and reopen!

@uvasag  I have same question. Could you please tell me how you fix the problem? I use VS2015.


Escape HTML characters in IOU message

Details

Fixed Issues

$ https://github.com/Expensify/App/issues/15026
PROPOSAL: https://github.com/Expensify/App/issues/15026#issuecomment-1428125573
Tests

Go to chat with user_A
Request money from user_A and add a message Test <> Request money
Send money to user_A and add a message Test <> Send money
Go to press green ➕ button
Split bill with user_A and add a message Test <> Split bill
Verify <> is displayed normally in chat messages of Request money, Send money and Split bill
Verify <> is displayed normally in Details pages of Request money and Send money

[ ] Verify that no errors appear in the JS console

Offline tests

N/A
QA Steps

Same as Tests

[ ] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is correct English and approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Web

Mobile Web - Chrome

Mobile Web - Safari

Desktop

iOS

Android

Uh oh, I think there are multiple things happening with this. There is another PR here: https://github.com/Expensify/App/pull/15315 which is supposed to prevent inputs from having HTML put into them. Could you please check out that branch and see if you're still allowed to add HTML tags in the IOU message? If not, then I think that's good and we can just close this PR (and issue).

Uh oh, I think there are multiple things happening with this. There is another PR here: #15315 which is supposed to prevent inputs from having HTML put into them. Could you please check out that branch and see if you're still allowed to add HTML tags in the IOU message? If not, then I think that's good and we can just close this PR (and issue).

Hi @tgolen , thanks for pointing out that PR.
I tested the patch of https://github.com/Expensify/App/pull/15315 locally and still can add HTML tags in the IOU message. I also confirmed the source code that we don't use form to validate the IOU message. I think it makes sense to allow users to add IOU message like Tokyo <> Mexico as mentioned in the original GH issue.
Below is the frontend component hierarchy snapshot FYI

We are still discussing this in Slack about how we want to approach this problem, but it's also been mentioned that the original problem can no longer be reproduced.

I am going to close this for now. If we decide to go through with it, this can be reopened and merged.


Reset messages with status dispatched when server (re)starts.

If a message was dispatched to a consumer and the server dies before it received the acknowledgement, this message will be stuck in status dispatched forever.
When the server (re)starts, set all messages with status dispatched to undispatched so consumer will get the message again and can acknowledge it.

Implemented. Documentation follows.


Homebrew installation

TL;DR
Would you consider adding support for installation by Homebrew?
goreleaser has built in support for creating taps
Detailed design
No response
Additional information
No response

@eriweb great idea. @nadav-legit is working on it

Hi @eriweb thanks for the idea.
We've added support for brew, currently only for x86 macos.
We weren't able to use goreleaser brew extension because we're singing macos binaries which cannot be combined with the rest of our release workflow and brew.


"IndexError: too many indices for tensor of dimension 1" error when text is empty

Hi, thank you for the great work with this model, it is really appreciated.
We are using 0.1.7 version of GLiNER with urchade/gliner_multi-v2.1 model. However, when the text is empty or has only whitespaces the model fails with the following error:
IndexError: too many indices for tensor of dimension 1
This is a problem especially when running the batch_predict_entities method. We can workaround the issue by filtering the texts with a content before feeding them to the model, but it would be definitely easier to avoid doing it.
Thank you!
Below the stacktrace:
IndexError: too many indices for tensor of dimension 1
File <command-3871930245476660>, line 7
----> 7 df["entities"] = basic_cleaner.ner.batch_predict_entities(df[TARGET_COL].tolist(), labels=labels)
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/gliner/model.py:304, in GLiNER.batch_predict_entities(self, texts, labels, flat_ner, threshold)
    301     all_end_token_idx_to_text_idx.append(end_token_idx_to_text_idx)
    303 input_x = [{"tokenized_text": tk, "ner": None} for tk in all_tokens]
--> 304 x = self.collate_fn(input_x, labels)
    305 outputs = self.predict(x, flat_ner=flat_ner, threshold=threshold)
    307 all_entities = []
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/gliner/modules/base.py:115, in InstructBase.collate_fn(self, batch_list, entity_types)
    113     class_to_ids = {k: v for v, k in enumerate(entity_types, start=1)}
    114     id_to_classes = {k: v for v, k in class_to_ids.items()}
--> 115     batch = [
    116         self.preprocess_spans(b["tokenized_text"], b["ner"], class_to_ids) for b in batch_list
    117     ]
    119 span_idx = pad_sequence(
    120     [b["span_idx"] for b in batch], batch_first=True, padding_value=0
    121 )
    123 span_label = pad_sequence(
    124     [el["span_label"] for el in batch], batch_first=True, padding_value=-1
    125 )
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/gliner/modules/base.py:116, in <listcomp>(.0)
    113     class_to_ids = {k: v for v, k in enumerate(entity_types, start=1)}
    114     id_to_classes = {k: v for v, k in class_to_ids.items()}
    115     batch = [
--> 116         self.preprocess_spans(b["tokenized_text"], b["ner"], class_to_ids) for b in batch_list
    117     ]
    119 span_idx = pad_sequence(
    120     [b["span_idx"] for b in batch], batch_first=True, padding_value=0
    121 )
    123 span_label = pad_sequence(
    124     [el["span_label"] for el in batch], batch_first=True, padding_value=-1
    125 )
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/gliner/modules/base.py:45, in InstructBase.preprocess_spans(self, tokens, ner, classes_to_id)
     42 spans_idx = torch.LongTensor(spans_idx)
     44 # mask for valid spans
---> 45 valid_span_mask = spans_idx[:, 1] > length - 1
     47 # mask invalid positions
     48 span_label = span_label.masked_fill(valid_span_mask, -1)

Sorry for the late reply,
I will fix this in the next release

Fixed now

weird. does it work if at least on the input is not empty ?


Recursive WorkGraph with Graph Builder can only be run for the amount of redefinition of it

Because @giovannipizzi mentioned that for a project that need to use recursive workflows I wanted to do an example for recursive workflows. It kind of works, but there are some problems with referring to the the graph builder itself
from aiida_workgraph import task, WorkGraph
from aiida import load_profile
load_profile()

@task.calcfunction
def identity(x):
    return x.clone()

@task.calcfunction
def multiply(x, y):
    return x*y

@task.graph_builder(outputs = [{"name": "result", "from": "multiply.result"}])
def recursive(n):
    wg = WorkGraph()
    if n == 1:
        mytask = wg.add_task(identity, name="multiply", x=n)
        return wg
    t = wg.add_task(recursive, name="recursive", n=n-1)
    wg.add_task(multiply, name="multiply", x=n, y=t.outputs["result"])
    return wg

I have to rerun the recursive definition the amount of time I want to use it in the recursion
wg = WorkGraph()
mytask = wg.add_task(recursive, n=3) # I have to run the above cell defining recursive 3 times so this works
wg.run()

The error message is otherwise
  File "/Users/alexgo/code/aiida-workgraph/.pixi/envs/docs-env/lib/python3.12/site-packages/plumpy/process_states.py", line 228, in execute
    result = self.run_fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexgo/code/aiida-workgraph/aiida_workgraph/engine/workgraph.py", line 285, in run
    return self._do_step()
           ^^^^^^^^^^^^^^^
  File "/Users/alexgo/code/aiida-workgraph/aiida_workgraph/engine/workgraph.py", line 300, in _do_step
    self.continue_workgraph()
  File "/Users/alexgo/code/aiida-workgraph/aiida_workgraph/engine/workgraph.py", line 705, in continue_workgraph
    self.run_tasks(task_to_run)
  File "/Users/alexgo/code/aiida-workgraph/aiida_workgraph/engine/workgraph.py", line 1109, in run_tasks
    wg = self.run_executor(executor, [], kwargs, var_args, var_kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexgo/code/aiida-workgraph/aiida_workgraph/engine/workgraph.py", line 1450, in run_executor
    return executor(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/lh/d5j2y3816xg0qffzv9bqlx2c0000gn/T/ipykernel_70299/3807790678.py", line 7, in recursive
    t = wg.add_task(recursive, name="recursive", n=n-1)
                    ^^^^^^^^^
NameError: name 'recursive' is not defined

Here is another working version; however, I am not very clear why it works.
from aiida_workgraph import task, WorkGraph
from aiida import load_profile
load_profile()

@task.calcfunction
def identity(x):
    return x.clone()

@task.calcfunction
def multiply(x, y):
    return x*y

def recursive(n):
    wg = WorkGraph()
    if n == 1:
        mytask = wg.add_task(identity, name="multiply", x=n)
        return wg
    Task = task.graph_builder(outputs = [{"name": "result", "from": "multiply.result"}])(recursive)
    t = wg.add_task(Task, name=f"recursive_{n-1}", n=n-1)
    wg.add_task(multiply, name="multiply", x=n, y=t.outputs["result"])
    return wg

Task = task.graph_builder(outputs = [{"name": "result", "from": "multiply.result"}])(recursive)
wg = WorkGraph()
mytask = wg.add_task(Task, n=3, name="recursive_3")
wg.run()