Serializer used by MassTransit throwing Newtonsoft.Json.JsonReaderException on nulled field

Is this a bug report?
Yes
(Yes/no)
Can you also reproduce the problem with the latest version?
(Write your answer here.)
Yes
Environment
Dev

Dotnet version: 6.0
Operating system: Windows 10
IDE (Rider, VS Code, Visual Studio): Visual Studio 2022

Steps to Reproduce

(Write your steps here:)

Send a message from an app using NServiceBus with RabbitMQ transport to another app using MassTransit connected to RabbitMQ with a consumer that consumes the message sent by NServiceBus.
Observe that the mass transit throws a Newtonsoft.Json.JsonReaderException with message: "Error reading integer. Unexpected token: StartObject. Path 'ItemCount'.  I'm fairly certain that the issue is with the way the Newtonsoft serializer is serializing the XML sent by the NServiceBus application.  A subscribing app using NServiceBus has no issues with the same payload.  The problem is that the offending field is nullable with a value of null and the XML expresses this as an empty element with the xsi:nil attribute which the serializer chokes on.

Expected Behavior

(Write what you thought would happen.)
The mass transit would consume the message and publish an event to the NServiceBus app's endpoint
Actual Behavior

(Write what happened. Please add screenshots!)
The app using MassTransit threw a Newtonsoft.Json.JsonReaderException with message: "Error reading integer. Unexpected token: StartObject. Path 'ItemCount'.
Newtonsoft.Json.JsonReaderException: Error reading integer. Unexpected token: StartObject. Path 'ItemCount'.
at Newtonsoft.Json.JsonReader.ReadAsInt32()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
at MassTransit.Serialization.NewtonsoftSerializerContext.TryGetMessage[T](T& message) in //src/MassTransit.Newtonsoft/Serialization/NewtonsoftEnvelopeSerializerContext.cs:line 39
at MassTransit.Serialization.BodyConsumeContext.TryGetMessage[T](ConsumeContext1& message) in /_/src/MassTransit/Serialization/BodyConsumeContext.cs:line 70          at MassTransit.Middleware.OutputPipeFilter2.MassTransit.IFilter.Send(TInput context, IPipe1 next) in /_/src/MassTransit/Middleware/OutputPipeFilter.cs:line 43          at MassTransit.Middleware.DeserializeFilter.Send(ReceiveContext context, IPipe1 next) in //src/MassTransit/Middleware/DeserializeFilter.cs:line 40
at MassTransit.Middleware.RescueFilter2.MassTransit.IFilter<TContext>.Send(TContext context, IPipe1 next) in //src/MassTransit/Middleware/RescueFilter.cs:line 43
fail: MassTransit.ReceiveTransport[0]
R-FAULT rabbitmq://dpasd-rbt03.dcs.leidos.com/vhost-reedl3/nsb-mt-interop-mt db11474b-979d-4415-b6db-af0e0158cff3 00:00:00.1519192
Newtonsoft.Json.JsonReaderException: Error reading integer. Unexpected token: StartObject. Path 'ItemCount'.
at Newtonsoft.Json.JsonReader.ReadAsInt32()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
at MassTransit.Serialization.NewtonsoftSerializerContext.TryGetMessage[T](T& message) in //src/MassTransit.Newtonsoft/Serialization/NewtonsoftEnvelopeSerializerContext.cs:line 39
at MassTransit.Serialization.BodyConsumeContext.TryGetMessage[T](ConsumeContext1& message) in /_/src/MassTransit/Serialization/BodyConsumeContext.cs:line 70          at MassTransit.Middleware.OutputPipeFilter2.MassTransit.IFilter.Send(TInput context, IPipe1 next) in /_/src/MassTransit/Middleware/OutputPipeFilter.cs:line 43          at MassTransit.Middleware.DeserializeFilter.Send(ReceiveContext context, IPipe1 next) in //src/MassTransit/Middleware/DeserializeFilter.cs:line 40
at MassTransit.Middleware.RescueFilter2.MassTransit.IFilter<TContext>.Send(TContext context, IPipe1 next) in //src/MassTransit/Middleware/RescueFilter.cs:line 43
Reproducible Demo
Attached

(Paste the link to an example project and exact instructions to reproduce the issue.)

nsb-mt-interop-bug-rpt.zip

I love the fact that there's already a commit for this.  Any idea when the fix would be available in NuGet?

Develop (prerelease) packages are already available. If you could verify the fix in your code that would be great!

I've done some initial testing and have not found any issues.  I've had to divert my attention for a while so wanted let you know: so far, so good.  Love the quick turn-around and the unit test.  I'll get back to this as soon as possible. Thank you!

Hi. I am having the same issue.  Which version for masstransit did you use? i tried with latest stable and also with latest prerelease and the issue persists.


[cmake] Recover original CMAKE_C/CXX_COMPILER after changing them.

Seems that the change in the two variables was spilling into the other
target of the file, but returning it back to the original values seems
to avoid that issue.
This should unbreak the Android CI build. In it, the Linux static
library was changing to the host compiler, and that compiler was being
used for the Android runtime library, which would have never compile
that way (since the host compiler in CI is an old-ish Clang without the
necessary argument).
/cc @compnerd: It is way more of a hack than what you were proposing, but it works.

@drodriguez absolutely disgusting!  Yes, but, unfortunately, until we split the builds, this is the best we can do :-(.

CC: @gottesmm @Rostepher

@swift-ci please test

This probably is a good thing to test on Windows as well, but, I guess that we can do that post-commit as well.  At the very least, the nightly build will test this.


Updating redux to 2.0

Thanks!

thanks you :+1:

oh! did you upload to the npm package?

@sebas5384 I have just published it.

@lapanoid awesome!


address_utxos endpoint sometimes returns an empty list instead of the results

Describe the bug
I am trying to get the UTxOs for my address using the address_utxos endpoint for the preproduction testnet. This is the address:
addr_test1qp9s80tz7l3dxmg4wcsd6fwnjcxuqul6wy6x5pwt98hmhj00c7g4mfe8tn70pveep8eep4dzuu0f784unh4wk5p7jkasa3cym4
There are 2 UTxOs at the address. I am using this command:
curl -X POST "https://preprod.koios.rest/api/v1/address_utxos"  -H "accept: application/json" -H "content-type: application/json"  -d '{"_addresses":["addr_test1qp9s80tz7l3dxmg4wcsd6fwnjcxuqul6wy6x5pwt98hmhj00c7g4mfe8tn70pveep8eep4dzuu0f784unh4wk5p7jkasa3cym4"],"_extended":true}'

However, the endpoint returns an empty list about 30% of the time.
To Reproduce
Repeat the above command multiple times and see how often it returns an empty list instead of the two UTxOs. I am consistently seeing at least 2 or 3 empty lists out of every 10 queries.
Expected behavior
It should never return an empty list when there are in fact UTxOs at the address. It should either return the proper result or return an error that explains what happened.

Thanks for raising, looks like one of the instance was on fork - but not far enough from tip to be marked down.

Will adapt health check to prevent in future


Default indexing policy unclear

I'm a little confused whether CosmosDb is creating spatial indexes on GeoJSON properties of type 'Point' on documents in my database. This article on one hand says:
"The default indexing policy for newly created containers indexes every property of every item, enforcing range indexes for any string or number, and spatial indexes for any GeoJSON object of type Point."
But in the same doc says:
"Azure Cosmos DB, by default, will not create any spatial indexes. If you would like to use spatial SQL built-in functions, you should create a spatial index on the required properties. See this section for indexing policy examples for adding spatial indexes."
I'm not sure if I need to add "spatialIndexes" to my indexing policy or not?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d3aeab4a-343d-b492-f9dc-a3233a117101
Version Independent ID: 075ccfaf-9b5c-aa6d-8a35-a57b4344773c
Content: Azure Cosmos DB indexing policies
Content Source: articles/cosmos-db/index-policy.md
Service: cosmos-db
GitHub Login: @timsander1
Microsoft Alias: tisande

Hi @coffeymatt. Thank you for pointing this out! The following statement is correct: "Azure Cosmos DB, by default, will not create any spatial indexes. If you would like to use spatial SQL built-in functions, you should create a spatial index on the required properties. See this section for indexing policy examples for adding spatial indexes."
The earlier sentence used to be true but has since changed. The default indexing policy now just includes all paths with a range index (not spatial). If you want to use geospatial system functions, you can add on geospatial indexes separately.
Doc update should be live later today.

#please-close


oneMKL Specification Versioning Strategy wording

Summary
The wording for the MINOR count in the oneMKL Specification Versioning Strategy is confusing.
URLs
https://spec.oneapi.com/versions/latest/oneAPI-spec.pdf section <IP_ADDRESS>
Additional details
Wording for MINOR count mentions deleting APIs but "without breaking backwards compatibility". Should deleting APIs only occur with an increment in the MAJOR count?

Hi Edward,
What you describe "deleting APIs only occur with an increment in the MAJOR count" is the current intent of this statement.  Any deletion of apis or functionality should happen at MAJOR counts.  That means that when a case for removing an api comes up, we will mark it as deprecated (still a process we are working out for the spec) and then hopefully be able to align the removal with other changes that may occur and group them into a MAJOR uptick.  Does this make sense to you?  Do you have any concerns with this?
Spencer

We will update the language in the spec to be inline with this intent.

Hi Spencer,
That makes sense, thanks.
Edward

Changes merged. Closing.


Flashlight doesn't work.

Sony Xperia XZ1c
Android 9

Thank you for reporting a bug. I gonna refactor the app to use the new CameraX API, and hopefully this issue will be solved.

Also, the flashlight doesn't work for android 10.

Same with an Xperia XZ1 G8341.

The flashlight is working fine for me on Android 10 (Pixel 2 XL)


feat: copy App Studio annotations from ITS to PR

Copies any App Studio annotations, ones that contain the string appstudio.openshift.io/ in their key, to the Tekton PipelineRun.
Ref. https://issues.redhat.com/browse/HACBS-2548
Maintainers will complete the following section

[x] Commit messages are descriptive enough (hints)
[x] Code coverage from testing does not decrease and new code is covered
[x] Controllers diagrams are updated when PR changes controllers code  (if applicable)

It should be named feat not chore, we are adding new behavior here

It should be named feat not chore, we are adding new behavior here

Just this small change in commit and it's perfect!

/retest

Merging this as it passed all tests and has the needed reviews.


fix: Add isolation between string and hyperloglog( issue#2719)

Description:
related issue https://github.com/OpenAtomFoundation/pika/issues/2719
This pull request addresses the issue of lack of isolation between HyperLogLog and String types in the key-value database. The changes ensure that HyperLogLog operations are properly distinguished from string operations.
Changes Made:
Introduced a New Data Type for HyperLogLog:
A new data type specifically for HyperLogLog has been added to distinguish it from string types. This change is crucial to ensure that operations intended for HyperLogLog do not inadvertently apply to string keys.
Implemented HyperlogValue Class:
A new HyperlogValue class has been created to handle the encoding of HyperLogLog values. This class ensures that the type field in HyperLogLog value is different to String one.
value format of string and hyperloglog
| type | value | reserve | cdate | timestamp |
|  1B  |       |   16B   |   8B  |     8B    |
Added Functions for HyperLogLog Operations:
New functions have been introduced to handle HyperLogLog-specific operations, including:
Expiry handling (HyperlogExpire, HyperlogExpireat)
Deletion (HyperlogDel)
Persistence (HyperlogPersist)
Time-to-live management (HyperlogTTL)
Basic HyperLogLog function (HyperlogGet, HyperlogSet)
Scanning HyperLogLog keys (ScanHyperlog)
These functions ensure that HyperLogLog keys are managed correctly within the database, providing full support for the new data type.

Summary by CodeRabbit

New Features

Added support for Hyperloglog data type, including new commands and methods for handling Hyperloglog operations.

Improvements

Enhanced logging for better debugging and tracking of PfMergeCmd command execution.
Configuration updates for replication settings to improve performance and reliability.

Refactor

Optimized storage methods to specifically handle Hyperloglog data type instead of generic methods.

Tests

Introduced new test files covering Hyperloglog functionality and Redis transactions to ensure robustness and correctness.

PR Reviewer Guide 🔍

 ⏱ Estimated effort to review [1-5]
4

 🧪 Relevant tests
Yes

 🔒 Security concerns
No

 ⚡ Key issues to review

Possible Bug: The implementation of HyperloglogGet and other similar methods should ensure that they handle the type checks and stale data checks correctly. The current implementation could potentially return incorrect errors or data if the type or stale checks fail.

Error Handling: The methods such as HyperloglogSet, HyperloglogDel, etc., should ensure that they handle errors gracefully and provide clear and informative error messages to the users.

Performance Concerns: The addition of multiple new methods and the changes to existing methods could potentially impact the performance of the system. It would be beneficial to perform performance testing to ensure that these changes do not negatively affect the system's performance.

@CodiumAI-Agent /review

Persistent review updated to latest commit https://github.com/OpenAtomFoundation/pika/commit/15b3000284d185639dd03479b934f782c3b4f665

@CodiumAI-Agent  improve

PR Code Suggestions ✨
CategorySuggestion                                                                                ScoreSecurity

Add a safety check before memcpy to prevent potential buffer overflow

The memcpy operation in IsHyperloglogObj does not check if the remaining size is less than kSuffixReserveLength, which could lead to buffer overflow. Add a check to ensure safe copying.
src/storage/src/redis_hyperloglog.cc [130]
-memcpy(reserve, internal_value_str->data() + offset, kSuffixReserveLength);
+if (internal_value_str->size() - offset >= kSuffixReserveLength) {
+    memcpy(reserve, internal_value_str->data() + offset, kSuffixReserveLength);
+} else {
+    // Handle error or set default values
+}

Suggestion importance[1-10]: 10
Why: Adding a safety check before memcpy is crucial to prevent potential buffer overflow, which is a significant security concern.
10
Possible issue

Verify and correct the command arity for kCmdNamePfMerge if necessary

The command arity for kCmdNamePfMerge was changed from -3 to -2. Ensure that this change is intentional and correctly handles the command's expected arguments, as this affects how the command parses and expects its arguments.
src/pika_command.cc [706]
-kCmdNamePfMerge, -2, kCmdFlagsWrite | kCmdFlagsHyperLogLog | kCmdFlagsSlow
+kCmdNamePfMerge, -3, kCmdFlagsWrite | kCmdFlagsHyperLogLog | kCmdFlagsSlow

Suggestion importance[1-10]: 9
Why: Verifying and correcting the command arity is important to ensure that the command handles its expected arguments correctly, which affects its functionality and reliability.
9
Possible bug

Verify and correct the data type setting in HyperloglogValue constructor

The HyperloglogValue constructor sets type_ as Kstring which might be incorrect or misleading if the data structure is intended to be distinct from regular strings. Verify and correct the data type setting to ensure that it reflects the actual data structure type being manipulated.
src/storage/src/strings_value_format.h [48]
-memcpy(dst, &type_, sizeof(type_));//set type as Kstring
+// Correctly set the type for HyperLogLog data structures
+type_ = DataType::kHyperLogLog; // Assuming kHyperLogLog is a valid DataType for HyperLogLogs
+memcpy(dst, &type_, sizeof(type_));

Suggestion importance[1-10]: 9
Why: This suggestion addresses a potential bug where the data type might be incorrectly set, which could lead to incorrect behavior or data corruption. Correcting this is crucial for the proper functioning of the code.
9

Add safety checks and documentation when manipulating reserve_ in HyperloglogValue

The reserve_ array is directly manipulated by setting its first byte to 0x80 without any preceding checks or comments explaining the rationale. This could lead to unexpected behavior or bugs if reserve_ is used elsewhere in the code. Consider adding safety checks and documentation for this operation.
src/storage/src/strings_value_format.h [54-55]
-reserve_[0] = 0x80;
-memcpy(dst, reserve_, kSuffixReserveLength);//copy reserve
+// Ensure reserve_ is not used uninitialized and explain the magic number usage
+if (reserve_.size() >= kSuffixReserveLength) {
+    reserve_[0] = 0x80; // Set the first byte to 0x80 to indicate a special condition or flag
+    memcpy(dst, reserve_, kSuffixReserveLength);//copy reserve
+} else {
+    // Handle error or initialize reserve_
+}

Suggestion importance[1-10]: 8
Why: Adding safety checks and documentation helps prevent potential bugs and makes the code more robust and understandable. This is important for maintaining code quality and preventing future issues.
8
Maintainability

Use a defined constant or dynamic size for the reserve array to enhance maintainability

The function IsHyperloglogObj uses a hard-coded size for the reserve array which might not be flexible for future changes. Consider defining this size as a constant or using a dynamic approach based on kSuffixReserveLength.
src/storage/src/redis_hyperloglog.cc [125]
-char reserve[16] = {0};
+char reserve[kSuffixReserveLength] = {0};

Suggestion importance[1-10]: 8
Why: Using a defined constant or dynamic size for the reserve array enhances maintainability and future-proofs the code against changes in kSuffixReserveLength.
8

Refactor the meta value type check to use a more flexible and robust method

The function HyperloglogGet checks for the expected meta value type using a direct comparison, which might lead to issues if additional meta types are introduced. Refactor this to use a function or method that can handle various types more flexibly.
src/storage/src/redis_hyperloglog.cc [143]
-if (!ExpectedMetaValue(DataType::kStrings, meta_value)){
+if (!IsValidMetaValueForType(DataType::kStrings, meta_value)){

Suggestion importance[1-10]: 7
Why: Refactoring the meta value type check to use a more flexible method improves maintainability and robustness, especially if additional meta types are introduced in the future.
7

Add method comments to new HyperLogLog methods for better code readability and maintenance

Consider adding a method comment for HyperloglogGet and HyperloglogSet to describe their purpose, parameters, and expected behavior. This is important for maintaining code readability and understanding, especially for methods that might be less self-explanatory than standard CRUD operations.
src/storage/src/redis.h [157-170]
+// Retrieves the value associated with the key in a HyperLogLog data structure.
+// @param key The key for the HyperLogLog data.
+// @param value Output parameter to store the retrieved value.
+// @return Status of the operation.
 Status HyperloglogGet(const Slice& key, std::string* value);
+
+// Sets the value for a key in a HyperLogLog data structure.
+// @param key The key for the HyperLogLog data.
+// @param value The value to set.
+// @return Status of the operation.
 Status HyperloglogSet(const Slice& key, const Slice& value);

Suggestion importance[1-10]: 7
Why: Adding comments improves code readability and maintainability, especially for new methods that may not be self-explanatory. However, this suggestion addresses a minor issue and does not fix any bugs or security concerns.
7
Enhancement

Simplify and clarify the configuration comments for sync-binlog-thread-num

The configuration comments for sync-binlog-thread-num and rsync-timeout-ms are verbose and could be simplified for better readability. Additionally, ensure that the configuration advice is clear and does not contradict other settings or recommended practices.
tests/assets/default.conf [43-45]
-#[NOTICE] It's highly recommended to set sync-binlog-thread-num equal to conf item 'database'(then each DB cloud have a exclusive thread to write binlog),
-# eg. if you use 8 DBs(databases_ is 8), sync-binlog-thread-num is preferable to be 8
-# Valid range of sync-binlog-thread-num is [1, databases], the final value of it is Min(sync-binlog-thread-num, databases)
+# Set sync-binlog-thread-num equal to the number of databases to optimize binlog writing.
+# Ensure the value is within the valid range [1, databases].

Suggestion importance[1-10]: 6
Why: Simplifying and clarifying configuration comments improves readability and helps users understand the settings more easily. However, this is a minor enhancement and does not address any functional issues.
6


What should be listed in setup.py's install_requires?

My previous understanding of setup.py's install_requires
When using pip install, pip will look up the setup.py file in klifs_utils. In this setup.py file I list all package requirements.
So: I thought I am supposed to add all packages there that I am using in klifs_utils. Now I think I am misunderstanding why we list packages there in the first place :)
The source of my confusion
I added rdkit to klifs_utils's setup.py file.
Before (pip install -e klifs_utils is successfull):
install_requires=['pandas', 'bravado', 'biopandas']

Now (pip install -e klifs_utils fails):
install_requires=['pandas', 'bravado', 'biopandas', 'rdkit']

I understand (I think) that it was not a clever idea to add rdkit here since it does not offer a pip installation (it uses conda).
Here the full error message:
$ pip install git+https://github.com/volkamerlab/klifs_utils.git@lib-skeleton
Collecting git+https://github.com/volkamerlab/klifs_utils.git@lib-skeleton
  Cloning https://github.com/volkamerlab/klifs_utils.git (to revision lib-skeleton) to /tmp/pip-req-build-u8i12sro
  Running command git clone -q https://github.com/volkamerlab/klifs_utils.git /tmp/pip-req-build-u8i12sro
  Running command git checkout -b lib-skeleton --track origin/lib-skeleton
  Switched to a new branch 'lib-skeleton'
  Branch 'lib-skeleton' set up to track remote branch 'lib-skeleton' from 'origin'.
Requirement already satisfied: pandas in ./anaconda3/lib/python3.6/site-packages (from klifs-utils==0+untagged.38.g7dc2105) (0.23.0)
Collecting bravado
  Using cached bravado-10.5.0-py2.py3-none-any.whl (36 kB)
Collecting biopandas
  Using cached biopandas-0.2.5-py2.py3-none-any.whl (263 kB)
ERROR: Could not find a version that satisfies the requirement rdkit (from klifs-utils==0+untagged.38.g7dc2105) (from versions: none)
ERROR: No matching distribution found for rdkit (from klifs-utils==0+untagged.38.g7dc2105)

My question
Why do we list packages in the setup.py?

We cannot list all dependencies for the library if e.g. a package cannot be installed via pip.
Apparently the setup.py file in combo with pip will not install packages for us, since we need to install these packages manually anyways (in our case using conda-env). So what is install_requires for?

This is a mental note to myself to do more reading about what pip is doing:
I have multiple branches in my git folder. The setup.py files differ in these branches (e.g. master has no information about install_required).
Even though, pip install seems to run the setup.py from the lib-skeleton branch.
How can pip deal with multiple setup.py files?
I assume it is not choosing lib-skeleton's setup.py file over master's - why should it?

Outside data science, conda is not really popular and everything happens through PyPI and pip. install_requires is the pip-way to specify dependencies. But of course, these dependencies must be available on https://pypi.org/, a repository that has nothing to do with conda. Conda can use pypi though, thanks to the - pip: entry in environment files.
So: conda knows about pip, but not the other way around.
Why do we use both? Well, historic reasons... when we build a conda recipe, we do install our own package with pip, BUT we specify an additional flag to ignore setup_requires:
pip install <package> --no-deps

If you want, you can ignore install_requires for now and focus on the conda env file.
When you specify a git address (or a ZIP file in an online resource, which is usually faster), it defaults to master, so I don't know how it might be getting lib-skeleton. Are you sure?


Cannot read property "find" of undefined react-native

Hello!
I am trying to set up subscriptions on a react-native app but i am receiving the following error:

This is the subscription handler which works perfectly on the web side:
const SubscriptionSocket = async (url, logger) => {
  const API_TOKEN = await AsyncStorage.getItem('API_TOKEN');

  return AbsintheSocket.create(
    new Socket(`${WS_URL}/socket`, {
      logger,
      params: { jwt: API_TOKEN }
    })
  );
}

function subscriptionHandler(subscriptionConfig, variables, cacheConfig, observer) {
  const socket = new SubscriptionSocket(WS_URL, (kind, msg, payload) => {
    if (typeof payload !== 'undefined' && payload.subscriptionId) {
      observer.onNext({ data: payload.result.data });
    } else {
      if (process.env.NODE_ENV === 'development') {
        console.log(`${kind.toUpperCase()} ${msg}`, payload);
      }
    }
  });

  AbsintheSocket.send(socket, { operation: subscriptionConfig.text, variables });

  return {
    dispose: () => socket.phoenixSocket.disconnect(() => {
      if (process.env.NODE_ENV === 'development') {
        console.log(`DISPOSE \n${subscriptionConfig.text}`);
      }
    })
  };
}

This is currently a blocker for us so any help would be appreciated.

I was returning a promise from the SubscriptionSocket constructor, sorry for posting the issue.


change export

I had trouble making the translation works. It give me an error in export default. I replace it with module.exports and it works

@selini good point! I messed the local copy with the module copy and I never checked thinking it is ok since it is forked from aor-language-french

ok great !


Contributing.md file

👨💻 Changes proposed

✔ Check List (Check all the applicable boxes)

[✔ ] My code follows the code style of this project.
[✔ ] This PR does not contain plagiarized content.
[✔ ] The title of my pull request is a short description of the requested changes.

📄 Note to reviewers

📷 Screenshots

No activity closing the PR


"Block not downloaded" for cloning an archive still being creating

Share a folder with large number of files:

$ du -sh
645M	.

$ dat share .
dat v13.6.0
Created new dat in /Users/clkao/src/st/stconfig/git/jirs/.dat
dat://e05ca4d7034e00f3769360010f153dc85d56ebe29e45c1033759eda9ea7653a5
Sharing dat: 39636 files (216 MB)

0 connections | Download 0 B/s Upload 0 B/s

Creating metadata for 77034 files (177 KB/s)
[===================-----------------------] 46%
ADD: 200910/  臺中地方法院刑 8,簡690,20091006,1.json (11 KB)

wait until the initial file count is done.

In a second terminal, clone the archive:

dat clone dat://e05ca4d7034e00f3769360010f153dc85d56ebe29e45c1033759eda9ea7653a5 jirs
dat v13.6.0
Created new dat in /private/tmp/jirs/.dat
Cloning: (empty archive)

1 connection | Download 1.2 MB/s Upload 0 B/s

Downloading updates...
[=-----------------------------------------] 4.31%

Ctrl+C to Exit
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: Block not downloaded
    at Feed.get (/Users/clkao/.nvm/versions/node/v6.10.3/lib/node_modules/dat/node_modules/hypercore/index.js:785:48)
    at Tree._getAndDecode (/Users/clkao/.nvm/versions/node/v6.10.3/lib/node_modules/dat/node_modules/append-tree/index.js:541:13)
    at Tree.head (/Users/clkao/.nvm/versions/node/v6.10.3/lib/node_modules/dat/node_modules/append-tree/index.js:354:47)
    at Tree.list (/Users/clkao/.nvm/versions/node/v6.10.3/lib/node_modules/dat/node_modules/append-tree/index.js:108:8)
    at /Users/clkao/.nvm/versions/node/v6.10.3/lib/node_modules/dat/node_modules/append-tree/index.js:475:16
    at onnodes (/Users/clkao/.nvm/versions/node/v6.10.3/lib/node_modules/dat/node_modules/append-tree/index.js:126:7)
    at get (/Users/clkao/.nvm/versions/node/v6.10.3/lib/node_modules/dat/node_modules/append-tree/index.js:564:10)
    at _combinedTickCallback (internal/process/next_tick.js:83:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)

I know what is causing this. Will try and push a fix tmw

@clkao can you reproduce this?

@mafintosh yes this is quite consistent with a large archive still being shared.  Or do you mean that I should try any updated modules?

@clkao Okay cool, thats good. I have a hard time reproducing myself. Could you share a video of what you do to make this happen?

@clkao follow up: try this: when it fails kill the source process and run the clone again (with no other dat process) and see if the clone throws an error

If the clone fails consistently could you zip up the .dat folder and share with me?

Reinstall node_modules. There is a high chance that this is fixed in hyperdrive 9.4.1 which I just released.

Hopefully this is fixed. Please reopen if still a bug.


Update named functions to arrows functions and remove unecessary bind…

…ings
Move all named functions and avoid manual bindings in constructor.
http://egorsmirnov.me/2015/08/16/react-and-es6-part3.html

looks clean. thank you so much for this PR. appreciate it. 👍


nixos-install doesn't work when home-manager is used as nixos module

The title says it all. I am upgrading my machines to SSD and nixos-18.09, therefore I am using nixos-install. I also have home-manager as nixos module in my configuration. nixos-install fails with the following error:
[0.1/0.0 MiB DL] downloading 'https://github.com/rycee/home-manager/archive/master.tar.gz'nix: src/libexpr/primops.cc:56: void nix::EvalState::realiseContext(const PathSet&): Assertion `store->isStorePath(ctx)' failed.
/run/current-system/sw/bin/nixos-install: line 98:  1416 Aborted               nix build --out-link "$outLink" --store "$mountPoint" "${extraBuildFlags[@]}" --extra-substituters "$sub" -f '<nixpkgs/nixos>' system -I "nixos-config=$NIXOS_CONFIG"

It's pretty unfortunate, as I dreamed of completely automatic installation of my system straight from nixos iso. Is there any fix or workaround or am I missing something obvious?

Actually, if I use builtins.fetchGit instead of builtins.fetchTarball, it works. Maybe we could change instructions on using home-manager as a module?

Apologies for taking so long! Yeah, this seems to be a known issue in Nix. I'm hoping to rework the manual before too long and will make sure to note this.

No problem! Thanks for the reply.

@rycee Any way we could fix this?

The issue in question is https://github.com/NixOS/nix/issues/2405. It will take a while before the fix is merged into any release, as it seems.
Until then a viable workaround is to use builtins.fetchGit. The disadvantage is that it is slightly slower and puts more load on GitHub. Or keep the home-manager repo as a git submodule, as @balsoft does, which is in some ways equivalent.


Fix erroneously wrapped URLs in links

Some of the links in this document contained hard-wrapped URL portions, which causes Markdown to insert a literal URL encoded \n inside the link, thus breaking them.

Can you please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "patch-1" git@github.com:bradleywright/docker.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

This will update the existing PR, so you do not need to open a new one.

Well spotted, thanks!

LGTM

LGTM

LGTM


fix import cwd if no cwd

Fixes https://github.com/Textualize/rich/issues/2197

Codecov Report

Merging #2212 (5100a2b) into master (46b1c95) will decrease coverage by 3.15%.
The diff coverage is 80.57%.

@@            Coverage Diff             @@
##           master    #2212      +/-   ##
==========================================
- Coverage   99.07%   95.91%   -3.16%
==========================================
  Files          72       73       +1
  Lines        7555     7563       +8
==========================================
- Hits         7485     7254     -231
- Misses         70      309     +239

Flag
Coverage Δ

unittests
95.91% <80.57%> (-3.16%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

rich/_windows_renderer.py
0.00% <0.00%> (-100.00%)
:arrow_down:

rich/layout.py
100.00% <ø> (ø)

rich/markup.py
100.00% <ø> (ø)

rich/__init__.py
93.54% <60.00%> (-6.46%)
:arrow_down:

rich/progress.py
92.57% <79.45%> (-0.95%)
:arrow_down:

rich/progress_bar.py
99.00% <85.71%> (-1.00%)
:arrow_down:

rich/console.py
97.18% <86.36%> (-2.03%)
:arrow_down:

rich/_export_format.py
100.00% <100.00%> (ø)

rich/control.py
100.00% <100.00%> (ø)

rich/file_proxy.py
100.00% <100.00%> (ø)

... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0ed9309...5100a2b. Read the comment docs.


Add firstSuccess function to list

I think is a useful function when you have a operation that might for some parameters fail and have to return the first one that works.
import java.security.InvalidParameterException

import scala.annotation.tailrec
import scala.util.{ Failure, Try }

object ListOps {

  implicit class ToListOps[+A](list: List[A]) {

    /**
     * Returns the first success by the processor. It will process each element of the list
     *
     * @param processor A function that will process each parameter of the list until the first success.
     * @param onError   The try value that will be used in case of every process failing.
     * @tparam B The kind of result returned by the processor function.
     * @return The first processor success as a Try[B]. If onError is defined, return its
     *         value in case of failing all toProcess list, otherwise, returns the last failure.
     *         Returns a InvalidParameterException failure in case of a empty toProcess list.
     */
    def firstSuccess[B](processor: A ⇒ B, onError: Option[Try[B]] = None): Try[B] = {
      @tailrec
      def loop(l2: List[A], lastFailure: Try[B]): Try[B] = {
        l2 match {
          case head :: tail ⇒
            val tryResult = Try(processor(head))
            if (tryResult.isSuccess) tryResult else loop(tail, onError.getOrElse(tryResult))
          case _ ⇒ lastFailure
        }
      }

      loop(list, Failure(new InvalidParameterException("Empty list is not allowed")))
    }
  }

}

//Usage exemple:
//val lines = Settings.supportedUploadEncodings.firstSuccess(encoding ⇒ Source.fromFile(file, encoding).getLines().toList).get

Ah nice function. Maybe we can add it in an "ops" package? like: purity.ops.ListOps


[HELP] ESP32C3 sleep after software reset

Description
I tested LUATOS ESP32C3 and ESP32-C3-mini
I tried these

boardctl(BOARDIOC_RESET, 0);
esp_restart();
esp_rom_software_reset_cpu(0);

But it sleep always.
I tried esp_restart from esp-idf project, it works normally.
So how to know where is the bug? NuttX or MCUboot ?
esp32c3-legacy works normally also.
Verification

[X] I have verified before submitting the report.

Hi,
Please check esp32c3-generic:sta_softap defconfig. It enables the reboot command (just type reboot). It uses the boardctl(BOARDIOC_RESET, 0) interface for that.
If it still does not work, please enable the following configs in menuconfig:
CONFIG_SYSTEM_NXDIAG=y
CONFIG_SYSTEM_NXDIAG_COMP_FLAGS=y
CONFIG_SYSTEM_NXDIAG_CONF=y
CONFIG_SYSTEM_NXDIAG_ESPRESSIF=y
CONFIG_SYSTEM_NXDIAG_HOST_PATH=y

And run in NuttX nxdiag --all (sharing your results here).
Also, please double-check which hardware exactly you are using (Is it a module? a board? Which Espressif module/chip is being used?)

Thank you
I found the problem is CONFIG_ELF now.
I will try where the problem is.

Skip this, reboot works.
#ifdef CONFIG_ELF
  ret = elf_initialize();
  if (ret < 0)
    {
      berr("ERROR: elf_initialize failed: %d\n", ret);
    }
#endif

But skip here, it will sleep
int elf_initialize(void)
{
  int ret;

  /* Register ourselves as a binfmt loader */

  binfo("Registering ELF\n");

  ret = register_binfmt(&g_elfbinfmt);
  if (ret != 0)
    {
      berr("Failed to register binfmt: %d\n", ret);
    }

  return ret;
}

Very strange...

I close this issue, it's very strange, and now it works normally...

I guest the problem is mcuboot
the ESP32-C3 generic is supported mcuboot only.
And if firmware is over than 1MB
It stuck at mcuboot (after the esp32 rom bootloader)

Hi @metarutaiga ,
esp-idf is not required to build MCUboot. Can you share exactly the steps you are doing? Also, can you please post your defconfig (run  make savedefconfig and a new defconfig will be generated)? (If not possible, please share a reduced version of it with the same error)

I used the mcuboot repository to build it directly.
Any MCUboot version is not affect the result.
I known it's hang not sleep now.
I am sure that if binary size is over than 1MB, it hangs when reboot.

I used the mcuboot repository to build it directly. Any MCUboot version is not affect the result.
I known it's hang not sleep now. I am sure that if binary size is over than 1MB, it hangs when reboot.

Right, can you provide the steps to reproduce and a failing branch of nuttx and nuttx-apps, please?

https://github.com/NyankoLab/esp32c3-nuttx
if it enabled wget and uiP web Client, it would be over 1MB.

Hi @metarutaiga , I couldn't reproduce your issue. I need clearer and more detailed instructions.
I tried:

esp32c6-devkitc:sta_softap, enabling wget, uiP web client and mbedTLS (with example applications). Firmware is over 1MB and the device boots successfully and reboot commands reboots the device successfully.
esp32c6-devkitc:sta_softap, enabling wget, uiP web client and mbedTLS (with example applications) and MCUboot (CONFIG_ESPRESSIF_BOOTLOADER_MCUBOOT=y). I adjusted the slots accordingly (default slot size is 1MB): CONFIG_ESPRESSIF_OTA_SCRATCH_OFFSET=0x410000,  CONFIG_ESPRESSIF_OTA_SECONDARY_SLOT_OFFSET=0x210000 and CONFIG_ESPRESSIF_OTA_SLOT_SIZE=0x200000 to 2MB. Same result: device boots as expected and reboots successfully.

Hmmm, it's almost the same instructions.
I give up to find the problem.


Plan to drop support for Python 2.7

Python 2.7 (and therefore all Python 2) reaches EOL on January 1st, 2020.

https://github.com/python/devguide/pull/344
https://pythonclock.org
http://python3statement.org

OSMViz 3.x will drop support for Python 2.7 before that, perhaps this year. There will still be an installable Python 2.7 version on PyPI (OSMViz 2.x), but no further updates will be made for 2.7.

Here's the pip installs for OSMViz from PyPI for May 2018:

python_version
percent
download_count

2.7
52.27%
46

3.5
18.18%
16

3.4
18.18%
16

3.6
11.36%
10

Total

88

Source: pypinfo --start-date 2018-05-01 --end-date 2018-05-31 --percent --markdown osmviz pyversion

Pre-drop:

[x] setuptools >= 24.3
[x] pip >= 9.0.1
[x] Update docs and scripts to use pip
[x] Runtime warning on master, link here
[x] Pre-drop release

Drop:

[x] Fail early at import time
[x] Fail early in setup.py
[x] Remove Python 2 code
[x] Update python_requires and Trove classifiers
[x] Change deprecation warning to error
[x] Bump major version to 3.0
[x] Release

OSMViz 2.1.0 with a deprecation warning for Python 2 is released.
Python 2 support has been removed from master and will be released as OSMViz 3.0.0 at some point.

OSMViz 3.0.0 released.


Support for entering data as series and for multiple series

feature1.rb and feature2.rb both work correctly now.
I chose to store the data internally as an array of points, where each point has one x-coordinate followed by arbitrarily many y-coordinates corresponding to the number of series displayed on the graph. By default, all series are represented by Xs.

God, you're so impatient!

😢


Add DeviceRef::new_texture_with_descriptor_and_iosurface

Add DeviceRef::new_texture_with_descriptor_and_iosurface

https://developer.apple.com/documentation/metal/mtldevice/1433425-newtexturewithdescriptor

IOSurfaceRef is already defined in:
https://github.com/servo/core-foundation-rs/tree/main/io-surface
So importing as dependency.

The comment and link in your description are wrong, those are already there. In this PR you're adding:
https://developer.apple.com/documentation/metal/mtldevice/1433378-newtexturewithdescriptor
with signature:
- (id<MTLTexture>)newTextureWithDescriptor:(MTLTextureDescriptor *)descriptor
                                 iosurface:(IOSurfaceRef)iosurface
                                     plane:(NSUInteger)plane;`

Renamed commit message and PR title/message.

Also, ideally IOSurface is included in core-foundation-types...


Pmr results mixed updated

These commits are included in https://github.com/ABI-Software/map-sidebar/pull/81


Target Encoder Giving Nan values

Expected Behavior
Target Encoder giving Nan values for few inputs
Same Issue:

https://stackoverflow.com/questions/68261917/why-is-target-encoder-encoding-some-values-as-nan

https://www.kaggle.com/questions-and-answers/204970

Version: Latest Pip Install Feb 2022 version
Platform: Linux
Subsystem: N/A

Hi @shauryauppal
Could you please also provide a dataset or even better a self-contained reproducible (minimal) example? Neither in the stackoverflow nor in  the kaggle post the dataset is mentioned. (except for a reference to the kaggle housing prices competition which I can't seem to find)

Maybe sigmoid function is numerically unstable in some cases, maybe using something like
scipy.special.expit(stats['count'] - self.min_samples_leaf) / self.smoothing)
in here
https://github.com/scikit-learn-contrib/category_encoders/blob/02a20aa96c5f1f234ec89a0f781980622e3b193a/category_encoders/target_encoder.py#L170
could be beneficial, both in terms of speed and stability. It will introduce dependency on scipy though.
But without minimal reproducible example it's a needle in a haystack.

neither stats["count"] nor self.smoothing should be 0. The former cannot even be 0 while for the second the documentation clearly states The value must be strictly bigger than 0. Without a reproducible example by @shauryauppal we cannot do anything here


deployApps - version infos from MANIFEST are not taken into account

Deploying libraries with deployApps.sh (Version 1.9.5) online works not correctly for libraries.
Extension-Name, Implementation-Version and Specification-Version from the MANIFEST.mf are not attached to the library name during the deployment as it is done for apps.
The MANIFEST.mf:
Manifest-Version: 1.0
Created-By: Maven WAR Plugin 3.3.0
Build-Jdk-Spec: 11
Extension-Name: xyz_lib_v1_jdk11_4.0
Implementation-Version: 4.0
Specification-Version: 1.0

The model:
appDeployments:
    Library:
        'xyz_lib_v1_jdk11':
            SourcePath: 'wlsdeploy/sharedLibraries/xyz_lib_v1_jdk11.war'
            Target: 'xxx_cluster'
            ModuleType: war
    Application:
        'xyz_v1_jdk11':
            SourcePath: 'wlsdeploy/applications/xyz_v1_jdk11.war'
            Target: 'xxx_cluster'
            ModuleType: war

The deploy command:
/home/oracle/weblogic-deploy/bin/deployApps.sh -oracle_home /u01/app/oracle/middleware -domain_type WLS -domain_home /u01/app/oracle/middleware/user_projects/domains/xyz_domain -archive_file xyz_v1.zip -model_file deploy_app.yaml -admin_url t3://n.nnn.nnn.nnn:nnnn -admin_user ABC`

The error message
 1. WLSDPLY-09015: deployApps deployment failed: deployApplication(xyz_lib_v1_jdk11, (), {'targets': 'xxx_cluster', 'path': '/u01/app/oracle/middleware<IP_ADDRESS>.0/user_projects/domains/xyz_domain/wlsdeploy/sharedLibraries/xyz_lib_v1_jdk11.war', 'libraryModule': 'true'}) failed: Error occurred while performing deploy : Target exception thrown while deploying application: Error occurred while performing deploy : Deployment Failed. : Error occurred while performing deploy : Deployment Failed.

The deployment with the following work around is successful
appDeployments:
    Library:
<EMAIL_ADDRESS>            SourcePath: 'wlsdeploy/sharedLibraries/xyz_lib_v1_jdk11.war'
            Target: 'xxx_cluster'
            ModuleType: war
    Application:
        'xyz_v1_jdk11':
            SourcePath: 'wlsdeploy/applications/xyz_v1_jdk11.war'
            Target: 'xxx_cluster'
            ModuleType: war

In this case Extension-Name, Implementation-Version and Specification-Version from the MANIFEST.mf are specified explicitly in the library name.
The problem is located in applications_deployer.py in function __deploy_app_online which is used for app and lib deployment in the following lines:
        # if options is not None and 'libraryModule' in options and string_utils.to_boolean(options['libraryModule']):
        computed_name = self.__get_deployable_application_versioned_name(source_path, application_name)
        application_name = computed_name

Here the function for getting the versioned library name is not called. I fixed this line locally but then came another problem: the program did not detect that the library was already deployed.

The changes to prevent the redeployment are going to require some refactoring, and will take longer.
Your workaround of putting the version information in the library name in the model seems to compensate for this as well, but is not ideal if the underlying version of the library changes.

I checked out your branch and tested it with the above configuration but leaving out the version configuration in the library name. It's working now when deleting the lib before running the script.
With the latter we can cope in our environment. Probably you will change it in a future release ...


allow use of notification matcher with async verifier in constructs l…

…ike [[@"my notification" shouldEventually] bePosted]; or even shouldNotEventually by keeping observer alive when evaluted multiple times until first notification received
Resolve #592

+1 for this PR. I need that in my project. I had to create a custom matcher for this purpose.


URL not valid.

Under this section:
https://docs.microsoft.com/en-us/windows-hardware/drivers/wdf/accessing-umdf-metadata-in-wer-reports#wudfunhandledexception-fields.
The link of EXCEPTION_RECORD is not valid.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a7111c17-937a-6da3-c4f3-1627d4a3e131
Version Independent ID: 057b3ac8-0ddf-abb6-6312-132aac418566
Content: Accessing UMDF Metadata in WER Reports - Windows drivers
Content Source: windows-driver-docs-pr/wdf/accessing-umdf-metadata-in-wer-reports.md
Product: windows-hardware
Technology: windows-devices
GitHub Login: @tedhudek
Microsoft Alias: tedhudek

Thanks for reporting this, @ivangan-cpl
And thanks for the PR with the fix, @RAJU2529
Eliot


Fix linting from prior PR

Honestly, both of these linting rules are annoying to me, but we have followed them elsewhere so I'm adding this for the sake of consistency

Suddenly I'm seeing that linting is passing in the pipeline on main, but failing locally... what the heck is going on...

Interesting... for some reason I had a newer version of golangci-lint in my .hermit/ directory. @alecthomas are there any docs around what the .hermit/ directory is and the DX? (briefly poked around here but didn't see anything https://cashapp.github.io/hermit/usage/) I presume it's a cache of sorts, but odd that it gets out-of-whack with what was pinned in the bin/ directory.
I've gone ahead and updated our golangci-lint in this PR's most recent commit
For proof that these changes are needed https://github.com/TBD54566975/web5-go/actions/runs/8068904849

That directory is where binaries installed by go install end up. Something (not Hermit) on your system installed golangci-lint explicitly. It's nothing to do with Hermit except that it overrides GOBIN and adds it to PATH.


Unable to reach ganache-cli in a docker container from docker compose

I asked about this on gitter: [X]
Link to my question on gitter: https://gitter.im/ConsenSys/truffle?at=5f47a4035580fa092b2896e0

Expected Behavior
Both memento and explorer must be connected to ganache-cli.
Current Behavior
For Memento(docker logs):
time="2020-08-27T12:24:40Z" level=info msg="[eth] starting best block tracker"
time="2020-08-27T12:24:40Z" level=info msg="[eth] setting up websocket connection"
time="2020-08-27T12:24:41Z" level=warning msg="error connecting to server: dial tcp <IP_ADDRESS>:8546: connect: connection refused "

For Explorer(in-browser console):
App.ts:25 Couldn't load application config SyntaxError: Unexpected token / in JSON at position 84
   at JSON.parse (<anonymous>)
   at _i.<anonymous> (HttpRequest.ts:101)
   at Generator.next (<anonymous>)
   at a (HttpRequestError.ts:2)

It means they cannot connect properly to  ganache-cli.
Steps to Reproduce (for bugs)

Launch up the network: docker-compose -f docker-compose-explorer.yml up
Review the docker-compose logs to see the memento container logs(you can use Kitematic).
Go to the localhost in browser and check the developer console.

Context
This is my docker compose file:
version: "3.7"
services:
  redis:
    container_name: memento-redis
    image: redis:5.0.5-alpine
    restart: always
    command: ["redis-server", "--appendonly", "yes"]
    volumes:
      - .volumes/redis:/data
  postgres:
    container_name: memento-postgres
    image: postgres:11.5-alpine
    restart: always
    environment:
      - POSTGRES_USER=memento
      - POSTGRES_DB=memento
      - POSTGRES_PASSWORD=password
    volumes:
      - .volumes/postgresql:/var/lib/postgresql/data
  ganache-cli:
    image: trufflesuite/ganache-cli
    container_name: ganache-cli
    ports:
      - 8545:8545
      - 8546:8546
  memento:
    container_name: memento
    image: alethio/memento:latest
    restart: always
    depends_on:
      - redis
      - postgres
    ports:
      - <IP_ADDRESS>:3001:3001
      - <IP_ADDRESS>:3000:3000
    environment:
      - PG_PASSWORD=password
    volumes:
      - .volumes/memento:/config
  explorer:
    container_name: memento-lite-explorer
    image: alethio/ethereum-lite-explorer:latest
    restart: always
    depends_on:
      - memento
      - ganache-cli
    ports:
      - <IP_ADDRESS>:80:80
    environment:
      - APP_NODE_URL="http://<IP_ADDRESS>:8545"
    volumes:
      - .volumes/lite-explorer/config.json:/usr/share/nginx/html/config.json

Also, i opened an issue in memento repository and i found the next issue.
Your Environment

Version used:v6.10.1 (ganache-core: 2.11.2)
Version of Truffle/Remix/Other tools used:
NodeJS Version: [ ] 6.x, [ ] 7.x (unsupported), [ ] 8.x, [ ] 9.x
Operating System and version (include distro if Linux):

it seems that I can't reproduce this as there is missing configuration. What do your various config files look like? make sure to sensor out any sensitive information

I have this in memento configuration file:
# api-related fields
api:
  # The port on which the API will be exposed (default:3001)
  port: 3001

  # Enable/disable development CORS
  dev-cors: true

  # Allowed hosts for HTTP API development CORS
  dev-cors-host: "*"

# Dashboard-related fields
dashboard:
  # The port on which the Dashboard will be exposed (default:3000)
  # WARNING: it must not be the same as api.port!
  port: 3000

  config-management:
    enabled: true

# database fields
db:
  # Database host
  host: "memento-postgres"

  # Database port
  port: 5432

  # Database sslmode
  sslmode: "disable"

  # Database name
  dbname: "memento"

  # Database user (also allowed via PG_USER env var)
  user: "memento"

# ethereum-related fields
eth:
  client:
    # HTTP endpoint of a web3-compatible node (mandatory)
    http: "http://<IP_ADDRESS>:8545"

    # Websockets endpoint of JSON-RPC enabled node (optional)
    # if this is present, the task manager will use a newHeads subscription to watch for new blocks
    # if not present, the task manager will use polling to get the highest block
    ws: "ws://<IP_ADDRESS>:8546"

    # The interval at which the task manager will poll the node for best block (default:"15s")
    # optional, only used if `ws` url is not specified
    poll-interval: "15s"

# feature flags
feature:
  # Backfilling
  backfill:
    # Enable/disable the backfilling feature
    enabled: false

  # Lag feature
  lag:
    # Enable/disable the lag feature
    enabled: true

    # The amount of blocks to lag behind the tip of the chain if the lag feature is enabled
    value: 10

  # Automatic migrations
  automigrate:
    # Enable/disable the automatic migrations
    enabled: true

  # Uncles scraping
  uncles:
    # Enable/disabled the uncles scraping
    enabled: false

# Control what to be logged using format "module=level,module=level"; `*` means all other modules
logging: "*=info"

# redis-related fields
redis:
  # URL of redis server (default:"localhost:6379")
  server: "memento-redis:6379"

  # The name of the redis list the scraper will use for task management (default: "todo")
  list: "todo"

And this in the explorer configuration file:
{
  "GOOGLE_ANALYTICS_ID": null,
  "HOTJAR_ID": null,
  "SENTRY_ENABLED": false,

  // "nodeUrl": "http://<IP_ADDRESS>:8545",
  // "APP_NODE_URL": "http://localhost:8545",

  "pluginsUrl": "plugins",

  "plugins": [
    {
      "uri": "plugin://aleth.io/eth-common?v=4.0.0"
    },
    {
      "uri": "plugin://aleth.io/3box?v=1.1.1",
      "config": {
        "ipfsUrlMask": "https://ipfs.infura.io/ipfs/%s"
      }
    },
    {
      "uri": "plugin://aleth.io/eth-memento?v=2.0.2",
      "config": {
        "apiBasePath": "http://localhost:3001/api/explorer"
      }
    }
  ],

  "pages": [
    {
      "def": "page://aleth.io/block",
      "children": {
        "sidebar": [
          {
            "def": "context://aleth.io/block/list",
            "children": [{ "def": "module://aleth.io/block/list" }]
          }
        ],
        "content": [
          {
            "def": "module://aleth.io/memento/block/basic",
            "pageCritical": true,
            "children": {
              "confirmations": [
                { "def": "module://aleth.io/block/confirmations" }
              ]
            }
          },
          {
            "def": "module://aleth.io/memento/block/txs",
            "pageCritical": true
          },
          {
            "def": "module://aleth.io/memento/block/advanced",
            "pageCritical": true
          },
          {
            "def": "module://aleth.io/memento/block/logs-bloom",
            "pageCritical": true
          }
        ]
      }
    },
    {
      "def": "page://aleth.io/uncle",
      "children": {
        "content": [
          {
            "def": "module://aleth.io/memento/uncle/details",
            "pageCritical": true
          }
        ]
      }
    },
    {
      "def": "page://aleth.io/tx",
      "children": {
        "sidebar": [
          {
            "def": "context://aleth.io/memento/tx/parentBlock",
            "children": [
              {
                "def": "module://aleth.io/tx/list"
              }
            ]
          }
        ],
        "content": [
          {
            "def": "context://aleth.io/memento/tx/parentBlock",
            "pageCritical": true,
            "children": [
              {
                "def": "module://aleth.io/memento/tx/basic",
                "children": {
                  "blockConfirmations": [
                    {
                      "def": "context://aleth.io/memento/tx/parentBlock",
                      "children": [
                        {
                          "def": "module://aleth.io/block/confirmations"
                        }
                      ]
                    }
                  ]
                }
              },
              {
                "def": "module://aleth.io/memento/tx/advanced"
              },
              {
                "def": "module://aleth.io/memento/tx/summary"
              },
              {
                "def": "module://aleth.io/memento/tx/payload"
              }
            ]
          }
        ]
      }
    },
    {
      "def": "page://aleth.io/account",
      "children": {
        "identicon": [
          {
            "def": "module://aleth.io/3box/profile",
            "children": {
              "fallback": [{ "def": "module://aleth.io/account/identicon" }]
            }
          }
        ],
        "top": [
          {
            "def": "module://aleth.io/memento/account/details",
            "pageCritical": true
          }
        ],
        "bottom": [
          { "def": "module://aleth.io/memento/account/txs" },
          { "def": "module://aleth.io/memento/account/contract" }
        ]
      }
    },
    {
      "def": "page://aleth.io/dashboard",
      "children": {
        "content": [
          { "def": "module://aleth.io/search" },
          {
            "def": "context://aleth.io/dashboard/latestBlockRange",
            "children": [
              {
                "def": "module://aleth.io/dashboard/charts",
                "children": {
                  "content": [
                    {
                      "def": "module://aleth.io/dashboard/blocksChart",
                      "children": {
                        "children": [
                          {
                            "def": "context://aleth.io/block/latest",
                            "children": [
                              {
                                "def": "module://aleth.io/dashboard/latestBlockInfo"
                              }
                            ]
                          }
                        ]
                      }
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    }
  ],

  "rootModules": {
    "toolbarTop": [
      {
        "def": "module://aleth.io/toolbar/search"
      }
    ],
    "toolbarBottom": [
      {
        "def": "module://aleth.io/toolbar/feedback"
      }
    ],
    "topbar": [
      {
        "def": "module://aleth.io/topbar/search"
      },
      {
        "def": "module://aleth.io/topbar/feedback"
      }
    ]
  }
}

I followed the tutorials, i copied the default's configuration and I changed just what i need to.

Ganache listens only on a single port at a time (8545 by default). The config here looks like it expects http to be over 8545 only and websockets to be over 8546 only. Try changing the ws: "ws://<IP_ADDRESS>:8546" line to ws: "ws://<IP_ADDRESS>:8545"`.

@SalahAdDin In addition to the 8545/8546 port stuff @davidmurdoch mentioned, it looks like the core of your issues circulates around incorrect docker configuration/misunderstanding how docker works rather than issues with ganache or memento
In both the explorer and memento services, you point to ganache using <IP_ADDRESS>. Each of these servers should be considered as separate machines on the same network. Ganache isn't running on the memento machine for example, it's running on the ganache-cli machine/service
Docker lets you refer to them by name, so you just need to change .volumes/memento/config.yml to have http://ganache-cli:8545 for both the http and ws urls. You also need to change your APP_NODE_URL environment variable for the explorer service in your docker compose to - APP_NODE_URL="http://ganache-cli:8545"
Further, the error that you receive in the webapp:
App.ts:25 Couldn't load application config SyntaxError: Unexpected token / in JSON at position 84
   at JSON.parse (<anonymous>)
   at _i.<anonymous> (HttpRequest.ts:101)
   at Generator.next (<anonymous>)
   at a (HttpRequestError.ts:2)

Is happening because your config.json is not valid JSON. JSON doesn't contain comments, so the JSON parser is like "wut" when it sees the // on line 6 of your config file. Removing those lines fixes that error
After removing those lines, I do end up getting an error {"data":"sql: no rows in result set","status":500}, but this is because there aren't any blocks yet in the memento db. After I mined 10 blocks (because you have the lag configured), I was able to see something in the UI
Hope this helps!

@seesemichaelj don't close this yet, please.
I made the changes you said and even so:
time="2020-08-29T16:53:42Z" level=info msg="[eth] starting best block tracker"
time="2020-08-29T16:53:42Z" level=info msg="[eth] setting up websocket connection"
time="2020-08-29T16:53:43Z" level=warning msg="error connecting to server: dial tcp <IP_ADDRESS>:8545: connect: connection refused "
time="2020-08-29T16:53:50Z" level=info msg="[taskmanager] setting up redis connection"
time="2020-08-29T16:53:50Z" level=info msg="[taskmanager] connected to redis successfully"
time="2020-08-29T16:53:50Z" level=info msg="[core] connecting to postgres"
time="2020-08-29T16:53:50Z" level=fatal msg="[core] dial tcp <IP_ADDRESS>:5432: connect: connection refused"
time="2020-08-29T16:53:53Z" level=info msg="[eth] starting best block tracker"
time="2020-08-29T16:53:53Z" level=info msg="[eth] setting up websocket connection"
time="2020-08-29T16:53:53Z" level=info msg="[taskmanager] setting up redis connection"
time="2020-08-29T16:53:53Z" level=info msg="[taskmanager] connected to redis successfully"
time="2020-08-29T16:53:53Z" level=info msg="[core] connecting to postgres"
time="2020-08-29T16:53:53Z" level=info msg="[core] attempting automatic execution of migrations"
2020/08/29 16:53:53 OK    00001_create_table_blocks.go
2020/08/29 16:53:53 OK    00002_create_table_uncles.go
2020/08/29 16:53:53 OK    00003_create_table_txs.go
2020/08/29 16:53:53 OK    00004_create_table_log_entries.go
2020/08/29 16:53:53 OK    00005_create_table_account_txs.go
2020/08/29 16:53:53 OK    00006_delete_block_function.go
2020/08/29 16:53:53 goose: no migrations to run. current version: 6
time="2020-08-29T16:53:53Z" level=info msg="[core] database version is up to date"
time="2020-08-29T16:53:53Z" level=info msg="[core] connected to postgres successfuly"
time="2020-08-29T16:53:53Z" level=info msg="[core] got highest block from db" block=0
time="2020-08-29T16:53:53Z" level=info msg="[core] got highest block from network" block=0
time="2020-08-29T16:53:53Z" level=info msg="[core] skipping backfilling since feature is disabled"
[GIN] 2020/08/29 - 16:54:37 | 500 |   22.157933ms |      <IP_ADDRESS> | GET      /api/explorer/block/latest
[GIN] 2020/08/29 - 16:54:45 | 200 |   11.596716ms |      <IP_ADDRESS> | GET      /
[GIN] 2020/08/29 - 16:54:45 | 200 |   67.940617ms |      <IP_ADDRESS> | GET      /web/assets/css/index.css
[GIN] 2020/08/29 - 16:54:45 | 200 |   94.735523ms |      <IP_ADDRESS> | GET      /web/assets/css/tailwind-overwrite.css
[GIN] 2020/08/29 - 16:54:45 | 200 |   63.449152ms |      <IP_ADDRESS> | GET      /web/assets/js/progressbar.min.js
[GIN] 2020/08/29 - 16:54:45 | 200 |   63.153452ms |      <IP_ADDRESS> | GET      /web/assets/images/logo.svg
[GIN] 2020/08/29 - 16:54:48 | 200 |     309.287µs |      <IP_ADDRESS> | GET      /web/assets/images/favicon.ico
[GIN] 2020/08/29 - 16:55:27 | 200 |   17.991228ms |      <IP_ADDRESS> | GET      /queue
[GIN] 2020/08/29 - 16:55:32 | 200 |    2.684777ms |      <IP_ADDRESS> | GET      /config
[GIN] 2020/08/29 - 16:55:57 | 200 |    4.410092ms |      <IP_ADDRESS> | GET      /
[GIN] 2020/08/29 - 16:56:10 | 500 |       865.1µs |      <IP_ADDRESS> | GET      /api/explorer/block/latest
time="2020-08-29T16:57:48Z" level=warning msg="[eth] WS connection closed"
time="2020-08-29T16:57:48Z" level=info msg="[eth] setting up websocket connection"
time="2020-08-29T16:57:48Z" level=warning msg="error connecting to server: dial tcp <IP_ADDRESS>:8545: connect: connection refused "
time="2020-08-29T16:57:48Z" level=info msg="[main] Got stop signal. Finishing work."
time="2020-08-29T16:57:48Z" level=info msg="[core] closed best block tracker"
time="2020-08-29T16:57:48Z" level=info msg="[core] closed db connection"
time="2020-08-29T16:57:48Z" level=info msg="[core] closed task manager"
time="2020-08-29T16:57:48Z" level=info msg="[main] Work done. Goodbye!"
time="2020-08-29T16:58:49Z" level=info msg="[eth] starting best block tracker"
time="2020-08-29T16:58:49Z" level=info msg="[eth] setting up websocket connection"
time="2020-08-29T16:58:49Z" level=warning msg="error connecting to server: dial tcp <IP_ADDRESS>:8545: connect: connection refused "
time="2020-08-29T16:58:54Z" level=info msg="[taskmanager] setting up redis connection"
time="2020-08-29T16:58:54Z" level=info msg="[taskmanager] connected to redis successfully"
time="2020-08-29T16:58:54Z" level=info msg="[core] connecting to postgres"
time="2020-08-29T16:58:54Z" level=info msg="[core] attempting automatic execution of migrations"
2020/08/29 16:58:54 goose: no migrations to run. current version: 6
time="2020-08-29T16:58:54Z" level=info msg="[core] database version is up to date"
time="2020-08-29T16:58:54Z" level=info msg="[core] connected to postgres successfuly"
time="2020-08-29T16:58:54Z" level=info msg="[core] got highest block from db" block=0
time="2020-08-29T16:58:54Z" level=info msg="[core] got highest block from network" block=0
time="2020-08-29T16:58:54Z" level=info msg="[core] skipping backfilling since feature is disabled"
[GIN] 2020/08/29 - 16:59:51 | 500 |     388.819µs |      <IP_ADDRESS> | GET      /api/explorer/block/latest

Connection refused by the ganache-cli.

@SalahAdDin Unfortunately this isn't an issue with ganache-cli, so I'm going to keep this issue closed. The issue lies in either your memento or docker configuration. However, I did publish my full configuration at https://github.com/seesemichaelj/memento-docker-compose in hopes that you can figure out what it is that you have configured wrong. This configuration works perfectly for me. Hope it helps!

@SalahAdDin man, yours does not work for me hahahahaha
I cloned it and postgresql gets a problem because it has .gitignore file inside, explorer is not accessible, and memento does not show any track, but ganache is getting transactions, it is something.

Ah, truthfully, I didn't test it after I added those .gitignore files 😬
Also, I changed the exposed port for explorer to be 8080, so try going to http://localhost:8080 instead.  I had to create at least 10 blocks before any of them started getting logged

If you wanted to get past the 10 block thing you can disable lag altogether or change the number of blocks here https://github.com/seesemichaelj/memento-docker-compose/blob/master/.volumes/memento/config.yml#L61-L66

@seesemichaelj i tested your configuration and i got that bug, with your own configuration, i didn't touch it.

I don't know what bug your'e talking about. If you're talking about the .gitignore files, feel free to remove them before you try running it. all that matters is the configuration files. They work for me, but your machine may vary. Unfortunately we don't support generic Docker or Memento configuration

THis is really very tricky, now i have a problem with the explorer:
app.bundle.js?34631def42855790e829:477 Couldn't load application config ProgressEvent
(anonymous) @ app.bundle.js?34631def42855790e829:477
/config.json:1 Failed to load resource: net::ERR_CONNECTION_REFUSED

Unfortunately, they are not answering issues in the main repo, It's sad that ganache-cli doesn't offer its own web interface.


Update webdriverio default version

Motivation/Description of the PR

Update webdriverio default version to 6.12.1. As it is pointing to a deprecated version, that makes the setup not working.

Applicable helpers:

[x] WebDriver
[ ] Puppeteer
[ ] Nightmare
[ ] REST
[ ] FileHelper
[ ] Appium
[ ] Protractor
[ ] TestCafe
[ ] Playwright

Applicable plugins:

[ ] allure
[ ] autoDelay
[ ] autoLogin
[ ] customLocator
[ ] pauseOnFail
[ ] coverage
[ ] retryFailedStep
[ ] screenshotOnFail
[ ] selenoid
[ ] stepByStepReport
[ ] wdio
[ ] subtitles

Type of change

[ ] :fire: Breaking changes
[ ] :rocket: New functionality
[ ] :bug: Bug fix
[ ] :clipboard: Documentation changes/updates
[ ] :hotsprings: Hot fix
[ ] :hammer: Markdown files fix - not related to source code
[ ] :nail_care: Polish code

Checklist:

[x] Tests have been added
[x] Documentation has been added (Run npm run docs)
[x] Lint checking (Run npm run lint)
[x] Local tests are passed (Run npm test)

Thanks!


Provide an option to use a common Ceph credential across CephFS subvolumes

Currently, each CephFS volume gets a sub-directory under the CephFS root and further gets a user created with rw access to the directory (and a few other rights as required to access the directory). For each stage and publish request of such a sub-directory to a node, the CephFS root volume is mounted, and then the sub-directory is bind mounted into the requested directory, using the user credentials created for this volume.
In certain situations where the volume counts are high it is desirable to not have the additional per directory Ceph user, and instead leverage a single user that can mount the sub-directory, leaving kuberetes (or other container orchestration systems) to deal with access restrictions on bind mounted paths.
The above scheme also lends itself to staging the CephFS root once during staging the volume the first time on a node, and bind mounting sub-directories for each volume that recieves further publish requests on the same node. Again at scale, such a scheme helps, as it reduces the number of CephFS mounts on a node.
Thoughts on implementation:

The option itself can be provided as a run-time option to the CSI plugin, thus reducing overhead on one more parameter to specify in the StorageClass and can also optionally be controlled by the StorageClass.

The volume_options in CreateVolume response can carry the boolean regarding the nature of secret to use to mount the volume, and hence node level plugin can make a dynamic decision regarding the same.

I had a similar idea in mind, but wanted to wait till we have cluster IDs in place - those could be used to simplify things down further.

:) totally agree, this can wait. I am just filing away stuff that is in my notes for consideration. Thanks!

@ShyamsundarR can we close this issue now?

Fixed, closing.


Add metrics with extra non-scalar arguments

Some metrics take extra non-scalar arguments (e.g. vectors and arrays). There will need to be some adjustments both to the library and the tests to support this behavior. Have listed the metrics not yet implemented below.

[x] mahalanobis ( https://github.com/jakirkham/dask-distance/pull/58 )
[x] seuclidean ( https://github.com/jakirkham/dask-distance/pull/56 )
[x] wminkowski ( https://github.com/jakirkham/dask-distance/pull/57 )

With seuclidean and wminkowski, the functionality provided by these can be easily achieved by simply rescaling using the weights before passing it to euclidean or minkowski. So there is a fair question as to whether these are worth handling at all.

Have some WIP PRs for these three functions (mahalanobis, seuclidean, wminkowski) linked by the cross references above. These all need tests and may need some optimization.

All are now added with tests. This is complete.


treewide: Remove fetchurl of SourceForge snapshots

Description of changes
I have a machine which builds everything without substituters, to test that nixpkgs actually works. I discovered that sox failed to build because snapshots from https://sourceforge.net/code-snapshots are generated by interaction with the web UI and they go away after a while. This PR changes every fetchzip against https://sourceforge.net/code-snapshots/... to fetch from the upstream source repo. I have confirmed that the source hashes did not change.
Things done

Built on platform(s)

[X] x86_64-linux

[X] Confirmed that source hashes did not change.

These are niche packages and the hash of their source did not change. Can someone explain OfBorg's high rebuild counts to me?

This is a dupe of #240766 and someone else fixed enblend-enfuse, so I'll close this PR.

What a coincidence!

These are niche packages and the hash of their source did not change. Can someone explain OfBorg's high rebuild counts to me?

@endgame, since you asked:
While the hash did not change, the store path did.
fetchzip: /nix/store/06d2xkrl3h1pbyg3m1whz1mvhafqgsqq-source
fetchgit: /nix/store/82jj7c66cwwp6pf6vi4p97j9isl7a2mq-code-42b3557
Which is what changes the build graph and thus triggers the rebuild of sox.
And a lot of packages directly or indirectly depend on that, thus resulting in a huge amount of rebuilds.
This can be prevented by making this a no-op, by telling fetchgit to use source instead of code-42b3557 in the store path.
See https://github.com/NixOS/nixpkgs/pull/240766#discussion_r1248976876 and the follow-up #241106 :)
Thank you for your contribution nonetheless!

@emilylange Thank you for the explanation.


add percent button

on the symbols page of the keyboard you have two '}' buttons and no '%' button.

Thanks for pointing this out.  Fixed


config.now retained

config.now (current time) is retained even when calling datepicker.destroy().
I think current time should be set when calling datepicker.open().
https://jsfiddle.net/ajk96poe/

Hi @ile,
Could you expand on why you need config.now? It seems that any time it's set, it becomes stale quite quickly (eg if the user has the calendar open for 2 minute). Maybe just using a new Date() might be a more appropriate solution

I am closing this due to inactivity. I'm sorry we couldn't provide an answer sooner but, if you still have questions, please feel free to reopen this issue.


Max/windows tmpfile

re-opening a tempfile.NamedTemporaryFile on windows fails with permission error

Codecov Report

Merging #16 (f266ae6) into main (0d073d6) will decrease coverage by 0.34%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##             main      #16      +/-   ##
==========================================
- Coverage   59.71%   59.37%   -0.35%
==========================================
  Files          11       11
  Lines         993      987       -6
==========================================
- Hits          593      586       -7
- Misses        400      401       +1

Impacted Files
Coverage Δ

doltcli/dolt.py
55.82% <85.71%> (-0.40%)
:arrow_down:

doltcli/types.py
81.13% <0.00%> (-0.18%)
:arrow_down:

doltcli/utils.py
93.96% <0.00%> (-0.06%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0d073d6...f266ae6. Read the comment docs.


Fix rendering of imported notes

This PR does:

Only show note author if one exists (Camden's imported call logs have no author)
Slightly improve formatting of note metadata (category/date/author)
Fix sorting of notes (let me know if there's a more railsy way to do this)

 Support notes that have no author

 Improve rendering of imported note contents


parquet-tools command to get rowcount & size

Parquet files contain metadata about rowcount & file size. We should have new commands to get rows count & size.
These command can be added in parquet-tools:

rowcount : This should add number of rows in all footers to give total rows in data.
size : This should give compresses size in bytes and human readable format too.
These command helps us to avoid parsing job logs or loading data once again to find number of rows in data. This comes very handy in complex processes, stats generation, QA etc..

Reporter: Swapnil
Note: This issue was originally created as PARQUET-196. Please see the migration documentation for further details.

Shannon Carey:
For those who are looking for this feature, when you run the parquet-tools "meta" command, you can find the row count after "RC:", for example: "row group 1: RC:2040356 ..."

Prateek Rungta:
https://github.com/apache/incubator-parquet-mr/pull/172

Haohui Mai:
It's very useful feature. What's the status of this jira? I can help on addressing the feedbacks.

Uma Maheswari:
Can I know the difference between pretty and detailed size?
Gives same results. Code also looks same.
Is that something like size -pretty means only datablock size and size -detailed means completed parquet file size (header+datablocks+footer)?

Julien Le Dem / @julienledem:
Issue resolved by pull request 406
https://github.com/apache/parquet-mr/pull/406


Unable to build with Xcode 13 GM

When I build the package for release (using swift build -c release) I get the following errors:
/Users/ci/Downloads/PathKit-master/Sources/PathKit.swift:594:12: error: value of optional type 'UnsafeMutablePointer<CChar>?' (aka 'Optional<UnsafeMutablePointer<Int8>>') must be unwrapped to a value of type 'UnsafeMutablePointer<CChar>' (aka 'UnsafeMutablePointer<Int8>')
      free(cPattern)
           ^
/Users/ci/Downloads/PathKit-master/Sources/PathKit.swift:594:12: note: coalesce using '??' to provide a default when the optional value contains 'nil'
      free(cPattern)
           ^
                    ?? <#default value#>
/Users/ci/Downloads/PathKit-master/Sources/PathKit.swift:594:12: note: force-unwrap using '!' to abort execution if the optional value contains 'nil'
      free(cPattern)
           ^
                   !
/Users/ci/Downloads/PathKit-master/Sources/PathKit.swift:625:12: error: value of optional type 'UnsafeMutablePointer<CChar>?' (aka 'Optional<UnsafeMutablePointer<Int8>>') must be unwrapped to a value of type 'UnsafeMutablePointer<CChar>' (aka 'UnsafeMutablePointer<Int8>')
      free(cPattern)
           ^
/Users/ci/Downloads/PathKit-master/Sources/PathKit.swift:625:12: note: coalesce using '??' to provide a default when the optional value contains 'nil'
      free(cPattern)
           ^
                    ?? <#default value#>
/Users/ci/Downloads/PathKit-master/Sources/PathKit.swift:625:12: note: force-unwrap using '!' to abort execution if the optional value contains 'nil'
      free(cPattern)
           ^
                   !
/Users/ci/Downloads/PathKit-master/Sources/PathKit.swift:626:12: error: value of optional type 'UnsafeMutablePointer<CChar>?' (aka 'Optional<UnsafeMutablePointer<Int8>>') must be unwrapped to a value of type 'UnsafeMutablePointer<CChar>' (aka 'UnsafeMutablePointer<Int8>')
      free(cPath)
           ^
/Users/ci/Downloads/PathKit-master/Sources/PathKit.swift:626:12: note: coalesce using '??' to provide a default when the optional value contains 'nil'
      free(cPath)
           ^
                 ?? <#default value#>
/Users/ci/Downloads/PathKit-master/Sources/PathKit.swift:626:12: note: force-unwrap using '!' to abort execution if the optional value contains 'nil'
      free(cPath)
           ^
                !

Is this a straightforward patch that can be applied?

Running into this same error with UnsafeMutablePointer

Running into this same error with UnsafeMutablePointer

Have same issue when trying to build

Hi @kylef is this fix sufficient for the release?

Oh sorry, didn't see the comment about releasing today. Thanks a lot

🥳 https://github.com/kylef/PathKit/releases/tag/1.0.1

Hi,
I'm still getting a compilation error with version 1.0.1
~/Library/Developer/Xcode/DerivedData/ios-ci-helpers-dorskpgcnqncbihcmayjbhhgveqs/SourcePackages/checkouts/XcodeProj/Sources/XcodeProj/Extensions/Path+Extras.swift:30:18: Value of optional type 'UnsafeMutablePointer<CChar>?' (aka 'Optional<UnsafeMutablePointer<Int8>>') must be unwrapped to a value of type 'UnsafeMutablePointer<CChar>' (aka 'UnsafeMutablePointer<Int8>')
Anyone else?

SourcePackages/checkouts/XcodeProj/Sources/XcodeProj/Extensions/Path+Extras.swift

This source file is not from PathKit

This has been resolved in 1.0.1 thanks to everyone who helped out here with propsed changes and investigations!

Opps 😬, apologies, I'm really sorry.


[bug] audio recording via getUserMedia permission error

Describe the bug
This is only an issue in production build and when the app is notarized.
It throws error when I call mediaDevices.getUserMedia({audio: true}):
The request is not allowed by the user agent or the platform in the current context

Reproduction
here's the repo I created using create-tauri-app: https://github.com/egoist-bot/tauri-audio-bug
here's the changes: https://github.com/egoist-bot/tauri-audio-bug/commit/b580280c151259111e3c5bf67d324a2cd2a8e2c6 i already added plist.info and it still doesn't work
see build.mjs to configure all the env variables needed for the build, then run node build.mjs
Expected behavior
should start recording audio
Platform and versions
[✔] Environment
    - OS: Mac OS 14.1.1 X64
    ✔ Xcode Command Line Tools: installed
    ✔ rustc: 1.73.0 (cc66ad468 2023-10-03)
    ✔ cargo: 1.73.0 (9c4383fb5 2023-08-26)
    ✔ rustup: 1.26.0 (5af9b9484 2023-04-05)
    ✔ Rust toolchain: stable-aarch64-apple-darwin (default)
    - node: 18.16.0
    - pnpm: 8.10.0
    - yarn: 1.22.19
    - npm: 9.5.1
    - bun: 1.0.11

[-] Packages
    - tauri [RUST]: 1.5.2
    - tauri-build [RUST]: 1.5.0
    - wry [RUST]: 0.24.6
    - tao [RUST]: 0.16.5
    - @tauri-apps/api [NPM]: 1.5.1
    - @tauri-apps/cli [NPM]: 1.5.6

[-] App
    - build-type: bundle
    - CSP: unset
    - distDir: ../dist
    - devPath: http://localhost:1420/
    - bundler: Vite

Stack trace
No response
Additional context
No response

Can you rename the plist file to an upper case i ? like Info.plist
I'm not sure if that's really relevant but i've only ever seen it with an upper case i so it's worth a try imo.

@FabianLars I realized that but Info.plist still doesn't work

Another Tauri app (with confirmed microphone use in prod) just uses this in the Plist:
	<key>NSMicrophoneUsageDescription</key>
	<string>Some reasoning</string>

they do not have these:
    <key>com.apple.security.device.microphone</key>
    <true/>
    <key>com.apple.security.device.audio-input</key>
    <true/>

Can you run tccutil reset Microphone com.demo (btw, i recommend adding a third component to that bundle-id) and try again?
If that doesn't work, maybe try the same but with sudo 🤔
Lastly, change the bundle identifier (basically a hard reset)

Another Tauri app (with confirmed microphone use in prod)

@denjell-crabnebula can I try that app to see if it's some settings in my OS causing this? I couldn't find a tauri app that uses microphone 😅

Last idea, can you try moving the sandbox properties into a seperate entitlements file like this one? https://github.com/tauri-apps/tauri/blob/1.x/examples/updater/entitlements.plist (needs to be added to tauri.conf here https://github.com/tauri-apps/tauri/blob/1.x/examples/updater/src-tauri/tauri.conf.json#L36

can I try that app to see if it's some settings in my OS causing this? I couldn't find a tauri app that uses microphone 😅

it's closed source so idk what exactly it does, but https://noor.to/ uses tauri

Another is here: https://www.meetsteno.com/ (closed source and paid)

Did you try removing these keys?
    <key>com.apple.security.device.microphone</key>
    <true/>
    <key>com.apple.security.device.audio-input</key>
    <true/>

Last idea, can you try moving the sandbox properties into a seperate entitlements file like this one? 1.x/examples/updater/entitlements.plist (needs to be added to tauri.conf here 1.x/examples/updater/src-tauri/tauri.conf.json#L36

you are the legend, that's what it took to make it work

Did you try removing these keys?
    <key>com.apple.security.device.microphone</key>
    <true/>
    <key>com.apple.security.device.audio-input</key>
    <true/>

I think these are required for sandboxed app
Anyways it's fixed using the solution from @FabianLars , thanks for the help 🫡

is that for desktop or mobile, i am looking for desktop solution

desktop, more specifically macOS

@FabianLars what about windows and linux ? i try to check official tauri docs but no answers there

@pierd i had do the same and it working on mac, as we can build one code on tauri should add permissions for win and linux


Footer is being rendered twice

I see that we set the section tag here, it looks like it's being rendered twice somehow.

Did some research:
Since DOMContentLoaded only fires on the initial page and not on any subsequent page changes, I can see why we added the following event listener twice to ensure the footer is loaded:
https://github.com/bullet-train-co/bullet_train-core/blob/9227d144096d0990d5abb4a90be994c2faeeb77f/bullet_train-themes-light/app/views/showcase/engine/_head.html.erb#L43-L44
In the placeThemeSelector method, we render the footer template on the page by using cloneNode and then appending it as a child to the document which is the correct way to render template tags.
From the following documentation, it makes sense to call this method for turbo:load as well:

You may be used to installing JavaScript behavior in response to the window.onload, DOMContentLoaded, or jQuery ready events. With Turbo, these events will fire only in response to the initial page load, not after any subsequent page changes.

However, turbo:load also gets fired on the initial initial page load as well (reference):

Turbo Drive triggers a series of events during navigation. The most significant of these is the turbo:load event, which fires once on the initial page load, and again after every Turbo Drive visit.

I'm still working through how to fix this, so I'll submit a PR if I make any progress.


Receiving keyvault error when deploying app services

Both IIOT web and service apps fail with an error during provisioning.
Several errors are spawning regarding keyvault.
Here is the output of the default_docker log:
2023-06-24T05:10:20.051185826Z [05:10:19 INF] No managed service identity configured for this service.
2023-06-24T05:10:22.662100535Z [05:10:22 FTL] Host terminated unexpectedly
2023-06-24T05:10:22.662135534Z System.AggregateException: One or more errors occurred. (A keyvault uri was provided could not access keyvault at the address. If you want to read configuration from keyvault, make sure the keyvault is reachable, the required permissions are configured on keyvault and authentication provider information is available. Sign into Visual Studio or Azure CLI on this machine and try again.)
2023-06-24T05:10:22.662145634Z  ---> Microsoft.Azure.IIoT.Exceptions.InvalidConfigurationException: A keyvault uri was provided could not access keyvault at the address. If you want to read configuration from keyvault, make sure the keyvault is reachable, the required permissions are configured on keyvault and authentication provider information is available. Sign into Visual Studio or Azure CLI on this machine and try again.
2023-06-24T05:10:22.662152334Z  ---> Microsoft.Azure.KeyVault.Models.KeyVaultErrorException: Operation returned an invalid status code 'Unauthorized'
2023-06-24T05:10:22.662157634Z    at Microsoft.Azure.KeyVault.KeyVaultClient.GetSecretWithHttpMessagesAsync(String vaultBaseUrl, String secretName, String secretVersion, Dictionary`2 customHeaders, CancellationToken cancellationToken)
2023-06-24T05:10:22.662162934Z    at Microsoft.Azure.KeyVault.KeyVaultClientExtensions.GetSecretAsync(IKeyVaultClient operations, String vaultBaseUrl, String secretName, CancellationToken cancellationToken)
2023-06-24T05:10:22.662168134Z    at Microsoft.Extensions.Configuration.ConfigurationEx.KeyVaultConfigurationProvider.ValidateReadSecretAsync(String secretName) in D:\a\1\s\common\src\Microsoft.Azure.IIoT.Auth.ActiveDirectory\src\Extensions\ConfigurationEx.cs:line 232
2023-06-24T05:10:22.662174034Z    at Microsoft.Extensions.Configuration.ConfigurationEx.KeyVaultConfigurationProvider.CreateInstanceAsync(IConfigurationRoot configuration, Boolean allowInteractiveLogon, String keyVaultUrlVarName, Boolean lazyLoad) in D:\a\1\s\common\src\Microsoft.Azure.IIoT.Auth.ActiveDirectory\src\Extensions\ConfigurationEx.cs:line 197
2023-06-24T05:10:22.662179834Z    --- End of inner exception stack trace ---
Here is the output of the pull and warm up request that continuously fails:
2023-06-24T05:55:14.034Z INFO  - Initiating warmup request to container rjs935iiotdemo_0_31937273_msiProxy for site rjs935iiotdemo
2023-06-24T05:55:14.168Z INFO  - Container rjs935iiotdemo_0_31937273_msiProxy for site rjs935iiotdemo initialized successfully and is ready to serve requests.
2023-06-24T05:55:14.170Z INFO  - Initiating warmup request to container rjs935iiotdemo_0_31937273 for site rjs935iiotdemo
2023-06-24T05:55:19.023Z ERROR - Container rjs935iiotdemo_0_31937273 for site rjs935iiotdemo has exited, failing site start
2023-06-24T05:55:19.109Z ERROR - Container rjs935iiotdemo_0_31937273 didn't respond to HTTP pings on port: 80, failing site start. See container logs for debugging.
2023-06-24T05:55:20.158Z INFO  - Stopping site rjs935iiotdemo because it failed during startup.
Tried in multiple subscriptions with same issue.  Also tried an older release: 2.8.4.
Please advise
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Hi, could you please provide repro steps, including the exact command you used when you deployed.?
Meanwhile 2 notes:

The deployment script in main line deploys latest I believe, please try setting the version to "2.9.0-preview4" to see if this is the cause.
The -type app deployment currently does not work, issue is filed (engineering tool not working)

Tested with 2.9.0-preview4 and scripts in main and it worked fine. Closing until more info is available.


Implement MetadataViews.ResolverCollection in HybridCustody.Manager

Given the fact ChildAccount and OwnedAccount both implement MetadataViews.Resolver and Capabilities for both are contained in the Manager, it would be a familiar pattern for the Manager to implement MetadataViews.ResolverCollection.
Currently, Manager implements Resolver, but AFAIK there are no plans for the resource to resolve metadata views itself. Implementing ResolverCollection would make an easier entrypoint for wallet providers and the like to resolve metadata for all of a user's child and owned accounts.

Curious, how would ResolverCollection work if we're not keying by id?
    pub resource interface ResolverCollection {
        pub fun borrowViewResolver(id: UInt64): &{Resolver}
        pub fun getIDs(): [UInt64]
    }

I guess we iterate over every child account until the uuid matches?

Ohhh right...🤔 I don't think that's worth it. Child/owned accounts are referenced by their Address everywhere else in the contract. IMO better to just close this out


Add group "statistics" page

The contents could be the player's overall gains, the number of maxed players, and whatever else makes sense.

Seeing the group's average player would also be neat


Add Support for Compute Shaders

Add support for data textures, compute shaders, etc. This is actually going to be quite a lot of work.

done as of 0.4.0


Update the nordigen-node gem to get new endpoints

Update to the new released version to get https://github.com/nordigen/nordigen-node/pull/57
It looks like actual gocardless users only have until the 20th to rollout this change.
The email:
Dear user,

We are contacting you to make sure you are using an up-to-date base_url for the GoCardless Bank Account Data API.
This small but important change which should be easy to make for anyone in your team who has access to the production setup of your service. The new connection string should be:
https://bankaccountdata.gocardless.com/api/v2/

The legacy URLs such as [ob.nordigen.com](http://ob.nordigen.com/) and [ob.gocardless.com](http://ob.gocardless.com/) will stop working on the 20th of February.

If you are using any of our public libraries, they are updated and can be found here:
https://developer.gocardless.com/bank-account-data/libraries

We would also like to inform you that there is maintenance window (with downtime) scheduled for 20.02.2024 between 03:00 and 05:00 UTC. You follow its progress on our statuspage:
https://status.nordigen.com/

If you have any questions, feel free to contact support by replying to this mail and will be happy to assist you.

Working on an ever better service,
Bank Account Data Team

It's now February 20th, is there any chance to get this fix released?


Review for Assignment 2

Hey Team,
I was about to start the review for assignment2.
Please share your Jenkins credentials with me on Slack.

Hey Team, I am unable to log into the website. There are no logs/errors shown to me about the login. Please guide me here.

Hey Team, I am facing issues in logging in and viewing the image after upload. Can you please share the logs of this issue? or help me with the issue?

Hi Nawaz,
The server was off since we are working on Milestone 3 as well. It should all work fine now. Please review and let us know.

Hey Team, I am sorry I forgot to share my feedback, so here it goes.

There were few things missing at the time of grading of assignment 2, I have listed them below:

I did not find any test cases in your modules, please do write test cases for your modules so that continuous integration shows right results.
The services were being built on the same instance where they were being deployed. IMO, there should be one dev environment used to build a system and deploy it to production servers. Please do implement that.
Documentation pertaining to Assignment 2 was not very detailed. Please mention the tasks achieved using zookeeper and the steps zookeeper does to support your system.
We are also looking for functionality improvement, so if there was any functionality improved over as the part of sprints, do include it in your wiki.

However, I really like the project concept and the work put behind it. Keep up the good work!


Feature/semantic segmentation

Please do not create a pull request without creating an issue first.
Changes need to be discussed before proceeding, pull requests submitted without linked issues may be rejected.
Please provide enough information so that others can review your pull request. You can skip this if you're fixing a typo – it happens.

[ ] I have added tests to tests to cover my changes.
[ ] I have updated docs/, if necessary.
[ ] I have updated the README.md, if necessary.

What existing issue does this pull request close?
Put closes #issue-number in this pull request's description to auto-close the issue that this fixes.
How are these changes tested?
This pull request includes automated tests for the code it touches and those tests are described below. If no tests are included, reasons why must be provided below.
These changes are tested with [...]
Demonstration
Demonstrate your contribution.
For example, what are the exact commands you ran and their output, related screenshots, screen-recordings, test runs, anything that can showcase.
Provide additional context.
Provide as much relevant context as you like.

Codecov Report

Merging #547 (155943a) into main (7cc6604) will decrease coverage by 4.35%.
The diff coverage is 8.26%.

@@            Coverage Diff             @@
##             main     #547      +/-   ##
==========================================
- Coverage   89.25%   84.91%   -4.35%
==========================================
  Files         147      157      +10
  Lines       11613    12242     +629
==========================================
+ Hits        10365    10395      +30
- Misses       1248     1847     +599

Impacted Files
Coverage Δ

dataquality/clients/objectstore.py
44.61% <ø> (ø)

...ity/integrations/cv/torch/semantic_segmentation.py
0.00% <0.00%> (ø)

dataquality/integrations/torch.py
89.10% <ø> (ø)

...lity/loggers/model_logger/semantic_segmentation.py
0.00% <0.00%> (ø)

dataquality/schemas/semantic_segmentation.py
0.00% <0.00%> (ø)

dataquality/utils/semantic_segmentation/errors.py
0.00% <0.00%> (ø)

dataquality/utils/semantic_segmentation/lm.py
0.00% <0.00%> (ø)

...ataquality/utils/semantic_segmentation/polygons.py
0.00% <0.00%> (ø)

dataquality/utils/semantic_segmentation/utils.py
0.00% <0.00%> (ø)

dataquality/core/finish.py
98.46% <50.00%> (-1.54%)
:arrow_down:

... and 7 more

... and 7 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Great job. It would be good to know the speed impact in google colab on something like coco. End to end is upcoming I assume. Proposing to create an evaluate function instead of watch. Which is calling dq.finish at the end.

@franz101 maybe we could connect on this would love to hear your thoughts