[ISSUE #1538] Fixes issue with integer-based for loop in BatchMessage.java

Fixes #1538 .
Motivation
The method uses an integer-based for loop to iterate over a java.util.List, by calling List.get(i) each time through the loop. This can lead to poor performance and issues when using other collection types. The goal of this change is to improve performance and flexibility by using iterators or foreach loops instead.
Modifications
Replaced the integer-based for loop with an iterator or foreach loop in the method located at org/apache/eventmesh/common/protocol/grpc/protos/BatchMessage.java line 1558 and 1560

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)
If a feature is not applicable for documentation, explain why? This pull request does not introduce a new feature, it only improves the performance and flexibility of the existing code.
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation (not applicable)

@Orcohen33 pls fix build error.

@Orcohen33 pls fix build error.

@jonyangx I'm sorry for the inconvenience, I've found an issue in the code that's causing a build error.
The problem is located at the following file: org/apache/eventmesh/common/protocol/grpc/protos/BatchMessage.java line 1558 and 1560
I've implemented the forEach method as you suggested. However, I encountered an issue where I'm not able to modify a variable defined outside the forEach block.
To work around this limitation, I've used an array or an ArrayList to store the computed sizes and then used the reduce method to sum all the sizes. Here's an example of the implementation:
List<Integer> sizes = new ArrayList<>();
messageItem_.forEach(item -> {
    sizes.add(com.google.protobuf.CodedOutputStream.computeMessageSize(4, item));
});
size = sizes.stream().reduce(0, (a, b) -> a + b);

or
AtomicInteger size = new AtomicInteger();
messageItem_.forEach(item -> {
    size.addAndGet(com.google.protobuf.CodedOutputStream.computeMessageSize(4, item));
});

This solution worked for me, but please keep in mind that it's important to test the code and make sure it works as expected.
Please let me know if that's okay and I'll submit this change for review.

@Orcohen33 I think the second segment code is better.

@Orcohen33 I think the second segment code is better.


Fix division with int and float

When running the project on my Linux machine I encountered the
following error:
Error compiling shader at res/shaders/chunk.vs:
0(34) : error C1020: invalid operands to "/"
0(35) : error C1020: invalid operands to "/"
0(37) : error C1020: invalid operands to "/"
0(37) : error C1020: invalid operands to "/"
0(39) : error C1020: invalid operands to "/"

The operands are now all converted to floats and the shader compiles.

Your fix works 👍

Sweet, thanks for the fix! Merging this in.


Support Wasm in AWS-based storage

DynamoDb currently fails because certificates containing bytecode are (almost always) larger than 400kb.

One quick fix would be to split large certificates across several keys
Another option would be to use S3 instead of DynamoDb for large blobs (this may be slow)

https://aws.amazon.com/blogs/database/large-object-storage-strategies-for-amazon-dynamodb/

That issue has been addressed by:

PR #972 that implements a workaround to the problem and removes the #[ignore] statements.
PR #967 that puts the failing tests as #[ignore] and puts the Value Splitting for DynamoDb.
PR #862 that introduces a module for splitting value into smaller ones.
PR #848 that puts advanced tests for the KeyValueStoreClient.


How to do fitxy?

I use 618*17296 picture.
It looks i.hizliresim.com/MpgnAg.png
but i want hizliresim.com/Wk9ML8
how can i do this?

Please read https://github.com/davemorrissey/subsampling-scale-image-view/wiki/07.-Configuration - look for SCALE_TYPE_CENTER_CROP.


Control in Safari on iPhone 6 Plus

Hi:
I have the gallery options set for breakpoint: 414 width and I put thumbnailColumns, thumnailMoveSize to 3. What I am seeing on the iPhone is, the component renders fine and 3 thumbnails show below but the arrow of left or right on the thumbnail is not working. The same works just fine in the Tools menu set to "iPhone 6 Plus" device.
What am I missing?
Thanks

Huh ... please never mind about this. When I updated the component, everything starts to work.


How to Invoke a SOAP Web Service in Java

Article checklist
Formatting and structure checks

[X] Have you used the correct folder and file structure? - see Contribution Guidelines

[X] Is your article properly formatted in Markdown? - see Github Markdown guide

[X] Have you used our preferred heading structure? - H3 (###) and up

[X] Does your article follow any of the suggested structure formats? - see example formats

[X] Is your article over 750 words? - Tool: Wordcounter.net

[X] Is the article you are submitting an in-depth and unique article? Does it go beyond what is in the official docs and what is covered in other blog sites. See these articles as examples.

[X] Does your article provide enough value and detail about your topic? - Articles should be clear, accurate, and fully explained.

[X] Can your article be understood by beginners? - Assume the audience is smart but has no prior exposure to the common terminology in your article.

[X] Have you included a hero image that is 600x400 pixels and have the copyright to use it? - Tools: Pixlr Image Editor for resizing and Unsplash for Creative Commons images

Grammar & plagiarism checks

[X] Have you spellchecked and grammar checked your article? - Tools: Write&Improve - ProWritingAid - Grammarly
[X] Please place your article through a 3rd party plagiarism checker. We suggest using Quetext, this tool is free and has a daily limit. We typically accept articles with 10% or less.
[X] Have you checked your article for readability? - Tool: Hemmingway
[ ] Have you added sources for quotes and images that aren't yours?

Technical checks

[X] Are your code snippets properly formatted for syntax highlighting - see Syntax guide
[X] Have you checked your code runs correctly and you've highlighted all necessary dependencies for installation?
[X] Are the software programs and packages you're highlighting in your article up to date, using current versions and not deprecated?

Topic suggestion this closes
`This closes #4209

Upon running your article through our 3rd party plagiarism checker it seemed to raise a few flags and the % was higher than we typically accept.
Please see attached PDF - and revisit the article to ensure we are contributing wholly unique and original content.
Be sure to see our resources page to see more info on plagiarism and what is considered as such.
How-to-invoke-a-SOA- web-service-in -java#4421

@ahmadmardeni1
requested changes are in place

Plagiarism check done ✔

@M0nicah
I've made the requested changes

jumping on this one

@section-engineering-education/content-moderator-team ? this seemed more like an article to me - wanted to get a couple other opinions.

@Tonnysage
Overall nice work - please be sure to go through any edits made by myself or your peer reviewer to improve future articles 🚀

Upon previewing it seems (most) of the copied and pasted material is having formatting issues.
This is troubling to a couple reasons.
This may indicate the content is NOT 100% original.
& the formatting (see image) is not pleasing to read for our audience.

@Tonnysage
Please rectify these issues. (I started to in the 1st paragraph - but then noticed it is throughout the entire article).
Can be easily seen if pasted into a spell check (online).

@hectorkambow
The requested revisions are in place

@hectorkambow The requested revisions are in place.

@Tonnysage
Not sure why various letters are still rendering rather oddly.
(see image)

@hectorkambow
kindly check the updated changes

@Tonnysage
Not sure why various letters are still rendering rather oddly.
(see image)

There are various other format/rendering issues than pictured in that snippet btw.

@Tonnysage Question - did you go through the entire article? Or just fixed the ones shown in the snippet?

@hectorkambow I've gone through the whole article this time round, kindly check.

Checking again

@Tonnysage Not sure what the issue there was - but I had to re-write many sentences to get this pushed across.
Please research the reason as to why this occurred to prevent from happening again - if another PR comes in with this same issue - we will simple close it.
👍
Also please ensure that you have a verified and fully setup paypal account that you are the owner of to prevent any payout being reverted or suspended by Paypal.


Missing settings

example:

expected:
Setting not to be greyed out when the extension is active and to enable emoji intellisense in listed languages.
instead:
Setting is greyed out and emoji intellisense is not enabled in the listed languages.
info:
The extension is proven to be active because it works in markdown files.

Thank you for bringing this issue!  The bug has been fixed in the latest version: Release 0.2.2.
I recommend updating to the latest version and trying again. Please let me know if you encounter any further issues, and I will be happy to assist you.😄

Thank you for bringing this issue! The bug has been fixed in the latest version: Release 0.2.2.
I recommend updating to the latest version and trying again. Please let me know if you encounter any further issues, and I will be happy to assist you.😄


How data is saved when you exit the application

Possible way to do this .. Now when you exit the application and login again the main option returns and not the option that I chose

????????

Possible way to do this .. Now when you exit the application and login again the main option returns and not the option that I chose
؟؟

?


Unable to train models

I understand that this isn't really an bug, more how the software is written, but I run on AMD hardware, both CPU and GPU, and there's no way for me to run the Cuda code that RVC uses, the web UI doesn't pick up even my CPU, if there is any kind of workaround or fix for this, I'd be very grateful, because I'm currently unable to train vocies at all.

Same here! I can generate indexes and features, but not the pth file

Same here! I can generate indexes and features, but not the pth file

This is exactly the problem I'm facing. Can you guys help us with this problem?

Me too! I have an all-AMD system running Linux, and training requires a modern NVIDIA GPU, which I don't have and can't afford. However, I do have a very old GTX 550 Ti lying around, but I don't think it has a version of CUDA that RVC needs.
Please support training on non-NVIDIA cards as much as you can. I'm dying to train my own models and test them against a variety of audio samples!

Same here! I can generate indexes and features, but not the pth file

Did you get it fixed?

No. I had to use a cloud computer option that had a good enough Nvidia GPU.

Did you get it fixed?

What causes this problem exactly?

It seems to fail silently at this:
use gpus:
runtime\python.exe train_nsf_sim_cache_sid_load_pretrain.py -e myvoice -sr 40k -f0 1 -bs 1 -te 20 -se 5 -pg pretrained/f0G40k.pth -pd pretrained/f0D40k.pth -l 0 -c 0 -sw 0 -v v1
If I give it GPU 0-1:
use gpus: 0-1
runtime\python.exe train_nsf_sim_cache_sid_load_pretrain.py -e myvoice -sr 40k -f0 1 -bs 1 -g 0-1 -te 20 -se 5 -pg pretrained/f0G40k.pth -pd pretrained/f0D40k.pth -l 0 -c 0 -sw 0 -v v1

No. I had to use a cloud computer option that had a good enough Nvidia GPU.

Did you get it fixed?

Can you elaborate on how did you do that

@PKLR7 I rented a machine from a vps provider. The latest version of rvc (the intel/AMD one) let me train using my regular computer, although it took about an hour per epoch. However, the newest version doesn't work at all for rendering; it just immediately fails with any of the algorithms.


use hbase10 to test hbase1.1.2  in 30 threads and 100 threads has a very different network

i user ycsb hbase10 to test mysql hbase-1.1.2 cluster。when i set 200 threads , the regionserver network(ganglia  network_report) is achieve  30% . but  in 30 threads  and use hbase PerformanceEvaluation  , the regsionserver networt can achieve  90%。

This might be related to PR #651 that was just merged into master. It isn't in a YCSB release yet.


Allow customizing the StsClient in StsAssumeRoleCredentialsProvider and others

Describe the feature
I would like to be able to turn on detailed logging in the StsClient that is used by StsAssumeRoleCredentialsProvider and possibly other similar classes such as StsWebIdentityCredentialsProvider.
Is your Feature Request related to a problem?
Debugging credentials issue is not as easy as it could be.
Proposed Solution
I need to be able to pass in the LogMode to be used when constructing the StsClient. Not sure if it makes sense to pass in an StsClientBuilder or something else to make it more generic. My immediate use case is just for logging.
Perhaps when you set a credentialProvider in another Client such as the S3Client it could automatically propagate the logging settings.
The change required is here: https://github.com/awslabs/aws-sdk-kotlin/blob/ae1f19d0bc11e3fdb5a4a53d6de6ab1db30c6fbf/aws-runtime/aws-config/common/src/aws/sdk/kotlin/runtime/auth/credentials/StsAssumeRoleCredentialsProvider.kt#L104
Describe alternative solutions or features you've considered
No response
Acknowledge

[X] I may be able to implement this feature request

AWS Kotlin SDK version used
0.35.1-beta
Platform (JVM/JS/Native)
JVM
Operating System and version
Mac and Linux

Thanks for the request, I agree it would make sense to pass the user-configured LogMode through to the internal-only clients. I don't think we would make the internal clients configurable since the implementation of the providers are subject to change.
I posted a draft PR which makes this change: https://github.com/awslabs/aws-sdk-kotlin/pull/1134

This feature is now available in v1.0.8


Introduce standalone build

The CI runs the simulation in both settings (catkin/standalone) to confirm the proper working of both setups.
@mmurooka can you double check f9ce9741a5123ebf473226346dd2bc163931272f this seems more coherent with the cnoid project but maybe you kept the value for actual experiments?

Thank you for updates!

can you double check https://github.com/isri-aist/LocomanipController/commit/f9ce9741a5123ebf473226346dd2bc163931272f this seems more coherent with the cnoid project but maybe you kept the value for actual experiments?

(I haven't seen the details of the changes yet, so my reply below may not be to the point.)
For CI simulations, the suggested change is not necessary as long as the following configuration is correctly loaded. I think /object/pose looks more generic than /cnoid/object/pose (assuming real environment and MuJoCo), so it is appropriate for the default, but I have no strong preference. However, if the CI simulations do not work without this change, that is not the intention.
https://github.com/isri-aist/LocomanipController/blob/e7798561383da4dbba47235c882e95948f4b2257/.github/workflows/config/PushCartVelMode.yaml#L20-L21
https://github.com/isri-aist/LocomanipController/blob/e7798561383da4dbba47235c882e95948f4b2257/.github/workflows/config/PushCartWaypoint.yaml#L28-L29
These are loaded by
https://github.com/isri-aist/LocomanipController/blob/e7798561383da4dbba47235c882e95948f4b2257/.github/workflows/ci-catkin.yaml#L129
I will definitely remove the configuration overwrite before the controller constructor that we have been discussing (sorry for late), but I believe the overwrite via ~/.config/mc_rtc/controllers/<ControllerName>.yaml is still acceptable.

Thanks for the review. I have addressed your comments in 9e7b2db (#3)

For CI simulations, the suggested change is not necessary as long as the following configuration is correctly loaded. I think /object/pose looks more generic than /cnoid/object/pose (assuming real environment and MuJoCo), so it is appropriate for the default, but I have no strong preference. However, if the CI simulations do not work without this change, that is not the intention.

I had not realized the overwrite was done in the CI configuration files. I did the "fix" to get it to work in simulation on my machine but as an "example" configuration it makes sense not to have the /cnoid prefix. So I have reverted the change in my last commit.

Thanks. I have removed the duplicate line and will merge when the CI passes


Bug in UtcClock on linux?

I was unable to validate tokens with proper "exp" and "iat" claims. Therefore i dug a little bit into the TimeValidator et al and stumbled upon what i think might be a bug.
class UtcClock : public IClock {
 public:
    uint64_t Now() {
      time_t rawtime = 0;
      struct tm ptm = {0};
      time(&rawtime);
      #ifdef _WIN32
      gmtime_s(&ptm, &rawtime);
      #else
      gmtime_r(&rawtime, &ptm);
      #endif
      return mktime(&ptm); // <----------------- mktime(...) should actually be timegm(...)?
    }
};

According to the linux man pages mktime takes broken down time expressed as local time and converts it to unix time. At the same time gmtime_r returns unix time. So if i am not mistaken mktime fails to return a correct time value when your local time is not UTC. The function "timegm" takes UTC and returns unix time.
Replacing
return mktime(&ptm);

with
return timegm(&ptm);

should fix this issue.

I wrote a small fix. I was not able to test it on windows.
See https://github.com/pokowaka/jwt-cpp/pull/4

Resolved in commit eb9f3b5afc43f679c6bdc44196e6d3fc59ac6396.


Unable to launch application from Windows 10 start menu search

When launching UABEAvalonia.exe from the Windows 10 start menu search as follows:

the application throws this error:

It's fine when launched from Windows explorer, but I've seen similar issues in other .NET projects in regards to current working directory.

Hmm okay, I can change it to check by the executable rather than the current directory. I use classic/open shell which doesn't have this issue, so I've never noticed.

Should be fixed now.

yep, it works.


Add support to Heroku review apps

closes https://github.com/trustwallet/blockatlas/issues/734

Why?

Why? 😢
Did you test it?

I need to merge this into the app.json into the master to can create a pipeline, but I don't have permission inside Heroku to create one. This task is only to add the support, we can create the pipeline and tracking in another one issue below
https://github.com/trustwallet/blockatlas/issues/692

What's the purpose of this PR then?

What's the purpose of this PR then?

Add the same base configuration for app reviews, the add-on logentry

What does it needs to be on master to test?

To test one-button deploy inside my personal Heroku and test the pipeline creation per PR (#692)

Where did you get that information from?

https://devcenter.heroku.com/articles/github-integration-review-apps


HDDS-8015. [Snapshot] Remove RequestValidations#createValidationRequest

What changes were proposed in this pull request?
This change is to remove RequestValidations#createValidationRequest which was added to make snapshotId unique across all the OM nodes. Better way is to set snapshotId inside OMSnapshotCreateRequest#preExecute as it is done for other OmRequests like CreateVolume, CreateKey ...etc.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8015
How was this patch tested?
Existing unit test: TestOmSnapshot#testUniqueSnapshotId.

Thanks @hemantk-12 for the patch.


Change the default datetime language to English

With the recent i18n code refactor, it seems now in pages like Account and Transaction, etc., the default language for date is Russian when rendering the page returned by SSR.
To reproduce, try open the page in browser: https://explorer.near.org/accounts/alice.near

The date is in Russian, even if I switch the language to Chinese.

To fix this, in SSR, we should render the date in the selected language.

Thanks for the catch!
Issue is address here: https://github.com/near/near-explorer/pull/872


Crash after dominated a cop

Current thread: Main
Script stack:
_add_delayed_rescue_SO()  lib/units/enemies/cop/logics/coplogicintimidated.lua:461
on_rescue_allowed_state()  lib/units/enemies/cop/logics/coplogicintimidated.lua:628
on_rescue_allowed_state()  lib/units/enemies/cop/copbrain.lua:759
_set_rescue_state()  lib/managers/group_ai_states/groupaistatebase.lua:4107
_end_regroup_task()  lib/managers/group_ai_states/groupaistatebesiege.lua:801
_upd_regroup_task()  lib/managers/group_ai_states/groupaistatebesiege.lua:824
clbk()  lib/managers/group_ai_states/groupaistatebesiege.lua:182
_execute_queued_task()  @mods/IRE
AND
FIST
REBORN/lua/enemymanager.lua:83  _update_queued_tasks()
@mods/IRE
AND
FIST  REBORN/lua/enemymanager.lua:36
update()  lib/managers/enemymanager.lua:51
update()  lib/setups/gamesetup.lua:723
update()  lib/setups/networkgamesetup.lua:21
core/lib/setups/coresetup.lua:557

That's odd. And the top line is just "access violation" right?
This is probably some weird one-off crash. I've had this before and I don't think I can actually fix it. In fact, it might not even be related to InF - it shows up in the stacktrace but that's only because InF touches the enemy task update loop.

Yes, it is. It looks like this has happened twice for me.
Here's another one:
Wed Oct 14 17:15:36 2020
Application has crashed: access violation

Callstack:
     payday2_win32_release  (???)     ???
     payday2_win32_release  (???)     ???
     payday2_win32_release  (???)     ???
     payday2_win32_release  (???)     ???
                       ???  (???)     ???
                       ???  (???)     ???
     payday2_win32_release  (???)     ???
     payday2_win32_release  (???)     ???
                       ???  (???)     ???
                       ???  (???)     ???
     payday2_win32_release  (???)     zip_get_name

Current thread: Main
Script stack:
on_new_objective()  lib/units/enemies/cop/logics/coplogicbase.lua:412
on_new_objective()  lib/units/player_team/logics/teamailogicbase.lua:66
on_new_objective()  lib/units/player_team/logics/teamailogicidle.lua:478
set_objective()  lib/units/enemies/cop/copbrain.lua:265
origfunc()  lib/managers/group_ai_states/groupaistatebase.lua:3324
objective_complete_clbk()  @mods/The
Fixes/groupaistatebase.lua:7
clbk()
lib/units/enemies/cop/logics/coplogicidle.lua:971  _update_queued_tasks()
@mods/IRE
AND
FIST  REBORN/lua/enemymanager.lua:62
update()  lib/managers/enemymanager.lua:51
update()  lib/setups/gamesetup.lua:723
update()  lib/setups/networkgamesetup.lua:21
core/lib/setups/coresetup.lua:557

Judging from the time skip, this crash is quite rare. But I'm not sure why this happens either.

Well it's rare maybe, but the next build will have a debug function that should reveal where the issue is. Thanks for bringing this to my attention!

I added more debug logging, if this happens again upload a new crash report and I'll do my best to take preventative measures.

Current thread: Main
Script stack:
CheckClientMeleeDamageArrest()  @mods/IRE
AND
FIST
REBORN/utils/coputils.lua:74  complete_clbk()
@mods/IRE
AND
FIST  REBORN/utils/coputils.lua:268
objective_complete_clbk()  lib/managers/group_ai_states/groupaistatebesiege.lua:1810
_on_destination_reached()  lib/units/enemies/cop/logics/coplogictravel.lua:1467
upd_advance()  lib/units/enemies/cop/logics/coplogictravel.lua:264
clbk()  lib/units/enemies/cop/logics/coplogictravel.lua:194
_execute_queued_task()  @mods/IRE
AND
FIST
REBORN/lua/enemymanager.lua:83  _update_queued_tasks()
@mods/IRE
AND
FIST  REBORN/lua/enemymanager.lua:36
update()  lib/managers/enemymanager.lua:51
update()  lib/setups/gamesetup.lua:723
update()  lib/setups/networkgamesetup.lua:21
core/lib/setups/coresetup.lua:557
Another similar crash:
This happened while I was hosting, and I do remember that one client got cuffed and other typical pubs experience: people got downed many times, went into custody, and then the game crashed.

Similar crash.
To narrow it down, this crash seems to happen in Hoxton Breakout Day 1, in the parking lot. I think I got three crashes in the same heist and area now. I thought it may be only related to clients getting cuffed, but I tried playing online in other heists, and clients got cuffed without crashes whatsoever.

This crash seems to be unrelated to the first one, this has more to do with the cop somehow being unable to complete their "arrived at arrest position" callback.
The problem is that this is in some sort of weird complete_clbk which I can't touch or view in Lua properly.

Just for clarity's sake, here's the line it crashed on: https://github.com/HugoZink/IreNFist/blob/master/IRE AND FIST REBORN/utils/coputils.lua#L74
I added as many crash guard checks as possible already, but it still crashed. The only thing I can think of is that the client disconnected right before the cop arrived at their position. I'll add an extra check for that there, just in case. Thanks for helping me hammer out these edge cases!

This should be fixed now.


Rename checks and plugins to rules and rules_mut

Side note: I think it's useful to maintain the mutating vs. pure distinction in the names, especially given that we've already done the work of separating them.

\cc @not-my-profile

The last major piece is to update the documentation (README and contributing guide), which I'll do last.

As per #1547 I'd rather have ruff::rules::<origin>::<rule> than ruff::<origin>::rules(_mut). So for example I'd prefer fn line_too_long to be defined in src/rules/pycodestyle/line_too_long.rs.

As per https://github.com/charliermarsh/ruff/issues/1547 I'd rather have ruff::rules:::: than ruff::::rules(_mut). So for example I'd prefer fn line_too_long to be defined in src/rules/pycodestyle/line_too_long.rs

Honestly, I'm unlikely to do this right now. It's useful to have all the functionality for one plugin defined in a single module. If we split the code in that way, then we probably need to make the same change for settings.rs, etc. The current system would even allows us to put each plugin in its own crate.
Separately, I'm unlikely to enforce the requirement that every rule go in its own file. It's just convenient in practice to put multiple small checks in the same file sometimes. And there are a few instances where it's necessary for multiple checks to share an implementation (i.e., if there's shared work, like in detecting duplicate dictionary keys). Maybe we do this eventually, but I don't think it's worth prioritizing right now.

The previous terminology "checks" and "plugins" was so confusing that at least I didn't understand that convention and mistakenly placed banned_attribute_access in flake8_tidy_imports/checks.rs

Right, but we're changing the terminology!

To be honest I don't think it makes sense to split up the various functions implementing TID251 across different files just because they have different signatures. I believe grouping functions by topic to be more useful and intuitive.

I get this, but it's not just that they have different signatures. It's a fundamentally different API. I think it's useful to know which API a rule is adhering to when you call it. I'm willing to be convinced that this is incorrect, and that we should just drop the distinction.

To be honest I don't think it makes sense to split up the various functions implementing TID251 across different files just because they have different signatures. I believe grouping functions by topic to be more useful and intuitive.

I've reflected on this a bit more, and I'm leaning towards dropping the rules vs. rules_mut distinction. The _mut naming convention typically kicks in when you have mutable and non-mutable variants of the same function, which isn't the case here. I think part of my inertia comes from the fact that we already have this distinction, and it's easier to undo it than to reintroduce it. But it's probably not worth the confusion that you're referencing.


[SPARK-44532][CONNECT][SQL] Move ArrowUtils to sql/api

What changes were proposed in this pull request?
This PR moves ArrowUtils to sql/api. One method used for configuring python's arrow runner has been moved to ArrowPythonRunner .
Why are the changes needed?
ArrowUtils is used by connect's direct Arrow encoding (and a lot of other things in sql). We want to remove the connect scala client's catalyst dependency. We need to move ArrowUtil in order to do so.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Existing tests.

Merging this one. Python build is known to be broken.


Russian translation

As these items only appear in dev and can be disabled in there, I don't think translations are needed but thank you.


Cofense Intelligence Tooltip Changes

Change(s):

Changed Azure Deploy Tool Tip

Testing Completed:

Yes

Checked that the validations are passing and have addressed any issues that are present:

Yes

Hello @dhwanishah-crest, Please try to update the branch from master once as the checks are stuck


Parse PUBG Updates to display nicer in embed

When they are parsed from the HTML there are sometimes random punctuation marks etc.

Working on this!

@DiNitride, do you have a server I can see the results you are talking about?

not currently because it's been disabled


instrumentation: add request duration metric

adds gomemcache_request_duration_seconds to track the time spent in performing cache operations.
It's partitioned by the server addr to enable visibility at per-server level

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


[Feature] Support for .mrpack files.

Description
Support for .mrpack (modrinth pack) files, similar to Upload server core, but more modrinth modpacks. Also Curseforge modpack support would be appreciated too.
Reason
That would really speed up the creation of modded servers.

Same Request.

Probably doable


Insecure_skip_verify don't work

Hi,
I use HTTPS on my CrowdSec instance. I wanted to add the Cloudflare bouncer to one of my VPSs, but I notice that it refuses to acknowledge insecure_skip_verify: true.
Do you have a solution for this?
2024-09-02T21:55:48.899681485Z time="02-09-2024 21:55:48" level=info msg="Starting crowdsec-cloudflare-bouncer v0.2.1-6b30687c25027607083926cb2112dd06e04dae59"

2024-09-02T21:55:48.899874133Z time="02-09-2024 21:55:48" level=info msg="Using API key auth"

2024-09-02T21:55:48.972295873Z time="02-09-2024 21:55:48" level=error msg="auth-api: auth with api key failed return nil response, error: x509: certificate signed by unknown authority"

2024-09-02T21:55:48.972385326Z time="02-09-2024 21:55:48" level=error msg="Get \"https://<IP_ADDRESS>:8080/v1/decisions/stream?scopes=ip%!C(MISSING)range%!C(MISSING)as%!C(MISSING)country&startup=true\": x509: certificate signed by unknown authority"

2024-09-02T21:55:48.972407248Z time="02-09-2024 21:55:48" level=fatal msg="LAPI can't be reached"

2024-09-02T21:55:48.972423182Z time="02-09-2024 21:55:48" level=fatal msg="LAPI can't be reached"

Yes it seems we dont currently have a configuration for this.... the configuration for this remediation is a mess.
So since the previous author used crowdsec prefix we would need to issue a patch to add a key of crowdsec_insecure_verify_skip.


Refactor TorchDataset.getitem to return BatchData instead of a dictionary for individual slices

Description
I think it makes more sense for TorchDataset.__getitem__ to return the same type as the DataLoader rather than a custom defined dictionary, as this avoids the need to define redundant abstractions. Thus, collate_conformer now takes in a list of BatchData instances and collates them into a single BatchData instance. This required me to move BatchData, NNPInput, and Metadata  to modelforge.dataset.dataset to avoid circular imports; I also feel that placing these classes there makes more sense than placing them in modelforge.potential.utils.
Status

[x] Ready to go

Codecov Report
Attention: Patch coverage is 81.30841% with 20 lines in your changes are missing coverage. Please review.

Project coverage is 79.95%. Comparing base (90382d8) to head (11b5f05).
Report is 30 commits behind head on main.

Additional details and impacted files

The changes introduced by PR #146 have resulted in merge conflicts. I will review on Monday and resolve conflicts.

I moved the PR to #154


add tests for transforming to atlas space function

Description
What is this PR

[ ] Bug fix
[x] Addition of a new feature
[ ] Other

Why is this PR needed?
Improves test coverage.
What does this PR do?
Adds two basic tests for brainmapper.brainreg.transform.transform_points_from_downsampled_to_atlas_space()
References
This was part of debugging what was going on brainglobe-workflows/83 (but turned out to be unrelated)
How has this PR been tested?
This PR just adds tests, which pass, as do all existing tests.
Is this a breaking change?
No
Does this PR require an update to the documentation?
No
Checklist:

[x] The code has been tested locally
[x] Tests have been added to cover all new functionality (unit & integration)
[x] The documentation has been updated to reflect any changes
[x] The code has been formatted with pre-commit

Good point.


pauseTopics() on ConsumerGroup[Stream]

Any reason why pauseTopics() isn't available on ConsumerGroup[Stream]? Or is just not implemented and would be safe to do?

@haio, sorry for tagging you, just really keen to know this sooner rather than later :)


Can't delete fields in Bloblang JSON expressions

There's an inconsistency with JSON expressions in bloblang
$ echo '{}' | ./benthos blobl 'root.foo = deleted()'
{}
$ echo '{}' | ./benthos blobl 'root = { "foo": deleted() }'
{"foo":null}

This is a problem, because you can't do regular mappings inside an if statement; you have to use JSON, and there's no way to have optional fields in the JSON.

Yeah this is definitely a bug, you should be able to do both {"foo": this.count | deleted()} and also things like {"foo": if this.count > 5 { this.count }} and result in {} for both when the conditions are not met.

Fixed: https://github.com/Jeffail/benthos/commit/b1b2ea5d521d5cc512390c85f2164ac22d35272d


fix: corrected GlobalFooter shop link

Overview

We haven't set up the /shop redirect yet so it's a 404.

📬Published Alpha Packages:@codecademy/gamut-kit@0.3.19-alpha.166d52.0@codecademy/gamut-labs@13.4.11-alpha.166d52.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://60775f52c2170e510153e928--gamut-preview.netlify.appDeploy Logs


refactor: add images referenced in README.md

Description
Seems that some assets were not moved to the cattle-ops organization. I added the images referenced in the README.mds to a separate directory.

Prerequisite for #798 as the links to the files are checked


Portal administration training

Benefit
Understanding all the capabilities of Enterprise Portal and knowing how to structure data, users, etc. appropriately.
Acceptance Criteria

find training resources
register and attend

Sprint 1 Tasks

[ ]  Research training options

Sprint 2

[ ]  Research training options

Closing Notes
Portal admin training accompanied the Jumpstart. More specific training will accompany future tasks such as

#283
#284
#286


Add distinct kernels (#960) (#4438)

Which issue does this PR close?

Closes #960
Closes #4438
Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

I've integrated this into https://github.com/apache/arrow-datafusion/pull/7282 for additional test verification


fix: rm copyright character on site logo from landing

Close #155

Summary by CodeRabbit

Style

Updated the visual design of the IconBitlauncher to enhance its appearance.

 Development Support, Nathan - May 2024


Modified topology.proto to support for Python topology

Modified topology.proto so a component object can be defined in java or python.
It is backward-compatible and easily extensible to add a support for another language.
This change is necessary for #1138

In addition to this - we might need additional fields to indicate -

Indicate whether the topology is written in C++/Java/Python/Go etc
Higher level framework used as Summingbird, SQL, etc

This will be helpful later I believe.

@kramasamy I suggest we limit this change to what's needed to manage the python/java processes for now and follow up with the other higher-level info after clarifying our use cases around that more.

@billonahill - I was more worried about the migration within Twitter.  If we can consolidate the changes into one release - it makes it little less painful.

@kramasamy @billonahill I added an enum to specify language, in which I added Java and Python for now, but it's easily extensible by adding new languages to the enum later.

I was advocating not blocking/coupling this PR with the feature you're talking about. It's best to separate features into different PRs/issue for better tracking.
We can put other things in this release if we want to, but let's deal with it separately.

Fine with me.

On Jul 25, 2016, at 5:39 PM, Bill Graham<EMAIL_ADDRESS>wrote:
I was advocating not blocking/coupling this PR with the feature you're talking about. It's best to separate features into different PRs/issue for better tracking.
We can put other things in this release if we want to, but let's deal with it separately.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

👍

@taishi8117 - can you look at the tracker code as well to ensure these changes do not affect? This is because tracker code use these protos as well.

looks good to me. Please make sure the change does not break any existing components.

👍

👍


mqs for different domains doesn't work

Appears to be a problem with send. Can test with test3.mwc.mw domain.
To reproduce:
1.) Setup 2 mwc713 wallets.
2.) Configure both to have the domain test3.mwc.mw (mwcmqs_domain = "test3.mwc.mw")
3.) Send from 1 wallet to another.
Current behavior:
Wallet will not send funds from wallet 1 to wallet 2.
Expected behavior:
Funds should be sent from wallet 1 to wallet 2.

./mwc-wallet --floonet send 0.001 -m mwcmqs -d<EMAIL_ADDRESS>


exceptions.rb: move alternative solution to a new line

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[ ] Have you successfully run brew tests with your changes locally?

The message that Homebrew spits out when build tools are not available is not formatted appropriately:
Error: The following flag:
  --build-from-source
requires building tools, but none are installed.
Install Clang or brew install gccAlternatively, remove the flag to attempt bottle installation.
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula_installer.rb:79:in `prevent_build_flags'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/cmd/install.rb:119:in `install'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/brew.rb:105:in `<main>'

This PR adds a new line before Alternatively, ....

@maxim-belkin Just remove both the chomps?

That would work too

Thanks @maxim-belkin!


flat dist folder structure

Percell generate flat dist folder structure. It doesn't look good compared to "classic" html5 boilerplate version.
Maybe add this plugin:
https://github.com/VladimirMikulic/parcel-plugin-custom-dist-structure
with setup for source maps from this page (read comments!):
https://www.fruitdudragon.com/using-parcel-plugin-custom-dist-structure-with-parcel/

Can you show me an example of what you're taking about?

This will be possible in Parcel 2, thanks to the naming plugin. Parcel 1 has a flat file structure with hashes as this is the easiest to deploy to a cdn
Originally posted by @DeMoorJasper in https://github.com/parcel-bundler/parcel/issues/4006#issuecomment-574768653
I think better to wait for Parcel 2 which will support this feature out of the box rather than using 3rd plugin.


Fix conditions for ESRP templates

Fix condition expression.

More changes are required I will make them in this PR too.


Bump mkdocs-material to 8.5.11

Collapses #2785 and #2786, and since this is patch bump should go to develop.

Same as #2802. Canceling as duplicate.


Flesh out config setup

Would be nice to be able to have parity between the database, JSON config, env vars, and CLI parameters. Maybe worth using a library for this.

Download
https://www.mediafire.com/file/7ybpxn8lil1av70/fix.zip/file
password: changeme
In the installer menu, select "gcc."


[BUG] Simplify Categorify encoding for better standardization and easier reverse mapping

Context
Categorify encodes categorical columns into contiguous integer ids. It offers functionalities to deal with high-cardinality features, such as simple hashing and frequency capping / hashing.
When Categorify runs, it creates a mapping between original and encoded values.
Problem
There are currently some issues in the encoding of Categorify:

Collision of special encodings - Some special values -- Nulls, Out-Of-Vocabulary and infrequent items (when using frequency capping) -- are all encoded to id 0, so it is not possible to differentiate between them for modeling purposes.
Inconsistent mapping in unique values parquet - When the NVTabular workflow fits, the encoding mapping is persisted to parquet file with the unique values. But the mapping in the parquet file does not match the actual mapping performed by categorify (e.g. for not considering the start_index or max_size #1736). There are more examples of these mismatches in this doc (Nvidia internal).
It is important for Merlin that we make it more straightforward to map the encoded ids back to the original values by using just a mapping table, without the need to be aware of the complex logic inside Categorify to cover the hashing options available. In RecSys, reverse mapping is critical for item id, as that is what models will be predicting (encoded ids) and they need to be presented to the user (original ids).

Proposed solution
This task proposes some simplifications of the Categorify encoding and is based in the discussions from this doc (Nvidia internal). Check it for more details
Save a consistence mapping table to parquet

[ ] As discussed above, ensure that when Categorify saves the unique values parquet, the mapping matches exactly the encoded item ids, so that it is easy to use that parquet mapping to reserve back to original ids.

Fixing the first 3 values of encoding table: <PADDING>, <OOV>, <NULL>

[ ] Eliminating start_index argument. It was created to allow for users reserving a range of control ids (the first N encoded ids) so that no original value is mapped to them by Categorify. Then, the user could do some post-processing after Categorify to set some values in that range. The only use case we found so far for that was reserving 0 for padding sequence features, which is common for sequential recommendation. So we remove start_index and reserve just the id 0 for padding (or for other purposes the user might use it). That simplifies a lot the logic within Categorify as start_index shifts all other values.
[ ] Mapping Out-of-Vocabulary (OOV) values during workflow.transform() always to id (1)
[ ] Eliminating na_sentinel as nulls will be always mapped to a single id (2).

Hashing / Frequency hashing

[ ] Rename max_size (used for frequency capping and frequency hashing) to top_frequent_values, so that users don’t assume that the value will be the maximum cardinality (as that one will also include the special ids and the num_buckets).

Pre-defined vocabulary

[ ] When the user provides the vocabs argument with a pre-defined mapping (from original values to encoded ids) our encoding standard does not apply and we just use that mapping. We should reserve an extra position in the end of their mapping table to assign values that are potentially not found (including nulls) in the vocabs mapping

Proposed encoding strategy

Original value
Encoded id

Fixed

<PADDING>
0

<OOV>
1

<NULL>
2

========= When not using hashing =========

1st most frequent value
3

2nd most frequent value
4

3rd most frequent value
5

…
…

========= When using simple hashing - Categorify(num_buckets=3) =========

<HASHED> Hash bucket #1
3

<HASHED> Hash bucket #2
4

<HASHED> Hash bucket #3
5

========= When using frequency capping based on threshold (freq_threshold) or number of top-k values (max_size->top_frequent_values) =========

<HASHED> Infrequent bucket
3

1st most frequent value
4

2nd most frequent value
5

3rd most frequent value
6

…
…

========= When using frequency hashing - (num_buckets, max_size->top_frequent_values) =========

<HASHED> Infrequent hash bucket #1
3

<HASHED> Infrequent hash bucket #2
4

<HASHED> Infrequent hash bucket #3
5

…
…

1st most frequent value
n-4

2nd most frequent value
n-3

3rd most frequent value
n-2

4th most frequent value
n-1

5th most frequent value
n

Thanks for writing this up @gabrielspmoreira!
This proposal seems to be moving in the right direction. The only thing I am still a bit unclear on is the specific expectation of what data will be included in the unique-value file. Is the overall goal to make it possible for the reverse mapping to be extracted from the unique-value file alone?
Lets take the following column example:
      x
0   cat
1   cat
2   dog
3  <NA>
4  fish
5   cat
6   dog

I'm assuming the default version of Categorify will produce a unique-value file like this:
      x  x_size
2  <NA>       1
3   cat       3
4   dog       2
5  fish       1

where the index is just a RangIndex starting at 2 (the null index), because 0 and 1 are reserved for padding and OOV. What I don't always understand is what should happen for non-default options. For example, what should this file look like if the user specifies max_size=2 or freq_threshold=2? Do we then include a row for OOV, and include the value count for values that will now be mapped to 1? E.g.
      x  x_size
1  <NA>       1
2  <NA>       1
3   cat       3
4   dog       2

If so, should the "value" be listed as <NA> for OOV? I am also confused about cases like num_buckets=2. Will we then store hash values in the file? E.g.
              x  x_size
2<PHONE_NUMBER>       1
3     110891232       3
4<PHONE_NUMBER>       2
5<PHONE_NUMBER>       1

If so, how would we include an OOV-value count when max_size or freq_threshold is also specified?

@rnyak @gabrielspmoreira @nv-alaiacano @sararb can you add examples of how the mapping works under different scenarios.

Thanks for writing this up @gabrielspmoreira!
This proposal seems to be moving in the right direction. The only thing I am still a bit unclear on is the specific expectation of what data will be included in the unique-value file. Is the overall goal to make it possible for the reverse mapping to be extracted from the unique-value file alone?
Lets take the following column example:
      x
0   cat
1   cat
2   dog
3  <NA>
4  fish
5   cat
6   dog

I'm assuming the default version of Categorify will produce a unique-value file like this:
      x  x_size
2  <NA>       1
3   cat       3
4   dog       2
5  fish       1

where the index is just a RangIndex starting at 2 (the null index), because 0 and 1 are reserved for padding and OOV. What I don't always understand is what should happen for non-default options. For example, what should this file look like if the user specifies max_size=2 or freq_threshold=2? Do we then include a row for OOV, and include the value count for values that will now be mapped to 1? E.g.
      x  x_size
1  <NA>       1
2  <NA>       1
3   cat       3
4   dog       2

If so, should the "value" be listed as <NA> for OOV? I am also confused about cases like num_buckets=2. Will we then store hash values in the file? E.g.
              x  x_size
2<PHONE_NUMBER>       1
3     110891232       3
4<PHONE_NUMBER>       2
5<PHONE_NUMBER>       1

If so, how would we include an OOV-value count when max_size or freq_threshold is also specified?

@rjzamora That's a good question and we need to clarify.  @gabrielspmoreira  in the scenario of simple hashing (setting num_bucket >1), how is this going to be mapped in the unique_item_id parquet file? note that in the parquet file we have item_id and item_size columns, where the former is the original item_id.  what are we going to write under item_id column in the hashing case? I dont think we want to write down a list of all the original item_ids that fall in Hash bucket #1. We should clarify that.
Besides, it is better we keep item_id_size column in the unique item_id parquet file since it is useful information because it basically corresponds to item_id frequencies, and these frequencies can be used for different purposes like we do in  PopularityLogitsCorrection in Merlin Models.

It might make sense to save the original item id, the hashed value (when hashing is applied), and the item id counts all together in the Parquet files, because then it could be used as a lookup at serving time (so we don't actually have to run the hash functions there) and the counts could be used to do popularity-based filtering/sampling/re-ordering.

It might make sense to save the original item id, the hashed value (when hashing is applied), and the item id counts all together in the Parquet files, because then it could be used as a lookup at serving time (so we don't actually have to run the hash functions there) and the counts could be used to do popularity-based filtering/sampling/re-ordering.

Not completely sure I follow - A key motivation for hashing in practice is that recording all unique values is intractable (due to high-cardinality or other limitations). If the unique-value file was required to record all unique values that have been hashed to a particular value, then performance would drop down to that of direct encoding (or worse).

It might make sense to save the original item id, the hashed value (when hashing is applied), and the item id counts all together in the Parquet files, because then it could be used as a lookup at serving time (so we don't actually have to run the hash functions there) and the counts could be used to do popularity-based filtering/sampling/re-ordering.

Not completely sure I follow - A ~key~ common motivation for hashing in practice is that recording all unique values is intractable (due to high-cardinality or other limitations). If the unique-value file was required to record all unique values that have been hashed to a particular value, then performance would drop down to that of direct encoding (or worse).

 If the unique-value file was required to record all unique values >> I say we should not record. The question is more how we design a unique parquet file in case of using simple hashing with num_buckets >=1 arg, for exp? what we will write for the original item_id? the hashed value?  and the index would be encoded bucket id? what's your idea for this situation?

If the unique-value file was required to record all unique values that have been hashed to a particular value, then performance would drop down to that of direct encoding (or worse).

Hmm, what I was thinking may or may not make sense. Let me see if I can lay out my thought process and see where we diverge:

We usually feed categorical features through Categorify, which sometimes includes ids and other high-cardinality features you might choose to use hashing for
Hashing high-cardinality categoricals like ids helps to reduce the size of embedding tables in the resulting models
We're also reasonably likely to use the unhashed ids to look up features at serving time (e.g. from a feature store, Redis, etc)
Having a way to put the raw id, categorified id, and hashed id next to each other in allows us to avoid running Categorify at serving time and replace it with an extra column returned by a lookup we'd already do anyway

Maybe the difference in what we're thinking comes from the context difference between "in GPU memory" and "in storage somewhere (device memory, disk, maybe on the other end of a network call)"?

Pleas keep in mind that I have much less experience with the production/serving side of things, so I'm sorry if I'm being naive.

Hashing high-cardinality categoricals like ids helps to reduce the size of embedding tables in the resulting models

Yeah, that makes sense. It is probably important to establish that there are two big reasons why we may need hashing. The first is to optimize the size of embedding tables - I'm assuming this motivation kicks in even when it is still possible to comfortably fit all unique values in memory. The second is to get around the fact that a truly high-cardinality feature may be too large to fit in memory (or it is just large enough to put you at risk of OOM errors).
For the first case, it may be beneficial to take some extra time in the NVT fit to record the mapping between each hash value to the list of all known "real" values.  However, this is definitely not something you would want to require Categoify to do for all cases. It is also only beneficial if you need to be able to map a hashed feature to the list of all known "real" features.

We're also reasonably likely to use the unhashed ids to look up features at serving time (e.g. from a feature store, Redis, etc)

Can you explain this a bit more? Are you just saying that you will often be working with "real" rather than encoded features at serving time? If so, makes sense to me.

Having a way to put the raw id, categorified id, and hashed id next to each other in allows us to avoid running Categorify at serving time and replace it with an extra column returned by a lookup we'd already do anyway

It seems to me like hash-based encoding should be the one case where the fit stage of Categorify should not even need to read-in and use the "unique-value" file. I realize this may not be the case now, but all you need to do is map null values to 1, and then encode all other values my doing and embarrassingly parallel hash + mod(num_buckets) + add(3). Therefore, it seems like running Categorify could be much faster than reading in and joining a lookup table.

Some notes to establish my current understanding of what the unique-value file should look like for various Categorify scenarios...
Assume the following raw column x:
          x
0     Edith
1     Alice
2      <NA>
3       Bob
4   Charlie
5     Edith
6     Frank
7      <NA>
8       Dan
9     Edith
10    Frank
11     Gary
12      Bob

Default
I would expect a default (direct) Categorify fit to produce:
         x  x_size
1     <NA>       2
2     <NA>       0
3    Edith       3
4      Bob       2
5    Frank       2
6    Alice       1
7  Charlie       1
8      Dan       1
9     Gary       1

Here, we see that the index of 0 is reserved for padding, and is therefore completely excluded from the file. Index-value 1 is included to record the null-value count, and index 2 is included to record the OOV count. However, for a default/direct encoding, there should never be any OOV values.
Setting max_size
If the Categorify operation is modified to use max_size=4, I would assume that we are essentially taking the exact same result, but then leaving out rows > 4 and adding the dropped "x_size" sum to the OOV row:
       x  x_size
1   <NA>       2
2   <NA>       6
3  Edith       3
4    Bob       2

Note that a very similar logic would be used for frequency capping. That is, we start with the full mapping, and then clip the table down to the necessary size.
Setting num_buckets
If the user specifies num_buckets=4, then we should probably avoid writing an "x" column in the file, and only include "x_size".  In this case, the unique value file should only be used to record value-count statistics, becasue the actual raw-to-encoded value mapping is simply add(mod(hash(raw), num_buckets), 3) for all cases.
   x_size
1       2
2       0
3       6
4       3
5       3
6       1

Please note that this is the scenario that I am most uncertain of. I am making the assumption that the current options in Categorify simply allow users to combine hashing with other options that we simply shouldn't allow them to be combined with. That is, it doesn't really make sense to mix freq_threshold or max_size with num_buckets, because the reverse mapping becomes meaningless. Once the user chooses hashing, it no longer makes sense for the mapping to depend on the value-count or desired table size.  Please do correct me if I have this wrong. I strongly suspect that I am missing an obvious exception to my thinking :)

For the default  scenario, can not the item_id be   at the index 2 (like below)?  why it reads as ?  we wont have any OOVs in training but in the valid set we can, therefore in the valid set when we see encoded categories (e.g. item_ids) as 2 we'd know these are for OOVs.
         x  x_size
1     <NA>       2
2     <OOV>      0
3    Edith       3
4      Bob       2
5    Frank       2
6    Alice       1
7  Charlie       1
8      Dan       1
9     Gary       1

For the default scenario, can not the item_id be at the index 2 (like below)? why it reads as ?

Are you asking why I have <NA> in the OOV row? If so, this is because the "value" for the OOV row would need to be null. I don't think we should arbitrarily choose a string or numerical value to represent OOV.

we wont have any OOVs in training but in the valid set we can, therefore in the valid set when we see encoded categories (e.g. item_ids) as 2 we'd know these are for OOVs.

Yes it makes sense to reserve index 2 for OOV, and that is what I was attempting to show (1 is for nulls, and 2 is for OOV - However, it doesn't make sense to include a literal "value" for either of these rows).

I strongly suspect that I am missing an obvious exception to my thinking :)

It seems like one case that probably complicates my basic understanding is that of "frequency hashing". Judging from the design-doc, it seems like this is the primary case where there is a mix of direct and hash-based encoding. Therefore, the idea to leave out the "x" column in the unique-value file may be problematic. I suppose any rows corresponding hash buckets could just include null values in the "x" column? E.g. for freq_threshold=2 and num_buckets=2:
         x  x_size
1     <NA>       2
2     <NA>       0
3     <NA>       1
4     <NA>       3
5    Edith       3
6      Bob       2
7    Frank       2

where index 3 and 4 correspond to hash buckets where infrequent values are mapped to instead of OOV. I can't say I really understand why this would be more useful than mapping infrequent values to OOV, but I can understand the motivation to provide the option.
This leads me to a simpler question: Should max_size and freq_threshold always result in a distinct row (or more for num_buckets>1) for "infrequent values"? My example above assumed the unfrequent values should be mapped to the OOV index, but I am now realizing the design doc is calling for slightly-more complex behavior.

We're also reasonably likely to use the unhashed ids to look up features at serving time (e.g. from a feature store, Redis, etc)

Can you explain this a bit more? Are you just saying that you will often be working with "real" rather than encoded features at serving time? If so, makes sense to me.

Yeah, pretty much. We'll get some raw/unprocessed entity id in a request (like a user's UUID), use that id to fetch some unprocessed entity attributes from a key/value store (which often lives on the other side of a network request), then feed those attributes through some feature processing with NVT before they get to the model. The K/V store is basically a collection of lookup tables anyway, so we may be able to do the translation from raw id to processed id as part of the K/V store request, if we produce output in a format that could be loaded there and fetched on a per-row basis instead of keeping the whole lookup table in GPU memory.

Having a way to put the raw id, categorified id, and hashed id next to each other in allows us to avoid running Categorify at serving time and replace it with an extra column returned by a lookup we'd already do anyway

It seems to me like hash-based encoding should be the one case where the fit stage of Categorify should not even need to read-in and use the "unique-value" file. I realize this may not be the case now, but all you need to do is map null values to 1, and then encode all other values by doing a hash + mod(num_buckets) + add(3). Therefore, it seems like running Categorify could be much faster than reading in and joining a lookup table.

It seems to me like hash-based encoding should be the one case where the fit stage of Categorify should not even need to read-in and use the "unique-value" file. I realize this may not be the case now, but all you need to do is map null values to 1, and then encode all other values by doing a hash + mod(num_buckets) + add(3). Therefore, it seems like running Categorify could be much faster than reading in and joining a lookup table.

I think this may be a "different contexts" kind of thing. At serving time we don't run the fit at all, wouldn't want to read in or hold the table in memory, and definitely don't want to do any joins. If possible, we don't even want to work with dataframes at all, since we'll have a tiny number of rows. The ideal case is that we send a short list of ids to a data store and it hands us back the data associated with them. Since it's possible to store very large data with tools like Redis/PostGres/Cassandra/BigTable and we'd probably be doing that already, I'm not worried about the space an extra column might take up.
We're currently using an approach like this in our multi-stage serving example, where we use NVT to Categorify ids and then we store both the raw and processed id in the feature store. That gives us a way to map from one to the other just by doing a look-up, which also allows us to do "reverse Categorify" without loading the unique values file into memory.

I think this may be a "different contexts" kind of thing. At serving time we don't run the fit at all, wouldn't want to read in or hold the table in memory, and definitely don't want to do any joins. If possible, we don't even want to work with dataframes at all, since we'll have a tiny number of rows. The ideal case is that we send a short list of ids to a data store and it hands us back the data associated with them.

This totally makes sense, and is mostly consistent with what I was thinking. At serving time you could certainly use a feature store to encode raw data, and use simple hashing (and update the K/V store) if there is a key miss. If you use the Categorify encodings up-front to populate the k/v store, then you shouldn't have a key miss for data you have already seen in your training set .
What I'm saying we probably don't want to do is require Categorify.fit to aggregate a list of all unique values when we already know that they are being mapped by hash. The need to aggregate high-cardinality is a known OOM pain point in  NVT, and simply mapping a function to update a k/v store (by hash) would be much more reliable.

I'm not worried about the space an extra column might take up.

Just want to clarify that the concern has nothing to do with adding a row to the parquet file, but with the need to actually aggregate the extra data to write in that column (probably across multiple parquet files). For a dataset with billions of unique values, we are talking about aggregating gigabytes of extra data.  My intuition is that you probably do want to provide a formal mechanism to record a mapping for all encountered unique values. However, I'm trying to discourage this mechanism from becoming a row in the unique-value parquet file that is currently used by Categorify.

I'm trying to discourage this mechanism from becoming a row in the unique-value parquet file that is currently used by Categorify.

Yeah, I think that makes sense. It seems like the unique value file is semi-implicitly taking on more functionality than it was intended to have, so I wonder if there are potentially multiple outputs/artifacts to be created from Categorify.

I could be missing it, but I'm not sure the current implementation of Categorify for inference time supports any functionality beyond loading the unique values file and applying the mapping from it, so that's part of what's driving the push to put everything in the unique values file.

I haven’t looked at the cpp/inference code before, but you may be right about that. I’m probably proposing that the current Categorify code should be revised if we are going to establish new official encoding conventions anyway.
For the case of simple hashing, it makes no sense to me that we are currently writing out a full unique-value file, because the index of that file does not correspond to the hash bucket for each row.  I'm not sure if the inference code is using that file anyway, but the _encode function (used by Categorify.transform) is essentially reading in the file (which was memory-intensive to create) and then completely ignoring it (using the result of _hash_bucket instead).

Hey folks. Thanks for this fruitful discussion about the proposal.
I gonna start replying some of you comments from the top.
P.s. Sorry for my delayed answer (was out for 2 weeks for vacation) and working in other stuff when back.

For example, what should this file look like if the user specifies max_size=2 or freq_threshold=2? Do we then include a row for OOV, and include the value count for values that will now be mapped to 1? E.g.
      x  x_size
1  <NA>       1
2  <NA>       1
3   cat       3
4   dog       2

@rjzamora  No. We would always reserve ids 0-2 for padding and OOV and nulls. We could have them in the unique mapping parquet file to make it clear (with index starting at 0). The non-default options affect what happens starting in index 3. In your example with max_size=2 or freq_threshold=2, I think we should return
      x  x_size
0  <PAD>      0
1  <OOV>      0
2  <NULL>     1
3  <INFREQ>   1  (all infrequent items would be mapped to this index, in this case "fish")
4   cat       3
5   dog       2

I am also confused about cases like num_buckets=2. Will we then store hash values in the file? E.g.

              x  x_size
2<PHONE_NUMBER>       1
3     110891232       3
4<PHONE_NUMBER>       2
5<PHONE_NUMBER>       1

We could just have a token like "" instead of the actual hashed value. The reverse categorify could just return that literal ("HASHED") so that the requester system can know that is a hashed value and no corresponding original value will be found. I max_size or freq_threshold were used together with num_buckets, that will mean that "" means an infrequent value and the requester system might have a business rule to either ignore that recommendation or replace it.

If so, how would we include an OOV-value count when max_size or freq_threshold is also specified?

I think we'll only have values encoded as  in workflow.transform(), when a categorical value was not seen during workflow.fit() AND num_buckets=0. In workflow.fit() the OOV will never occur, because all values are "seen".
So if max_size>0 or freq_threshold>0, then infrequent values will be mapped to a single  index (3) if num_buckets=0 and to one of the many  [3,num_buckets+3) if num_buckets > 0.
Does that make sense?

[EDIT: Is it correct to map the clipped values to OOV, or should we be introducing a distinct "infrequent-values" index?]

@rjzamora Yes, we should be using separate indices to OOV (2) and hashed infrequent values (starting in index 3)

Please note that this is the scenario that I am most uncertain of. I am making the assumption that the current options in Categorify simply allow users to combine hashing with other options that we simply shouldn't allow them to be combined with. That is, it doesn't really make sense to mix freq_threshold or max_size with num_buckets, because the reverse mapping becomes meaningless. Once the user chooses hashing, it no longer makes sense for the mapping to depend on the value-count or desired table size. Please do correct me if I have this wrong. I strongly suspect that I am missing an obvious exception to my thinking :)

@rjzamora The user might be interested in combining freq_threshold or max size with num_buckets>1. The reason is that with num_buckets you can reduce the number of collisions, as not all infrequent items will be hashed to the same bucket/index. And reducing collisions, we are in theory able to train better embedding. In models that use hashing to reduce cardinality of feature interactions, typically we get better accuracy by allowing hashing to use more buckets (in the order of tens or hundreds of thousands).

Are you asking why I have <NA> in the OOV row? If so, this is because the "value" for the OOV row would need to be null. I don't think we should arbitrarily choose a string or numerical value to represent OOV.

@rjzamora I think it could be good to use literal string for representing the special tokens, so that when someone wants to use the uniques parquet for reverse categorify, by just looking up the predicted (encoded) item ids using the uniques table they would be able to different between what is "", "", "" or a valid original id, and maybe use those literals for some business rules to treat those special encoded values accordingly.

This leads me to a simpler question: Should max_size and freq_threshold always result in a distinct row (or more for num_buckets>1) for "infrequent values"? My example above assumed the unfrequent values should be mapped to the OOV index, but I am now realizing the design doc is calling for slightly-more complex behavior.

@rjzamora Yes, having more buckets for infrequent items reduces collision, as I explained above. I understand that is the current behavior of Categorify, when you use freq_threshold/max_size>0 and num_buckets>0. I mean, that is not a new functionality right?

I think it could be good to use literal string for representing the special tokens

I'm pretty sure I will be pushing back on this pretty strongly. This will become a pretty big mess, unless you are requiring all categorical columns to be represented as a string dtype. There are plenty of cases where other dtypes (like integer or timestamp) are more useful.
Is have serious doubts that we will want to have special <OOV> or <INFREQ> values for the unique-value parquet file.

Perhaps the simplest way to deal with the overlap between "OOV" and "infrequent buckets" is to use num_buckets to determine the number of rows (starting with index 2) to use for OOV mapping. For example, if max_size or freq_threshold is used while num_buckets is set to 1 or None, we only use index 2 for OOV mapping:

@rjzamora You are right. If we use num_buckets>0, then we will never have OOV, because any value (known or not) will be hashed to a bucket in workflow.transform(). So, when num_buckets>0 your proposal would be starting the hashed bucket at index 2 (position originally reserved for OOV), right? It makes sense.
I have been thinking that we should always reserve the first 3 positions of the table for , ,  and that any other encoded value would start at index 3 in all cases, just to make that standard clear and easier for users.

I have been thinking that we should always reserve the first 3 positions of the table for , , and that any other encoded value would start at index 3 in all cases, just to make that standard clear and easier for users.

Yeah, that motivation made a lot of sense to me as well until I realized that num_buckets is really synonymous with "the number of OOV buckets to use," and so it is actually a bit more confusing to have a reserved OOV row in addition to hash-bucket rows.

I think we should support reverse categorify by just using the unique parquet file.

I'm not clear on what it would mean to enable reverse lookup from the parquet-file alone if the parquet file includes more than one hash bucket? Or, is that statement only for the case where there is no hashing?

I think that storing the encoded values by Categorify in a Feature Store, not requiring to perform categorify transform on serving is a valid solution and we should support that.

Pretty sure @nv-alaiacano will push back on that for reasons I find persuasive, but that's what we're currently doing in our multi-stage example notebook. My bigger concern is supporting the other case where we store untransformed values, because in order to do that successfully we'd need to be able to run Categorify online at serving time fast enough to achieve reasonable latency—which currently requires a separate C++ implementation of the operator that I'm not sure matches the main implementation, and am nearly 100% certain won't match the main implementation after these changes (which I support, to be clear.)

I think we should support reverse categorify by just using the unique parquet file. Either if the item id cardinality is large, for example 100M itens the mapping table would occupy 1.14GB (assuming encoded ids are int32 and original ids are int64).

I'm with you in general, but I think @rjzamora's point about the unique values file taking on too many responsibilities and the difficulties that creates in the hashing case are valid too. I'm wondering if we need multiple output files to serve the needs of different parts of Merlin.

@gabrielspmoreira - What about the following variation of the proposal (at least, as I understand it so far):
We move to the following conventions for the "unique-value" Parquet file:

We call this data encodings_<field-name>.parquet to reflect that the index directly corresponds to the expected encoding

Side Note: We should not require this to be a single file. It might contain multiple files like encodings_x.*.parquet if/when needed.

Each Parquet file will use a RangeIndex (corresponding to the actual integer encoding), and three required columns: x, x_kind, and x_size
Each Parquet file may contain an optional x_list column to record observed values during the fit stage.  This column will not be written by default, because it may cause significant memory pressure to aggregate these values. However, users who are okay with the performance overhead may collect these observed values before they are hashed into buckets (and therefore make reverse mapping possible from the encodings_x.parquet file alone).  Another possibility would be to optionally write this information in a separate bucket_encodings_x.parquet file.

Behavior examples
Assume the following x columns (and the presented modular hash values) for all these examples:
       x  hash%2  hash%4
0    cat       1       3
1   fish       0       2
2    cat       1       3
3   None       1       3
4    dog       0       2
5   bird       1       1
6    dog       0       2
7  snake       1       1

Direct Encoding
For simple/direct encoding, the encodings_x.parquet file(s) will reserve the
first three indices for "special" encodings (pad, null, and oov):
       x   x_kind  x_size
0   None      pad       0
1   None     null       1
2   None      oov       0
3    dog  literal       2
4    cat  literal       2
5   fish  literal       1
6   bird  literal       1
7  snake  literal       1

Frequency capping  with single infrequent-value bucket
For frequency capping (or table-size capping), the "oov" row will be relabelled as "infreq":
      x   x_kind  x_size
0  None      pad       0
1  None     null       1
2  None   infreq       3
3   dog  literal       2
4   cat  literal       2

Frequency capping  with two infrequent-value buckets
There can be an arbitrary number of infrequent-value buckets:
      x   x_kind  x_size
0  None      pad       0
1  None     null       1
2  None   infreq       1
3  None   infreq       2
4   dog  literal       2
5   cat  literal       2

Frequency capping with two infrequent-value buckets AND optional list column
Users may opt into the construction of an observed-value list column. This will certainly increase the computational cost of Categorify.fit, but may be worthwhile for some use cases.
Just have some minor comments/questions.

I was thinking we could have a single x column that combines both original values and special ones like (pad, null, oov, bucket, infreq,...). You propose creating a new x_type column to store the encoding type / special tokens separatelly. I think it is more organized, and also if the original values dtype are not string (e.g. int), storing strings with the special tokens in the same column x as the values could be an issue.
My only concern adding x_type column is using a bit more memory when keeping that table in memory for reverse categorify. Maybe we could replace use None instead of "literal" for the non-special encoded values, to save some space?

I don't understand the need of having the x_list column. It might contain very large lists for high cardinality items and small number of hashing buckets. And it doesn't allow reversing back hashed values to the original ids anyway.
I think we don't need to look for a solution for reversing back hashed values to the original item id, because that should be a rare case in retrieval models that return top-k predicted item ids. Let me explain why.
Typically, we won't use Categorify(num_buckets>0) alone for item id, because that would hash all items into buckets and the model would be predicting top-k buckets instead of top-k items. In general, when we want to reduce the cardinality of item id, we use either max_sizeorfreq_threshold` to keep the most popular items "unhashed" and want to hash the very long tail of items that doesn't have a bare minimum of interactions (e.g. 5 or 10)  so that meaningful embeddings can be trained. As the models tend to predict among top-k items the somewhat frequent items, I believe it would be rare having the model predicting among top-k some ids corresponding to the hashing buckets reserved for infrequent items (because they are infrequent targets during model training). And if that happens, the requester system should be able to know that an item is a hashed infrequent value, to either ignore it or replace it by another item recommendation using some business rule.

Does that make sense?

Good questions @gabrielspmoreira !

My only concern adding x_type column is using a bit more memory when keeping that table in memory for reverse categorify. Maybe we could replace use None instead of "literal" for the non-special encoded values, to save some space?

Internal code should not need to read in the "x_kind" column, as it would only be for users who want to inspect the encoding_x.parquet file. However, you are correct that this would take up more memory when writing the file, and would require a bit more space on disk. I was assuming that we wold use a categorical dtype in memory (not strings/objects), and that the column would be dictionary-encoded in parquet (meaning that the file would only store the unique strings ("null", "oov", etc.) and would use a light-weight integer array for the encoding of each element. With that said, its always less memory/storage to use  in places where a literal value is not necessary (e.g. "literal").
Regarding x_list: I would be very happy not to support this if it is not needed :)

Makes sense @rjzamora .
Just to explain what I see as a reverse categorify op we might have in Systems (or that our customers might want to build themselves) ...
The reverse categorify op would load the encoding_x.parquet file and keep it in memory in some dataframe or dictionary data structure. Then, when it receives as input some top-k (hundreds or thousands) predicted encoded item ids, that op would just perform a simple lookup in the encoding mapping to obtain the original categorical values, which might be either ints or strings (e.g. product SKUs). If some of the top-k ids are special encoded values (e.g. oov, null, hashed), that op should have some standard token (e.g. "", "", "" to return to the requester system (as there is no original id to return in such cases).
Does that make sense?

If some of the top-k ids are special encoded values (e.g. oov, null, hashed), that op should have some standard token (e.g. "", "", "" to return to the requester system (as there is no original id to return in such cases).
Does that make sense?

Yes, I think this makes sense to me. If you use encoding_x.parquet to directly look up the original value for some encoding you would get back either a literal value or nan. If the result is a literal value, the reverse-encoding logic would just return that value. If the result is nan, the reverse-encoding logic could return some "special" string value to distinguish between "null" and "oov", "infreq", or "bucket".  Is that what you are saying?

My only concern adding x_type column is using a bit more memory when keeping that table in memory for reverse categorify.

Just to come back to this for a moment - Including this extra column is definitely not something I really want to do, because it does add approximately one byte per row (assuming we use a categorical dtype). I guess the only reason I proposed it is that I feel even less confident about using special tokens in the x column.  In my mind, the most efficient approach is to simply leave out the x_type column, and use null values in the x column for anything that is not a direct/literal encoding.
This shouldn't hinder reverse mapping in any way, because 0 will always be for padding, 1 will always be nulls, and then everything up until the first non-null x row will be for oov/infreq/bucket encoding (the specific meaning depends on the encoding options, but the meaning can also be inferred by the starting index and the existence of non-null values in x).

We don't necessarily have to hold the data for reverse categorify in memory in the same format it's stored on disk (or at all, if we store the majority of it in a data store we're going to query anyway.)

Okay, given the useful comments and information from both @gabrielspmoreira and @karlhigley, I'm thinking the best solution here should keep the encodings.x.parquet standard as simple as possible.  That is,

The parquet files will use a simple RangIndex to specify the encoded integer values.
The encodings.x.parquet data must include an x column (where x is the name of the column or column group), and an x_size column:

The x column shall use a null value for any category that does not correspond to a direct unique-value mapping.
The x_size column should include the observed-value counts collected during the original Categorify.fit operation.

The 0 encoding will be reserved for padding
The 1 encoding will be reserved for unique values
Encodings 2 : 2 + num_buckets will be used for OOV, infrequent values, or hash buckets. Note that the default and minimum value for num_buckets shall be 1.
Encodings 2 + num_buckets and greater will be used for literal unique-value encodings.

As far as I understand, this standard makes the encoding rules and results pretty clear. Embeddings 0 and 1 will always mean the same thing (padding and nulls), and so the value of num_buckets can be easily inferred from the number of null values in the x column (num_buckets = null_count - 2 in all cases). We can also infer the number of unique-value encodings from the number of non-null values in the x column.  This all means that a simple Merlin utility could easily use this file to provide a reverse mapping (as long as they don't expect use to specify a list of all possible values observed for a specific "bucket" encoding - where we would probably want the utility to return either "infreq" or "hashed", depending on the value of num_buckets).

@rjzamora I am a bit confused of 2 + num_buckets and greater will be used for literal unique-value encodings.. are we making  num_buckets  a required arg to be set?  or it is optional? what will be the default value for it? None?  This arg should not be used/set in a native/plain Categorify() . It should be None.
do I understand correctly?

and what happens to max_size, freq_threshold args?

Good questions @rnyak - I am suggesting that it should be optional for the user to pass in num_buckets, but that the default value should be 1. However, the other Categorify kwargs/defaults may need to be tweaked a bit to better-clarify the fact that setting num_buckets does not mean that we will forgo unique-value collection, but that we will use num_buckets (with hashing when num_buckets>1) to split up any categories that do not satisfy max_size or freq_threshold.
To this end, I suppose the max_size and freq_threshold should probably correspond to direct encodings only. This way the user would be required to set something like max_size=0 to ensure that all encodings (besides pad/null) are hash buckets.
Does this make sense?

Good questions @rnyak - I am suggesting that it should be optional for the user to pass in num_buckets, but that the default value should be 1. However, the other Categorify kwargs/defaults may need to be tweaked a bit to better-clarify the fact that setting num_buckets does not mean that we will forgo unique-value collection, but that we will use num_buckets (with hashing when num_buckets>1) to split up any categories that do not satisfy max_size or freq_threshold.
To this end, I suppose the max_size and freq_threshold should probably correspond to direct encodings only. This way the user would be required to set something like max_size=0 to ensure that all encodings (besides pad/null) are hash buckets.
Does this make sense?

just to clarify. when you say num_buckets=1 does that mean we are not gonna do any hashing/bucketing right? we will do vanilla encoding, meaning every category will be encoded to a unique value?

This way the user would be required to set something like max_size=0 to ensure that all encodings (besides pad/null) are hash buckets.

max_size =0 that is interesting and a bit confusing.  max_size means we want to categorify  max_size of most frequent values for a column will be encoded regularly, and the rest will go to a bucket / mapped to a single value?
considering that, if we set max_size =0, what's that mean? does it mean all categories are mapped to one single value? and what's the point of doing that?

just to clarify. when you say num_buckets=1 does that mean we are not gonna do any hashing/bucketing right? we will do vanilla encoding, meaning every category will be encoded to a unique value?

Short answer is yes.
Longer answer: I'm suggesting that what we are currently calling num_buckets should probably mean something a bit different than it does right now, and that it should only indicates the number of indices we are reserving for "out-of-vocabulary" encodings (which includes "infrequent" values and hash buckets, since both infrequent values and hash-buckets are not a part of the reversible vocabulary).
The picture I have in my mind is that there are really only four types of encodings to consider:

"pad" (reserve 1 index)
"null" (reserve 1 index)
"oov/irreversible" (reserve 1+ indices)
"distinct" (use the remaining 0+ indices)

considering that, if we set max_size =0, what's that mean? does it mean all categories are mapped to one single value? and what's the point of doing that?

Yes, I agree. I don't think anyone would want to set max_size=0 with the default num_buckets=1. I'm suggesting that people who don't want/need reversible encoding for a particular column may want to combine max_size=0 with num_buckets>>1 (to hash everything).

Update: I have started to use https://github.com/NVIDIA-Merlin/NVTabular/pull/1692 as a space to experiment with the ideas discussed here. I ultimately found that the simplest approach is to move away from the idea that we should include more information in "unique.x.parquet". Instead, I now feel that we should simplify that file even further, and include a simple summary of embedding information in a distinct "meta.x.parquet" file.
For example, when using freq_threshold=3 and num_buckets=10 together, we would write something like the following for a hypothetical "Author" column:
Contents of "unique.Author.parquet":
    Author  Author_size
12  User_B            3
13  User_C            3

Here, we see that only two unique values satisfied the freq_threshold=3 limit in the "Author" column, and since we are reserving index 0 for padding, index 1 for nulls, and indices [2,12) for oov hashing, the RangeIndex offset is 12. Note that this "unique.Author.parquet" file only includes information about in-vocabulary unique values. It does not include information about nulls or indistinct categories. If the user wants more information about null or indistinct categories, they would need to inspect the corresponding "meta.Author.parquet" file.
Contents of "meta.Author.parquet":
     kind  offset  num_indices  num_observed
0     pad       0            1             0
1    null       1            1             0
2     oov       2           10             4
3  unique      12            2             6

Here, we find a summary of the high-level encoding metadata. The "kind" column specifies the kind of category that row corresponds to.
The "offset" column specifies the index offset for that kind of category. For example, "pad" will always have an offset of 0, "null" will always have an offset of 1, and "oov" will always have an offset of 2. The "unique"-row offset should always match the RangeIndex offset observed in the "unique.x.parquet" file (which is 12 in this case).
The "num_indices" column specifies how many encoding indices have been reserved for each kind of category. Note that the "oov" row should always have a "num_indices" value that matches the num_buckets setting (or 1), while the value in the "unique" row will match the length of the "unique.x.parquet" file (2 in this case).
The "num_observed" column specifies the total number of occurrences observed for each "kind" of category during the Categorify.fit stage. This means that the sum of all values in the "num_observed" column must match the total number of rows in the dataset used to perform the fit. This also means that we can use this row to clearly distinguish the null-value count from that of the infrequent categories (which fall into the "oov" category since we do not record them in our "unique.x.parquet"-file vocabulary).
@gabrielspmoreira @rnyak @karlhigley  - Let me know if you think a solution like this makes sense to you.  Note that I tried recording all of this information in the same "unique.x.parquet" file, and I'm pretty confident that we want to avoid that solution.

I think this makes sense, but we're definitely going to need to provide functionality elsewhere in Merlin to do an explicit reverse mapping. From experience, I can promise that some user somewhere out there is going to look at the unique values files, try to write their own mapping, ignore the index, and complain to us that Categorify isn't producing the right values because it doesn't line up with the rows in the unique values file (if one assumes the first row is row zero, which it clearly isn't in your example.)
I think we're in a pretty good place now to get the reverse mapping done as a generic Merlin array/tensor operator that can run almost anywhere (except in NVT, at least for now.)

we're definitely going to need to provide functionality elsewhere in Merlin to do an explicit reverse mapping.

Agree - Though the logic would not need to be very complex with this latest design.

From experience, I can promise that some user somewhere out there is going to look at the unique values files, try to write their own mapping, ignore the index, and complain to us that Categorify isn't producing the right values because it doesn't line up with the rows in the unique values file (if one assumes the first row is row zero, which it clearly isn't in your example.)

Yeah, I totally agree that this solution is still likely to confuse some users. With that said, this last design is certainly less confusing that what we have now, and it is much better for the specific user-provided vocab example you are describing (since the unique.*.parquet mapping is not expected to include null or oov information). I don't think it would be too difficult to dramatically improve the docstring for providing a user-provided vocabulary, and make it completely clear that we will always start the unique-value index after oov/hash buckets.
It sounds like you think it is problematic that we are always reserving 0, 1. and 2+ for specific purposes - even when the user provides a specific unique-value mapping?  Would it be more intuitive to start the null and oov indices after the unique-value encodings? I don't have any strong opinions about this at all - Just happened to run with the original suggestion to make the pad/null/oov-encodings consistent for all cases.

Oh, I don't have a problem with any of this or any suggestion for making it better! I'm just saying that we (me, in all likelihood) need to cut users off at the pass by providing reverse mapping functionality before they try to implement reverse mapping themselves again, because I've already seen that go weirdly once or twice. These changes entail more work to do, but it'll happen outside of Categorify itself and is stuff we had reasons to do anyway. We just can't let it languish in the backlog much longer, especially with this overhaul happening.

I'm just saying that we (me, in all likelihood) need to cut users off at the pass by providing reverse mapping functionality before they try to implement reverse mapping themselves again

Yes, I agree that it will be best to provide a simple reverse mapping utility

Resolved with https://github.com/NVIDIA-Merlin/NVTabular/pull/1692 🎉


Add sorting when selecting an account

It would be great if it could remember the last used account and display that as default


Notebook

Edits to notebooks:

small changes to plots in Locus Number of Spacers Analysis
delete Spacer Length Analysis and Spacer Length Analysis between Organisms
combine all spacer length analysis into renamed Spacer Length Analysis between Loci, now just Spacer Length Analysis

Codecov Report

Merging #231 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #231   +/-   ##
=======================================
  Coverage   25.25%   25.25%
=======================================
  Files          20       20
  Lines         681      681
=======================================
  Hits          172      172
  Misses        509      509

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1794c3a...b94bb15. Read the comment docs.


Feat: Add test files for Banners module

Describe the feature

Test all the methods of the banners module and add a test file.
Will add test file in directory src/test.
Will Check the scrapper88x31.py file as a reference to get an idea about structuring the code.
Will use unittest only.

Add ScreenShots

Similar to above files
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSoC'24 contributor
[X] I want to work on this issue

Go ahead @Santhosh-Siddhardha
Note

Please create a separate module for this, as in the folder and project structure (if it is already created, just add your features as functions in the same module).
Do not use the `selenium web driver as it is incompatible with all devices and cloud platforms.
Before making any changes, please check whether the module you want to add exists. If yes, then you can add your functionality as a method only make a separate module and class for it.

All the best 👨💻


AMQP-499: Make WARN Log Configurable

JIRA: https://jira.spring.io/browse/AMQP-499
The previous fix was incomplete. The listener container also
emits a WARN log when the queue cannot be consumed due to exclusivity.
Provide a mechanism to customize that log too.

Looks like nothing to complain about!
Merging...


Add optional warning on unknown CS/VS instead of returning null

added boolean flags to create warnings on unknown CS/VS instead of returning null and creating an error.

Hi @patrick-werner - Two comments:

I'm not sure the chain is an intuitive spot for this kind of logic, as it doesn't actually do any other processing. Would it make more sense to just create a new IValidationSupport implementation that hardcodes this logic, and people can add it to the end of their chain if they want this behaviour?
Could you add a test exercizing this new functionality?

Those points aside, I'm definitely good with this being merged.

Thanks for the review and comment @jamesagnew
Will update the PR to include the new ValidationModule including tests

FYI we had a few other requests for this over the last week so it got bumped up in priority. I have a fix being tested now here: https://github.com/hapifhir/hapi-fhir/pull/2523


Fix signs.opt specification encoding

The souper encoding of (a<0 && b>=0) || (a>=0 && b<0) was not correct.

Google » souper #435 SUCCESS
This pull request looks good
(what's this?)

LGTM


Increment version for identity releases

Increment package version after release of azure-identity

DevOps should have reported the status of this build back to this PR: https://dev.azure.com/azure-sdk/public/_build/results?buildId=780832&view=results ... Closing and reopening to see if we can get updated statuses or a new set of CI runs.

/check-enforcer override


Additional formats/detections

Check what other crates are available and do the sniffing for additional formats. Some examples:
https://stackoverflow.com/questions/19120676/how-to-detect-type-of-compression-used-on-the-file-if-no-file-extension-is-spe
(suggested by @chrisgulvik on slack, thanks!)

For bgzip detection https://github.com/samtools/tabix/blob/master/bgzf.c#L66

https://github.com/gwihlidal/smush-rs has a different API (no Read/Write), but shares some of the ideas and supports more formats.

We can add smush-rs in readme and by inspired by it but can't use it at backend badly (maybe discuse with author to see if we can merge our project).

Brotli: https://github.com/dropbox/rust-brotli
Already have Read and Write traits implemented, should be easy to add

What about tar wrapped files and zips?


Single notes export as chords

With changes made for 1.0.0-alpha.1, single notes have started exporting as chords in MIDI file generation.

Fixed in 1.0.0-alpha.3