Will the data be released?

Hi, thanks for the interesting work! Do you plan to publish the data you used? (ie: annotations, splits etc)
Moreover, I don't fully understand how are the sound and speech samples "mixed". Do you actually superimpose the speech samples with, for example, the sound of a car?

The CSV files for the dataset used to train VoiceLDM have been uploaded to the following location on GitHub:
https://github.com/glory20h/voiceldm-data
Yes, the samples from the speech segments were simply "added" with the audio from non-speech segments with a signal-to-noise ratio randomly drawn from a uniform distribution.
However, this only applies to speech segments from CommonVoice, since speech segments from AudioSet and VoxCeleb is already "noisy".


Allow loading environment variables from given path

Component
Forge, Cast, Chisel
Describe the feature you would like
Now, the .env file is only loaded from the project root. A CLI flag can be added to load the .env file from another directory. This will be helpful if Foundry is used within a monorepo that uses a single .env file.
The lines that load the .env file can be seen here:
https://github.com/foundry-rs/foundry/blob/8dd7ad2b22a8d7aca13f2e9d73c121a41da28baf/crates/cli/src/utils/mod.rs#L181-L203
I would also like to work on this.
Additional context
The code can be seen below:
pub fn load_dotenv() {
    let load = |p: &Path| {
        dotenvy::from_path(p.join(".env")).ok();
    };

    // we only want the .env file of the cwd and project root
    // `find_project_root_path` calls `current_dir` internally so both paths are either both `Ok` or
    // both `Err`
    if let (Ok(cwd), Ok(prj_root)) = (std::env::current_dir(), find_project_root_path(None)) {
        load(&prj_root);
        if cwd != prj_root {
            // prj root and cwd can be identical
            load(&cwd);
        }
    };
}

I'm interested in this solution because I have a monorepo structure with the .env file placed in the root folder. Is there any workaround until this is released?

Hi @altugbakan thanks for your suggestion! Would you still be willing to work on this?

Hey, I would like to!

Awesome! Assigning to you

Opened #8346 for this

Hey, I tried my hand, but don’t currently have the capacity to work on this. Anyone who wants to solve this can check this out.

No problem @altugbakan, thanks for your time and effort

marking as dupe of https://github.com/foundry-rs/foundry/issues/3711 and closing it, will keep track of request there. thank you


resultMap Bug when SQL using "as" aliases for column names

I think I've found a bug related to resultMaps and SQL that include "as" aliases.  I've created a PR with tests that expose the problem.  Please see that PR for details.  I think the bug was created with https://github.com/mybatis/mybatis-3/pull/895, but I'm not sure.
MyBatis version
Broken starting with 3.4.3.
Database vendor and version
N/A
Test case or example project, Steps to reproduce, Expected result, Actual result
See https://github.com/mybatis/mybatis-3/pull/1100

Discussion and resolution at https://github.com/mybatis/mybatis-3/pull/1100


Log proxy does not install promtail on charm upgrade

Title says it all. This should probably be done at the same time as we merge the two libraries.

This actually happens anytime a pod is redeployed. We should probably fix this by observing the on_pebble_ready event in the lib.

This actually happens anytime a pod is redeployed. We should probably fix this by observing the on_pebble_ready event in the lib.

We probably should use on_pebble_ready, though with a guard so that the code only runs if the relevant relation is established?

Solved in #74


Make power data in item stacks persist

This PR includes the data of the power upon being applied to an item stack. This is essential for storing important power data, such as resource values, stored items, cooldowns, etc. The stack power is updated only on equipment change (at the same time powers are removed from the entity upon being removed from its specified equipment slot)

(at the same time powers are removed from the entity upon being removed from its specified equipment slot)

It seems that this doesn't account for dropping items or picking items up using the cursor :thinking: I might update the power data of the stack power constantly (if the data of the power and the stack power doesn't match), though I'm not sure if that would be optimal

Closing this for now. I'll come back for this issue in the near future!


Error when erasing a mouse-selected text

When I select all the text with the mouse cursor and then press Backspace, the following erros occurs:
VM7873:37 TypeError: Cannot read property 'commonAncestorContainer' of undefined
    at HTMLDivElement.<anonymous> (fate-editor.js:979)
    at ZoneDelegate.invokeTask (zone-evergreen.js:399)
    at Object.onInvokeTask (core.js:41632)
    at ZoneDelegate.invokeTask (zone-evergreen.js:398)
    at Zone.runTask (zone-evergreen.js:167)
    at ZoneTask.invokeTask [as invoke] (zone-evergreen.js:480)
    at invokeTask (zone-evergreen.js:1621)
    at HTMLDivElement.globalZoneAwareCallback (zone-evergreen.js:1647)

I'm using version 2.3.4

Deep testing, I noticed this occurs when I drag selection outside the target, causing component to not detect mouseup event and assing selectionRange

:tada: This issue has been resolved in version 2.3.5 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Why get result when QRCode is far out of the scan area ?

Can we exactly set up scan area ?
Thanks

+1


Running into errors while using --device gpu while running inference using MONAI

Hi,
I am running into errors while using run_inference_single_image.py under the monai folder which is now part of the main branch.
I am using the command in the format:
python run_inference.py --path-json <path/to/json> --chkp-path <path/to/checkpoint> --path-out <path/to/output> --model <unet/nnunet> --crop_size <48x160x320> --device cpu

When I am using cpu for the value of --device, the inference runs fine. But when I try to use the value as gpu, the script runs into an error mentioned below (not pasted the whole traceback, can paste if needed):
RuntimeError: applying transform <monai.transforms.post.dictionary.Invertd object at 0x7f4b4bf0c970>
To reproduce this, you can run:
python /home/GRAMES.POLYMTL.CA/robana/duke/temp/rohan/fmri_sc_seg/monai/run_inference_single_image.py --path-img /home/GRAMES.POLYMTL.CA/robana/duke/temp/rohan/fmri_sc_seg/nnUNet_raw/Dataset013_spinefmri/imagesTs/sc_superset_geneva_ss_sub-genevaR105_001_0000.nii.gz --chkp-path /home/GRAMES.POLYMTL.CA/robana/duke/temp/muena/contrast-agnostic/final_monai_model/nnunet_nf=32_DS=1_opt=adam_lr=0.001_AdapW_CCrop_bs=2_64x192x320_20230918-2253 --path-out /home/GRAMES.POLYMTL.CA/robana/duke/temp/rohan/fmri_sc_seg/results/monai_results --device gpu"

Is this a known issue?
TIA

Hey, I think it's becase of the cupy package. I created a branch nk/fix-gpu-inference. There's a new requirements file for gpu inference.
Could you install those reqs and report back if it is working? thanks!

I installed the requirement and it works as expected when gpu is used. Thanks of the quick fix!

Great, thanks for confirming! 🙌


React Native support

Is there a plan to add support for react-native? What is the best way to use winterfell in react-native app? Should we fork it and refactor components to be compatible with react-native?
Thanks

Hi @mikaoelitiana.
I don't plan to support React Native right now. But always open to seeing some work around that done. Go ahead.
Thanks.


Execute spark scala/python remotely.

Right now we have to ssh code over to cluster and run it "locally".
Need to look at alternative:
https://developer.ibm.com/hadoop/blog/2016/04/26/submit-spark-jobs-via-rest-iop-4-1-using-livy/
or also see https://issues.apache.org/jira/browse/KNOX-516

+1 for being able to execute spark jobs remotely.
Rohan pointed me at livy a few days back.  Although it looks really cool, I'm not sure I'd want the ability to run spark jobs remotely have a dependencies on users installing livy.
Another option is to push engineering to fix the OozieWorkflowSpark example.  It's not good that there isn't an option to remotely run spark scripts out-of-the-box on BigInsights.
At the moment, I see the development workflow as:

developing spark jobs over ssh which gives you lots of output, and then
migrate them to oozie when you are ready to productionise them

(assuming the issue with oozie is going to get fixed)

I've had some feedback that 4.2 will fix the issues with webhcat and oozie.

@pregazzoni are there any other options for executing spark jobs remotely without having to install something on the cluster?

Need some research, have not found one yet.

So i can do remote execution like this, so you get the idea:
vagrant@vagrant-ubuntu-trusty-64:~$ sudo -u ambari-qa \
HADOOP_CONF_DIR=/home/vagrant/hadoop-conf-hdtest102  \
spark-1.5.1-bin-hadoop2.6/bin/spark-submit \
  --master yarn-cluster --num-executors 2 --executor-cores 1 --executor-memory 1G \
  --class "org.apache.spark.examples.WordCount" \
  /tmp/scala-examples.jar /tmp/LICENSE

Now to same with kerberize cluster we will need to setup kerberos client/conf (REALM). Get a valid keytab.


Revert "Added entityPrefix and entitySuffix to GenerateOptions.Rename"

This reverts commit 324ab3e6f87d6204571204332f7ca6270c10d79d.
Former PR contained unintentional change which I was working on different PR. Sorry 😢

No worries.


Context.CreateNamespace hides error from lower level calls

I spotted this while running util/csi-sanity.sh against unified mode stand-alone pmem-csi in drivermode=ndctl (i.e. bypassing LVM2).
Volume creation sometimes fails in that test like this:
GRPC request: name:"sanity-node-full-836087B2-33695F1A" volume_capabilities:<mount:<> access_mode:<mode:SINGLE_NODE_WRITER > >
CreateVolume: Name: sanity-node-full-836087B2-33695F1A, req.Required: 4194304 req.Limit; 0
CreateVolume: Special create volume in Unified mode
setting namespace sector size ...: 0
setting pfn
rpc error: code = Internal desc = CreateVolume: failed to create volume: Failed to create namespace

Here the only information given is "Failed to create namespace" without pointing out which phase of creation actually failed. In Region.CreateNamespace() there is some trace ("setting..." prints) but real error remains hidden, although it is available in err that returned from lower calls.

Olev Kartau<EMAIL_ADDRESS>writes:

Seems we lose original lower-layer error in ndctl.go in Context.CreateNamespace
where return stmt creates a new error message, losing the err that was
returned from failed call.

https://github.com/pkg/errors is a solution for that: always add
additional information to existing errors with errors.Wrap() when
returning a new error.

resolved via fixes coming from merge of direct-nvdimm mode in #133


Polymorphic association dropdown

Hi! I’m upgrading from rails_admin 0.0.5 to 3.1.2 and I’m having issues with polymorphic select forms/dropdown.
It worked great in rails 3, but not in rails 7.
I have these 4 models with polymorphic relations:
class Source < ActiveRecord::Base
  belongs_to :resource_item, polymorphic: true, inverse_of: :sources
end

class SourceItem < Source
end

class SheetMusicPiece < ActiveRecord::Base
  has_many :sources, as: :resource_item, inverse_of: :resource_item
end

class SourceCollection < Source
  has_many :sources, :as => :resource_item, :inverse_of => :resource_item
end

The way it used to work was that when I created a new SourceItem in the admin, it would generate a dropdown containing model names like "SheetMusicPiece" and "SourceCollection". Once I chose the model name, it would then generate a dropdown of instances of that model from which I could choose one to associate a new object with
This is a rails admin config in initializer:
config.model SourceItem do
    list do
      field :resource_item do
        help 'Required'
      end
    end
    edit do
      field :resource_item
    end
end

Expectation (rails 3):

Rails 7:

My Gemfile:
source 'http://rubygems.org'
ruby '3.0.1'

gem 'activemerchant', '~> 1.99.0'
gem 'active_shipping'
gem 'cancan'
gem 'ckeditor', '4.3.0'
gem 'devise', '~> 4.9.2'
gem 'erubis'
gem 'exception_notification', '~> 4.5'
gem 'execjs'
gem 'jquery-rails', '~> 4.5.1'
gem 'jquery-ui-rails', '~> 6.0', '>= 6.0.1'
gem 'haml', '~> 6.1', '>= 6.1.1'
gem 'kaminari'
gem 'kt-paperclip', '~> 7.1', '>= 7.1.1'
gem 'mysql2' # required by thinking-sphinx
gem 'nio4r', '~> 2.4.0'
gem 'nokogiri', '~> 1.14', '>= 1.14.2'
gem 'paper_trail', '~> 14.0'
gem 'pg', '~> 1.4.6'
gem 'pdf-core', '~> 0.9.0'
gem 'pdf-reader'
gem 'prawn', '~> 2.4'
gem 'rack-ssl-enforcer'
gem 'rails', '~> 7.0', '>= <IP_ADDRESS>'
gem 'rails_admin', '~> 3.0.0'
gem 'rails_admin_import', '~> 3.0.3'
gem 'responders', '~> 3.1'
gem 'rexml', '~> 3.2', '>= 3.2.5'
gem 'rubyzip', '~> 1.0'
gem 'sprockets-rails', '~> 3.4', '>= 3.4.2'
gem 'therubyracer'
gem 'thinking-sphinx', '~> 5.5', '>= 5.5.1'
gem 'tilt'
gem 'ttfunk', '~> 1.7.0'
gem 'unicorn-rails', '~> 2.2'
gem 'unicorn', '~> 6.1'

# Install iconv gem to prevent errors after Ruby 2.1 upgrade
gem 'iconv', '~> 1.0.4'

# Gems used only for assets and not required in production environments by default.
group :assets do
  gem 'sass-rails', "~> 5.1.0"
  gem 'coffee-rails', "~> 5.0.0"
  gem 'uglifier', '~> 3.0'
end

group :development, :test do
  gem 'byebug', '~> 10.0'
end

group :development do
  gem 'spring', '~> 3.1.1'
end

gem 'rack-utf8_sanitizer'
gem "sassc-rails"

I did a bit of digging and found that the potential issue is in rails_admin-3.1.2/lib/rails_admin/config/fields/types/polymorphic_association.rb and how the entire form is being constructed,
specifically when setting associated_model_config:
def associated_model_config
   @associated_model_config ||= association.klass.collect { |type| RailsAdmin.config(type) }.reject(&:excluded?)
end

when running this in the console for my polymorphic association:
irb(main):003:0> Source.reflect_on_association(:resource_item).klass
/home/user/.rbenv/versions/3.0.1/lib/ruby/gems/3.0.0/gems/activerecord-<IP_ADDRESS>/lib/active_record/reflection.rb:423:in `compute_class': Polymorphic associations do not support computing the class. (ArgumentError)

so when it comes to constructing the form in rails_admin-3.1.2/app/views/rails_admin/main/_form_polymorphic_association.html.erb
The type_collection variable is nil, which is why there are no dropdown options in the DOM.
So my question is: Is this a bug, or is the functionality I'm expecting no longer supported? Please, advice what should I do in this situation, I’m getting kind of desperate trying to make it work
Thanks!

I could confirm the issue and made the fix as https://github.com/railsadminteam/rails_admin/commit/2a89ebcfa96243697988f6570b9c9be19a7a01b5.
Thank you for reporting, and also being a long-time user!

Thank you!


Enable compression when serving

Considering wasm files are often large, it would be nice to have the contents compressed by the server.  I'm happy to send a pull request if this is a useful feature.
Design question: automatically compress based on Accept-Encoding from the request, or flag guard automatically compressing based on Accept-Encoding?

Sorry for the late reply.
I appreciate the interest, however I don't think this feature is actually desirable. (: cargo web start is not meant to be used in production - it's only meant for development, so I don't really a use case here for supporting compression.
Thanks!


Project set up description is wrong or  1 css file is not included.

When setting up project SVG icons manual include is mandatory.
require('../node_modules/ui-bootstrap4/src/svg-icon.css');
Otherwise, SVG icons will be messed up a lot.

@mindej You're right; that file needs to be included in the default build. It's an external css dependency that one of the forks I built this from used to handle the fact that glyphicons was no longer included in Bootstrap 4.
But, everything should work "out of the box", and that includes icons.

This is a duplicate of #23.


[Bug] SQLSTATE[42000]: Syntax error or access violation

Describe the bug
Whenever I try to install I get this error.

And when I click again, I get this error:

So, the database installed with all the tables correctly.
When I remove the install directory to see if I can access, and navigate to the admin panel, I get this:

I have no idea what is that!
How to reproduce
All you have to do is just download and follow the "Download the latest preview build" steps.
I tried many times with different things to download:

latest release
preview build
git version (with composer install and other source code steps)

So to reproduce the issue:

Download the preview build / latest release.
Unzip to your public web directory.
go to /install in your browser.
Enter credentials.
Click next. And it'll show.

Expected behavior
Installation should complete.
Screenshots
No response
FOSSBilling version
preview
Module version
No response
Device
No response
Information

OS: Centos 7.9.2009
Web Server: Apache 2.4.55
MySQL version: MariaDB 10.4.28

Additional context
Note: I had the same error with your old fork BoxBilling!

The SQL installation process happens with 3 distinct steps:

It creates the empty tables based off the structure.sql file. At this stage, all tables are completely empty and do not have any of their required rows
It imports the content.sql file. At this stage, most of the required SQL work is done, but still not enough to have a functional FOSSBilling instance.
Finally, it replaces with default USD currency in the database with whatever you entered during the install process and creates a new entry in the admin table to create your admin account with the credentials you entered.

Judging by the error messages you get, the installer is able to create the empty tables, but it doesn't have permissions to actually write to the tables, and then fails at the 2nd step.
You need to check the permissions that user has for the database, it sounds like it can create tables, but it cannot write to them.
I see you have CentOS for the OS, are you using CWP? From what I remember (from a few years back), the initial DB user set up didn't actually display all of the available permissions on the setup screen and I had to go back in and add in the few remaining permissions before it worked.

@BelleNottelling

I see you have CentOS for the OS, are you using CWP? From what I remember (from a few years back), the initial DB user set up didn't actually display all of the available permissions on the setup screen and I had to go back in and add in the few remaining permissions before it worked.

Yes for this I'm using CWP.. It does have permission setup screen now, but if it's only problem with permission I'll try to solve it manually from the command line and see what will happen.
I'll reply back with the results.

@BelleNottelling I can confirm now that it's working as expected.
I noticed that the CWP adding the permissions to wrong database name.. for example the name of the database contains underscore _, it get escaped to be like \_, so it gives the permission to a wrong database.
After I fixed this issue, everything worked correctly.
Thanks.

@BelleNottelling I can confirm now that it's working as expected.
I noticed that the CWP adding the permissions to wrong database name.. for example the name of the database contains underscore _, it get escaped to be like \_, so it gives the permission to a wrong database.
After I fixed this issue, everything worked correctly.
Thanks.

Awesome, glad you got it figured out


Proposal: Generic Filter Functionality

Description
We are suffering from Feature Creep on the filtering side of GraphQL API implementation, and we can still not cover most of the use cases (we cannot filter blocks by the proposer, for example).
I did a quick research and it is possible to implement a plugin on gqlgen that can generate the filter objects we need to be able to filter by all the fields we want.
Proposal
Implement a gqlgen plugin
We can create a plugin implementing CodeGenerator interface:
type CodeGenerator interface {
	GenerateCode(cfg *codegen.Data) error
}

That will allow us to scan all existing objects in the schema, get their fields with their types, and generate the needed objects for applying the filtering.
Proposed Filter structure
A filter will contain And filters, Or filters, Not filter, and field filters. They will be composed as follows (we are using Block filter as an example):
type FilterBlock struct {
	And []*FilterBlock
	Or []*FilterBlock
	Not *FilterBlock

	hash *FilterString

	num_txs *FilterNumber

	txs *NestedFilterBlockTransaction
}

type FilterString struct {
	Exists bool
    Eq       string
    Neq    string

    Like    string
    Nlike  string

	// TODO: maybe there are more useful ones?
}

That will allow us to create filters like:

{
   "_and":[
      {
         "num_txs":{
            "gt":10,
            "lt":100
         }
      },
      {
         "proposer_address_raw":{
            "eq":"ADDR"
         },
         "_or":[
            {
               "tx":{
                  "hash":{
                     "eq":"TXHASH"
                  }
               }
            }
         ]
      }
   ],
   "_or":[
      {
         "consensus_hash":{
            "eq":"HASH"
         }
      }
   ]
}

That will be the same as:
WHERE (((num_txs > 10 AND num_txs < 100) AND proposer_address_raw = "ADDR") OR tx_hash="TXHASH") OR consensus_hash = "HASH"

Schema tagging for filter creation
We can scan the existing graphql schema to find specific annotations, and from that, generate the needed filters. Example:

# plugin:filter
type Block {

  # plugin:filter
  hash: String!

  # plugin:filter
  height: Int!

  version: String!

  # plugin:filter
  chain_id: String!

  # plugin:filter
  time: Time!

[...]

That will allow us to create filters only from the types and fields we want/need. In the future we can be more specific, like only creating eq filter for a field: (# plugin:filter(types:[eq])))
Special cases
We have filters that are used outside the filtering logic, for example, to query the storage (block height). We need to expose this values to retrieve them. One idea (still thinking if it is the best) will be add a special tag to these values to expose a method to retrieve the max and min values from the filters, something like:

  # plugin:filter(expose:[minmax])
  height: Int!

Generating the following method in the generated filter struct:

func (f *FilterBlock) GetMinMaxHeight()(min int, max int) {
	// TODO check height filters values (eq, gt, lt and so on) in a recursive way, and return the min value and max value
}

I love this, I think it's great 💯
I was bummed to find out we need to directly modify the existing objects in order to index specific block fields, like the proposer, but if this is a long term solution then I'm all for it
@jinoosss what do you think?

@zivkovicmilos @ajnavarro ,
I think that's a really good suggestion, too.
We can provide consistent and intuitive filters.

Closed by #90


Updated to v0.13.0

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

@conda-forge-admin, please rerender

@ndmaxar @oblute @rluria14 Can we please merge this if it looks ok? The last published recipe is ancient, from prehistorical versions of nevergrad, this one points to the latest just-released version. 🙏


Cannot find parameter named A in message named B

Hey 👋
It seems there is an issue with lexical part of the ICU parser which cannot handle properly variations of Intl.message and Intl.gender.
// localizations.dart
class MyLocalizations {
  String test(String gender) => Intl.message(
    "Hi ${Intl.gender(gender, female: 'women', other: 'men')}",
    name: "test",
    args: [gender],
  );
}

Running the command to extract data from above file into arb:
flutter pub run intl_translation:extract_to_arb \
	--locale pl \
	--output-dir=lib/l10n/autogenerated \
	--warnings-are-errors \
	localizations.dart

New files has been generated under /lib/l10n/autogenerated/intl_messages.arb:
{
  "@@locale": "pl",
  "@@last_modified": "2019-11-28T16:25:05.527968",
  "test": "Hi {gender,select, female{women}other{men}}",
  "@test": {
    "type": "text",
    "placeholders": {
      "gender": {}
    }
  }
}

So far, so good, but when performing the last step the intl crashes:
flutter pub run intl_translation:generate_from_arb \
	--output-dir=lib/l10n/autogenerated \
	--no-use-deferred-loading \
	localizations.dart \
	lib/l10n/autogenerated/intl_*.arb

Unhandled exception:
Invalid argument(s): Cannot find parameter named 'WOMEN' in message named 'test'. Available parameters are [gender]
#0      VariableSubstitution.index (package:intl_translation/src/intl_message.dart:396:7)
#1      VariableSubstitution.variableName (package:intl_translation/src/intl_message.dart:411:57)
#2      VariableSubstitution.toCode (package:intl_translation/src/intl_message.dart:416:21)
#3      CompositeMessage.toCode.<anonymous closure> (package:intl_translation/src/intl_message.dart:352:41)
#4      MappedListIterable.elementAt (dart:_internal/iterable.dart:415:29)
#5      ListIterable.join (dart:_internal/iterable.dart:165:22)
#6      CompositeMessage.toCode (package:intl_translation/src/intl_message.dart:352:51)
#7      MainMessage.addTranslation (package:intl_translation/src/intl_message.dart:520:39)
#8      MessageGeneration.generateIndividualMessageFile (package:intl_translation/generate_localized.dart:87:18)
#9      generateLocaleFile (file:///usr/local/opt/flutter/.pub-cache/hosted/pub.dartlang.org/intl_translation-0.17.7/bin/generate_from_arb.dart:162:14)
#10     main.<anonymous closure> (file:///usr/local/opt/flutter/.pub-cache/hosted/pub.dartlang.org/intl_translation-0.17.7/bin/generate_from_arb.dart:115:5)
#11     _LinkedHashMapMixin.forEach (dart:collection-patch/compact_hash.dart:377:8)
#12     main (file:///usr/local/opt/flutter/.pub-cache/hosted/pub.dartlang.org/intl_translation-0.17.7/bin/generate_from_arb.dart:114:20)
#13     _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:303:32)
#14     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:172:12)

It works well when changing the sentence "Hi ${Intl.gender(gender, female: 'women', other: 'men')}" to one of the followings:

"${Intl.gender(gender, female: 'women', other: 'men')} Hi"
"Hi ${Intl.gender(gender, female: 'ą', other: 'ć')}"

The only feasible solution for now which works is to omit the Intl.message part and use only the Intl.gender - but it introduces repeated amount of text. Like so:
// fixed_localizations.dart
class MyLocalizations {
  String test(String gender) => Intl.gender(
    gender,
    female: "Hi women",
    other: "Hi man",
    name: "test",
    args: [gender],
  );
}

Well that's peculiar, and definitely seems like a bug. However, I note that it's generally recommended practice with translations to include the context of the full message, even if it introduces redundancy. In some languages, the translation of just 'women' standalone, might be different than 'Hi women.'  e.g. in French Google translates women as "femmes" , "hi" as "salut", and "hi women" as "salut les femmes".

Closing in favor of #86 for the issue of nesting calls in general.


Crash on stop in completeOnNewConnectionWithInfo

How to reproduce:

Enable full logging to have more time
Start protection.
Open some yandex app.
In 1-3 seconds, press stop protection.

core/pull-requests/345


Parsing different NlAttrType Enums for different received Cmds

Hey,
I'm sorry if the response to this is somewhere in the docs, but i searched them and read a lot of issues but still could not find a solution for my problem.
Issue #50 seems to be relevant, but could not find an exact solution after reading it.
Neli version: 0.6.2
I am using the async Socket neli::socket::tokio::NlSocket to wait for messages from the socket via:
let mut buffer: Vec<u8> = Vec::new();
let response =  self.socket.recv::<NlTypeWrapper, Genlmsghdr<MYCMDS, ATTRIBUTES_COMMAND_ONE>>(&mut buffer);

Now i am forced to give socket the type of the attributes. But i would like to parse the attributes of the Genlmsghdr depending on the command i got in MYCMDS.
Right now my attribute enums and commands look something like this:
#[neli_enum(serialized_type = "u8")]
pub enum MYCMDS {
    COMMAND_ONE = 0,
    COMMAND_TWO = 1,
}
impl Cmd for MYCMDS {}

#[neli_enum(serialized_type = "u16")]
pub enum ATTRIBUTES_COMMAND_ONE {
    ATTRIBUTE_ONE_A = 0,
    ATTRIBUTE_ONE_B = 1,
}
impl NlAttrType for ATTRIBUTES_COMMAND_ONE {}

#[neli_enum(serialized_type = "u16")]
pub enum ATTRIBUTES_COMMAND_TWO {
    ATTRIBUTE_TWO_A = 0,
    ATTRIBUTE_TWO_B = 1,
    ATTRIBUTE_TWO_C = 2,
    ATTRIBUTE_TWO_D = 3,
}
impl NlAttrType for ATTRIBUTES_COMMAND_TWO {}

And i would like to do something like this (maybe not correct code. just as an example):
let response =  self.socket.recv::<NlTypeWrapper, Genlmsghdr<MYCMDS, u16>>(&mut buffer);
let message = response.unwrap()
let attr_handle = match nlmessage.nl_payload.get_payload().unwrap().cmd{
    MYCMDS::COMMAND_ONE => {
        nlmessage.get_payload().unwrap().get_attr_handle_as(ATTRIBUTES_COMMAND_ONE)
    }
    MYCMDS::COMMAND_TWO => {
        nlmessage.get_payload().unwrap().get_attr_handle_as(ATTRIBUTES_COMMAND_TWO)
    }
  }

Hope you understand my use case.
Thanks in advance!

Hi @Alexandero89! Your use of u16 and then conversion to the correct type after matching on MYCMDS is generally the way to do it. It is hard to come up with a better way to handle the general case because netlink has such a wide range of formats that have to be supported by neli. Does that code cause errors that I can help you with or did you just want to check if this was the best way to do it? I'm happy to help with either.

Hey John,
Thanks for your response!
Actually this were some function calls I thought are non existent. Because I could not find a function with an enum as an input value (like the get_attr_handle_as example I used).
Because this kind of implementation would be completely fine for me.
But I will have another look and try to implement it like this.

Okay, I think I see what you're missing now. So the current way to do this is to get the field nla_type and parse the u16 as ATTRIBUTES_COMMAND_ONE or ATTRIBUTES_COMMAND_TWO manually using ATTRIBUTES_COMMAND_TWO::from(attribute.nla_type.nla_type). This is admittedly not very nice so let me look into adding an API like the one you specified above.

@Alexandero89 Does the most recent commit in #182 solve this issue for you?


Geo fence status

Running a eufy floodlight camera pro 2, which has a very reliable geo fence status. Would be great to be able to access this as a trigger for HA.

need clarification, what is exactly needed here?

in homebase, you can have arm-home, arm-away, but there is no geofencing and schedule for arm, how to add?

You need to call services for geofencing, it is not a state of alarm station, it is a rule config.
Check eufy_security services, you will find them there


Windows terminal使用vim打开文件鼠标上下滚动会出现terminal页面滚动，而不是vim编辑文档内容滚动

Windows terminal使用vim打开文件鼠标上下滚动会出现terminal页面滚动，而不是vim编辑文档内容滚动

Just based on the images, and a machine translation of the text, I suspect this is the same issue as described in #4088. And as I mentioned there, I think it's a dup of #381.

Yep! Thanks @j4james. /dup #381.


[FEATURE] Create Async SDKClient

Is your feature request related to a problem?
Currently, SDKClient which sits on top of opensearch-java client only supports synchronous request calls. This issue talks about adding Async functionality for SDKClient.
What solution would you like?
A clear and concise description of what you want to happen.
What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.
Do you have any additional context?
Add any other context or screenshots about the feature request here.

I think we should give users options:

Async client (preferred, recommended)
Sync client (an option for simplicity)
High Level Rest Client (deprecated, provided for compatibility with existing plugins, not recommended)


Changed invalid nginx error_log level to warn

Resolved incorrect nginx.conf error_log level as per issue #101

Thanks! Fwiw, this has been resolved in another branch I'm working on but hasn't been merged into main yet. Thanks for fixing.

No worries, good to know.
I forgot to say it is preventing the published docker hub image from starting without intervention.
We're planning on trialing this at our makerspace, SparkCC. You might see a few other small fixes come through as we start testing here.

Awesome, sounds great! We've been running it at hsbne.org (Hackerspace Brisbane) for nearly two years now and it's been an awesome improvement to our onboarding workflow. I've put a lot of work into it recently to make it easier for others to run, but I think there's still a handful of rough edges. Do let me know if you run into any other problems and I'll happily assist/update docs, etc. :)

Also feel free to jump into our Discord channel here if you want to chat.


I have question about twa.

How can I call CloseButton in a header when the deck is finished

The close button simply calls screenStore.back() internally, and you can invoke it from anywhere in the code.


Add persistent storage to prevent data extinction

Describe the bug
After Home page refresh, user data context is being deleted.
To Reproduce
Steps to reproduce the behavior:

Log in,
Wait until Home page shows up,
Refresh page,
See error on header bar - no user's nickname there,

Expected behavior
User data context should be preserved until logout.
Screenshots
Desktop (please complete the following information):
Smartphone (please complete the following information):
Additional context

This will not be resolved, as there is no appropriate way of doing this. Data persistance is job for a database. Local storage of web browser is not working, as React is doing server-side rendering first. Using effect hook is not working too. Context API is only preserving data till re-rendering of page.


libtokenizers.a path should not depend on build source directory

By relatively locating the library, one can just drop the pre-built library in their own project to make it work.

When I imported the module and dropped the prebuilt library at the root of my project, I had the following error:
$ go run main.go
# command-line-arguments
/usr/lib/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: cannot find<EMAIL_ADDRESS>No such file or directory
collect2: error: ld returned 1 exit status

With the change in this commit, it just works.

Yes, it will work if you just have one package, but if your repo has multiple packages that use tokenizers, they will all look for the binary in the root of each package. So making the path relative to the module source decouples it from user's repo structure.

Yes, it will work if you just have one package, but if your repo has multiple packages that use tokenizers, they will all look for the binary in the root of each package. So making the path relative to the module source decouples it from user's repo structure.

Fine. Maybe a better way would be to make the library built during the installation by using the //go:generate macro somehow, maybe that could work and avoid the burden of searching for the exact path where the module is located in the go directory to drop the prebuilt library. WDYT?

Maybe a better way would be to make the library built during the installation by using the //go:generate macro somehow

If you could make it work, sure.

avoid the burden of searching for the exact path where the module is located

isn't that always under $GOPATH/pkg/mod, unless you vendor.

Maybe a better way would be to make the library built during the installation by using the //go:generate macro somehow

If you could make it work, sure.

avoid the burden of searching for the exact path where the module is located

isn't that always under $GOPATH/pkg/mod, unless you vendor.

Nop, as you can see in the error above, the library should land in<EMAIL_ADDRESS>I mean that's not so bad but that's definitely a barrier that we can get rid of to increase adoption of the project in my opinion.

As I mentioned, under proposed solution once your project has two packages using this module you'd have to copy the lib to two places .

I think this proposed change is an improvement, as we build our application via docker CI/CD, and have a complex, version dependent, pre-setup step of having to initialize a full path<EMAIL_ADDRESS>in order to drop in libtokenizers.a before compiling our app.  At least with this change we can just place libtokenizers.a in the root of our app before compiling.  Alternative is to actually commit this file to the repo, so it will be there without the copy (though committing artifacts is a bit nasty)?

I tried the generate method but I did not manage to make it work.
I was in the same situation as @RJKeevil , that's why I made the change in the first place. However, I think that commiting the file to the repo is not the right approach because the lib could be misaligned with the code. I would not recommend it.

@clems4ever @RJKeevil

I found a simple but not best way to solve this problem, refer here: https://github.com/sunhailin-Leo/tokenizers#import-this-package-in-your-projects

Alternative is to actually commit this file to the repo, so it will be there without the copy (though committing artifacts is a bit nasty)?

Artifacts are platform (OS/CPU) dependent so there is no single artifact to satisfy all.

At least with this change we can just place libtokenizers.a in the root of our app before compiling.

You'd need to copy it to the directory of each module that uses this dependency, which means N copies, while making it source agnostic allows a single copy. Does that make sense?

The main issue is having the version number in the path, in reality it can be placed anywhere, as long as the version number isnt part of the path.

I bet you can write a script to extract the version from go.mod since it will specify it, then that becomes the only place that requires bumping the version.

Yes it's definitely possible and no big issue, just might be easier for future users if it can be an argument, something like how it is done in this project https://github.com/yalue/onnxruntime_go

Ok one additional issue with the lib being in the go path - it prevents you from using the docker run target feature in goland/intellij.  This feature dynamically compiles and runs your dev code in an existing docker image. But it fails to compile as go mod sees that the tokenizer folder exists in the image, and assumes the go code for the dependency has already been fetched. So I would need to also copy the tokenizer go code in my Docker build process, which is very messy.

Yes it's definitely possible and no big issue, just might be easier for future users if it can be an argument, something like how it is done in this project https://github.com/yalue/onnxruntime_go

Thanks for suggestion, was interesting read. They have a different problem where the lib they are trying to load is outside of their control and named differently on different platforms. So they end up loading the library at runtime, which brings its own complications with packaging and releasing. This library produces static build that is self sufficient.

Hello, in understanding cgo a little better I perhaps have a simple solution to this discussion.  I see that you can switch cgo directives using build tags.  This means that if you replace the directive statement with something like
#cgo redirect_tokenizer LDFLAGS: /usr/lib/libtokenizers.a -ldl -lm -lstdc++ #cgo !redirect_tokenizer LDFLAGS: ${SRCDIR}/libtokenizers.a -ldl -lm -lstdc++
Then using go build -tags redirect_tokenizer would instead point to the fixed path at compile time.  This solves my main issues (allows library to be used in Goland's docker run targets, simplifies image build). It's probably a nice answer for supporting different architectures in future, e.g. we could now package linux, darwin versions of libtokenizer.a and switch based on build flags.

@RJKeevil that's cool, want to send a PR? I think we should the directive something like tokenizers_srcdir_relative

Thanks @daulet , submitted a PR to https://github.com/daulet/tokenizers/pull/15 .  As mentioned in the notes this does change the default behaviour, let me know if this is ok or not.

The thing is, the go module cache is read-only by design... long story short- this shouldn't be there, and it's probably should be configurable....

I think https://github.com/daulet/tokenizers/pull/18 allows for the modularity that is being described here.

I believe #18 solves everyone's use cases.


Create .gitignore to comply with Godot Asset Library requirements

The Asset Library application for FuncGodot has been rejected because the repo lacks a .gitignore. I have added the template .gitignore file from the official docs. After this change the package can be resubmitted for inclusion.

Looks good. Sorry for the oversight!


Project Status

Hey @iximiuz hope all is great 👋
Despite being parked for a little bit, I still think pq is amazing and I was pondering the idea of using as a DuckDB Extension now that we have a rust framework available. Do you consider it abandonware or still worth maintaining if there are contributions/contributors?
Thanks!

Hi @lmangani! This project is de facto abandoned, but I haven't lost the hope to return to it one day. It was born out of necessity - I used to deal with large volumes of raw log data, and I desperately needed to compute some ad-hoc metrics on it in a handy way (i.e., w/o writing N disposable python scripts a day). The nature of my work changed, and this need is gone now. Without it, I lost the motivation to work on the project. If the need is back or a miracle happens and I have some spare time, I'd gladly get back to pq.


Make nodes required in debug fork choice endpoint

I don't think it makes sense for the response to omit fork_choice_nodes, it should be a required property.
Ref https://github.com/ethereum/beacon-APIs/pull/232

Do all client implementations provide fork_choice_nodes in their current output?  If not, this would appear to be a breaking change.

Do all client implementations provide fork_choice_nodes in their current output?

I don't see a scenario where those would be omitted, even pre-genesis, the fork choice should contain the genesis block. In case a client has not yet initialized the fork choice when the api is queried, it seems better to throw an error instead (if that's even something that can happen).

It's a debug endpoint so i'm less against altering it, but it is technically breaking if anyone has any scenario where this isn't returned.... we can let it break but its worth being aware of for any consumers... @tbenr may have an opinion...

The good news is that the four nodes I run all provide this already.  There are some minor differences in output, but I assume that as this is already in use then clients are coping with this.
So I don't see any reason not to go ahead with this change (unless anyone looking at the output below thinks that it's a major problem).
Lighthouse:
    {
      "slot": "2749152",
      "block_root": "0xbaca99eb66bf41e3ed4a50170b8b25813e8ff321bb535bef77972f051ea48ed0",
      "parent_root": "0xa7dfbe53532ad7394bdcdb54069e0ec21de5538003dc7acfa12a4cb80c204b99",
      "justified_epoch": "85910",
      "finalized_epoch": "85909",
      "weight": "51527308612500000",
      "validity": "valid",
      "execution_block_hash": "0xee03e3fadd47860dfe37f23ecd133820771e8e8de82785c77c296aefdb5936f6"
    }

Nimbus:
    {
      "slot": "2749152",
      "block_root": "0xbaca99eb66bf41e3ed4a50170b8b25813e8ff321bb535bef77972f051ea48ed0",
      "parent_root": "0x0000000000000000000000000000000000000000000000000000000000000000",
      "justified_epoch": "85910",
      "finalized_epoch": "85909",
      "weight": "51589164612500000",
      "validity": "VALID",
      "execution_block_hash": "0xee03e3fadd47860dfe37f23ecd133820771e8e8de82785c77c296aefdb5936f6",
      "extra_data": {
        "justified_root": "0x7639a595aa81a824038fabbf6b3bef45dbdd25e74ba87cbe28fd9ba4ff79989a",
        "finalized_root": "0x222a116a5ed54c1f8e398ee4946204d4f178d050de09b5e56ccb33dad18979c7",
        "best_child": "0xcc7e5e924a2f3d57b718d7f0b72ce1c9b62ed8c66be2f86871024ea8079e4525",
        "best_descendant": "0x07d54d8da166030343286ead4b499e61a9c750f50f6a2f7ac529bfde6536d724"
      }
    }

Prysm:
    {
      "slot": "2749152",
      "block_root": "0xbaca99eb66bf41e3ed4a50170b8b25813e8ff321bb535bef77972f051ea48ed0",
      "parent_root": "0x0000000000000000000000000000000000000000000000000000000000000000",
      "justified_epoch": "85910",
      "finalized_epoch": "85909",
      "weight": "51585068612500000",
      "validity": "valid",
      "execution_block_hash": "0xee03e3fadd47860dfe37f23ecd133820771e8e8de82785c77c296aefdb5936f6",
      "extra_data": {
        "unrealized_justified_epoch": "85910",
        "unrealized_finalized_epoch": "85909",
        "balance": "2624000000000",
        "execution_optimistic": false,
        "timestamp": "1728892227"
      }
    }

Teku:
    {
      "slot": "2749152",
      "block_root": "0xbaca99eb66bf41e3ed4a50170b8b25813e8ff321bb535bef77972f051ea48ed0",
      "parent_root": "0xa7dfbe53532ad7394bdcdb54069e0ec21de5538003dc7acfa12a4cb80c204b99",
      "justified_epoch": "85910",
      "finalized_epoch": "85909",
      "weight": "51595564612500000",
      "validity": "VALID",
      "execution_block_hash": "0xee03e3fadd47860dfe37f23ecd133820771e8e8de82785c77c296aefdb5936f6",
      "extra_data": {
        "state_root": "0xae813eee2c3a18baa90eafda56275eebc7fa4308ca8ab10845b96cefa9ead08d",
        "justified_root": "0x7639a595aa81a824038fabbf6b3bef45dbdd25e74ba87cbe28fd9ba4ff79989a",
        "unrealised_justified_epoch": "85910",
        "unrealized_justified_root": "0x7639a595aa81a824038fabbf6b3bef45dbdd25e74ba87cbe28fd9ba4ff79989a",
        "unrealised_finalized_epoch": "85909",
        "unrealized_finalized_root": "0x222a116a5ed54c1f8e398ee4946204d4f178d050de09b5e56ccb33dad18979c7"
      }
    }

The good news is that the four nodes I run all provide this already.

Thanks for checking @mcdee :pray:

There are some minor differences in output

seems to only be the extra_data which is free-form (and optional) as per spec

Also looks like Nimbus and Teku don't follow the spec for validity casing, should be all lower-case
https://github.com/ethereum/beacon-APIs/blob/02d595daf39943430f54aa978f88a5c50d957e64/types/fork_choice.yaml#L23


Where does the gotv demo saves?

Hi, I just tried out your plugin and it's awesome!btw  I dont getwhere the gotv saves his demo, can you help me out pls?

Hey Joele!
The demos are recorded to the file location of the game server itself. The only thing the bot does for you is automatically "tv_startrecord" and give it a name. Make sure to have GOTV enabled (tv_enable 1) and have FTP access to the server. :)

I watched every where but I'm not able to find it. Gotv is enabled


⚠ Thumbnails API Endpoint has degraded performance

In b7149ef, Thumbnails API Endpoint (https://thumbnails.roblox.com/v1/assets?assetIds=82738847&format=Png&isCircular=false&size=30x30) experienced degraded performance:

HTTP code: 200
Response time: 1378 ms

Resolved: Thumbnails API Endpoint performance has improved in 6beed5a.


How to continue the training process from a checkpoint

Hi @mkshing,
Your work is amazing, and I have tried to experiment with SVDiff on Google Colab with your repo.
But I actually got a little problem. Because the time for a Google Colab session is very limited, so I can not complete the training process in one Google Colab session. When I tried to resume the training process with "--resume_from_checkpoint", I coped with the following error:
FileNotFoundError: [Errno 2] No such file or directory: /path/to/checkpoints/pytorch_model.bin
If I understand correctly, your repo only saves spectral_shifts_te information, but accelerator.load_state() needs more (u-net/text encoder weight). Any solution for that?
Btw, thank you again for your repo. It is great when you publicize the repo.

@LeCongThuong yeah unfortunately —resume_from_checkpoint is not supposed now and waiting until the training finishes is only you can do atm.
I’ll fix it asap.

Maybe this helps:
    if args.resume_from_checkpoint:
        path = os.path.basename(args.resume_from_checkpoint)

        if path is None:
            accelerator.print(
                f"Checkpoint '{args.resume_from_checkpoint}' does not exist. Starting a new training run."
            )
            args.resume_from_checkpoint = None
        else:
            # accelerator.print(f"Resuming from checkpoint {path}")
            # accelerator.load_state(os.path.join(args.output_dir, path))
            global_step = int(path.split("-")[1])

            resume_global_step = global_step * args.gradient_accumulation_steps
            first_epoch = global_step // num_update_steps_per_epoch
            resume_step = resume_global_step % (num_update_steps_per_epoch * args.gradient_accumulation_steps)

    if args.train_text_encoder:
        if args.resume_from_checkpoint:
            text_encoder, trained_token_embeds = load_text_encoder_for_svdiff(args.pretrained_model_name_or_path,
                                                        subfolder="text_encoder", revision=args.revision,
                                                        spectral_shifts_ckpt=args.resume_from_checkpoint)
        else:
            text_encoder, trained_token_embeds = load_text_encoder_for_svdiff(args.pretrained_model_name_or_path, subfolder="text_encoder", revision=args.revision)
    else:
        text_encoder = CLIPTextModel.from_pretrained(args.pretrained_model_name_or_path, subfolder="text_encoder", revision=args.revision)

    vae = AutoencoderKL.from_pretrained(args.pretrained_model_name_or_path, subfolder="vae", revision=args.revision)
    if args.resume_from_checkpoint:
        unet = load_unet_for_svdiff(args.pretrained_model_name_or_path,
                                    subfolder="unet", revision=args.revision, low_cpu_mem_usage=True,
                                    spectral_shifts_ckpt=args.resume_from_checkpoint)
    else:
        unet = load_unet_for_svdiff(args.pretrained_model_name_or_path, subfolder="unet", revision=args.revision, low_cpu_mem_usage=True)

Thanks @dome272 !
Could you possibly make a PR for it? Then, I will merge it.

Hi, @dome272
Thank you for proposing the solution. But I have 2 questions

Firstly, what is the meaning of the "trained_token_embeds" variable? In the original code, the load_text_encoder_for_svdiff() function only returns CLIPTextModel, why do you need that variable?
Secondly, after removing the "trained_token_embeds" variable, I coped with another error:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and CPU! at the following line:
encoder_hidden_states = text_encoder(batch["input_ids"])[0]
I have not known much about the Accelerate library, but I think that the cause of the error is switching the loading models with Accelerate.load_state() into the custom load. So, I try to add ".to(accelerator.device, dtype=weight_dtype)" into Unet, TextEncoder, and AutoencoderKL but the OOM error happens. Might you have the same problem?

Thank you!

Sorry this took so long. I actually wanted to make a pull-request and add all my changes. But I'm quite busy atm and my codebase became very different from this one here. I've just uploaded all my stuff from my hacked repo here https://github.com/dome272/SVDiff-pytorch. Feel free to take out the necessary stuff you need for loading a checkpoint. I'm also training a separate token embedding for the object that is finetuned. I think I'm getting better results than without. But I didn't do much tests, maybe it was just luck and it's not so important.


Source of initial guess for the pose extrapolator if no odometry and IMU data is provided?

The documentation for Cartographer ROS says the following:
''Local SLAM inserts a new scan into its current submap construction by scan matching using an initial guess from the pose extrapolator. The idea behind the pose extrapolator is to use sensor data of other sensors besides the range finder to predict where the next scan should be inserted into the submap.''
My question is the following
If I disable use_odometry and use_imu_data, what does the extrapolator base its initial guess on ?
FYI - My robot has a laser scanner, an IMU and has wheel based odometry.

I believe the speed is simply estimated in PoseExtrapolator as the difference of the last couple of positions divided by time.

Thanks @ojura .  Would you also happen to know if using odometry and IMU as data sources, act only as initial guess to the scan matcher or do they affect the final pose estimate too and how can each scenario be configured ?

The odometry and the IMU data is used to unwarp the scan (e.g. account for motion during the lidar sweep) prior to scan matching. There is nothing to configure except the use imu data/odometry settings you mentioned. Perhaps the translation and rotation weights of the scan matcher, which control how much the prior is trusted.


vieb: 5.0.0 -> 5.1.0

Motivation for this change

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 126387 at d5ead7ab run on x86_64-linux 1

1 package built successfully:

vieb

2 suggestions:

warning: build-tools-in-build-inputs
rsync is a build tool so it likely goes to nativeBuildInputs, not buildInputs.
Near pkgs/development/tools/yarn2nix-moretea/yarn2nix/default.nix:294:7:
    |
294 |       buildInputs = [ yarn nodejs rsync ] ++ extraBuildInputs;
    |       ^

warning: build-tools-in-build-inputs
yarn is a build tool so it likely goes to nativeBuildInputs, not buildInputs.
Near pkgs/development/tools/yarn2nix-moretea/yarn2nix/default.nix:294:7:
    |
294 |       buildInputs = [ yarn nodejs rsync ] ++ extraBuildInputs;
    |       ^

Thanks!


Create rules to load external dependencies

Commit Message:
Use repositories.bzl to load the dependencies and
dependency_import.bzl to load the dependencies of the dependencies
Additional Description:
The user can use
load("@bazel-rules-fuzzing//:repositories.bzl", "fuzzing_rule_dependencies")
fuzzing_rule_dependencies()
load("@bazel-rules-fuzzing//:dependency_import.bzl", "fuzzing_dependency_imports")
fuzzing_dependency_imports()

to complete our dependencies importing.
Testing:
Local testing and CI testing
Docs Changes: N/A
Fix #49
Signed-off-by: tengpeng<EMAIL_ADDRESS>modified:   WORKSPACE
new file:   dependency_import.bzl
new file:   repositories.bzl

As https://docs.bazel.build/versions/0.22.0/external.html#transitive-dependencies pointed out, the user is responsible for loading our dependencies. In this case, I will close this PR and open a new PR to solve #49 .

As https://docs.bazel.build/versions/0.22.0/external.html#transitive-dependencies pointed out, the user is responsible for loading our dependencies. In this case, I will close this PR and open a new PR to solve #49 .

In https://docs.bazel.build/versions/master/skylark/deploying.html#dependencies, it recommends to use a macro to declare the dependencies.
In this PR, I created a macro in repositories.bzl to download the dependencies, and the loading transitive dependencies is done in WORKSPACE.


applying TestOfAttribute to unit tests

This pull request contains the following changes:

marking unit tests with [TestOfAttribute] in order to indicate the method under test
small code cleanup of unit tests
fixing a typo (matric_exp => matrix_exp) and marking the original version as obsolete

@MovGP0 -- wow. that's a lot of editing! I'm curious, what was your motivation for the TestOfAttribute?

@MovGP0 -- wow. that's a lot of editing! I'm curious, what was your motivation for the TestOfAttribute?

There are multiple advantages:

finding tests that test a specific method in search
only executing those tests instead of the whole test suit for faster iterations
easier navigation from the test to the method in the IDE


Button to resume practice from where last left off

Prominent button to resume practice at http://www.freecodecamp.com/map
This will help users to continue practice on a single button click.

This is what the Learn button on the main navigation does :smile:
Thanks and happy coding!


ATProto => ActivityPub: mentions aren't prefixed to replies

Normally, on the fediverse, replies have mentions at the start of their body. When replying, there are mentions implicitly filled in when you begin authoring a reply.
These mentions are, on Sharkey (and probably others? doesn't really matter, because it is broken on at least one fedi instance type), for public posts, the only thing controlling notification delivery.
By default, Bluesky doesn't fill in mentions in the post body, but it still deliver notifications nonetheless.
When a Bluesky account, which is otherwise bridged, replies to me on the Fediverse, I get no notification. The replies are delivered to my instance, I can see them under the original post, but as I am never mentioned, I do not receive a notification.
This is essentially the inverse of #1207, as that one is about trimming replies at the start of a post when delivering to Bluesky.

I'm also somewhat surprised that there is no existing issues for this? I tried looking, but no open issues seem relevant?

Hm, those should be added already, just only as tags and not as links in the post body: https://github.com/snarfed/bridgy-fed/blob/e375b5a8bfff3585f8fedee5fe2de85ee32afff2/activitypub.py#L725-L739
Do you have an example of a message that didn't generate a notification?
If the mention is missing or incorrect then that's a regression in Bridgy Fed, otherwise it's an issue that should be fixed in Sharkey.

@Tamschi is right. @sodiboo you're right that these @-mention "canoes" in fediverse replies are common, but they're not required by AP, and most fediverse servers still notify the original post's author without them. (Notification triggers aren't really standardized as part of AP, they're just convention right now.)

 I would also have added this reply by my own account, but the strange thing is that I could find these on the notifications tab of my Sharkey just now?? but I swear I couldn't when they were newer. the first one also definitely has a notification out of order.
however, that second thread does have a reply that semantically I should be included in, but never did receive a notification. it's actually from Mastodon replying to a post by Bridgy Fed, which only tags my Bluesky account and never delivers a notification to fedi. it's this one by zy. That reply shouldn't deliver a notification, because the canonical way to remove someone from a thread on Mastodon-likes is to no longer mention them in the start of every post, but the reason my fediverse account was excluded from mentions is because bsky doesn't add mentions.
The post that caused me to create this issue is this one by theycallhermax which seems to have never federated? My instance can't show me that reply. I'm mutuals with the reply author, so I'd expect that to cause it to show up at least. Federation is working from Bridgy Fed => my instance, because if you look at the boats you will see<EMAIL_ADDRESS>in there. This is another reply from the same user that never showed up on my instance
My own user's posts, including replies to myself, also seem to rarely federate. This is the user page in question, as viewed from my instance. It notably includes boosts from an hour ago, but it's missing this post from 3 days ago about Rust Civilization and this thread I made to test notification delivery. This is despite me being mutuals with this account since its creation.
Any issues with my own Bluesky account are likely due to that profile being cursed in Bridgy Fed (all links on that page go to 404), but it can't be the only factor because I'm not seeing Max's replies either and her profile is fine (she can also boost but she can't meaningfully reply since they are half-broken)
I think the biggest weighing factor is the fact that mastodon mentions and replies are the same physical property, and cannot be separated. When such a user replies to a bridged post, the Fediverse instance overwrites any activitypub purism and gives you an Outright incorrect answer (of dropping the ancestor's author from the thread)
also uhh, I am sort of sleep deprived as I write this. the last paragraph was difficult to formulate in a sensible way, I apologize in advance for any troubles in comprehending it.

Sorry for all the difficulties and confusion here! Lots going on, including inherent fediverse delivery inconsistency, differences between fedi software and how they each handle replies and notifications, and probably actual Bridgy Fed bugs. Sorry about those.

I think the biggest weighing factor is the fact that mastodon mentions and replies are the same physical property, and cannot be separated.

One key distinction here is between Mention tags in the underlying AS2 objects, which Mastodon does use to trigger notifications, and user-visible [@username] links in content, which it doesn't. Bridgy Fed includes the latter but not the former in its replies, and they still generate notifications in Mastodon fine.
Also, one useful thing the user-visible @-mention links in replies do is trigger notifications for all previous authors in the thread - with caveats, as you mention - not just for the immediate in-reply-to author. Bridgy Fed doesn't do that right now, so only the immediate in-reply-to author gets notified. https://github.com/snarfed/bridgy-fed/issues/1218 is somewhat related here.

Fwiw, I'm not really inclined to add the common user-visible @-mention link canoes to all previous authors, but I could definitely consider adding Mention tags for them to the underlying AS2 reply Notes.


&retro is not working

error.txt
when using this command this error shows up

I was worried that it would stop working at some point with how long it's been since the API module for it was worked on. The original developer seems to have abandoned the module, so I might have to manage it myself from now on. Gonna look into how I can fix it.

From an investigation into this issue I have come to the conclusion that the service used to provide these images has blanket banned IP ranges relating to certain hosting providers. For example, on a Hetzner-provided dedicated server it returns 403 Forbidden whilst on a Contabo VPS it returns the expected 302 Found. The solution would be to rent some cheap server to use as a proxy, although I doubt they'll like that very much and will end up banning it.
I'm not sure if there's a practical fix for this that self-hosters and Essem can do without circumventing bans.

An alternative implementation that doesn't rely on Photofunia has been introduced, note however that said implementation is vastly different from the old one.


add: disabled dates options

Add options to disable dates

Can I have you add an example of the disabled dates to the README :)


The list_item_job_instances() fails when trying to return Job history for dataflows

I have tried to use the function list_item_job_instances() to display job history for dataflows in my workspace, but I'm told the item doesn't exist - which is incorrect.

Currently, dataflows (gen1 or gen2) are not supported item types in the Fabric APIs.
https://learn.microsoft.com/en-us/rest/api/fabric/core/items/list-items?tabs=HTTP#itemtype


Fix E2E setup script failure

Recent changes to install subm on the brooker cluster caused a
failure when waiting for the subm pods to be ready b/c no
gateway node is configured so the engine isn't deployed. Removed
the code to wait for the pods.
Signed-off-by: Tom Pantelis<EMAIL_ADDRESS>

@Jaanki


feat(context): introduce c.var

This PR introduces two new features: the ability to set variables using Middleware generics and accessing them via c.var.
Currently, you can globally define variable types using generics for Hono.
type Client = 'foo client'

const app = new Hono<{
  Variables: {
    client: Client
  }
}>()

Alternatively, you can specify them for a specific middleware using ContextVariableMap:
// JWT middleare
declare module '../../context' { // Or you can write `declare module 'hono'`
  interface ContextVariableMap {
    jwtPayload: any
  }
}

The challenge here is that using ContextVariableMap makes it globally available. For instance, after importing the JWT middleware, you can access jwtPayload in any handler.
With this PR, you can confine middleware-defined variables to specific handlers where the middleware is applied.
const oneApiMiddleware: MiddlewareHandler<{
  Variables: {
    client: Client
  }
}> = async (c, next) => {
  c.set('client', new Client())
  await next()
}

app.get('/foo', oneApiMiddleware, (c) => {
  const client = c.get('client') // `client` is accessible only in this handler
  // ...
  return c.json({})
})

This offers a scoped approach to variable type availability.
The second feature is c.var. While the conventional method to access the actual value of a variable is c.get():
const client = c.get('client')

With c.var, you can use a more intuitive syntax:
const result = c.var.client.oneMethod()

The combination of these two features will provide a powerful tool for developers:
const app = new Hono()

const echoMiddleware: MiddlewareHandler<{
  Variables: {
    echo: (str: string) => string
  }
}> = async (c, next) => {
  c.set('echo', (str) => str)
  await next()
}

app.get('/echo', echoMiddleware, (c) => {
  return c.text(c.var.echo('Hello!'))
})

This was discussed in issue #414 and is now being realized.

Hi @usualoma , @ThatOneBro , and others.
What do you think about this feature? I think this is good because it will achieve big things with minimal change of the APIs.

I think this makes a lot of sense overall. 👍 LGTM

With this commit, I've set the properties to read-only. This is to prevent confusion, emphasizing that users should exclusively use c.set() to assign values.


Added null check to stacktrace

Hi there, is there any chance you could do a new release with this fix?


Prevents overwriting with duplicate slug names

- Summary
Fixes #1377 & #1164, piggy backing on PR #1239.
getEntry in github's implementation was returning a 404, and the error wasn't being resolved correctly in checkOverwrite in backend.js.
- Test plan
I created a github back end and tested. It worked.
- Description for the changelog
Prevents duplicate slugs from overwriting files.
- A picture of a cute animal (not mandatory but encouraged)

Deploy preview for cms-demo ready!
Built with commit a3b883dd9902eebd3bc70aca20cd5996452bb0d9
https://deploy-preview-1381--cms-demo.netlify.com

Deploy preview for netlify-cms-www ready!
Built with commit a3b883dd9902eebd3bc70aca20cd5996452bb0d9
https://deploy-preview-1381--netlify-cms-www.netlify.com

We'll need a maintainer to sponsor this one, closing as stale for now.


Update x/mod dependency for the new golang version format

Update dependency for parsing go.mod file. Now it can parse go.mod with the new golang version format 1.23.3 instead of the old format 1.21

Hi, @Arlet2 thanks for the PR, can you check the pipeline that breaks? it seems to be related to x

Hi @Arlet2 also can you please remove vendor package from the PR, and put that in .gitignore

Hi, @parvez3019! I pushed your suggestions to PR

LGTM


new Vibrant() on  is a really bad idea

Causes Firefox and Chrome to grind to a complete halt :\
"But why would you hide the image that you're trying to accentuate I don't even what"
I'm doing some dynamic styling in the browser and I was hoping to hide the image element itself, using it only as the source to script in a background-image with background-size: cover and dynamic background/text overlay color values (provided by Vibrant.js)
But that's besides the point, checking the image dimensions or display properties before copying the image to a new canvas element might be a good idea idk
Love the library, ran into this just now, thought I'd say something =)

Did you ever resolve this? I am actually trying to do the same thing and am running into all kinds of problems.

If you're asking me that question, then yes I resolved it by not doing that and instead re-inventing the wheel.


Broken Search Box on Discover page

When I type "industrial" into the text search box on https://www.hydroshare.org/search/ I should have this resource returned: https://www.hydroshare.org/resource/4800f6fafe7e43bc969a3770ea817738/
Only when I narrow resources down by owner or author does this resource appear.
Seems the text search functionality is broken.

@Lizabrazil @dtarb @martinseul @mjstealey @cbcunc
My initial theory is that the nightly updates to the SOLR catalog aren't happening. This should show up if they were up to date.

My first suggestion is to first run the manual SOLR catalog building procedure that Michael used to build the catalog from scratch, and see if that addresses this particular problem.
If that fixes the problem, I have the ability to reproduce it by creating a new resource.
If that does not fix the problem, then there is a schema problem. This is very, very unlikely.

@dtarb @martinseul @Lizabrazil @mjstealey @cbunc The manual catalog update process does not seem to be running now. We should turn that back on. However, the bug seems to have two parts:

The overnight update is needed and isn't running.
The immediate post of the resource to SOLR when it's made public does not include enough metadata for basic discovery.

A reminder of how this is supposed to work.

When a resource is made public, a half-hearted SOLR record is generated and pushed to SOLR. It is not necessarily a full record. Mostly we did this to save development time. It's enough to discover the resource but not to discover it in all ways.
Theoretically, overnight, these records are updated to be full records.
When a resource is made private, its SOLR record is immediately deleted.

So the nightly update needs to be turned back on, and the temporary push to SOLR needs to be made a bit more complete so that basic textbox searching works immediately.

@phuongdm This is where we're discussing the SOLR problems.

I have enabled Jenkins->Housekeeping->nightly-rebuild-solr-index. It's set to run between midnight and 1am nightly. But I just manually, initiated it. I don't know how long it should take to run. But it seemed to be sitting and spinning, as well as running a Django shell, so I killed the job and investigated. hsctl It appears to only be running docker exec hydroshare python manage.py rebuild_index --noinput in response to a reubuild_index subcommand. This doesn't seem quite right to me. I would have thought it would be talking to the solr container.

There is no indication this job has ever run before through Jenkins. How have the index rebuilds been occurring all these years?

@cbcunc I have no idea!
This may have been dropped from Jenkins for some reason.

@dtarb @Lizabrazil @martinseul @aphelionz I analyzed the code and it is really easy to duplicate the behavior that @Lizabrazil describes:

Create a resource with the wrong name.
Make it public.
Change the title to have a desired keyword that doesn't appear anywhere else.

That keyword will not be discoverable using the subject box. The reason is that SOLR data isn't updated except when something is made public or private. For now, a simple fix is to make the resource private and then make it public again. This will push the change to SOLR. Longer term, we need to insure that SOLR is updated for any change of a public resource.
There are other deeper issues: SOLR is updated when a user explicitly changes something to private, but is not updated when a resource is forced to be private due to inappropriate edits. Solution to this is to write a definitive routine for making a resource private that always updates SOLR, and then call it everywhere one wants something to become private.

Thanks for tracking this down.

@alvacouch:
From 10:00am to 3:00pm this Wednesday, I and @yxiao did some investigations on the "SOLR unsecurity link" and found that "Last Modified" field of "Collection1" Core were changed several times. And, I agree with @cbcunc that the Jenkins's task didn't run on that time (in fact it never run before).
Is that the case you wrote: "SOLR is updated when a user explicitly changes something to private"?

@alvacouch , @cbcunc

./hsctl managepy rebuild_index ===> asked me to make sure delete all docs or not ===> we can't call it in Jenkins task

./hsctl rebuild_index  ===> run without asking  ===> could call in Jenkins task

On cuahsi-dev-1.hydroshare.org, it took me around 12 minutes to finish. I will manually run it tonight on www.hydroshare.org and see how long will it take then decide to run the task daily or weekly.

./hsctl rebuild_index is the correct command to run (no managepy). Running it does not fix the problem in this issue though. I spoke with @mjstealey and rebuild_index has only ever been run by ./hsctl rebuild when rebuilding the site. It doesn't appear the intention was to run it every day or on a regular schedule, but maybe on demand from Jenkins.

Make sense, thanks

I still cannot find the resource...
I need this to work next week for a class exercise. I typed "industrial" @alvacouch
Thanks

@Lizabrazil - Oddly typing in a portion of the word, like "indus" seems to find it.

@Lizabrazil for the workshop can you fix it by making it private then public as @alvacouch suggested.

@dtarb That did not work. I need this fixed by October 6th or I'll have to take out the HydroShare portion of the exercise.

@alvacouch If i search just for "plant" then i can find the resource.

@lizabrazil That suggests that the search query is not matching the first word in the title. Could you do me a big favor and see if that happens for other resources?

This line is very, very suspicious
line 19 of hs_core/discovery_form.py
This is the line:
sqs = self.searchqueryset.filter(text__startswith=self.cleaned_data.get('q'))\
               .filter(discoverable=True)\
               .filter(is_replaced_by=False)

The line includes the query from the search box (named 'q'). The query asserts that the text field "starts with" the query q. Since the text field is a catchall field, it contains all text from all fields in a non-deterministic order. Thus startswith is most definitely the wrong operator.
Looking deeper, the text field is:
"text": " \n    4800f6fafe7e43bc969a3770ea817738\n \n \n\n    Industrial Plants PA\n \n \n    This layer  displays the coordinates of several industrial waste plants throughout Pennsylvania.\n \n\n\n\nMay 4, 2017, 5:36 p.m.\n\n \n \n \n    eng\n \n\n    GenericResource\n \n \n    Generic\n \n \n \n \n \n\n    \n        liza brazil\n     \n\n\n\n    \n        pennsylvania\n     \n\n    \n        industrial\n     \n\n    \n        industrial plants\n     \n\n    \n        water quality\n     \n\n\n    \n        CUAHSI\n     \n\n\n    \n<EMAIL_ADDRESS>  \n\n\n\n\n\n\n\n\n\n\n    application/json\n\n\n    hydroShareIdentifier\n\n\n\n\n    lizabrazil\n\n\n    liza brazil\n\n\n    liza brazil\n\n\n    \n\n\n    lizabrazil\n\n\n    liza brazil\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
This is really ugly and could have been cleaned up by deleting \ns but it is clear that "Industrial Plants PA" is not the first string!
Thus this would result in the behavior we see.

Several things here are suspicious.
The code startswith would be fine if the text field were tokenized. However, it is not tokenized; it is a generic character field:
Line 16 of search_indexes.py
Fixing that requires changing the declarations and re-indexing.
To fix this temporarily without re-indexing, see the following page:
http://django-haystack.readthedocs.io/en/master/searchqueryset_api.html#field-lookups
We need to replace startswith with something more reasonable.
Let's weigh the options:

contains will match substrings of words, e.g., "logical" matches "hydrological". Users didn't like that before, which is how we got this solution.
fuzzy works like contains but matches near misspellings. Even more strange results.

For now, a simple solution is to use contains'. A better solution would be to switch text to tokenized and then use startswith`, which will do what we really want.
My guess is that we got to this particular solution by not reading a stackoverflow post carefully enough.

@dtarb @Lizabrazil @martinseul The above proposal is not going to fix this. Somehow, the keywords are being tokenized and stemmed internally. I tried other words and the situation was consistent. For example, "variat" matches "variations" but "variation" does not. Likewise, "indus" matches "industrial" but "industria" does not. So, the search is stemming the words that come into text and then matching the stems.
"indus plant" works but "indus plants" does not. "plant" is the stem of "plants".
Solution is to turn stemming off.

@dtarb @Lizabrazil @martinseul
Here is the root of the problem:
    <fieldType name="text_en" class="solr.TextField" positionIncrementGap="100">
      <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.StopFilterFactory"
                ignoreCase="true"
                words="lang/stopwords_en.txt"
                enablePositionIncrements="true"
                />
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.EnglishPossessiveFilterFactory"/>
        <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
        <!-- Optionally you may want to use this less aggressive stemmer instead of PorterStemFilterFactory:
          <filter class="solr.EnglishMinimalStemFilterFactory"/>
        -->
        <filter class="solr.PorterStemFilterFactory"/>
      </analyzer>

This is applied to the text field (on beta). This reads that all words appearing in the whole resource are transformed by removing possessives, reducing to the stem of the word, etc.  The problem is that when you make a query, including one of these unmodified words doesn't match.

@dtarb @martinseul @Lizabrazil There is still something strange about this. Theoretically, both the query and the text should be stemmed. This doesn't seem to be happening. It seems that the query is not being so transformed. Perhaps this is the result of "startswith"; I don't know.
Just so you understand a bit more, here is some background.
"Stemming" is the process of deriving a root word from a complex word. Some stemmers are simple, and some are quite complex. The simplest transform "papers" to "paper" (removing plurals) and "plastering" to "plaster" (or "plast").
The point is that everything works properly if both the query and the text to be matched are stemmed in the same way. Somehow,

the text is getting stemmed.
the query is not getting stemmed.

This is happening automagically in Haystack. Queries direct to SOLR are working properly.
So the whole scheme is breaking down.
The root of the problem may be here:
line 19 of hs_core/discovery_form.py
In this line, the query "startswith" is used rather than "contains". I think that might have caused this behavior (because stemming a prefix doesn't make sense).
Could someone try "startswith" => "contains" to see if that addresses the issue?

May be better to use icontains (https://docs.djangoproject.com/en/dev/ref/models/querysets/#icontains)

@pkdash icontains is not implemented for haystack. Haystack is an incomplete implementation of the queryset model for Django.

@martinseul @dtarb @Lizabrazil @pkdash
My research suggests that the simplest fix is to replace text__startswith= with text=. This will invoke the default mechanism for matching in SOLR, which is term-by-term aggressive stem matching.
What this will do: Everything will be tokenized and matched word-to-word (both queries and titles):

"Industrial" matches "Industrial"
"Indus" doesn't match "Industrial"
"Indust" does match "Industrial"

because stem("Industrial") == "Indust".
What we really want is "text= or text__startswith=". Django Q expressions do not work for Haystack. Instead, Haystack supports "SQ" expressions. The appropriate expression might be
cleaned_q = self.cleaned_data.get('q')
sqs = self.searchqueryset.filter(SQ(text__startswith=cleaned_q) | SQ(text=cleaned_q))\
                .filter(discoverable=True)\
                .filter(is_replaced_by=False)

What this does:

query
stem(query)
stored
stem(stored)
matches?
how?

Industrial
indust
Industrial
indust
Yes
=

Indust
indust
Industrial
indust
Yes
=

Indus
indus
Industrial
indust
Yes
startswith

Industr
industr
Industrial
indust
No
not prefix or exact

So, things match unless one specifies a match term that is between the stem length and the total length. Then matches fail.
Other than this, we are getting beyond what Haystack was designed to do and have to take more drastic measures.

@Lizabrazil @martinseul @aphelionz @dtarb
The proposed fix is running on cuahsi-dev-2.hydroshare.org. Here is the current behavior (note the stem correction from the above):

query
stem(query)
stored
stem(stored)
matches?
how?

Industrial
industri
Industrial
industri
Yes
=

Industria
industria
Industrial
industri
No
neither prefix nor =

Indus
indus
Industrial
industri
Yes
prefix of stem but not =

Industri
industri
Industrial
industri
Yes
prefix of stem but not =

So, "Industrial" and "Industri" match, but "Industria" does not match due to the aggressiveness of the stemming algorithm.
I will submit a PR.

@dtarb @Lizabrazil @martinseul @aphelionz
If this isn't good enough, I suggest using the weak stemmer instead of the aggressive stemmer that is the default in SOLR. This requires patching the SOLR generator in Haystack. This is common enough a task that there are instructions on the web as to how to do it. This then generates a SOLR configuration file that uses a weaker stemmer, that stems "planned" => "plan" and "grapes" => "grape" and "Alva's" to "Alva", but does not stem "industrial" to "industri".
Then all prefixes of "Industrial" will match "Industrial".

This is fixed in 1.13.1.

@alv


Improve Documentation #2845 - lexical/list

This PR adds documentation to the lexical/list module @ files: formatList.ts, LexicalListItemNode.ts, LexicalListNode.ts, and utils.ts.

Thank you! I will say, I was a bit worried about over explaining the logic, there are a lot of nested if checks and I didn't know if it was more helpful to be specific or general in some cases.


Removed unnecessary steps when building docker image with plugins (OFBIZ-12876)

Thanks Daniel, sounds good to me +1


Stresstest sample causing issues

Last I checked, the stresstest sample does not complete in a reasonable amount of time. Previously it would give its first errors after 5-7 seconds, now it doesn't give anything after at least 20 seconds.

I can not repro this after the recent improvements


Returning 500 YARP, returns 200 manually through Service Fabric reverse proxy

Describe the bug
I've set up a solution using YARP as a gateway reverse proxy between external requests and the reverse proxy intended to be utilized internally by Service Fabric. When running a 5-node cluster in local debug, I can issue a request to the project housing YARP and see the logging indicating that it's proxying the request to the appropriate endpoint, but it yields a 500.
Sending a request via Postman to the same address YARP uses works with a 200.
To Reproduce
I've created a sample solution SfYarp.zip that demonstrates the issue. You'll need to run it on a machine with the Service Fabric SDK installed, but otherwise just hit F5 in Visual Studio to start it up.

Initiate a request to http://localhost:8603/weather and you'll see something like the following in debug:

Microsoft.AspNetCore.Hosting.Diagnostics: Information: Request starting HTTP/1.1 GET http://localhost:8603/weather
Microsoft.AspNetCore.Routing.EndpointMiddleware: Information: Executing endpoint 'forecast'
Microsoft.ReverseProxy.Middleware.LoadBalancingMiddleware: Information: Operation started: ReverseProxy.PickDestination
Microsoft.ReverseProxy.Middleware.LoadBalancingMiddleware: Information: Operation ended: ReverseProxy.PickDestination, 3.1505ms, success
Microsoft.ReverseProxy.Middleware.ProxyInvokerMiddleware: Information: Operation started: ReverseProxy.Proxy
Microsoft.ReverseProxy.Service.Proxy.HttpProxy: Information: Proxying to http://localhost:19081/SfYarp/OtherService/WeatherForecast
Microsoft.ReverseProxy.Middleware.ProxyInvokerMiddleware: Information: Operation ended: ReverseProxy.Proxy, 115.7036ms, success
Microsoft.AspNetCore.Routing.EndpointMiddleware: Information: Executed endpoint 'forecast'
Microsoft.AspNetCore.Hosting.Diagnostics: Information: Request finished in 165.0948ms 500

Note that line 6 contains the URL it's intended to proxy to: http://localhost:19081/SfYarp/OtherService/WeatherForecast This is as expected, but you'll note that it concludes with a 500.

Initiate a GET request manually to this URL and you'll get a 200 OK with a JSON response.

When the request is performed manually, it works fine. When performed via YARP, it fails.
There are no exceptions observed anywhere, just the 500 internal service error when attempting to use YARP.
Further technical details
Using .NET Core 3.1 on Windows 10 Pro using VS 16.6.3
Microsoft.ServiceFabric.AspNetCore.Kestrel - 4.1.417
Microsoft.ReverseProxy - 1.0.0-preview.2.20318.2

Are there any more hints of you turn the log level up to Debug?
What do the logs in the destination server say?

I'm not in terribly familiar territory here, so bear with me. Finally got some additional information to appear in the other service while debugging it and I see this error:

Microsoft.ReverseProxy.Middleware.ProxyInvokerMiddleware: Information: Operation ended: ReverseProxy.Proxy, 162.1257ms, success
Microsoft.AspNetCore.Routing.EndpointMiddleware: Information: Executed endpoint 'forecast'
Microsoft.AspNetCore.Hosting.Diagnostics: Information: Request finished in 225.4117ms 500
System.IO.IOException: Failed to bind to address http://[::]:8603: address already in use.
---> Microsoft.AspNetCore.Connections.AddressInUseException: Only one usage of each socket address (protocol/network address/port) is normally permitted.
---> System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted.
at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Bind(EndPoint localEP)
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
--- End of inner exception stack trace ---
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.<>c__DisplayClass21_01.<<StartAsync>g__OnBind|0>d.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)    --- End of inner exception stack trace ---    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)    at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.BindAsync(AddressBindContext context)    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.EndpointsStrategy.BindAsync(AddressBindContext context)    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(IServerAddressesFeature addresses, KestrelServerOptions serverOptions, ILogger logger, Func2 createBinding)
at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)

I recall noting in the Service Fabric documentation that "...unlike HTTP.sys, Kestrel doesn't support port sharing between multiple processes. Each instance of Kestrel must use a unique port."
Is it possible that's the issue here? I'm going to try swapping out the use of Kestrel with HTTP.sys and see if that remedies this.

Yep, that did it - simply switching out the use of Kestrel with HTTP.sys did the charm. Here's a working variant of the original solution:
SfYarp-httpsys.zip
Does this mean that Kestrel isn't a supported platform for YARP and that any implementations must use HTTP.sys instead?

There are multiple things at play, let's sort them out one at a time.
1. Port 8603 already in use

IOException: Failed to bind to address http://[::]:8603: address already in use

@WhitWaldo you must have something else running that already claimed that port. I see that you are (correctly) setting Gateway's InstanceCount to 1 when running locally, so it is not immediately clear who the culprit is. Sugg running netstat -abno from an elevated prompt to identify who is LISTENING on :8603 next time you see this error.
2. 500 status code
Since your localhost communication between Gateway and SF Reverse Proxy is unencrypted, you can easily capture it using a tool like Wireshark. Just compare that to a trace where you hit it with postman. you can also inspect SF Reverse Proxy diagnostics. see docs

Does this mean that Kestrel isn't a supported platform for YARP and that any implementations must use HTTP.sys instead?

Kestrel should work with YARP, that's what we use by default. Port conflicts are usually a configuration issue.

@davidni
netstat yields two listeners on 8603 when I launch the SF solution:

TCP    <IP_ADDRESS>:8603           <IP_ADDRESS>:0              LISTENING       92500
[Gateway.exe]
TCP    [::]:8603              [::]:0                 LISTENING       92500
[Gateway.exe]

From what I can find, this means that it's listening on all possible IPv4 addresses and the IPv6 loopback for port 8603. Does this count as multiple uses of the port?
I thought I'd see if I could just set the Kestrel options to use so it'd specifically just use <IP_ADDRESS> on 8603 instead of <IP_ADDRESS> or [::]
.UseKestrel(opt => {
  opt.Listen(IPAddress.Lookback, 8603);
});

But this just added another listener explicitly using <IP_ADDRESS> on top of the other two. Looking at the Wireshark traffic, it just led to the same binding error as posted before:

410	4.024712	<IP_ADDRESS>	<IP_ADDRESS>	HTTP	277	GET /weather HTTP/1.1
411	4.024763	<IP_ADDRESS>	<IP_ADDRESS>	TCP	44	8603 → 61259 [ACK] Seq=1 Ack=235 Win=10232 Len=0
417	4.070687	::1	::1	TCP	76	61345 → 19081 [SYN] Seq=0 Win=65535 Len=0 MSS=65475 WS=256 SACK_PERM=1
418	4.070769	::1	::1	TCP	76	19081 → 61345 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=65475 WS=256 SACK_PERM=1
419	4.070817	::1	::1	TCP	64	61345 → 19081 [ACK] Seq=1 Ack=1 Win=2618880 Len=0
420	4.071542	::1	::1	TCP	441	61345 → 19081 [PSH, ACK] Seq=1 Ack=1 Win=2618880 Len=377 [TCP segment of a reassembled PDU]
421	4.071570	::1	::1	TCP	64	19081 → 61345 [ACK] Seq=1 Ack=378 Win=2618880 Len=0
422	4.071714	::1	::1	HTTP	69	GET /SfYarp/OtherService/WeatherForecast HTTP/1.1
423	4.071729	::1	::1	TCP	64	19081 → 61345 [ACK] Seq=1 Ack=383 Win=2618880 Len=0
424	4.072506	::1	::1	HTTP	189	HTTP/1.1 500 Internal Server Error
425	4.072548	::1	::1	TCP	64	61345 → 19081 [ACK] Seq=383 Ack=126 Win=2618624 Len=0
426	4.086716	<IP_ADDRESS>	<IP_ADDRESS>	HTTP	169	HTTP/1.1 500 Internal Server Error

As a last attempt to get it to quit listening on multiple ports, I updated the .UseUrls(url) to use "http://localhost:8603" instead of the default being passed into it "http://+:8603" but netstat still indicates that it's listening on "127.0.01:8603" as well as "[::]:8603" and I'm not seeing how one might disable one or the other to reduce it to only a single listener.
I'm not really seeing anything special you have in your samples regarding Kestrel configuration and I'm not really doing anything special here apart from running everything within Service Fabric. Any thoughts on why this isn't working within an SF environment? I'd wait on that SF integration PR, but I don't see anything in there that the Kestrel configuration to demonstrate what's not right here.

Kestrel can be configured via code, config file, or UseUrls. Is it possible you have the same url defined in both code and the config file?

Don't think so. The only sections in my appsettings.json (zipped two sample solutions in the above responses) are Logging, AllowedHosts and ReverseProxy. My WebHostBuilder is equivalent to what the Service Fabric template kicks out + logging:
                        return new WebHostBuilder()
                            .UseKestrel()
                            .ConfigureServices(
                                services => services
                                    .AddSingleton<StatelessServiceContext>(serviceContext))
                            .ConfigureLogging(logging =>
                            {
                                logging.ClearProviders();
                                logging.AddDebug();
                            })
                            .UseContentRoot(Directory.GetCurrentDirectory())
                            .UseStartup<Startup>()
                            .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                            .UseUrls(url)
                            .Build();

The input URL just gets set to http://+:<port> where the port is set in the SF ServiceManifest:

And that's it - can't find anything else that deals with Kestrel configuration.


Run ECS task locally

Feature
Similar to how moto supports running Lambda Function as seen here, it would be great if moto had the ability to run local docker containers to simulate ECS task definitions. Ideally, a user would be able to send a RunTask request to the moto server and moto will be able to parse the ECS task definition that's registered within the moto server or AWS and run the task.
Ideas
A:
Steps:

User registers ECS task defintion
User executes the RunTask API with the moto endpoint URL
Moto responds with mock RunTask response
Moto server gets the task definition content via client.describe_task_definition()
Task definition is saved to tmp file (e.g /tmp-path/task-definition.json)
Using ecs-cli, run ecs-cli local create --force --task-def-file/tmp-path/task-definition.json --output $OUT_FILEPATH (OUT_FILEPATH can be a shared docker volume path so users can read the docker-compose file from other containers)
Moto server runs docker compose command to execute the generated docker-compose.yml file

If this seems within the scope of moto, I'd be more than happy to get a PR going!

Hi @marshall7m, thanks for raising this!, I've given this some thought, and although I like the approach and usefulness, I'm going to declare this out of scope.
The ecs-cli library is platform dependent, so executing that would have to be inside another Docker-container (or packaging the libraries for all three platforms...)
There are similar services where Moto veers from mocking territory into emulation land, but that workflow would become too intensive to justify it being part of a 'mocking library'.
It would also generate a lot of maintenance that I'm not ready to take on.
I'm going to close this, but feel free to continue the conversation if you have other idea/suggestions around this. topic

Hi @bblommers, thanks for taking the time to think about it! After you mentioned it, I agree with your point that this idea is out of the scope of moto.
In terms of other services that emulate running ECS tasks locally, could you mention what those are? From my research, I found Local Stack to be the only service that does, although it comes with the caveat that users need the Pro subscription in order to work with ECS.

Besides LocalStack, I'm not aware of any other libraries/services in this space, @marshall7m. There are other AWS test libraries, but they tend to focus specifcally on Serverless applications, i.e. AWSLambda/S3/etc - nothing more.