loading vocab for new language

Hi again,
I finally managed to build new vocab for my language ( using your init.py). But, it can't work. Here is the output for spacy.load().

spacy/vocab.pyx in spacy.vocab.Vocab.load (spacy/vocab.cpp:4866)()
105         with (path / 'vocab' / 'strings.json').open('r', encoding='utf8') as file_:
106             self.strings.load(file_)
--> 107         self.load_lexemes(path / 'vocab' / 'lexemes.bin')
108         return self
109
spaCy/spacy/vocab.pyx in spacy.vocab.Vocab.load_lexemes (spacy/vocab.cpp:9365)()
426
427             lexeme.vector = EMPTY_VEC
--> 428             py_str = self.strings[lexeme.orth]
429             key = hash_string(py_str)
430             self._by_hash.set(key, lexeme)
spaCy/spacy/strings.pyx in spacy.strings.StringStore.getitem (spacy/strings.cpp:2481)()
135                     return _decode(utf8str)
136                 else:
--> 137                     raise IndexError(string_or_id)
138         else:
139             if isinstance(string_or_id, bytes):
IndexError: 412564

Here showed that orth >= self.size. How is it possible?

Hi!
Having the same issue here loading a Spanish language model created with the init.py script as well:

  File "test_load_new_model.py", line 3, in <module>
    nlp = spacy.load("es", parser=False, tagger=False, entity=False,path="/usr/local/lib/python2.7/site-packages/spacy/data/")

  File "/usr/local/lib/python2.7/site-packages/spacy/__init__.py", line 42, in load
    return cls(**overrides)

  File "/usr/local/lib/python2.7/site-packages/spacy/language.py", line 269, in __init__
    if 'vocab' not in overrides \

  File "/usr/local/lib/python2.7/site-packages/spacy/language.py", line 56, in create_vocab
    tag_map=cls.tag_map, lemmatizer=lemmatizer)

  File "spacy/vocab.pyx", line 107, in spacy.vocab.Vocab.load (spacy/vocab.cpp:4871)

  File "spacy/vocab.pyx", line 428, in spacy.vocab.Vocab.load_lexemes (spacy/vocab.cpp:9370)

  File "spacy/strings.pyx", line 137, in spacy.strings.StringStore.__getitem__ (spacy/strings.cpp:2481)
IndexError: 416470

Did you manage to solve this issue?
Best wishes

Hi again:
Just to add some more information regarding why this happens.
We managed to track it down to the way in which the StringStore is loaded/dumped.
In our case our vocabulary contained some (probably undesired symbols), which are parsed  as 0-length strings by the ujson library (which is used for loading and dumping the strings.json to build the StringStore). Using the json library this does not happen.
The issue is that those 0-length strings are skipped when creating the store, allocating memory, increasing strings.self.size, etc. In fact, there is one 0-length string which is created in the language defaults, and that one is skipped but taken into account in the class string logic (it seems that reflecting this in the size property).
Problem is, if you have more 0-length strings in your strings.json this will lead some of your strings to be out of bounds. In our case, original size of the strings list was 416471, thus logically the index should not raise and IndexError for position 416470, but our list contained two 0 sized strings thus leading the size of the StringStore object to be 416470.
As far as I see there are two options:

Make sure that your corpus does not contain weird characters, that will likely influence the performance of your language models anyway.

We have a modification of the StringStore() class to account for this edge case (simplifying the handling of the self.size property, and allocating space even for 0-length strings. Of course it would need proper testing and a deeper understanding of the internals of the string store in spaCy, before even proposing/discussing a pull-request. We would be pleased to contribute to this great library and we can discuss this with @honnibal, but maybe is just intended to be this way.

In any case, we finally managed to properly load the Spanish lang resources into spaCy.
Best wishes,
Dani

Hi Dani,
Thanks a lot for the analysis!
First of all, there are no weird characters! Every string should be valid input, and there's only one 0-length string. So if the ujson library is ignoring some inputs and instead writing the empty string, that's a serious error!
The strings.json is assumed to be unique --- it should be index-aligned to the vocabulary.
So it sounds like there can be one of two problems:

Something is seriously wrong in the internals of ujson, and we should switch to e.g. cjson or something

(Most likely) I'm screwing up the serialisation of unicode strings, and ujson is receiving malformed input.

Hi Matt!
Thanks for your prompt response!
You are absolutely right.
Yesterday, I have experienced a somewhat related issue with ujson, in this case during the execution of the end_training method for saving the serialization.json after training the parser with a Spanish Corpus (UD_Spanish) with the train_ud. In this case, the json.dump method with Python 2.6 exhibits a weird behaviour as well with regards to unicode and strings, so the json could not be dumped. After some research, this is a known problem with ujson for Python 2. Switching to Python 3 solved this issue and training the parser and setting up the model worked like a charm with your scripts (congrats for your great job by the way!). This led me to think that maybe if we switch to Python3 for the full lang model generation will solve this previous issue as well.
We will during the coming days execute the full pipeline: create Spanish word vectors with 1B words corpus, generate brown clusters, word_freqs, etc. and train the parser. This time using Python 3 env. We will report on this as well. Also, we would be pleased to contribute the Spanish model, after proper testing and evaluation if you think is a good idea.
Thanks!
Dani

Great!
Thanks for the pointer re ujson. That's very disappointing — I'll switch libraries.
Btw a quick pointer re Spanish: I've done some preliminary experiments on on the AnCora corpus, and in fact it's my main testbed for the spaCy 2.0 models because it's one of the few UD's that's GPL, so everyone can use it.
A little trick I found effective in my preliminary experiments: try concatenating the dependency label onto the UD POS tag, to train the tagger. The UD tags are a little bit easy on Spanish (the model is getting ~98%). Making life a little harder for the tagger so that it learns more useful features for the parser seems to help.
There's probably a clever intersection of the morphology features you could use instead, but when I tried just using all morphology features the tagset was a bit large and the model didn't do so well. Ideal would be something like CCG supertags. But for a quick and dirty hack, augmenting with the dep label was giving me about +0.5%.

Guys, I think I am facing the same issue for Hungarian. @dvsrepo do you have any hints on how did you get rid off "weird" tokens?

Hi @oroszgy:
If I remember correctly, this issue was related to issues when using ujson using Python2. From that point on, we switched to Python3 for everything we do with spaCy. I recommend using virtualenv and Python3, although the issues might have been solved for Python2.
Can you share your precise environment and what issues you are a facing?
We can also chat in gitter if you like.
Dani

I've just tracked down the issue, and as I see (in my case) it is rather related to spacy's  StringStore utf8 encoding method and string handling differencies between Python version.
More concretely, there is a strange error with surrogate strings. The code below runs well in Python2 but throws an exception in Python3.
from spacy.strings import StringStore

ss = StringStore()
s = "\ud800"
ss[s]
ss.dump(open("./out", "wb"))

ss2 = StringStore()
ss2.load(open("./out", "rb"))

Adding this one line to my preprocessing script solves the problem:
word = ftfy.fix_text(word)

I beleive that something similar happened with ujson and Python2 in your case.


Support for 7.0.0 (Nougat)

Nougat seems to have squashed a number of dtf features. I just ran some tests and almost all modules fail. I'll have to investigate and start updating content.

Are far as dtf is concerned, Nougat + Nougat_R1 are supported. The modules are a different story. I added a small warning to let people know their API is not supported (by dtf). It supports through Nougat R1 (https://github.com/android-dtf/dtf/commit/8e179f17a4d94e14b7cc1bf8ecd9e89266d325b1)


Modify hard-coded image dimensions and train/val split

Image dimensions are currently hard-coded throughout cocopen.py (height = 1080, width = 1920). Need to replace these with a class variable self.height and self.width. It would be helpful to have a function that checks to make sure all images used to generate the dataset are the same dimensions before beginning the image combination - should throw an exception if image dimensions are not consistent. This function could be included in a dataset preprocessing script (i.e., not in coconut.py).
Note: This affects the scale_image, random_operations, and color_augmentation functions in coconut.py where image dimensions are hard-coded
Train/val split is currently hard-coded as 0.8 in two locations. It would be better to let the user define their train/val split (some users may want a 0.9/0.1 split, for example). Assign this to a class variable and/or read it in from run.py.
if 0 <= rand < 0.8:

Added dataset_parameters: train_split: 0.8 in ./config/parameters.yml file. Still need to read in image dimensions from ./config/parameters.yml and verify all images used in the data set have the same dimensions before performing combine().

#22


feat: random filter id

What this PR does / why we need it:
Filter IDs must be unpredictable so filters can't be touched/uninstalled by other users.
And this makes it possible to sticky load balancing filter requests.
Which toolchain this PR adaption:
No Breaking Change
Special notes for your reviewer:
NIL
CI Description

CI Name
Description

Chaos CI
Test the liveness and robustness of Axon under terrible network condition

Cargo Clippy
Run cargo clippy --all --all-targets --all-features

Coverage Test
Get the unit test coverage report

E2E Test
Run end-to-end test to check interfaces

Code Format
Run cargo +nightly fmt --all -- --check and cargo sort -gwc

Web3 Compatible Test
Test the Web3 compatibility of Axon

v3 Core Test
Run the compatibility tests provided by Uniswap V3

OCT 1-5 | 6-10 | 11 | 12-15 | 16-19
Run the compatibility tests provided by OpenZeppelin

CI Usage

Check the CI you want to run below, and then comment /run-ci.

CI Switch

[ ] Chaos CI
[x] Cargo Clippy
[ ] Coverage Test
[x] E2E Tests
[x] Code Format
[x] Unit Tests
[x] Web3 Compatible Tests
[ ] OCT 1-5 And 12-15
[ ] OCT 6-10
[ ] OCT 11
[ ] OCT 16-19
[ ] v3 Core Tests

rand is re-export from protocol::rand, it is useless to add to Cargo.toml.
The id field in FilterHub struct seems useless, it should be removed.

/run-ci

@driftluo Please review it.


exposes opts + bits

LGTM

thank you @noffle . Could you add a test with this PR?

Is it possible to supply less than 1024 or a non-scalar value of bits to this? I remember seeing this in go where they require the bits to at least be greater than 1024 as well as some tests that ensure it's multiple of 1024.
https://github.com/ipfs/go-ipfs/blob/6f8ea136d5d9d927016c59a232b712e7bd6ba795/repo/config/init.go#L92**

@diasdavid: Added a test. It's not very rigorous, but you aren't guaranteed a certain # of bytes in the resultant private key -- it varies from key to key.
@nginnever: You can generate smaller keys (it would seem). go-ipfs puts that limit for security, but I think that should be an IPFS-level concern (application) rather than a library concern.

LGTM, thank you @noffle and @nginnever for the input! :)

released on 0.6.1 :)


add option disableKeyDownBinding to partially fix #1444, #1387

Disable keyDown binding will stop monotoring keydown events, in some cases user only wants to use mouse/touch to select a date and leave the input as readonly.
the onKeyDown is a little too agressive and conflicts with other inputs/keystrokes outside of flatpickr, disabling the binding is a good alternative
this partially address issues #1387 and #1444 by providing a way to disable the keydown event listener
this also fixes an issue found in my lib (Angular-Slickgrid) by another user which is causing huge performance decrease because the Flatpickr keydown listener is listening for every input in the page (even non-Flatpickr inputs), see this comment

closing in favor of #2329


Uncaught TypeError: Cannot read property 'instrument' of undefined

After the last pull from master, I get following error on npm start:
Uncaught TypeError: Cannot read property 'instrument' of undefined
configureStore @ configureStore.js:15
(anonymous function) @ main.js:13
webpack_require @ bootstrap 11d11fa4d959c057fa47:550
fn @ bootstrap 11d11fa4d959c057fa47:98
(anonymous function) @ app.11d11fa4d959c057fa47.js:5
webpack_require @ bootstrap 11d11fa4d959c057fa47:550
webpackJsonpCallback @ bootstrap 11d11fa4d959c057fa47:21
(anonymous function) @ app.11d11fa4d959c057fa47.js:1
npm install and npm update didn't help.
Thanks!

I am getting the same thing

i was able to resolve this by replacing all instances of require(...).default with require(...). here's a diff. not sure what caused this, or if my fix is legit.

 The project seems to have updated to Babel 6 but if https://github.com/davezuko/react-redux-starter-kit/issues/526#issuecomment-181154312 works for you then you are using Babel 5.

I am getting the same error.

Same issue, same fix, from a fresh clone and npm install. npm ls reports babel-core 6.5.1.

This plugin causes behavior similar to Babel 5 (not needing to specify .default):
https://github.com/davezuko/react-redux-starter-kit/commit/c52510491c0c15e497909fa3e11e0b9bcdf36097
I don't know why it was added. Usually this plugin is used in libraries to slightly easier to consume from CommonJS environments but I don't see why one would want to use it in an app.
If you remove this plugin, .default should work as intended.

Hilariously, doing that results in the inverse problem: needing to add .default in a number of places. Here's a diff:
diff --git a/bin/server.js b/bin/server.js
index 4bfd764..3adfe15 100644
--- a/bin/server.js
+++ b/bin/server.js
@@ -1,7 +1,7 @@
 require('babel-register')

-const config = require('../config')
-const server = require('../server/main')
+const config = require('../config').default
+const server = require('../server/main').default
 const debug = require('debug')('app:bin:server')

 const port = config.server_port
diff --git a/config/index.js b/config/index.js
index e2c2e45..a8c2fbf 100644
--- a/config/index.js
+++ b/config/index.js
@@ -21,7 +21,7 @@ try {
 // the Node process to exit if an error occurs.
 let overrides
 if (hasOverridesFile) {
-  overrides = require(`./${overridesFilename}`)(config)
+  overrides = require(`./${overridesFilename}`).default(config)
 } else {
   debug(`No configuration overrides found for NODE_ENV "${config.env}"`)
 }
diff --git a/server/main.js b/server/main.js
index acb1cb4..a032082 100644
--- a/server/main.js
+++ b/server/main.js
@@ -29,11 +29,11 @@ if (config.env === 'development') {

   if (config.proxy && config.proxy.enabled) {
     const options = config.proxy.options
-    app.use(convert(require('./middleware/webpack-proxy')(options)))
+    app.use(convert(require('./middleware/webpack-proxy').default(options)))
   }

-  app.use(require('./middleware/webpack-dev')(compiler, publicPath))
-  app.use(require('./middleware/webpack-hmr')(compiler))
+  app.use(require('./middleware/webpack-dev').default(compiler, publicPath))
+  app.use(require('./middleware/webpack-hmr').default(compiler))

   // Serve static assets from ~/src/static since Webpack is unaware of
   // these files. This middleware doesn't need to be enabled outside

This is pretty much expected :-)
You would need to go either one way or another. Apparently the project is in the middle state right now, and this is why it's broken.
I'd go for ES6 modules everywhere (and thus explicit .default where needed).

Agreed.

What's weird is that it's been using require().default for months without problem, so there must have been a change in a dependency to cause this issue all of the sudden. I've been planning on removing the add-module-exports plugin for quite a while now but it's just been very low on the priority list. The plugin is a holdover from when this project switched to Babel 6 (which was quite early on).

Edit: add-module-exports was not even being used to transform the client-side code, it was only being used for the build system and server. Removing it either way, but I'm still at an absolute loss as to how this became an issue over night.

I ran an npm update and saw these packages changed:
<EMAIL_ADDRESS>/Users/zuko/projects/react-redux-starter-kit
├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>└──<EMAIL_ADDRESS>
Because we were expecting redux to be using CommonJS modules (and therefore using .default) the update broke the build. Will adjust accordingly. https://github.com/rackt/redux/commit/b27cb480bca24c0bff732370d6c2bfce4f894caa

Confirmed it's caused by a dependency change, was working for me until I ran npm update and the following changed:
<EMAIL_ADDRESS>/Users/zuko/projects/react-redux-starter-kit
├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>└──<EMAIL_ADDRESS>
Will be putting out a fix for this and then looking into what exactly broke it.

Fixed with https://github.com/davezuko/react-redux-starter-kit/pull/529.

Had the same issue, fixed with the suggestion from @schpet

Nice fix! I wonder which dep broke it.

I'm getting this problem now (in November)?  I'm not an expert with npm but I have tried npm install and npm update to try to get the fixed versions of dependences.  Any suggestions most appreciated.

same problem here but only on tests running on Codeship


Demo not working on Apple M1, 'NoneType' object has no attribute 'tfra_hkv_hash_table_of_tensors'

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS  14.4.1
TensorFlow version and how it was installed (source or binary):
tensorflow                     2.16.2
tensorflow-datasets            4.9.3
tensorflow-io-gcs-filesystem   0.37.1
tensorflow-macos               2.16.2
tensorflow-metadata            1.16.1
tensorflow-metal               1.1.0
tensorflow-recommenders-addons 0.8.1
install via pip3 and in the conda created virtual env
TensorFlow-Recommenders-Addons version and how it was installed (source or binary):
tensorflow-recommenders-addons 0.8.1
install via pip3 and in the same  conda created virtual env
Python version: Python 3.9.12
Is GPU used? (yes/no):
yes
Describe the bug
https://github.com/tensorflow/recommenders-addons/tree/master/demo/dynamic_embedding/movielens-1m-keras
AttributeError: 'NoneType' object has no attribute 'tfra_hkv_hash_table_of_tensors'

A clear and concise description of what the bug is.
it seems no table resource was created,
self._resource_handle = self._create_resource()
table_ref = hkv_ops.tfra_hkv_hash_table_of_tensors(
Code to reproduce the issue
`

import tensorflow as tf
import tensorflow_recommenders_addons as tfra
class DIN_KerasModel(tf.keras.Model):
def init(self, embedding_dim, num_items, num_features, name=None):
super(DIN_KerasModel, self).init(name=name)
self.embedding_dim = 32
# self.userID_embs = tfra.dynamic_embedding.keras.layers.BasicEmbedding(8)
self.userID_embs = tfra.dynamic_embedding.get_variable(
name="userid_dynamic_embeddings",
dim=self.embedding_dim,
initializer=tf.keras.initializers.RandomNormal(-1, 1)
)
din_keras_model = DIN_KerasModel(embedding_dim=8, num_items=10000, num_features=50)

`
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
Other info / logs
2024-11-25 10:07:24.160004: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2024-11-25 10:07:24.160027: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>) Traceback (most recent call last):   File "/Users/fanwei/work/moviebox/panlong_jobs/排序/test/tfra_mlen_1m.py", line 204, in <module>     app.run(main)   File "/Users/fanwei/miniforge3/envs/factormachine/lib/python3.9/site-packages/absl/app.py", line 308, in run     _run_main(main, args)   File "/Users/fanwei/miniforge3/envs/factormachine/lib/python3.9/site-packages/absl/app.py", line 254, in _run_main     sys.exit(main(argv))   File "/Users/fanwei/work/moviebox/panlong_jobs/排序/test/tfra_mlen_1m.py", line 194, in main     train()   File "/Users/fanwei/work/moviebox/panlong_jobs/排序/test/tfra_mlen_1m.py", line 120, in train     model = DualChannelsDeepModel(FLAGS.embedding_size, FLAGS.embedding_size,   File "/Users/fanwei/work/moviebox/panlong_jobs/排序/test/tfra_mlen_1m.py", line 54, in __init__     self.user_embedding = de.keras.layers.SquashedEmbedding(   File "/Users/fanwei/miniforge3/envs/factormachine/lib/python3.9/site-packages/tensorflow_recommenders_addons/dynamic_embedding/python/keras/layers/embedding.py", line 201, in __init__     self.params = de.get_variable(parameter_name,   File "/Users/fanwei/miniforge3/envs/factormachine/lib/python3.9/site-packages/tensorflow_recommenders_addons/dynamic_embedding/python/ops/dynamic_embedding_variable.py", line 1334, in get_variable     var_ = Variable(   File "/Users/fanwei/miniforge3/envs/factormachine/lib/python3.9/site-packages/tensorflow_recommenders_addons/dynamic_embedding/python/ops/dynamic_embedding_variable.py", line 666, in __init__     mht = self.kv_creator.create(   File "/Users/fanwei/miniforge3/envs/factormachine/lib/python3.9/site-packages/tensorflow_recommenders_addons/dynamic_embedding/python/ops/dynamic_embedding_creator.py", line 111, in create     return de.CuckooHashTable(   File "/Users/fanwei/miniforge3/envs/factormachine/lib/python3.9/site-packages/tensorflow/python/trackable/resource.py", line 103, in __call__     return previous_getter(*args, **kwargs)   File "/Users/fanwei/miniforge3/envs/factormachine/lib/python3.9/site-packages/tensorflow/python/trackable/resource.py", line 98, in <lambda>     previous_getter = lambda *a, **kw: default_resource_creator(None, *a, **kw)   File "/Users/fanwei/miniforge3/envs/factormachine/lib/python3.9/site-packages/tensorflow/python/trackable/resource.py", line 95, in default_resource_creator     obj.__init__(*a, **kw)   File "/Users/fanwei/miniforge3/envs/factormachine/lib/python3.9/site-packages/tensorflow_recommenders_addons/dynamic_embedding/python/ops/cuckoo_hashtable_ops.py", line 121, in __init__     self._resource_handle = self._create_resource()   File "/Users/fanwei/miniforge3/envs/factormachine/lib/python3.9/site-packages/tensorflow_recommenders_addons/dynamic_embedding/python/ops/cuckoo_hashtable_ops.py", line 155, in _create_resource     table_ref = hkv_ops.tfra_hkv_hash_table_of_tensors( AttributeError: 'NoneType' object has no attribute 'tfra_hkv_hash_table_of_tensors'
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

this issue solved by indicate the device:
self.user_embedding = de.keras.layers.SquashedEmbedding(
    user_embedding_size,
    initializer=embedding_initializer,
    name='user_embedding', devices=['/cpu:0'],)


Feature: locally server-side flag evaluations on applications

I found similar functionality in another project. I am discouraged that flagr does not support it.
It's really convenient. First, flags are cached. No dependency on service availability and network problems.
Local flag evaluation is guaranteed in case of flagr accessibility problems. The load on the network is reduced and the response time is increased.
https://docs.launchdarkly.com/sdk/concepts/client-side-server-side#server-side-sdks-1

When requested, server-side SDKs evaluate feature flags to determine a user's flag variation. To do this, they execute an in-process flag evaluation algorithm and return the resulting value.
Server-side SDKs can evaluate flags because these SDKs know your complete flag ruleset. These SDKs can determine users' flag variations without having to make requests to LaunchDarkly's servers for every evaluated flag.
Server-side SDKs get the complete ruleset associated with an SDK key when initializing a connection to LaunchDarkly's servers. Through this persistent connection, LaunchDarkly will continuously update the SDK's cached flag ruleset whenever flag rules change on LaunchDarkly.
You can think of each flag evaluation as a pure function, where we use the provided user context along with the complete ruleset to see if your user should be included in an evaluation or not. It is important to note that we pass along rule data not user data. If a user does not have the correct attributes locally, then it will not get evaluated by the rules that are cached by the SDK.

See examples of how it is implemented there:
https://github.com/launchdarkly/php-server-sdk
https://github.com/launchdarkly/go-server-sdk
https://github.com/launchdarkly/node-server-sdk
https://docs.launchdarkly.com/sdk/server-side

May be similar:
https://github.com/checkr/flagr/issues/350
https://github.com/checkr/flagr/issues/298


Fix iOS 12 deprecation warning

See more details here: https://forums.developer.apple.com/thread/62765

There's a bug in this — I'll close and resubmit.


[SQL Lab] Add hard time limit fallback for async queries

CATEGORY
Choose one

[x] Bug Fix
[ ] Enhancement (new features, refinement)
[ ] Refactor
[ ] Add tests
[ ] Build / Development Environment
[ ] Documentation

SUMMARY

According to https://github.com/celery/celery/issues/1958 celery only implements time_limit for the gevent pool. This PR adds a hard time limit 60 seconds after the soft one to clean up any tasks that the soft time limit isn't implemented for (or where the soft time exception is handled in such a way that never returns)
TEST PLAN

Pass CI
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

REVIEWERS
@michellethomas @john-bodley @mistercrunch @betodealmeida

Codecov Report

Merging #7783 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #7783      +/-   ##
==========================================
+ Coverage   65.71%   65.71%   +<.01%
==========================================
  Files         459      459
  Lines       21983    21984       +1
  Branches     2415     2415
==========================================
+ Hits        14446    14447       +1
  Misses       7416     7416
  Partials      121      121

Impacted Files
Coverage Δ

superset/sql_lab.py
77% <100%> (+0.12%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ebb7fbc...dbdc7f3. Read the comment docs.


Own demo project not working

I viewed your demo project and it's not able to register users.

Never mind. Just needed to click login as user.


Fix #628

Related issue(s)
Resolves #628
Description
Summary of changes
Because there is no handle_assign when a function call is used as argument of another function, we end-up using the array from the (inner) function. If the function is used twice (as first and second argument), then the array is overwritten, leading to the bug.
In order to solve this issue we call handle_assign when this case happen.
Dependency additions / changes
Test additions / changes
Regression test added to 7_function
Checklist

[X] I have tested the changes locally.
[X] I have formatted the changes with Prettier and/or cargo fmt with default settings.
[X] I have linked this PR to the issue(s) that it resolves.
[X] I have reviewed the changes on GitHub, line by line.
[X] I have ensured all changes are covered in the description.

Seems first class function PR added conflicts with this PR


Max send message len may exceed u32

Fix this issue.

usize is equal to u64 when word size is 64bit.

I also meet this problem. Seems ChannelArguments in grpc will truncate u64 as int32.

This is a bug of grpcio, let's fix it there.

Not yet.

Can we close it now? @BusyJay


fix(responsiveness,darkmode)

[x] responsiveness
[x] darkmode adaptation
[x] refactor
[x] adjust copytoclipboard in overview
[x] remove billing button
[x] set button to primary
[x] make use of startIcon in button


export file

NOT ready to merge
JSON file stuff (input/output)

Finished import and export
Did not know how to set the limit of the number of imported JSON files to 1.

Fixing some minor bugs rn

I improved export JSON. Import JSON is bugged. Doesn't seem to change smoothing options or threshold options. However, project is due in 4 minutes so yeah I'm merging and uploading this to pip.

I improved export JSON. Import JSON is bugged. Doesn't seem to change smoothing options or threshold options. However, project is due in 4 minutes so yeah I'm merging and uploading this to pip.

Yeah I found that bug as well. I know how to fix that bug. Hopefully this will not influence our whole project? I will fix it later.

I improved export JSON. Import JSON is bugged. Doesn't seem to change smoothing options or threshold options. However, project is due in 4 minutes so yeah I'm merging and uploading this to pip.

Yeah I found that bug as well. I know how to fix that bug. Hopefully this will not influence our whole project? I will fix it later.

I fixed it.

I improved export JSON. Import JSON is bugged. Doesn't seem to change smoothing options or threshold options. However, project is due in 4 minutes so yeah I'm merging and uploading this to pip.

Yeah I found that bug as well. I know how to fix that bug. Hopefully this will not influence our whole project? I will fix it later.

I fixed it (#82).

Got it. Thanks!


:dependabot: Add Dependabot configuration generator workflow

This pull request:

Adds a new workflow that regenerates Dependabot configuration on every commit and pushes it back if changes, this means any new components should be picked up automatically if the guidance is followed
It uses @moj-data-platform-robot's token so workflows can kick off again
BONUS: Adds concurrency to Terraform workflow

Signed-off-by: Jacob Woffenden<EMAIL_ADDRESS>

I'm unclear. Does that mean we no longer need to run the config generation locally? And if not, does that mean guidance needs an update?

Yes and yes


crash on list item tap

Steps to Reproduce
Create a list,
crash on tap of any list item (nothing happens on android, on tap not called)
##code snippet
class NewsListItem extends StatelessWidget{

  final NewsFeed newsFeed;
  NewsListItem(this.newsFeed);

  @override
  Widget build(BuildContext context){
    return new ListItem(
      isThreeLine: true,
      dense: true,
      title: new Text(newsFeed.title),
      subtitle: new Text(newsFeed.description),
      onTap: (){
        print("tapped");
      },
      onLongPress: (){
        print("Long tapped");
      },
    );
  }
}

Logs
CoreSimulatorBridge: Error in registering callback port Error Domain=NSPOSIXErrorDomain Code=22 "Invalid argument" UserInfo={NSLocalizedDescription=Unable to register pasteboard callback port., NSLocalizedFailureReason=Port is invalid.}
SpringBoard: [KeyboardArbiter] HW kbd: Failed to set (null) as keyboard focus
Application finished.
com.apple.CoreSimulator.SimDevice.215A9935-BD79-4FAE-8DDB-49BE6CCBDA19.launchd_sim[79148] (UIKitApplication:com.yourcompany.helloWorld[0x2015]: Service exited due to Broken pipe: 13
backboardd: [Common] Unable to get short BSD proc info for 82963: No such process
backboardd: [Common] Unable to get proc info for 82963: No such process
assertiond: notify_suspend_pid() failed with error 7
Flutter Analyze
[lint] Annotate overridden members (lib/NavDrawer.dart, line 7, col 10)
[hint] Unused import (lib/NewsFeedItem.dart, line 4, col 8)
[lint] Specify type annotations. (lib/NewsList.dart, line 76, col 5)
[lint] Specify type annotations. (lib/NewsList.dart, line 75, col 5)
[lint] Name non-constant identifiers using lowerCamelCase. (lib/NewsList.dart, line 17, col 10)
[lint] Name non-constant identifiers using lowerCamelCase. (lib/Routes.dart, line 3, col 17)
[lint] Name non-constant identifiers using lowerCamelCase. (lib/Routes.dart, line 4, col 17)
[lint] Name non-constant identifiers using lowerCamelCase. (lib/Routes.dart, line 5, col 17)
[lint] Annotate overridden members (lib/UserProfile.dart, line 8, col 10)
[hint] Unused import (lib/UserProfile.dart, line 2, col 8)
[lint] 137 public members lack documentation (ran in 34.1s)
Flutter Doctor
[✓] Flutter (on Mac OS, channel master)
• Flutter at /Users/mrugendrathattle/Research/Flutter/flutter
• Framework revision edbe034847 (5 days ago), 2016-10-19 16:10:46
• Engine revision 525c947ece
• Tools Dart version 1.21.0-dev.0.0
[✓] Android toolchain - develop for Android devices (Android SDK 24.0.3)
• Android SDK at /Users/mrugendrathattle/Library/Android/sdk
• Platform android-24, build-tools 24.0.3
• Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
[-] iOS toolchain - develop for iOS devices (Xcode 8.0)
• XCode at /Applications/Xcode.app/Contents/Developer
• Xcode 8.0, Build version 8A218a
x Brew not installed; use this to install tools for iOS device development.
Download brew at http://brew.sh/.
[✓] Atom - a lightweight development environment for Flutter
• flutter plugin version 0.2.6
• dartlang plugin version 0.6.42
[✓] Connected devices
• iPhone SE • 215A9935-BD79-4FAE-8DDB-49BE6CCBDA19 • ios (simulator)

For more information about diagnosing and reporting Flutter bugs, please see https://flutter.io/bug-reports/.

if its any help, this is my code,
3 files:
NewsFeed.dart: model
NewsFeedItem: renders an news list item <= this is where it crashes in onTap and onLongPress
NewsList: fetches the data from internet and creates a list
Archive.zip

flutter upgrade might fix.  iOS builds were busted last week for a couple days.  There isn't much that looks relevant from the logs.  It's possible that the Console.app shows a crash log for the Runner process?

I can confirm that the attached code works fine on Android. Must be just iOS issue.

@eseidelGoogle that worked. thank you


Amazon.nl: projector failed with error 0

Error message: Call to a member function parent() on null
Query string: action=display&bridge=Amazon&q=projector&sort=price-desc-rank&tld=nl&format=Html
Version: git.master.bb2329f

Thanks for reporting this issue. I can confirm this query failing on my machine.
It fails on line 76, likely because it couldn't find the h2 element a few lines before. I'm not sure why it is failing though, as the source data looks fine to me.
https://github.com/RSS-Bridge/rss-bridge/blob/3160e62293c3a06609390e1dfde035a32cd6c0c8/bridges/AmazonBridge.php#L69-L76
https://www.amazon.nl/s/?field-keywords=projector&sort=price-desc-rank


Adding Hidora to Partners page

Description
Adding Hidora (https://hidora.io/) as partners with OpenSearch
By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

@mattiaeleuteri Glad to have Hidora as part of the OpenSearch community!

@mattiaeleuteri Glad to have Hidora as part of the OpenSearch community!

we’re glad to contribute to this wonderful project !! :)


Improvement: Adding HTTP/S proxy support

This commit adds support for specifiying an HTTP/S proxy to the ClientBuilder class.

Seems like this fix is yet to be updated in the Maven repository.

Are there any plans of committing this to master?

As far as I know, this was already pulled into master, but never released?..


FS-3575: Add is_withdrawn flag and update queries

Change description

Adds a column is_withdrawn to assessment records, which acts as a flag to determine whether a record is marked as withdrawn
The queries have been updated to not pull back any withdrawn applications, so we essentially "hide" it from the user interface
This is done to match a support query to "withdraw" an application without losing the data, it would be good to have an option in the UI for this though to allow lead assessors to do this

[x] Unit tests and other appropriate tests added or updated
[x] README and other documentation has been updated / added (if needed)
[x] Commit messages are meaningful and follow good commit message guidelines

LGTM 👍 do we need to add the 'is_withdrawn' to the application models in the other services as part of this?

Yeah - I'm doing a change at the moment moving the round/fund into utils so we can share it across services, I'll include this!


lxd/instance/qemu: Set spawn=allow

spawn=allow is required when QEMU is asked to daemonize.
Previous QEMU versions would incorrectly block fork when spawn=deny
was passed while allowing clone to succeed.
This was then making it possible for us to use daemnize thanks to most
Linux distributions using the clone syscall to implement fork.
Current QEMU has fixed their seccomp profile to block both fork and
clone, preventing -daemonize when spawn=deny is passed.
Signed-off-by: Stéphane Graber<EMAIL_ADDRESS>

I think we initially mirrored the flags from libvirt or something like that which wasn't using -daemonize, so yeah, we were effectively relying on a bug.


YAML grammar files

This pull request partially fixes the issue at #3070.
The scopes for vendor/grammars/NSIS and vendor/grammars/awk-sublime were not recognized because the Sublime Text XML files were recently removed. The authors only kept the YAML files from which the XML files are generated.
Thus, this pull request adds support for YAML syntax definitions in script/convert-grammars.
A few new scopes were detected via .YAML-tmLanguage files. I added them in grammars.yml.
:warning: Of course, for this to work, GitHub backend for syntax highlighting must also recognize YAML grammar files.

/ cc @aroben incase he has a chance to give this a quick look.

Of course, for this to work, GitHub backend for syntax highlighting must also recognize YAML grammar files.

GitHub uses the JSON files this script creates rather than parsing the original files directly. So if this script can turn YAML into JSON then it should work fine on GitHub.

This seems reasonable to me 👍

@arfon Do you want to wait for a specific version to merge this or should I go ahead?
Show all checks

Nah, let's get this merged in.


feat: Adds new lines to newfile

Adds additional lines to newfile, this is a new feature, i promise!

You should maybe clarify why this was needed.

LGTM


add support for flutter using dart generator

Description
We use flutter a lot in my company and have been using arkana for our ios projects, so thought of adding support for flutter.
This builds on an infra similar to @husseinala's https://github.com/rogerluan/arkana/pull/53.
Arkana Arguments
The  --lang operator now supports dart.
Arkana YAML config
The existing result_path allows you to specify a subfolder path/location in the dart project.
Generated Assets
{namespace}.dart

{namespace}_environment.dart

{namespace}_tests.dart

Footnotes
The import path in tests file is currently relative although absolute/package imports are preferred. However, package imports would require us to

read the dart project's pubspec for namespace
or add a new parameter both of which I find to be overkill.

I'd appreciate any recommendations and love to hear your thoughts on this.

@rogerluan Any thoughts on this? Would appreciate any feedback

Hi @humblerookie ! Sorry for the radio silence, I haven't had time to review this yet as my weeks have been pretty hectic 🤕  I promise to come back to this as soon as I can! Meanwhile, you can always point your Gemfile to this PR if you need to use this code, like this:
gem 'arkana', git: 'https://github.com/humblerookie/arkana.git', branch: 'main'

And run bundle install to apply the changes.
I hope I'm not blocking your awesome work!
Again, I'll try to come back to this and give this a proper review some time soon 🙏

@rogerluan  No worries at all, and thanks for the suggestions.

@rogerluan Do you think you might get sometime this week?


Remove Headers, Footers, External Links and their related data

Hi, Thanks for this great work.
I have been playing around with this, to crawl webpages and get content in markdown format, which can be used to provide to LLMs for grounding. But when I used them to say get data for news articles, I get lots and lots of unnecessary data like headers, footers, nav options, other article links. More than 50% of the data is not the actual data.
Yes, I can use LLMExtraction, but that will increase the bill tremendously as the input are like 10-20 web articles, with each web article ranging around 5000-7000 tokens. I saw one option is to provide elements, but I want the crawler to be generic, which can work on any website, so I don't have any fix elements which I know can remove header, nav, footer information for sure. Any way where the playwright extractor, can get the actual content of the webpage. I understand this may not be perfect, but I want to reduce the extra content as much as possible while being generic across all websites.
P.S: I would love to contribute to this project, I am not much experienced in the JS/TS side, but I am pretty confident on the python side of things. So is there any discord/slack/telegram group where I can join to discuss on how to contribute?

I have the same need too.
Perhaps you can try things like this to get the main content, I used this for my own need now :
def clean_content(content):
    # Find the start of the content (first # title)
    start_index = content.find('#')
    if start_index == -1:
        return ""  # No title found

    # Find the end of the content (next ## title)
    end_index = content.find('##', start_index + 1)
    if end_index == -1:
        # If no ## title found, return until the end
        return content[start_index:]
    else:
        return content[start_index:end_index].strip()

article['markdown_content'] = asyncio.run(crawl_url(article['source_url']))
article['markdown_content'] = clean_content(article['markdown_content'])

I too started with this, but the problem is many times, the title is not h2(##), sometime is h1(#) or h3(###) etc. And also the ads, signup banners and other links too will have ##, causing all other content also to be read. It's difficult I think to get the content from generic pages deterministically without LLMs.

agreed.

@syed-al Thx for using Crawl4ai, Would you please share with me the URL you're trying to get the best markdown out of it? I think there are a bunch of things that I can share with you to help you achieve that, as it has been one of our goals to produce as much AI friendly data as possible without using LLM. Could you please give me the link, and I'll give it a try? I'd be more than happy to invite you to become one of our collaborators. Would you kindly share your email address? I can then send you an invitation to our Discord channel. Perhaps this could be one of the areas where you can help us and work together.

@syed-al Thx for using Crawl4ai, Would you please share with me the URL you're trying to get the best markdown out of it? I think there are a bunch of things that I can share with you to help you achieve that, as it has been one of our goals to produce as much AI friendly data as possible without using LLM. Could you please give me the link, and I'll give it a try? I'd be more than happy to invite you to become one of our collaborators. Would you kindly share your email address? I can then send you an invitation to our Discord channel. Perhaps this could be one of the areas where you can help us and work together.

I also need to remove unnecessary content. You should set additional parameters such as HTML selectors, excluding specific HTML tags that start with 'header', 'footer', and similar.

@syed-al Thx for using Crawl4ai, Would you please share with me the URL you're trying to get the best markdown out of it? I think there are a bunch of things that I can share with you to help you achieve that, as it has been one of our goals to produce as much AI friendly data as possible without using LLM. Could you please give me the link, and I'll give it a try? I'd be more than happy to invite you to become one of our collaborators. Would you kindly share your email address? I can then send you an invitation to our Discord channel. Perhaps this could be one of the areas where you can help us and work together.

Clean the HTML first and then convert the HTML to markdown.

@syed-al @mentaLwz @QuangTQV  There are multiple flags and parameters you can use to control the level of data-cleaning you may need. Please share with me your URL, then I give you an example. However I share one example that I think will answer your question:
async def main():
    async with AsyncWebCrawler(headless = True, sleep_on_close = True) as crawler:
        url = "https://janineintheworld.com/places-to-visit-in-central-mexico"
        result = await crawler.arun(
            url=url,
            # bypass_cache=True,
            word_count_threshold = 10,
            excluded_tags = ['form'], # Optional - Default is None, this adds more control over the content extraction for markdown
            exclude_external_links = False, # Default is True
            exclude_social_media_links = True, # Default is True
            exclude_external_images = True, # Default is False
            # social_media_domains = ["facebook.com", "twitter.com", "instagram.com", ...] Here you can add more domains, default supported domains are in config.py

            html2text = {
                "escape_dot": False,
                # Add more options here
            }
        )
        # Save markdown to file
        with open(os.path.join(__data, "mexico_places.md"), "w") as f:
            f.write(result.markdown)

    print("Done")

Here you can see some flags and parameters that allow you to focus more on the content relevant to what you're searching for. This is a website about places that can be visited in Mexico, where you can see the excluded_tags that you can pass. You can see that here we have parameters to exclude or include external links, and the same goes for social media links such as Facebook, Pinterest, and so on. You can also pass extra domains that you want to be excluded from the links, and the same thing goes for images.
Another thing you can see here is the concept of the word count threshold. By setting this amount to a high number, like 10, you are basically excluding any HTML blocks that contain text with less than 10 words. This is a very useful way of removing unnecessary text. You can also use this one, depending on your needs.
Finally, our markdown generator uses Html2Text and you can override some of the standard parameters to use it in an advanced way.
@QuangTQV I already shared same answer to your other issues, perhaps I close that issue and we continue here thx.

Here is the sample url to get the content:
https://www.hindustantimes.com/world-news/us-news/cnn-reporter-chokes-on-laughter-covering-trumps-x-rated-arnold-palmer-remark-live-watch-101729484003535.html
It is a new article, where are lot of links, extra information, other article information etc, which I would want to neglect and only get the main content of the article. The main article is around 30 lines, but the entire markdown I get is more than 600-700 lines.
 # Create an instance of AsyncWebCrawler
    async with AsyncWebCrawler(verbose=False) as crawler:
        # Run the crawler on a URL
        results = await crawler.arun_many(
            urls=urls,
        )

I am for now using the basic usage without any extra flags. Will try the flags you mentioned

@syed-al @mentaLwz @QuangTQV There are multiple flags and parameters you can use to control the level of data-cleaning you may need. Please share with me your URL, then I give you an example. However I share one example that I think will answer your question (This will be available in 0.3.72):
async def main():
    async with AsyncWebCrawler(headless = True, sleep_on_close = True) as crawler:
        url = "https://janineintheworld.com/places-to-visit-in-central-mexico"
        result = await crawler.arun(
            url=url,
            # bypass_cache=True,
            word_count_threshold = 10,
            excluded_tags = ['form'], # Optional - Default is None, this adds more control over the content extraction for markdown
            exclude_external_links = False, # Default is True
            exclude_social_media_links = True, # Default is True
            exclude_external_images = True, # Default is False
            # social_media_domains = ["facebook.com", "twitter.com", "instagram.com", ...] Here you can add more domains, default supported domains are in config.py

            html2text = {
                "escape_dot": False,
                # Add more options here
            }
        )
        # Save markdown to file
        with open(os.path.join(__data, "mexico_places.md"), "w") as f:
            f.write(result.markdown)

    print("Done")

Here you can see some flags and parameters that allow you to focus more on the content relevant to what you're searching for. This is a website about places that can be visited in Mexico, where you can see the excluded_tags that you can pass. You can see that here we have parameters to exclude or include external links, and the same goes for social media links such as Facebook, Pinterest, and so on. You can also pass extra domains that you want to be excluded from the links, and the same thing goes for images.
Another thing you can see here is the concept of the word count threshold. By setting this amount to a high number, like 10, you are basically excluding any HTML blocks that contain text with less than 10 words. This is a very useful way of removing unnecessary text. You can also use this one, depending on your needs.
Finally, our markdown generator uses Html2Text and you can override some of the standard parameters to use it in an advanced way.
@QuangTQV I already shared same answer to your other issues, perhaps I close that issue and we continue here thx.

Could you provide an example with this URL? 'https://tiki.vn/search?q=giày adidas

@QuangTQV For pages with repetitive patterns, I suggest using the JSON CSS extraction. Let's look at the following code, which crawls one page and turns it into a list of JSON. Your case does not require the extraction of the markdown.
async def main():
    print("\n--- Using JsonCssExtractionStrategy for Fast Structured Output ---")
    schema = {
        "name": "Tiki Shoes",
        "baseSelector": ".CatalogProducts__Wrapper-sc-1r8ct7c-0 > div",
        "fields": [
            {
                "name": "image",
                "selector": "picture > img",
                "type": "attribute",
                "attribute": "src",
            },
            {
                "name": "price",
                "selector": '.price-discount__price',
                "type": "text",
            },
            {
                "name": "brand",
                "selector": ".above-product-name-info",
                "type": "text",
            },
            {
                "name": "description",
                "selector": "h3",
                "type": "text",
            }
        ],
    }

    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)

    async with AsyncWebCrawler(verbose=True, headless = True) as crawler:
        result = await crawler.arun(
            url="https://tiki.vn/search?q=gi%C3%A0y%20adidas",
            extraction_strategy=extraction_strategy,
            bypass_cache=True,
            # delay_before_return_html = 1,
            wait_for = "css:.CatalogProducts__Wrapper-sc-1r8ct7c-0 picture > img", # Important: This make sure the dynamic data is loaded and available
            magic=True
        )

        items = json.loads(result.extracted_content)
        print(f"Successfully extracted {len(items)} items")
        print(json.dumps(items[0], indent=2))

        with open(os.path.join(__data__, "tiki_shoes.json"), "w") as f:
            f.write(result.extracted_content)

Output:
[{
  "price": "971.000\u20ab",
  "brand": "BITI'S",
  "description": "Gi\u00e0y Th\u1ec3 Thao Nam - N\u1eef Biti's Hunter X - 2K22 - Midnight III DSUH00502DEN (\u0110en)"
}, ...]

@QuangTQV For pages with repetitive patterns, I suggest using the JSON CSS extraction. Let's look at the following code, which crawls one page and turns it into a list of JSON. Your case does not require the extraction of the markdown.
async def main():
    print("\n--- Using JsonCssExtractionStrategy for Fast Structured Output ---")
    schema = {
        "name": "Tiki Shoes",
        "baseSelector": ".CatalogProducts__Wrapper-sc-1r8ct7c-0 > div",
        "fields": [
            {
                "name": "image",
                "selector": "picture > img",
                "type": "attribute",
                "attribute": "src",
            },
            {
                "name": "price",
                "selector": '.price-discount__price',
                "type": "text",
            },
            {
                "name": "brand",
                "selector": ".above-product-name-info",
                "type": "text",
            },
            {
                "name": "description",
                "selector": "h3",
                "type": "text",
            }
        ],
    }

    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)

    async with AsyncWebCrawler(verbose=True, headless = True) as crawler:
        result = await crawler.arun(
            url="https://tiki.vn/search?q=gi%C3%A0y%20adidas",
            extraction_strategy=extraction_strategy,
            bypass_cache=True,
            # delay_before_return_html = 1,
            wait_for = "css:.CatalogProducts__Wrapper-sc-1r8ct7c-0 picture > img", # Important: This make sure the dynamic data is loaded and available
            magic=True
        )

        items = json.loads(result.extracted_content)
        print(f"Successfully extracted {len(items)} items")
        print(json.dumps(items[0], indent=2))

        with open(os.path.join(__data__, "tiki_shoes.json"), "w") as f:
            f.write(result.extracted_content)

Output:
[{
  "price": "971.000\u20ab",
  "brand": "BITI'S",
  "description": "Gi\u00e0y Th\u1ec3 Thao Nam - N\u1eef Biti's Hunter X - 2K22 - Midnight III DSUH00502DEN (\u0110en)"
}, ...]

I want to extract with all URLs, this is just an example URL, how can I remove redundant data?

@syed-al I just added a new heuristic function that has the capability to produce a better markdown, which I call 'fit markdown.' The way they're going to use it is as follows: [code]. I also share with you the output of the markdown: it's much cleaner and I really like it. It's good for pages like the example you shared, and it will be soon released in the new version.
async def main():
    async with AsyncWebCrawler(verbos=True) as crawler:
        url = "https://janineintheworld.com/places-to-visit-in-central-mexico"
        result = await crawler.arun(
            url=url,
            bypass_cache=True,
            word_count_threshold = 10,
        )
        # Save markdown to file
        with open(os.path.join(__data__, "mexico_places.md"), "w") as f:
            f.write(result.fit_markdown)

    print("Done")

Everything remains the same. You simply pick up the fit_markdown property of the result crawl item. I also attach the markdown here. Result is very clean and exactly contains the main part. Btw Its under the test, perhaps this will be the first task you can help upon joining to Discord.
mexico_places.md

@QuangTQV For pages with repetitive patterns, I suggest using the JSON CSS extraction. Let's look at the following code, which crawls one page and turns it into a list of JSON. Your case does not require the extraction of the markdown.
async def main():
    print("\n--- Using JsonCssExtractionStrategy for Fast Structured Output ---")
    schema = {
        "name": "Tiki Shoes",
        "baseSelector": ".CatalogProducts__Wrapper-sc-1r8ct7c-0 > div",
        "fields": [
            {
                "name": "image",
                "selector": "picture > img",
                "type": "attribute",
                "attribute": "src",
            },
            {
                "name": "price",
                "selector": '.price-discount__price',
                "type": "text",
            },
            {
                "name": "brand",
                "selector": ".above-product-name-info",
                "type": "text",
            },
            {
                "name": "description",
                "selector": "h3",
                "type": "text",
            }
        ],
    }

    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)

    async with AsyncWebCrawler(verbose=True, headless = True) as crawler:
        result = await crawler.arun(
            url="https://tiki.vn/search?q=gi%C3%A0y%20adidas",
            extraction_strategy=extraction_strategy,
            bypass_cache=True,
            # delay_before_return_html = 1,
            wait_for = "css:.CatalogProducts__Wrapper-sc-1r8ct7c-0 picture > img", # Important: This make sure the dynamic data is loaded and available
            magic=True
        )

        items = json.loads(result.extracted_content)
        print(f"Successfully extracted {len(items)} items")
        print(json.dumps(items[0], indent=2))

        with open(os.path.join(__data__, "tiki_shoes.json"), "w") as f:
            f.write(result.extracted_content)

Output:
[{
  "price": "971.000\u20ab",
  "brand": "BITI'S",
  "description": "Gi\u00e0y Th\u1ec3 Thao Nam - N\u1eef Biti's Hunter X - 2K22 - Midnight III DSUH00502DEN (\u0110en)"
}, ...]

I want to extract with all URLs, this is just an example URL, how can I remove redundant data?

@unclecode Can you help me @@ ?

Hi @unclecode
When would the new version be released on pip?
Also, waiting for the discord invite<EMAIL_ADDRESS>

@syed-al tomorrow, Friday. Sorry for the delay, I will send the invitation before weekend, and you are most welcome.

@QuangTQV No worries, Let me first understand what exactly you're looking for. You said you want to do this with all URLs. Maynard, what do you refer to by 'all URLs'? When you're using the JsonCssExtractionStrategy, it's a mechanism that works exactly with one specific layout on the page. So I have to understand what you mean by 'all URLs'. Are you referring to any page or any form so that you can use the JsonCssExtractionStrategy, unless all the pages you're using have a similar HTML structure? So I guess I didn't get your question properly. You explained the task you have in a complete way rather than the example, and then I will try to illustrate a code for you to show you how it works.

@syed-al Hi, sorry for the delay, it has been very hectic last two weeks, but now all good, I just send the invitation link to you, looking forward to see you on the other side, work on this smart FIT markdown ;)


Make referencing Node/Npm projects consistent with python and dotnet

Background and Motivation
When adding a Python or Dotnet project, the API is builder.AddProject / builder.AddPythonProject - Npm/Node should probably be consistent and not AddNodeApp etc. Also I'm sure there is a reason they are two separate APIs, but... feels like we should have builder.AddJSProject("node") or something and abstract that
Proposed API
no
Usage Examples
builder.AddNodeProject("pls");

I find the naming a little confusing because the NodeAppResource and PythonProjectResource are of type ExecutableResource not ProjectResource, which means that from a manifest perspective they are treated differently (resulting in a different hosting model).
I would argue that it should be AddPythonApp to align with the current Node style and thus have a point of difference that they will be handled differently in the manifest for hosting.

after talking with @DamianEdwards I agree with you @aaronpowell !!! It should be "AddNodeApp" and "AddPythonApp" and we should reserve "Add<>Project" for things with a .csproj file!

We should add the new method (AddPythonApp) and obsolete the existing one in 8.2 and, remove the old one in 9.0


Correct interface order, and include all networks in metadata.

With this patch I end up with a correctly configured multi-nic centos.


Unable to Export Power Connections

Environment

Python version: 3.6
Nautobot version: 1.0b3

Steps to Reproduce

Create a power feed
Create a connection from a device to the feed
Go to /dcim/power-connections/ and click export

Expected Behavior
A .csv file is generated

Observed Behavior
Stack trace
builtins:AttributeError: 'PowerFeed' object has no attribute 'device'
Traceback(most recent call last):
File "/usr/local/lib/python3.6/site-packages/django/utils/deprecation.py", line 114, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 1206, in _wrapper
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
File "/usr/local/lib/python3.6/site-packages/django/utils/deprecation.py", line 114, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 1206, in _wrapper
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
File "/usr/local/lib/python3.6/site-packages/django/utils/deprecation.py", line 114, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 1206, in _wrapper
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
File "/usr/local/lib/python3.6/site-packages/django/utils/deprecation.py", line 114, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 1206, in _wrapper
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
File "/usr/local/lib/python3.6/site-packages/django/utils/deprecation.py", line 114, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 1206, in _wrapper
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
File "/usr/local/lib/python3.6/site-packages/django/utils/deprecation.py", line 114, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 1206, in _wrapper
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
File "/usr/local/lib/python3.6/site-packages/django/utils/deprecation.py", line 114, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 1206, in _wrapper
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
File "/usr/local/lib/python3.6/site-packages/django/utils/deprecation.py", line 114, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 1206, in _wrapper
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
File "/usr/local/lib/python3.6/site-packages/nautobot/core/middleware.py", line 127, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 1206, in _wrapper
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
File "/usr/local/lib/python3.6/site-packages/django/utils/deprecation.py", line 114, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 1206, in _wrapper
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
File "/usr/local/lib/python3.6/site-packages/django/utils/deprecation.py", line 114, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 1206, in _wrapper
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
File "/usr/local/lib/python3.6/site-packages/nautobot/core/middleware.py", line 111, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 1206, in _wrapper
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
File "/usr/local/lib/python3.6/site-packages/nautobot/core/middleware.py", line 97, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 1206, in _wrapper
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
File "/usr/local/lib/python3.6/site-packages/django/utils/deprecation.py", line 114, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 1206, in _wrapper
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 181, in _get_response
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 563, in wrapper
File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 70, in view
File "/usr/local/lib/python3.6/site-packages/nautobot/utilities/views.py", line 94, in dispatch
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 953, in wrapper
File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 98, in dispatch
File "/usr/local/lib/python3.6/site-packages/nautobot/core/views/generic.py", line 184, in get
File "/usr/local/lib/python3.6/site-packages/nautobot/dcim/views.py", line 2479, in queryset_to_csv

FWIW this appears to be an issue in NetBox 2.10 as well, though perhaps not a known issue.


Add lower bounds to bytestring

We need Builder, and also doubleDec (which seems to be only >=0.10.4 in
Data.ByteString.Lazy.Builder)
I already made revisions for formatting on Hackage.

Ta!


The terminal code is monolithic

All Arduino code is contained in one file, making it hard to read and maintain. Refactoring would be very beneficial.

mentioned in commit bde2fc3a690cecbdcb80f838e0696b9c374a1105

mentioned in commit 775df9d4b06bd49e9361b98b1aa630903cb9615c

mentioned in commit 541109328df4dfafbba47ececaf9e3b909aae26d

created branch 45-terminal-refactor to address this issue

mentioned in commit 1a782b95bd120eee913a49eaf5b5d3e48f561114

mentioned in commit 130e17b5eed67d0b908995b79579ea3537fdd280

mentioned in commit 607f955484a909dc89fcaef6514e4c2195c4d2e4

mentioned in commit e5fd3cd78ecc9cad91f7c0362831f1d53072bc14

mentioned in commit 99506a83c62fb9f181344b4a57bdef3f363e4280

mentioned in commit 41e9f79775ac5ae4aef69b14f1779943d8ed1493

mentioned in commit 37e4fe147ccb7e76c69a4ec1a8dd3be3ecea4ef7

mentioned in commit 6134b8892440a14a36e175f1d71d027b27d32e45

mentioned in commit 51827283f32753571888c339bc04bbd69d9bd2c9

mentioned in commit 84ecd58eae44c9e8647ba0f923c26db351d85c3b

In GitLab by @alexkhval on May 17, 2024, 15:39
mentioned in commit b2b509c0bc8891699da9c0694fa67cc45368907b

In GitLab by @albu-razvan on May 17, 2024, 23:40
mentioned in commit 287f62ec1b0961ac9bcc3a2aeded22952c63a5a4


Orchestrator: ChooseIntent (aka Duplicate Intent) Trigger doesn't work when using Orchestrator

Describe the bug
When using Orchestrator with a ChooseIntents trigger in a dialog, Orchestrator will not pop up the disambiguation prompt, even if all conditions are met.
Additional context
OrchestratorRecognizer supports this scenario, but we need to explicitly set the detectAmbiguousIntents flag in the recognizer for the dialog if it has a duplicate intent/ChooseIntent trigger.
Fix is to add this flag to orchestrator recognizer settings for every dialog that has a ChooseIntent trigger.

Is this still on track on R13? @taicchoumsft
Regarding the fix, should we add the flag only to every dialog that has an onChooseIntent trigger,  or we can just add it for all orchestrator recognizer? What will be the harm of adding it for all cases?

Yes one fix is to add flag only to dialog that has onChooseIntent trigger, and expose the recognizer thresholds in Composer somewhere.
I experimented with adding it for all, but then dialog will not work as expected when Orchestrator returns ChooseIntent and there is no onChooseIntent trigger to handle.

Yes one fix is to add flag only to dialog that has onChooseIntent trigger, and expose the recognizer thresholds in Composer somewhere.
I experimented with adding it for all, but then dialog will not work as expected when Orchestrator returns ChooseIntent and there is no onChooseIntent trigger to handle.

I feel like that is the expected behavior, if orchestrator returns ChooseIntent and there is no trigger, and thus nothing happens.
I think it's down to the question, do we think ChooseIntent from Orchestrator is "trustworthy" or not.  If it is, i feel like it makes more sense that we add this flag for all cases, and educate user that, "you know, not always one single intent win, you need to design for ambiguous case".   If not, probably, adding this flag conditionally is better, it would be less breaking.
what do you guys think? @scheyal @chrimc62


Rename CudnnScratchAllocator to DnnScratchAllocator

Rename CudnnScratchAllocator as the logic is applicable for not only Cudnn, but
also other DNN algorithm libraries such as MIOpen.

@timshen1 May I ask you to take a look at this PR? It's a precursor to upstream ROCm-specific logic for TensorFlow operators.
Failed CI targets are:

MacOS Contrib
MacOS Python2 and CC
Windows Bazel
Windows Bazel GPU

From the logs they seem to have nothing to do with this PR, but a configuration issue in CI test infrastructure. I'm wondering if it's possible to manually apply ready-to-pull label, after your review and approval?


Can not train generative transition model: Tensorflow complains about dimensions

After training the autoencoder model, I close the server used for that training and open a new server as specified.  I then follow the next step for training the generative transition model, but Tensorflow immediately complains about mismatched (or undefined?) shapes as can be seen in the last line of the trace-back.
>>>>>>: ~/Desktop/research$ ./train_generative_model.py transition --batch 64 --name transition
Using TensorFlow backend.
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node
zero

I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GTX 1060 6GB
major: 6 minor: 1 memoryClockRate (GHz) 1.7085
pciBusID 0000:01:00.0
Total memory: 5.93GiB
Free memory: 5.59GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:839] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0)
Traceback (most recent call last):
  File "./train_generative_model.py", line 159, in <module>
    g_train, d_train, sampler, saver, loader, extras = get_model(sess=sess, name=args.name, batch_size=args.batch, gpu=args.gpu)
  File "/home/kamal/Desktop/research/models/transition.py", line 65, in get_model
    T = transition(batch_size)
  File "/home/kamal/Desktop/research/models/transition.py", line 42, in transition
    activation="tanh", batch_input_shape=(batch_size, time, z_dim)))
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 276, in add
    layer.create_input_layer(batch_input_shape, input_dtype)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 371, in create_input_layer
    self(x)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 515, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 573, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 150, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/home/kamal/Desktop/research/models/layers.py", line 371, in call
    input_length=self.output_length)
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 1195, in rnn
    sequence_length=None)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn.py", line 1006, in _dynamic_rnn_loop
    output.set_shape(shape)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 405, in set_shape
    self._shape = self._shape.merge_with(shape)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_shape.py", line 570, in merge_with
    (self, other))
ValueError: Shapes (?, 64, 512) and (9, 2, 512) are not compatible

any ideas?

Sorry about the duplication in #22
Closing now.


Data race and concurrency problems of actor placement

In what area(s)?

/area placement
/area actor

What version of Dapr?

1.0.x

Expected Behavior

Codes can handle concurrency robustly.
Actual Behavior

Tether are some obvious data race and concurrency problems in implementation of ActorPlacement.  And there are lots of error when we run test with race detect.
serverIndex
Actual
Read:

https://github.com/dapr/dapr/blob/034022589c6fb4f9cf32aebc6278bd18d45e23b5/pkg/actors/internal/placement.go#L254

Write:

https://github.com/dapr/dapr/blob/034022589c6fb4f9cf32aebc6278bd18d45e23b5/pkg/actors/internal/placement.go#L158

Expected
Should use atomic operation or guarded by mutex.
shutdown
Actual
Read:

https://github.com/dapr/dapr/blob/034022589c6fb4f9cf32aebc6278bd18d45e23b5/pkg/actors/internal/placement.go#L144

Write:

https://github.com/dapr/dapr/blob/034022589c6fb4f9cf32aebc6278bd18d45e23b5/pkg/actors/internal/placement.go#L234

Expected
Should use atomic operation or guarded by mutex.
placementTables
Actual
Read:

https://github.com/dapr/dapr/blob/034022589c6fb4f9cf32aebc6278bd18d45e23b5/pkg/actors/internal/placement.go#L385

Write:

https://github.com/dapr/dapr/blob/034022589c6fb4f9cf32aebc6278bd18d45e23b5/pkg/actors/internal/placement.go#L361

Expected
Should be guarded by mutex
tableIsBlocked&unblockSignal
Actual
Read:

https://github.com/dapr/dapr/blob/034022589c6fb4f9cf32aebc6278bd18d45e23b5/pkg/actors/internal/placement.go#L374
Write:
https://github.com/dapr/dapr/blob/034022589c6fb4f9cf32aebc6278bd18d45e23b5/pkg/actors/internal/placement.go#L339

There both have data race and memory order problem.
Expected
Use condition variable to do wait and signal.
streamConnAlive&streamConnectedCh
Similiar to tableIsBlocked&unblockSignal
Steps to Reproduce the Problem

cd pkg/actors/internal
go test ./... -race

Release Note

RELEASE NOTE: FIX Data race and concurrency problems of actor placement.

Tkanhs

Do not close this.

@skyao Thanks for reviewing this. As a maintainer, please, don't forget to assign issues and add them to the milestone before merging the corresponding PRs.


Fix typo in documentation: Starring not Staring.

We aren't staring at our commits, we are adding stars.

I don't understand github's new "At least 1 approving review is required by reviewers with write access." maybe you don't have write access? I don't see any merge button anywhere, feel free to merge. I can't unless I'm missing something somewhere.

Ah, maybe you also don't have write access. Makes sense, ok.

Ah, maybe you also don't have write access. Makes sense, ok.

Indeed I don't.. don't really know how this repo is being run.. just figured doing a review would allow the owner(s) to merge this more quickly.

I will merge it when the CI is going through. Thank you for the contribution!


Documentation for parsing GeoJSON from a file

I am trying to read GeoJSON from a file and iterate through the features in the FeatureCollection with their geom and feature properties. I would have thought this would be a very common use case, but I don't see any documentation giving example code for how to do this. All the examples seem to assume you've got the string of a single geometry in memory already, or that you're using FlatGeobuf (*.fgb) files.
I've found two examples on GitHub, but:

geoq: Doesn't use either GeoJsonReader or FeatureProcessor, which seems like it should be the right way…
geoarrow: Uses this GeoTableBuilder thing which I think is part of Arrow, and thus not something someone who isn't using Arrow would use.

Have I missed the sample code for this? I'd expect a simple example in the overview on the GeoJsonReader page, and probably in the main README as well.

The GeoJsonReader accepts any input that implements Read. So you can pass in a File or, preferably, a BufReader<File>.

Thanks for the quick reply. I found this issue and adapted his code to do what I needed, along with this bit of their docs. I'm thinking that the answer to my question is "geojson is the a higher-level library, and is probably what you want to use for simply iterating through features in a geojson file." Is that safe to say?
For posterity, here's what I ended up with:
use geo::{MultiLineString, MultiPolygon};
use geojson::{Feature, GeoJson, Value};
use std::fs::File;

fn load_geojson(path: &PathBuf) -> Result<(), Box<dyn std::error::Error>> {
    let file = File::open(path)?;
    let geojson = GeoJson::from_reader(file)?;

    match geojson {
        GeoJson::FeatureCollection(collection) => {
            for feature in collection.features {
                let _ = process_feature(&feature)?;
            }
        }
        _ => println!("Unsupported GeoJSON type"),
    }

    Ok(())
}

fn process_feature(feature: &Feature) -> Result<(), Box<dyn std::error::Error>> {
    // String value of the "name" property, or an empty string
    let name = feature
        .property("name")
        .and_then(|v| v.as_str())
        .map_or(String::from(""), |s| s.to_string());

    let geom = feature.geometry.as_ref().unwrap();

    match &geom.value {
        Value::MultiPolygon(_) => {
            let p: MultiPolygon<f64> = geom.value.clone().try_into().unwrap();
            println!("{p:?}");
        }
        Value::MultiLineString(_) => {
            let p: MultiLineString<f64> = geom.value.clone().try_into().unwrap();
            println!("{p:?}");
        }
        _ => panic!("not a recognized feature type"),
    };

    Ok(())
}

I'm thinking that the answer to my question is "geojson is the a higher-level library, and is probably what you want to use for simply iterating through features in a geojson file." Is that safe to say?

I'd say the opposite. geojson is lower-level in the sense that you have to handle specifics about GeoJSON to handle input data. geozero is higher-level in the sense that GeoJSON input is just one type of input, but can export to any consumer. For example in geoarrow-rs GeoJsonReader::process just works even though geozero has no knowledge of the GeoArrow output format.
I'd say the real issue is that there's no "default" library in georust for handling geometries with attributes. You can parse to geo structs but then you lose the associated attributes. This is a main feature of geoarrow though; being really optimized about both the geometries and their attributes.

OK, that's interesting to know. If this is the higher level library, I think it's even more important to have some sample code of loading a GeoJSON file (ideally from disk, but at least from a string) and iterating its features. I never got that to work with geozero.
As for geometries and attributes, the geojson::Feature struct seems to handle that pretty well with feat.property("prop_key") and feat.geometry upon pulling them out of the file. But you're right, I had to create my own struct to store the converted geometry along with the properties I wanted. Providing a ready-made struct for that purpose would be a nice QoL improvement.

I never got that to work with geozero.

You'd need to impl your own GeozeroDatasource

As for geometries and attributes, the geojson::Feature struct seems to handle that pretty well with feat.property("prop_key") and feat.geometry upon pulling them out of the file

Sure, but that's storing attributes in the GeoJSON model, which is quite restrictive. For example, you can't store a date time in GeoJSON; you can only store a string.

Providing a ready-made struct for that purpose would be a nice QoL improvement.

It's not geozero's concern to provide those structs. Geozero focuses only on conversions between representations. I'm building a representation around Arrow, which enables storing properties quite efficiently, but does incur some large dependencies.

OK, well I filed this bug to document that I wanted to do something I thought was very common, and yet could find no sample code on how to do it. I've solved my problem at this point. If you want people to actually use this library for parsing geometry and features out of geojson files, especially given that it's nonobvious how to do that, then having some sample code would really help and I'd suggest this bug should stay open.
But if people looking to parse GeoJSON should just use geojson instead, which already has sample code for this, then that's fine too. I'd still suggest that it would be friendly to new devs to put a pointer about this the docs, but that's up to you.
Thanks for engaging with me on this, just trying to make things a little easier for the next guy or gal. :-)


[Feature] External cancellation of ...WaitFor... methods where feasible

Add external cancellation of ...WaitFor... methods where feasible.
Reasoning:
Wait for style methods have a timeout, but there are circumstances where external events could be the cause to cancel specific waits.
The current solution doesn't provide any cancellation except default/custom timouts.

Thanks for the feature request, unfortunately this is not compatible with the way Playwright, the driver and calls to the browser are working. If you sent once something to the browser, it can't be canceled. Since we don't plan on implementing this, I will close it for now.

Could you provide a complete example where you benefit from the cancellation token? I'd like to understand the use case better.


[FORMAT] Support for Patran neutral files (.neu)

How would you improve meshio?
Support for Patran neutral files (.neu)
What problem does it solved for you?
To begin with, just reading Patran files in Python. Eventually, plotting 2D meshes, determine if meshes are 2D/3D, etc.

It does not seem easy to find the specification for this format, but here is an unofficial description, it seems: http://www.g-boege.de/english/makrosae/Manual/PATRAN.htm


Add setting to allow using custom separator for js concatenation

Example.
File 1:
(console.log(1))

File 2:
(console.log(2))

File 3:
(console.log(3))

Concatenated result:
(console.log(1))
(console.log(2))
(console.log(3))

This js raises  Uncaught TypeError: undefined is not a function on second line.
Solution: use custom separator with ; or \n;\n.

Fixed in #470


Fix issue with pod install and new pre-release version of ReactiveSwift #trivial

This fixed the issue mentioned in #842.

Current coverage is 83.27% (diff: 100%)

Merging #878 into master will increase coverage by 8.20%

@@             master       #878   diff @@
==========================================
  Files            19         34     +15
  Lines           690       2769   +2079
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits            518       2306   +1788
- Misses          172        463    +291
  Partials          0          0

Powered by Codecov. Last update 6282a21...416cb23

Going to close this unless it can be reproduced.


SyncQueue::pushRaw not implemented

Laravel Version: 5.5.x - 6.0 (all?)
PHP Version: 7.3.x
Database Driver & Version: N/A

Description:
The SyncQueue driver is missing an implementation for QueueContract::pushRaw. This was encountered when trying to write tests covering interop with other services submitting 'raw' jobs ({job: FQCN, data: {...}}) to a Laravel app.
Steps To Reproduce:

Set queue driver to 'sync'
Push a raw job onto the queue using the Queue facade:Queue::pushRaw(json_encode([
        'job' => Your\Job::class,
        'data' => (object) [],
]));

Observe no handle method is called, no queue events are fired etc.

Expected Behaviour

Job's handle method is called
All relevant queue events are dispatched

Push raw is not available for the sync queue:
https://github.com/laravel/framework/blob/419253f67bd914cb8729a105ee847393c22fd9c5/src/Illuminate/Queue/SyncQueue.php#L134

Hi @driesvints, apologies I may be misunderstanding something here, but from my point of view, the fact it's not available in the sync queue is the reason I've opened this issue.
To meet the QueueContract correctly, I would expect this method to be implemented.
If it is not implemented by design, can you explain the reasoning behind that decision? In this case, I would expect the method to throw a NotImplementedException explaining why.
Thanks

@cqwyllie it's because there's no implementation for "raw" with a sync queue. There's no driver. No "raw" way of handling queued jobs.

Should it not still be possible to implement the 'raw' method?
If I read other drivers correctly, the difference seems to be that 'raw' treats what it's given as a 'payload' whilst other push methods call through 'createPayload' or similar prior to persistence/handling. In this case, it should be possible to run the 'raw' payload through the same handling pipeline after some deserialisation step (as with other drivers retrieving jobs from a persistence layer).
I appreciate this is perhaps an edge case, but it is surprising to find the driver silently ignoring calls to a method because it doesn't meet the implemented contract.
@driesvints is there any info to read about the 'raw' job stuff? It's something that's of interest to us because of interfacing with other non-laravel systems, but material seems to be a little sparse from our searches.
Thanks again for your time.

I don't think it's possible to implement a raw implementation on the sync queue sorry. There's no single way to handle all cases. If you still want this please open up an issue on the ideas repo.

Ok, cool, was unaware of the ideas repo.


Can't get '$' alias to work

Looking at the documentation, '$' is special and used for exact match: http://webpack.github.io/docs/configuration.html#resolve-alias
like
alias: {
'xyz$': 'a/b/c'
}
But is there a way to escape the $ and have an alias like this?
alias: {
'$': 'path/to/jqueryAlias'
}

I fixed this by using webpack.ProvidePlugin
https://webpack.github.io/docs/shimming-modules.html#plugin-provideplugin


 I'll work on them tomorrow.

@arjunguha, you forgot to answer the two comments I had about the PR. Can you answer them?


An updated version of utils/admin script with more options

Added options to change other user flags, in addition to the isSupport flag.
Maybe related to #953.
Changes in this pull request:

Update Dockerfile;
README explaining how to run locally;
backwards compatibility with update-support-user.sh;
support to change isSuper, isSupport, isLocked, isTest flags and show user info.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Investigate use of xlc17 for AIX builds

Related: https://github.com/adoptium/adoptium-support/issues/824#issuecomment-1753328083
We have some interest in building AIX with xlc17 instead of xlc16. The link above indicates that a customer believes they have some issues related to JNI when the JDK is built with xlc16. There has been some upstream resistance to the change to switch over to requiring C++17 (therefore xlc17) by default although it looks like SAP has successfully built the JDK with xlc17.
We should pre-empt a change in the future by looking at installing xlc17 and verifying that it works on our existing build systems, which could also allow us to supply pre-release test builds (unsupported, untested etc. for now) with that compiler to users who desire them.

Once you have a build requirement for xlc17 aka openXL - you will also have a requirement for AIX 7.2 TL5 as that is the base requirement for the compiler itself - and it's libraries depend on these base AIX libraries.
Trying to install on AIX 7.2 TL2 (would be nearly the same for TL4 obviously):
FAILURES
--------
  Filesets listed in this section failed pre-installation verification
  and will not be installed.

  Requisite Failures
  ------------------
  SELECTED FILESETS:  The following is a list of filesets that you asked to
  install.  They cannot be installed until all of their requisite filesets
  are also installed.  See subsequent lists for details of requisites.

    libc++.rte <IP_ADDRESS>                       # IBM XL C++ Runtime for AIX 7...
    libc++abi.rte <IP_ADDRESS>                    # IBM XL C++ Runtime for AIX 7...
    libunwind.rte <IP_ADDRESS>                    # IBM XL C++ Runtime for AIX 7...
    openxlCcmp.17.1.1 <IP_ADDRESS>                # XL C++ compiler
    openxlCcmp.17.1.1.bundle <IP_ADDRESS>         # Open XL C++ media defined bu...
    openxlCcmp.17.1.1.lib <IP_ADDRESS>            # Open XL C/C++ libraries
    openxlCcmp.17.1.1.ndi <IP_ADDRESS>            # Open XL C++ non-default inst...
    openxlCcmp.17.1.1.tools <IP_ADDRESS>          # Open XL C++ tools
    xlmass.10.1.1 <IP_ADDRESS>                    # IBM Mathematical Acceleratio...

  MISSING REQUISITES:  The following filesets are required by one or more
  of the selected filesets listed above.  They are not currently installed
  and could not be found on the installation media.

    bos.adt.base <IP_ADDRESS>                    # Fileset Update
    bos.adt.include <IP_ADDRESS>                 # Fileset Update
    bos.adt.lib <IP_ADDRESS>                       # Base Level Fileset
    bos.adt.libm <IP_ADDRESS>                      # Base Level Fileset
    bos.rte.bind_cmds <IP_ADDRESS>               # Fileset Update
    bos.rte.iconv <IP_ADDRESS>                     # Base Level Fileset
    bos.rte.libc <IP_ADDRESS>                    # Fileset Update
    bos.rte.libcur <IP_ADDRESS>                  # Fileset Update
    bos.rte.libpthreads <IP_ADDRESS>             # Fileset Update
    bos.rte.security <IP_ADDRESS>                # Fileset Update

  << End of Failure Section >>

OK so basically the existing machines are TL04 and TL05 is required for XLC17.
That means we can either set it up on the AIX 7.3 system that we have, or (since you mentioned that we have an adopt09 that isn't yet set up) we could set that up with 7.2TL05 for the purposes of experimenting with XLC17.
I'm happy with going for that option.

Most of ther existing systems are at 7.2 TL02 actually. There is one test system at TL04 iirc.

Yeah xlc13 is still being used for JDK8 so ideally we want all three.

system is up and ready for playbook configuration. o7

Assigning to Haroon on the basis that he's working on the other AIX 7.3 machine and once that's settled and has a working playbook we can test the output from that on this new machine.

Reopening as the inventory PR shouldn't have closed this (We haven't done any testing on a suitable machine yet)

@Haroon-Khel It looks like the AIX 7.2 build machine -3 has had the playbooks run on it - can you remember how far we got? It doesn't seem to have the jenkins user created yet so I guess it hasn't got too far ...

Potentially required to resolve https://github.com/adoptium/aqa-tests/issues/5169

Noting that the JDK23 builds now fail with the following - starting from teh 14th March (Was ok on the 7th):
20:11:52  configure: Using default toolchain clang (clang/LLVM)
20:11:52  configure: Will use user supplied compiler CC=xlclang
20:11:52  checking for xlclang... /opt/IBM/xlC/16.1.0/bin/xlclang
20:11:52  checking resolved symbolic links for CC... /opt/IBM/xlC/16.1.0/bin/xlclang++
20:11:53  configure: error: A clang compiler is required. Try setting --with-tools-dir.
20:11:53  configure: The C compiler (located as /opt/IBM/xlC/16.1.0/bin/xlclang) does not seem to be the required clang compiler.
20:11:53  configure: The result from running with --version was: "IBM XL C/C++ for AIX, V16.1.0  (5725-C72, 5765-J12)
20:11:53  Version: 16.01.0000.0011"

JDK22 is currently still able to build with XLC16 so is not yet affected

Related https://github.com/adoptium/temurin-build/pull/3822

Investigation done and OpenXL is installed on an AIX7.2TL05 machine and is now live for JDK22+


feat: make minio work

This is awesome! I didn't know minio was that seamless to work. I did run into a couple of issues though.

If I were to click on Check synchronization configuration it would error TypeError: options.endpoint is not a function.
If I synced my files to S3, then delete my local files on ~.config/joplindev-desktop, and try to sync back all the files from S3. It would error UriParameterError: Expected uri parameter to have length >= 1, but found "" for params.Bucket and not sync.

Sorry - should have said, this is a hilarious-ly untested version - it does work (basic sync) but I've not tested it beyond thatI'll have a go with it tomorrow at some pointOn 14 Jul 2020 23:03, alexchee<EMAIL_ADDRESS>wrote:
This is awesome! I didn't know minio was that seamless to work. I did run into a couple of issues though.
If I were to click on Check synchronization configuration it would error TypeError: options.endpoint is not a function.If I synced my files to S3, then delete my local files on ~.config/joplindev-desktop, and try to sync back all the files from S3. It would error UriParameterError: Expected uri parameter to have length >= 1, but found "" for params.Bucket and not sync.
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.

Closing this, will open a PR vs the main repo


[Crunchyroll] Error: Unable to extract title

Checklist

[ ] I'm reporting a broken site support issue
[x] I've verified that I'm running youtube-dl version 2019.06.21
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar bug reports including closed ones
[x] I've read bugs section in FAQ

Verbose log

E:\Crunchy>youtube-dl.exe https://www.crunchyroll.com/yu-gi-oh-gx --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.crunchyroll.com/yu-gi-oh-gx', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2019.06.21
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17134
[debug] exe versions: ffmpeg N-93726-g7eba264513
[debug] Proxy map: {}
[crunchyroll:playlist] yu-gi-oh-gx: Downloading webpage
ERROR: Unable to extract title; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmppgbwk_g4\build\youtube_dl\YoutubeDL.py", line 796, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmppgbwk_g4\build\youtube_dl\extractor\common.py", line 530, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmppgbwk_g4\build\youtube_dl\extractor\crunchyroll.py", line 666, in _real_extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmppgbwk_g4\build\youtube_dl\extractor\common.py", line 1014, in _html_search_regex
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmppgbwk_g4\build\youtube_dl\extractor\common.py", line 1005, in _search_regex
youtube_dl.utils.RegexNotFoundError: Unable to extract title; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

Trying to get this working with Crunchyroll Playlists. Not sure what I'm missing though. I've tried plugging in my username and password in it too. It does work if I put in the episode in the link (eg https://www.crunchyroll.com/yu-gi-oh-gx/episode-1-yugis-heir-684069) but the playlist itself does not and gives me Error matching title.
I've tried this on both my Windows and my Mac. Both updated using youtube-dl -U.

How is this issue a duplicate of https://github.com/ytdl-org/youtube-dl/pull/15840??
An explanation would be nice.

I've searched the bugtracker for similar bug reports including closed ones

I am seeing this problem too.  It looks like this may have been fixed as discussed in #21291, but I am having this issue and there are two more duplicate issue of people having the same problem.  Can we confirm that this was fixed please?