Add docker container and lua script for feeding data to Prometheus push gateway

Prometheus push gateway support for wrk2
This PR adds support for exporting data to a Prometheus push gateway. Both run-time data (during benchmark execution) as well as result data (latency histogram) will be exported.
The PR adds a LUA script, a wrk2 wrapper script (shell), and a build recipe for a docker container image.
How to use
For testing the PR it is recommended to build and run the benchmark container on the local developer machine, testing against a local HTTP endpoint, and feeding the benchmark data to a Prometheus push gateway in a kubernetes cluster.
Instructions below assume

a kubernetes cluster is available with
1.a Prometheus
1.b push gateway
1.c and Grafana installed and set up
A HTTP endpoint available on the local host for testing

Steps to use:

Build the docker container

$ docker docker build -t wrk2-prometheus -f docker/Dockerfile.benchmark-container-prometheus-export .

Create suitable port forwards to the push gateway and to Grafana, using the namespace and service names that match your cluster's. The below commands assume we use a monitoring namespace and Lokomotive's Prometheus oerator:

$ kubectl -n monitoring port-forward svc/prometheus-operator-grafana 3000:80 &
$ kubectl -n monitoring port-forward svc/pushgateway 9091 &

import the experimental wrk2 cockpit dashboard to Grafana. Open the imported dashboard. Set the time span (upper right hand side) to "last 5 minutes", and the refresh rate to "1s" or "5s".

Run the benchmark container on your local machine. It will use the push gateway port forwarded in step 2. to push metrics. In the below example the local machine provides a HTTP endpoint at http://<IP_ADDRESS>/index.txt. We'll run a test with 10 simultaneous connections and 1000 RPS for 60 seconds:

$ docker run --net=host -ti wrk2-prometheus -p http://<IP_ADDRESS>:9091/metrics/job/wrk2_bench_test/instance/test_run -c 10 -d 60 -r 1000 http://<IP_ADDRESS>/index.txt

Testing done
Ran the above for 60, 300, and 600 seconds, with 1000, 5000, and 10000 RPS, and 10, 50, 200 parallel connections on my local machine.

@t-lo The app always stops with non-zero exit code but what the error is, is not very visible:
...
  59997 requests in 1.00m, 48.63MB read
Requests/sec:   1000.30
Transfer/sec:    830.27KB
Total Requests: 59997
HTTP errors: 0
Requests timed out: 0
Bytes received: 50994470
Socket connect errors: 0
Socket read errors: 0
Socket write errors: 0

URL call count
http://<IP_ADDRESS> : 60010

$ echo $?
56

Hmmm, I suspect the Prometheus push gateway is unreachable from where you're running the test. The container should validate the push gateway before starting the benchmark - I've just added this.
Can you please re-test? Also, you may use the -p stdout command line option to just print prometheus metrics to stdout.

ping @surajssd


Feature request: ability to reorder pinned items by drag and drop

Thanks for a very useful extension!
Just one thing I'm missing is the ability to reorder the pinned items list, preferably by drag and drop.

Great idea, but it gonna take a while to add this feature. Because the Tree View of VSCode didn't provide the drag & drop API. So I'm thinking of creating a webview UI to let the user reorder the pinned items, but it needs time to do so. I'll leave this issue open until I finish the job.
Reference:

https://code.visualstudio.com/api/references/vscode-api#TreeView

Would it be possible to implement a 'Sort by: Name' or similar function in the 'More Actions...' menu?
Any sorting ability would be great. But otherwise awesome plugin!

@Lila-Blume I think it would be best to open a separate issue as it is a different feature

As a temporary workaround, you can manually modify .vscode/pinned-files.json. After, you need to reload the window and the new order will apply.


Load and use trained model with ktrain

I have a trained model in .h5 and preproc file format for racial recognition using ktrain library. How do I get to load and use the trained model at a later time.

You can use a Predictor instance, which is demonstrated in each of the tutorial notebooks and the example notebooks, as well.

Your tutorials and example notebooks were very helpful.
But I am still unable to load and use my trained model on a new notebook or
at a later time after training on the ktrain library, except i have to
retrain from scratch.
After training I got a .h5 and a preproc file, which I saved on my google
drive. How do I go about loading and using the trained model without having
to start training all over again. I have tried the predictor instance, but
it can only work on a new notebook at a later time. Please I need your
assistance. Thanks.
On Fri, Apr 9, 2021 at 11:01 PM Arun S. Maiya @.***>
wrote:

You can use a Predictor instance, which is demonstrated in each of the
tutorial notebooks
https://github.com/amaiya/ktrain/tree/master/tutorials and the example
notebooks https://github.com/amaiya/ktrain/tree/master/examples, as
well.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/amaiya/ktrain/issues/354#issuecomment-816994747, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AS6ZVQUSBSPR3FJUPFGRZ7TTH52NJANCNFSM42VU4JPQ
.

You stated the following:

I have tried the predictor instance, but it can only work on a new notebook at a later time.

It's not clear what you mean by this, so I don't understand the source of your confusion.  Once the .h5 model and .preproc Preprocessor instance is saved, you can reload and make predictions either within the same notebook or a new notebook (without retraining the model):
predictor = ktrain.load_predictor('/tmp/my_predictor')
predictor.predict(data)

See this Google Colab notebook for example.

Thank you very much Maiya. I have been able to use the preprocessor
instance without retraining the model. The notebook was helpful.
Now, i'm trying to redeploy the ktrain model which is for computer vision
(race/gender estimation) on flask. How do I go about it? I only came across
your notebook for text classification.
On Tue, Apr 20, 2021 at 2:12 AM abhus oses @.***> wrote:

Thank you, I will do that.
On Tue, Apr 20, 2021, 2:10 AM Arun S. Maiya @.***>
wrote:

You stated the following:
I have tried the predictor instance, but it can only work on a new
notebook at a later time.
It's not clear what you mean by this, so I don't understand the source of
your confusion. Once the .h5 model and .preproc Preprocessor instance is
saved, you can reload and make predictions either within the same notebook
or a new notebook (without retraining the model):
predictor = ktrain.load_predictor('/tmp/my_predictor')predictor.predict(data)
See this Google Colab notebook
https://colab.research.google.com/drive/1YxcceZxsNlvK35pRURgbwvkgejXwFxUt
for example.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/amaiya/ktrain/issues/354#issuecomment-822871280, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AS6ZVQT5P37YELRLYHHFUPTTJTA7HANCNFSM42VU4JPQ
.

You can take a look at these resources:
https://github.com/amaiya/ktrain/blob/master/FAQ.md#how-do-i-deploy-a-model-using-flask
https://blog.keras.io/building-a-simple-keras-deep-learning-rest-api.html

Thanks alot. I will check them out and get back to you.
On Tue, Apr 20, 2021, 4:46 PM Arun S. Maiya @.***>
wrote:

You can take a look at these resources:
https://github.com/amaiya/ktrain/blob/master/FAQ.md#how-do-i-deploy-a-model-using-flask
https://blog.keras.io/building-a-simple-keras-deep-learning-rest-api.html
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/amaiya/ktrain/issues/354#issuecomment-823334526, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AS6ZVQTPCKANNWK6YDTULM3TJWHUFANCNFSM42VU4JPQ
.

You stated the following:

I have tried the predictor instance, but it can only work on a new notebook at a later time.

It's not clear what you mean by this, so I don't understand the source of your confusion. Once the .h5 model and .preproc Preprocessor instance is saved, you can reload and make predictions either within the same notebook or a new notebook (without retraining the model):
predictor = ktrain.load_predictor('/tmp/my_predictor')
predictor.predict(data)

See this Google Colab notebook for example. Once the .h5 and .preproc are saved, you can transport it to a different machine (e.g., your local laptop) and load it using the code above to make predictions at a later time.
If you're still having trouble, please provide more details on why the above does not resolve your problem.

Good day Amaiya, I just discovered I cant redeploy the model on a  new notebook (i'm sure i dont know  how to go about it). I'm replicating the age estimation task on your github notebook. But  I couldn't redeploy the saved model for a later use on a fresh notebook.
I keep getting the "NameError: name 'learner' is not defined" after calling the predictor instance. Do i need to convert the learner.model to a json file?.

You stated the following:

I have tried the predictor instance, but it can only work on a new notebook at a later time.

It's not clear what you mean by this, so I don't understand the source of your confusion. Once the .h5 model and .preproc Preprocessor instance is saved, you can reload and make predictions either within the same notebook or a new notebook (without retraining the model):
predictor = ktrain.load_predictor('/tmp/my_predictor')
predictor.predict(data)

See this Google Colab notebook for example. Once the .h5 and .preproc are saved, you can transport it to a different machine (e.g., your local laptop) and load it using the code above to make predictions at a later time.
If you're still having trouble, please provide more details on why the above does not resolve your problem.

Good day Amaiya, I just discovered I cant redeploy the model on a  new notebook (i'm sure i dont know  how to go about it). I'm replicating the age estimation task on your github notebook. But  I couldn't redeploy the saved model for a later use on a fresh notebook.
I keep getting the "NameError: name 'learner' is not defined" after calling the predictor instance. Do i need to convert the learner.model to a json file?.

It sounds like this is coding error on your end.  You don't need a Learner instance to reload the Predictor instance in a deployment scenario to make predictions on new data. If you post your entire code, I can try to help further.

Okay, thanks. i will do just that.
On Fri, Apr 23, 2021 at 10:30 PM Arun S. Maiya @.***>
wrote:

It sounds like this is coding error on your end. You don't need a Learner
instance to reload the Predictor instance in a deployment scenario to
make predictions on new data. If you post your entire code, I can try to
help further.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/amaiya/ktrain/issues/354#issuecomment-825935295, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AS6ZVQULVIMDGPLVF6DW4C3TKHRFRANCNFSM42VU4JPQ
.

Here is the code I used for training training code
https://colab.research.google.com/drive/1U1Sg4Oyjy-zatdeSynDRpirTJFozoAHG#scrollTo=gorIaIoEnJa1
Here is a new notebook where i tried to load and use the saved model to
make further predictions load saved model
https://colab.research.google.com/drive/1NbmsPNWShfD9hWdZGJoThKmOPftpxp1s
Here is the folder containing the saved model (the .h5 and preproc file)
after training, just incase you need it
age_Amaya
https://drive.google.com/drive/folders/1f_pAgf3J1KLfBCcqZv6IvASqCg9s95kR
I will be anticipating for a positive response from you, so i solve solve
further problems.
Thanks alot.
On Sat, Apr 24, 2021 at 4:34 AM abhus oses @.***> wrote:

Okay, thanks. i will do just that.
On Fri, Apr 23, 2021 at 10:30 PM Arun S. Maiya @.***>
wrote:

It sounds like this is coding error on your end. You don't need a Learner
instance to reload the Predictor instance in a deployment scenario to
make predictions on new data. If you post your entire code, I can try to
help further.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/amaiya/ktrain/issues/354#issuecomment-825935295, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AS6ZVQULVIMDGPLVF6DW4C3TKHRFRANCNFSM42VU4JPQ
.

Access to your notebooks was denied

I'm sorry about that. I have given access now.
This is the trained model, containing the .h5 and preproc files
Amaya_age_predict.ipynb
https://colab.research.google.com/drive/1U1Sg4Oyjy-zatdeSynDRpirTJFozoAHG?ouid=106465896083804561239
Here is the code for training
age_Amaya
https://drive.google.com/drive/folders/1f_pAgf3J1KLfBCcqZv6IvASqCg9s95kR
Here is the new notebook where i tried to redeploy the model without
retraining.
Redeploy_trained_model.ipynb
https://colab.research.google.com/drive/1NbmsPNWShfD9hWdZGJoThKmOPftpxp1s?ouid=106465896083804561239
Thanks.
On Sat, Apr 24, 2021 at 3:03 PM Arun S. Maiya @.***>
wrote:

Access to your notebooks was denied
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/amaiya/ktrain/issues/354#issuecomment-826098012, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AS6ZVQRFZ4OLYNYRTRQ4P5TTKLFS3ANCNFSM42VU4JPQ
.

It sounds like you're not accessing your Google Drive folder in the correct way.  In other words, this is not an issue with ktrain.
After adding a shortcut to the folder you shared with me, I was able to load your predictor just fine:
predictor = ktrain.load_predictor('/content/drive/MyDrive/age_predictormaya')

I would recommend that you go through some tutorials on how to access Google Drive with Google Colab.

Thanks very much. I will do just that.
I hope I'm able to get it right this time.
On Sat, Apr 24, 2021, 7:39 PM Arun S. Maiya @.***>
wrote:

It sounds like you're not accessing your Google Drive folder in the
correct way. In other words, this is not an issue with ktrain.
After adding a shortcut to the folder you shared with me, I was able to
load your predictor just fine:
predictor = ktrain.load_predictor('/content/drive/MyDrive/age_predictormaya')
I would recommend that you go through some tutorials on how to access
Google Drive with Google Colab.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/amaiya/ktrain/issues/354#issuecomment-826127493, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AS6ZVQT2ZWAZWQSXAIVPXH3TKL66LANCNFSM42VU4JPQ
.

I have finally gotten it. I wasn't properly accessing my my drive from
google colab.
Thank you very much.
I will always get in touch with you as I progress.
On Sat, Apr 24, 2021 at 8:30 PM abhus oses @.***> wrote:

Thanks very much. I will do just that.
I hope I'm able to get it right this time.
On Sat, Apr 24, 2021, 7:39 PM Arun S. Maiya @.***>
wrote:

It sounds like you're not accessing your Google Drive folder in the
correct way. In other words, this is not an issue with ktrain.
After adding a shortcut to the folder you shared with me, I was able to
load your predictor just fine:
predictor = ktrain.load_predictor('/content/drive/MyDrive/age_predictormaya')
I would recommend that you go through some tutorials on how to access
Google Drive with Google Colab.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/amaiya/ktrain/issues/354#issuecomment-826127493, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AS6ZVQT2ZWAZWQSXAIVPXH3TKL66LANCNFSM42VU4JPQ
.

Good morning Amaiya. How are you doing?
Is it possible to augment my data with ktrain? I want to create several
samples of my image dataset at different angles.
If possible how do I go about it?
Thanks.
On Sat, Apr 24, 2021, 11:34 PM abhus oses @.***> wrote:

I have finally gotten it. I wasn't properly accessing my my drive from
google colab.
Thank you very much.
I will always get in touch with you as I progress.
On Sat, Apr 24, 2021 at 8:30 PM abhus oses @.***> wrote:

Thanks very much. I will do just that.
I hope I'm able to get it right this time.
On Sat, Apr 24, 2021, 7:39 PM Arun S. Maiya @.***>
wrote:

It sounds like you're not accessing your Google Drive folder in the
correct way. In other words, this is not an issue with ktrain.
After adding a shortcut to the folder you shared with me, I was able to
load your predictor just fine:
predictor = ktrain.load_predictor('/content/drive/MyDrive/age_predictormaya')
I would recommend that you go through some tutorials on how to access
Google Drive with Google Colab.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/amaiya/ktrain/issues/354#issuecomment-826127493,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AS6ZVQT2ZWAZWQSXAIVPXH3TKL66LANCNFSM42VU4JPQ
.

And also convert rgb images to grayscale.
On Mon, Jul 5, 2021, 7:29 AM Abhulimen Osekhonmen @.***>
wrote:

Good morning Amaiya. How are you doing?
Is it possible to augment my data with ktrain? I want to create several
samples of my image dataset at different angles.
If possible how do I go about it?
Thanks.
On Sat, Apr 24, 2021, 11:34 PM abhus oses @.***> wrote:

I have finally gotten it. I wasn't properly accessing my my drive from
google colab.
Thank you very much.
I will always get in touch with you as I progress.
On Sat, Apr 24, 2021 at 8:30 PM abhus oses @.***> wrote:

Thanks very much. I will do just that.
I hope I'm able to get it right this time.
On Sat, Apr 24, 2021, 7:39 PM Arun S. Maiya @.***>
wrote:

It sounds like you're not accessing your Google Drive folder in the
correct way. In other words, this is not an issue with ktrain.
After adding a shortcut to the folder you shared with me, I was able to
load your predictor just fine:
predictor = ktrain.load_predictor('/content/drive/MyDrive/age_predictormaya')
I would recommend that you go through some tutorials on how to access
Google Drive with Google Colab.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/amaiya/ktrain/issues/354#issuecomment-826127493,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AS6ZVQT2ZWAZWQSXAIVPXH3TKL66LANCNFSM42VU4JPQ
.


history: add

Handle hrefs globally

Current coverage is 100.00%

Merging #2 into master will not affect coverage as of d2649a3

@@            master      #2   diff @@
======================================
  Files            2       2
  Stmts           99      99
  Branches         0       0
  Methods          0       0
======================================
  Hit             99      99
  Partial          0       0
  Missed           0       0

Review entire Coverage Diff as of d2649a3

Powered by Codecov. Updated on successful CI builds.


Added Crafting Interpreters

Though is not yet finished it looks very promising!

@eshellman, given #2357, should @quobit update this PR with the 🚧 character/emoji? (The author of Crafting Interpreters is actively working on this and I'm enjoying it so far.)

as I feared, the linter wants 🚧 to be alphabetized differently.

@eshellman do you need any other action from my side?

@quobit no. will merge and see if people like it.

Nice!


Repeated styles are generated and appended to . CSS grows enourmously.

I have created a utility component for Flex called ''.  So, if we add a prop called, 'Center' to this component, it will center the child elements both horizontally and vertically. Please see the code below.

const getAlignmentProperties = (center, hCenter, vCenter) => {
let styles = ``;
if (center) {
styles += align-items: center; justify-content: center;;
} else {
if (hCenter) {
styles += justify-content: center;;
}
if (vCenter) {
styles += align-items: center;;
}
}
return styles;
};
const Flex = styled.div(
({
inline,
flexWrap,
center,
hCenter,
vCenter
}) => {
return             display: ${inline ? "inline-flex" : "flex"};             flex-wrap: ${flexWrap};             ${getAlignmentProperties(center, hCenter, vCenter)}               ;
}
);

When I added two flex elements, like below

I get repeated styles, the Flex, flex wrap is getting repeated in the head as shown below.
.dqIUYs {     display: -webkit-box;     display: -webkit-flex;     display: -ms-flexbox;     display: flex;     -webkit-flex-wrap: wrap;     -ms-flex-wrap: wrap;     flex-wrap: wrap; } .cKPPpA {     display: -webkit-box;     display: -webkit-flex;     display: -ms-flexbox;     display: flex;     -webkit-flex-wrap: wrap;     -ms-flex-wrap: wrap;     flex-wrap: wrap;     -webkit-align-items: center;     -webkit-box-align: center;     -ms-flex-align: center;     align-items: center;     -webkit-box-pack: center;     -webkit-justify-content: center;     -ms-flex-pack: center;     justify-content: center; } .duEPgs {     display: -webkit-box;     display: -webkit-flex;     display: -ms-flexbox;     display: flex;     -webkit-flex-wrap: wrap;     -ms-flex-wrap: wrap;     flex-wrap: wrap;     -webkit-box-pack: center;     -webkit-justify-content: center;     -ms-flex-pack: center;     justify-content: center; } .jEreaw {     display: -webkit-box;     display: -webkit-flex;     display: -ms-flexbox;     display: flex;     -webkit-flex-wrap: wrap;     -ms-flex-wrap: wrap;     flex-wrap: wrap;     -webkit-align-items: center;     -webkit-box-align: center;     -ms-flex-align: center;     align-items: center; }
Ideally, the flex, flexwrap properties, should be present only once in a class. Only the dynamically added properties alone should be present in the new class generated . For Ex: .jEreaw class should contain only
    -webkit-align-items: center;     -webkit-box-align: center;     -ms-flex-align: center;     align-items: center;
since, class 'dqIUYs' is already containing flex and flex wrap properties.
I also thought of adding margin or padding (gutters) to this  component. But I'm afraid whenever a new margin or padding is given, again and again all the styles will be appended to the head with a new class name:(
Let me know how to avoid repeated styles. Please help.

It's expected that a rule is inserted per variant of your component, so I'm not really sure what the question here is. They should never be duplicated however, but without a reproduction I can't really help.
Please refer with usage questions to Stack Overflow, as you're more likely to get quicker and helpful answers there ❤ Repository issues on GitHub are great for that, issues, things that need to be resolved in the library itself, or can't be resolved by anyone but the maintainers. While I (and many others on other repos) are happy to eventually answer usage questions, styled-components is a busy repo and mostly stable and slow moving towards new releases, so you're not likely to get fast responses here


implementing and testing starting player behavior when there is a tie

Current coverage is 97.65% (diff: 100%)

Merging #60 into master will increase coverage by 0.33%

@@             master        #60   diff @@
==========================================
  Files             8          8
  Lines           298        298
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits            290        291     +1
+ Misses            8          7     -1
  Partials          0          0

Powered by Codecov. Last update f777dd4...45b6fab


AttributeError: 'OneBranchMLP' object has no attribute 'use_multiGPU

Hi Guo-Jian_Wang!
I am Suresh Parekh. Recently,  I discovered your library colfi for cosmological parameter estimation. I was trying to install it and try it out. After installation, when I run the test code as mentioned on colfi.readthedocs.io, I get an error when I run the following block
nde_type = 'MNN'
chain_n = 3
num_train = 500
epoch = 2000
num_vali = 100
base_N = 500
predictor = nde.NDEs(sim_data, model, param_names, params_dict=params_dict, cov_matrix=None,
init_chain=None, init_params=init_params, nde_type=nde_type,
num_train=num_train, num_vali=num_vali, local_samples=None, chain_n=chain_n)
predictor.base_N = base_N
predictor.epoch = epoch
predictor.fiducial_params = [a_fid, b_fid]
predictor.randn_num = randn_num
predictor.fast_training = True
predictor.train(path='test_linear')
Error I encounterd is:
AttributeError: 'OneBranchMLP_G' object has no attribute 'use_multiGPU'

AttributeError                            Traceback (most recent call last)
 in <cell line: 23>()
21 # predictor. = False
22
---> 23 predictor.train(path='test_linear')
2 frames
/usr/local/lib/python3.10/dist-packages/colfi/models_g.py in train(self, repeat_n, showEpoch_n, fast_training)
270             self.optimizer.param_groups[0]['lr'] = lrdc.exp()
271
--> 272         if self.use_multiGPU:
273             self.net = self.net.module.cpu()
274         else:
AttributeError: 'OneBranchMLP_G' object has no attribute 'use_multiGPU'
Later, in the model_g.py, I added a line under the class OneBranchMLP_G to debug the error:
self.use_multiGPU = False
But now it gave one more error and I am unable to figureout how to run this code.
Error
AttributeError                            Traceback (most recent call last)
Cell In[14], line 23
20 predictor.fast_training = False
21 # predictor. = False
---> 23 predictor.train(path='test_linear')
File ~/.local/lib/python3.10/site-packages/colfi/nde.py:769, in NDEs.train(self, path, save_items, showEpoch_n)
767 self._train = eval("self.train%s"%nde_type_i)
768 print('\nThe neural density estimator used here is %s'%nde_type_i)
--> 769 chain_1, losses = self._train(self.train_set, self.vali_set, step=step, burnInEnd=burnInEnd, burnInEnd_step=self.burnInEnd_step,
770                               randn_num=randn_nums[step-1], save_items=save_items, showEpoch_n=showEpoch_n, params_space=prev_space)
771 #works for MDN in the case None ann chain obtained, see also models_ann --> auto_train
772 retrain_n = 0
File ~/.local/lib/python3.10/site-packages/colfi/nde.py:619, in NDEs._train_MNN(self, train_set, vali_set, step, burnInEnd, burnInEnd_step, randn_num, save_items, showEpoch_n, params_space)
616     exec('self.eco.%s = value'%(key))
618 if self.branch_n==1:
--> 619     self.eco.train(repeat_n=self.repeat_n, showEpoch_n=showEpoch_n, fast_training=self.fast_training)
620     # self.eco.train_AvgMultiNoise(repeat_n=self.repeat_n, showEpoch_n=showEpoch_n) #need further test
621 else:
622     self.eco.train(repeat_n=self.repeat_n, train_branch=self.train_branch, parallel=False, showEpoch_n=showEpoch_n, fast_training=self.fast_training) #reset parallel???
File ~/.local/lib/python3.10/site-packages/colfi/models_g.py:256, in OneBranchMLP_G.train(self, repeat_n, showEpoch_n, fast_training)
254 #vali_loss
...
--> 256     self.inputs_vali, self.target_valids.AddGaussianNoise(self.obs_vali,params=self.params_vali,obs_errors=self.obs_errors,cholesky_factor=self.cholesky_factor,noise_type=self.noise_type,factor_sigma=self.factor_sigma,multi_noise=self.multi_noise,use_GPU=self.use_GPU).multiNoisySample(reorder=True)
257     self.inputs_vali = self.normalize_obs(self.inputs_vali, self.obs_base_torch)
258     self.target_vali = self.normalize_params(self.target_vali, self.params_base_torch)
AttributeError: 'OneBranchMLP_G' object has no attribute 'inputs_vali'
Can you please help me to debug and run this code?

Hi Suresh Parekh,
Thanks for using the colfi code. Are you installing it using pip install colfi? If so, please re-install from the GitHub source since the use_multiGPU issue is fixed in the GitHub version. You can do it by using:

git clone https://github.com/Guo-Jian-Wang/colfi.git
cd colfi
sudo python setup.py install

https://github.com/Guo-Jian-Wang/colfi/blob/master/examples/linear/train_linear.py, this code works well on my side.
Best regards

Hi Guo-Jian_Wang!
I am Suresh Parekh. Recently, I discovered your library colfi for cosmological parameter estimation. I was trying to install it and try it out. After installation, when I run the test code as mentioned on colfi.readthedocs.io, I get an error when I run the following block
nde_type = 'MNN'
chain_n = 3 num_train = 500 epoch = 2000 num_vali = 100 base_N = 500
predictor = nde.NDEs(sim_data, model, param_names, params_dict=params_dict, cov_matrix=None, init_chain=None, init_params=init_params, nde_type=nde_type, num_train=num_train, num_vali=num_vali, local_samples=None, chain_n=chain_n) predictor.base_N = base_N predictor.epoch = epoch predictor.fiducial_params = [a_fid, b_fid] predictor.randn_num = randn_num predictor.fast_training = True
predictor.train(path='test_linear')
Error I encounterd is: AttributeError: 'OneBranchMLP_G' object has no attribute 'use_multiGPU'
AttributeError Traceback (most recent call last) in <cell line: 23>() 21 # predictor. = False 22 ---> 23 predictor.train(path='test_linear')
2 frames /usr/local/lib/python3.10/dist-packages/colfi/models_g.py in train(self, repeat_n, showEpoch_n, fast_training) 270 self.optimizer.param_groups[0]['lr'] = lrdc.exp() 271 --> 272 if self.use_multiGPU: 273 self.net = self.net.module.cpu() 274 else:
AttributeError: 'OneBranchMLP_G' object has no attribute 'use_multiGPU'
Later, in the model_g.py, I added a line under the class OneBranchMLP_G to debug the error:
self.use_multiGPU = False
But now it gave one more error and I am unable to figureout how to run this code.
Error
AttributeError Traceback (most recent call last) Cell In[14], line 23 20 predictor.fast_training = False 21 # predictor. = False ---> 23 predictor.train(path='test_linear')
File ~/.local/lib/python3.10/site-packages/colfi/nde.py:769, in NDEs.train(self, path, save_items, showEpoch_n) 767 self._train = eval("self.train%s"%nde_type_i) 768 print('\nThe neural density estimator used here is %s'%nde_type_i) --> 769 chain_1, losses = self._train(self.train_set, self.vali_set, step=step, burnInEnd=burnInEnd, burnInEnd_step=self.burnInEnd_step, 770 randn_num=randn_nums[step-1], save_items=save_items, showEpoch_n=showEpoch_n, params_space=prev_space) 771 #works for MDN in the case None ann chain obtained, see also models_ann --> auto_train 772 retrain_n = 0
File ~/.local/lib/python3.10/site-packages/colfi/nde.py:619, in NDEs._train_MNN(self, train_set, vali_set, step, burnInEnd, burnInEnd_step, randn_num, save_items, showEpoch_n, params_space) 616 exec('self.eco.%s = value'%(key)) 618 if self.branch_n==1: --> 619 self.eco.train(repeat_n=self.repeat_n, showEpoch_n=showEpoch_n, fast_training=self.fast_training) 620 # self.eco.train_AvgMultiNoise(repeat_n=self.repeat_n, showEpoch_n=showEpoch_n) #need further test 621 else: 622 self.eco.train(repeat_n=self.repeat_n, train_branch=self.train_branch, parallel=False, showEpoch_n=showEpoch_n, fast_training=self.fast_training) #reset parallel???
File ~/.local/lib/python3.10/site-packages/colfi/models_g.py:256, in OneBranchMLP_G.train(self, repeat_n, showEpoch_n, fast_training) 254 #vali_loss ... --> 256 self.inputs_vali, self.target_valids.AddGaussianNoise(self.obs_vali,params=self.params_vali,obs_errors=self.obs_errors,cholesky_factor=self.cholesky_factor,noise_type=self.noise_type,factor_sigma=self.factor_sigma,multi_noise=self.multi_noise,use_GPU=self.use_GPU).multiNoisySample(reorder=True) 257 self.inputs_vali = self.normalize_obs(self.inputs_vali, self.obs_base_torch) 258 self.target_vali = self.normalize_params(self.target_vali, self.params_base_torch)
AttributeError: 'OneBranchMLP_G' object has no attribute 'inputs_vali'
Can you please help me to debug and run this code?

Hi Suresh Parekh, I have fixed this bug, please re-install colfi and run examples in it. Thanks.

Hi Guo Jian Wang,
Thank you very much for your help! It worked! All 5 steps were completed successfully. Later during the plotting part, it generated an error with the coplot library. As again you are the developer of the same library, I thought to address it here.
While running the last block of the code,
predictor.get_steps() predictor.get_contour() predictor.get_losses() plt.show()
In the predictor.get_steps() it is giving me the following error:
######################### step 5/5 #########################
Updating parameter space to be learned ...
Learning range of a: [-1.715130, 5.051654] ~ [-5.0\sigma, +5.0\sigma]
Learning range of b: [2.259633, 2.733966] ~ [-5.0\sigma, +5.0\sigma]
Selecting samples from the mock data of the previous step ...
600 sets of samples of previous step are added to the training set
The neural density estimator used here is ANN
Training the network using CPU
The batch size is set to 500
Fast training the network, the iteration per epoch is 3, the batch size is 2500!
randn_num: 5.1157
(epoch:100/2000; train_loss/vali_loss:0.06154/0.10735; lr:0.00504661)
(epoch:200/2000; train_loss/vali_loss:0.06315/0.06557; lr:0.00252930)
(epoch:300/2000; train_loss/vali_loss:0.06230/0.06236; lr:0.00126765)
(epoch:400/2000; train_loss/vali_loss:0.05717/0.06067; lr:0.00063533)
(epoch:500/2000; train_loss/vali_loss:0.05969/0.06169; lr:0.00031842)
(epoch:600/2000; train_loss/vali_loss:0.06049/0.06456; lr:0.00015959)
(epoch:700/2000; train_loss/vali_loss:0.05932/0.05931; lr:0.00007998)
(epoch:800/2000; train_loss/vali_loss:0.06138/0.06170; lr:0.00004009)
(epoch:900/2000; train_loss/vali_loss:0.06129/0.06237; lr:0.00002009)
(epoch:1000/2000; train_loss/vali_loss:0.05914/0.06034; lr:0.00001007)
(epoch:1100/2000; train_loss/vali_loss:0.06080/0.06008; lr:0.00000505)
(epoch:1200/2000; train_loss/vali_loss:0.06011/0.05646; lr:0.00000253)
(epoch:1300/2000; train_loss/vali_loss:0.05919/0.05886; lr:0.00000127)
(epoch:1400/2000; train_loss/vali_loss:0.06066/0.06446; lr:0.00000064)
(epoch:1500/2000; train_loss/vali_loss:0.06431/0.06513; lr:0.00000032)
(epoch:1600/2000; train_loss/vali_loss:0.06104/0.06275; lr:0.00000016)
(epoch:1700/2000; train_loss/vali_loss:0.06736/0.06881; lr:0.00000008)
(epoch:1800/2000; train_loss/vali_loss:0.06497/0.05629; lr:0.00000004)
(epoch:1900/2000; train_loss/vali_loss:0.06159/0.06288; lr:0.00000002)
(epoch:2000/2000; train_loss/vali_loss:0.06116/0.06004; lr:0.00000001)
The directory "test_linear/chain_ann" is successfully created !
Time elapsed for the training process: 6.385 minutes
Traceback (most recent call last):
File "/home/suresh/colfi/examples/linear/train_linear.py", line 83, in
predictor.get_steps()
File "/usr/local/lib/python3.10/dist-packages/colfi-0.6.0-py3.10.egg/colfi/plotter.py", line 311, in get_steps
File "/home/suresh/.local/lib/python3.10/site-packages/coplot/plots.py", line 315, in plot
pls.PlotSettings().setting(location=[self.lon_n, self.lat_n, i*self.lat_n+j+1], lims=data['lims'], labels=data['labels'],
File "/home/suresh/.local/lib/python3.10/site-packages/coplot/plot_settings.py", line 109, in setting
tick.label.set_rotation(rotation)
AttributeError: 'XTick' object has no attribute 'label'. Did you mean: '_label'?
Can you please suggest what changes shall I make?

Hi Guo Jian Wang,
Thank you very much for your help! It worked! All 5 steps were completed successfully. Later during the plotting part, it generated an error with the coplot library. As again you are the developer of the same library, I thought to address it here.
While running the last block of the code,
predictor.get_steps() predictor.get_contour() predictor.get_losses() plt.show()
In the predictor.get_steps() it is giving me the following error:
######################### step 5/5 ######################### Updating parameter space to be learned ... Learning range of a: [-1.715130, 5.051654] ~ [-5.0\sigma, +5.0\sigma] Learning range of b: [2.259633, 2.733966] ~ [-5.0\sigma, +5.0\sigma]
Selecting samples from the mock data of the previous step ... 600 sets of samples of previous step are added to the training set
The neural density estimator used here is ANN
Training the network using CPU The batch size is set to 500 Fast training the network, the iteration per epoch is 3, the batch size is 2500! randn_num: 5.1157 (epoch:100/2000; train_loss/vali_loss:0.06154/0.10735; lr:0.00504661) (epoch:200/2000; train_loss/vali_loss:0.06315/0.06557; lr:0.00252930) (epoch:300/2000; train_loss/vali_loss:0.06230/0.06236; lr:0.00126765) (epoch:400/2000; train_loss/vali_loss:0.05717/0.06067; lr:0.00063533) (epoch:500/2000; train_loss/vali_loss:0.05969/0.06169; lr:0.00031842) (epoch:600/2000; train_loss/vali_loss:0.06049/0.06456; lr:0.00015959) (epoch:700/2000; train_loss/vali_loss:0.05932/0.05931; lr:0.00007998) (epoch:800/2000; train_loss/vali_loss:0.06138/0.06170; lr:0.00004009) (epoch:900/2000; train_loss/vali_loss:0.06129/0.06237; lr:0.00002009) (epoch:1000/2000; train_loss/vali_loss:0.05914/0.06034; lr:0.00001007) (epoch:1100/2000; train_loss/vali_loss:0.06080/0.06008; lr:0.00000505) (epoch:1200/2000; train_loss/vali_loss:0.06011/0.05646; lr:0.00000253) (epoch:1300/2000; train_loss/vali_loss:0.05919/0.05886; lr:0.00000127) (epoch:1400/2000; train_loss/vali_loss:0.06066/0.06446; lr:0.00000064) (epoch:1500/2000; train_loss/vali_loss:0.06431/0.06513; lr:0.00000032) (epoch:1600/2000; train_loss/vali_loss:0.06104/0.06275; lr:0.00000016) (epoch:1700/2000; train_loss/vali_loss:0.06736/0.06881; lr:0.00000008) (epoch:1800/2000; train_loss/vali_loss:0.06497/0.05629; lr:0.00000004) (epoch:1900/2000; train_loss/vali_loss:0.06159/0.06288; lr:0.00000002) (epoch:2000/2000; train_loss/vali_loss:0.06116/0.06004; lr:0.00000001) The directory "test_linear/chain_ann" is successfully created !
Time elapsed for the training process: 6.385 minutes Traceback (most recent call last): File "/home/suresh/colfi/examples/linear/train_linear.py", line 83, in predictor.get_steps() File "/usr/local/lib/python3.10/dist-packages/colfi-0.6.0-py3.10.egg/colfi/plotter.py", line 311, in get_steps File "/home/suresh/.local/lib/python3.10/site-packages/coplot/plots.py", line 315, in plot pls.PlotSettings().setting(location=[self.lon_n, self.lat_n, i*self.lat_n+j+1], lims=data['lims'], labels=data['labels'], File "/home/suresh/.local/lib/python3.10/site-packages/coplot/plot_settings.py", line 109, in setting tick.label.set_rotation(rotation) AttributeError: 'XTick' object has no attribute 'label'. Did you mean: '_label'?
Can you please suggest what changes shall I make?

Hi Suresh Parekh,
This might be caused by the version of matplotlib; please update your matplotlib; it works well on my side with both matplotlib=3.4.3 and matplotlib=3.7.2.
BTW. Please use coplot=0.1.3 if you are using a higher version.

Hi Guo Jian Wang,
Thanks a lot for the help! Yes, changing the version of matplotlib and coplot worked. Now it's working perfectly! Thank you very much.


Dial a Ride Problem

Please, if timefold tool applicable for a solution to the Dial-A-Ride Problem?

Good idea. We need to create a quickstart for it in timefold-quickstarts.


Create passlist.txt

Error : check your password list path in termux ....what does it mean..pls help me


Add raml org data testing

Added some integration tests for examples in raml.org's repo. There is still a lot of features needed for data types, but this starts to lay the ground for more complete integration testing based off of "official" examples.

Current coverage is 95.74% (diff: 100%)

No coverage report found for v0.2.0-dev at a46cc61.

Powered by Codecov. Last update a46cc61...7e8d795

mergin' it!


t2iadapter preprocs do not get listed in Deforum

I'm trying to use the t2iadapter_color_sd14v1 ControlNet model in Deforum (A1111). The corresponding preprocessor doesn't appear in the list, even though the yaml is in the models directory. None of the t2iadapters show up, although they appear in the drop down list in img2img.

Solved. They aren't missing, just renamed.


Show Action Button dosen't work , No exist [Encore\Admin\Message\MessageController::show()]

At message grid view, if I click title column, I can see detail of message.
but if I click [ Show Action Button ] at At action column, It doesn't work because No exist show method.
so I called  $actions->disableView(); at Encore\Admin\Message\MessageController.php

I didn't fix it.
I just removed view menu.because I click Title Column I can see message .


License

Hi,
Does this project have the same license as the older versions? eg. Microsoft Public License (Ms-PL). Would it be possible to specify it in the repository?
Thanks,
Mark

AFAICT very few people use this old technology. I'm busy with other
things now. Just tell me what you need and I'll try to make it happen
for you.
On 2016-08-20 18:01, Mark Nuttall-Smith wrote:

Hi,
Does this project have the same license as the older versions? eg.
Microsoft Public License (Ms-PL). Would it be possible to specify it
in the repository?
Thanks,
Mark
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub [1], or mute the
thread [2].

Links:
[1] https://github.com/keithalewis/xll12/issues/2
[2]
https://github.com/notifications/unsubscribe-auth/ABQqoiEJHkSWPIrWg2Zw4vXEqWU2XfFwks5qhyTSgaJpZM4JpGrH

Old technology? Are you referring to Excel, C++ or the XLL lib itself?! Pretty sure Excel and C++ will never go out of fashion, therefore something that helps join the two together will always have a place right? :) Anyhow, I'm investigating the library for a small project, but without a license it's pretty much a non-starter. So - thanks for adding the license file.
I come from a managed code background, and have used Excel-DNA with success in the past. However, for this project I need to integrate a (secure) zeromq transport layer to a backend service, so have decided to swallow the C++ red pill. I've actually got a plugin working using XLLPlus, but having a closed-source dependency makes me a little uncomfortable.
Couple more questions if you have a mo:
I notice there's no XLOPER subclassed in oper.h. Am I right in thinking that this means Excel <= 2003 is no longer supported? Not a massive deal for me, just curious.
Is there any other functionality that remains to be ported from the older versions?

I'm referring to the Excel SDK. I don't think Microsoft will commit any resources to this in the future.
I no longer support Excel 2003 and older.
No automatic help file generation is the main missing functionality.
Be sure to look into asynchronous functions if you are getting data over a network.


Missing imports on yew.rs code snippets

This is about:

Undocumented code

Problem

Most yew.rs code snippets don't include the required imports - this makes it difficult for new users to copy and paste code snippets and get them running. This is especially true when the import is not included in prelude.
An example can be seen on the introduction page for Components but there are many more.
This issue was mentioned in this Discord message and I may have forgotten to write up this issue until now,..🤦♂

Details about the solution you'd like (Optional)
Imports should be explicit, so sticking yew::prelude::* on every example wouldn't solve this issue. This should also be done for both v0.18.0 and on master/next.
Additional context (Optional)

Questionaire (Optional)

[x] I'd like to write this documentation
[ ] I'd like to write this documentation but I'm not sure what's needed
[ ] I don't have time to add this right now, but maybe later

I'd like to take this one step further and have some sort of a way to test the code snippets. Tokio's website does this (I haven't looked at how) so we might be able to do it as well.


No documentation about config for method client.containers.create(config, wait=False)

There is no documentation for the argument config of method client.containers.create(config, wait=False).
The documentation says:

This method requires the container config as the first parameter. The config itself is beyond the scope of this documentation. Please refer to the LXD documentation for more information.

What is the documentation that is refereed as "LXD documentation" ?
TODO: add a link in pylxd documentation to that documentation.

How did you find out?

Actually from the docs and from some trial and error :). I am not fully familiar and I can't use it like the command like but the general idea is a dictionary with keys image profile etc and values the settings you want to apply

that still need other config options
, 'source': {'type':'image', 'alias': 'DetectionSvc'}

whart are the ohter type of source?
I want to load from the file , but  without defination of source types  i don' know how to load images from backup exports.
PyLXD is ubuntu funded project and it is lacking a lot of stuff , documentation  , API . - Not professional.


Add translation for Nepali in docs: HOWTO

What does this PR do?
Improve repo
Description
Added Nepali translation of How-To in docs and linked it in Readme.md.
Why is this valuable (or not)?
It improves the language diversity of the repository.
Checklist:

[X] Read our contributing guidelines.
[X] Search for duplicates.
[X] Include author(s) and platform where appropriate.
[X] Put lists in alphabetical order, correct spacing.
[X] Add needed indications (PDF, access notes, under construction).
[X] Used an informative name for this pull request.

Follow-up

Check the status of GitHub Actions and resolve any reported warnings!

Thanks, and welcome to free-programming-books!


hal command log location in Linux?

Where can I find hal command logs if its running as a container?
other than the --debug output on the CLI, is there any place where I could see all the hal log files?

please anybody help..


Arabic tokens are suspiciously short

The new unicode tests show that things are mostly working in unicode. But I noticed one problem.
The token regex is designed for languages I speak; it gets increasingly out of whack the farther you get from them.
It's probably basically broken on Japanese or Chinese; I would suggest pre-parsing Japanese or Chinese text to be space-separated before inputting into a bookworm.
But it's also failing on Arabic, where it should have a fighting chance. The first line of Quran is ending up as 29 words, which seems probably too long; many "words" are only a character or so. (Though I dunno, maybe that's how Arabic looks).
If there are Arabic speakers out there, I'm curious where things are going wrong. Below It looks to me like possibly it's splitting at some kind of inflection character, or something of the sort? (Each line, it thinks, is a "word" in Arabic.)
It may be that some inflection characters aren't technically instances of unicode 'p{L}', but should be allowed to occur inside words?
In [1]: import bookwormDB.tokenizer

In [2]: verse = u"بِسْمِ اuللَّـهِ الرَّحْمَـٰنِ الرَّحِيمِ"

In [3]: tokens = bookwormDB.tokenizer.tokenizer(verse).tokenize()

In [4]: for token in tokens: print token
ب
ِ
س
ْ
م
ِ
اuلل
ّ
َ
ـه
ِ
الر
ّ
َ
ح
ْ
م
َ
ـ
ٰ
ن
ِ
الر
ّ
َ
ح
ِ
يم
ِ

judging by this, the first verse should have 4 tokens: http://corpus.quran.com/treebank.jsp?chapter=1&verse=1

I've asked a native speaker, it really should output 4 tokens.

Here's a few tokens to potentially add to your tests: 'मे', 'है', 'नही',
'होत', 'जात', 'किय', 'लिए', 'मै', 'हुए', 'कुछ', 'रूप', 'हुआ', 'हैं', 'दिय',
'होन', 'मेर', 'किस', 'कोई', 'साथ',  'जान', 'प्रकार', 'लिय', 'बात', 'वाल',
'हुई', 'क्य', 'फिर', 'द्वार', 'जैस', 'कारण', 'देत', 'जीवन', 'बहुत', 'देख',
'नाम', 'मुझ', 'होग'.
I'm not sure if this pertains to Bookworm, but I've had Python re stumble
over word characters that consist of two codepoints, as in Hindi and
Punjabi. It does not consider them word characters (i.e. \w) and fails
isalpha(). The 'regex' module has better support, which BW uses in most
places.
On Wed, Jan 25, 2017 at 8:19 AM Whadup<EMAIL_ADDRESS>wrote:
I've asked a native speaker, it really should output 4 tokens.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Bookworm-project/BookwormDB/issues/123#issuecomment-275135563,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADj74QaASKn5VnmlpJXqfjnsNe5DB9Mks5rV2f8gaJpZM4LtcYy
.

 Hindi?
So we need some tests, and then we need a regex that enforces the test.
It is clear that the catchall regex here: \p{L}+ is not catching some class of characters in Hindi and Arabic. But I don't understand these scripts well enough to fix this, so we need someone who speaks the language and can look up some regex class codes in them. It may be that there's a group capture for all inflection characters or something; it might be that there are just five or ten codepoints that would cover language spoken by a significant fraction of the world's population (if we're having trouble with Arabic and Hindi).
It's possible that we need language-aware tokenization rules. I would really rather not adopt this strategy, because language metadata and detection is very frequently wrong, particularly in historical documents.
import regex as re
token_regex = re.compile("\\p{L}+",re.UNICODE|re.IGNORECASE)
tokens = ['मे', 'है', 'नही', 'होत', 'जात', 'किय', 'लिए', 'मै', 'हुए', 'कुछ', 'रूप', 'हुआ', 'हैं', 'दिय', 'होन', 'मेर', 'किस', 'कोई', 'साथ',  'जान', 'प्रकार', 'लिय', 'बात', 'वाल', 'हुई', 'क्य', 'फिर', 'द्वार', 'जैस', 'कारण', 'देत', 'जीवन', 'बहुत', 'देख', 'नाम', 'मुझ', 'होग']
tokens = [t.decode("utf-8") for t in tokens]
tokenized = re.findall(token_regex,u" ".join(tokens))
print len(tokens)
print len(tokenized)

These are Sanskrit, Hindi looked similar but is less common. They're the top tokens in Extracted Features, and I don't know if they were properly tokenized to begin with.
The problem here seems to be what I encountered: that there are more code points per character than typical, because some glyphs use multiple 'characters' that can't be understood in isolation. In my case, re wasn't looking at the whole sequence when I asked for a character-based evaluation like \w. e.g. 'है'.encode('utf-8') is b'\xe0\xa4\xb9\xe0\xa5\x88', but re treated that as two characters, the second of which is not a word character to match '\w'. regex got it correct, but in your case, seems to be failing.
You can see hints at this with your code by looking at the bytes. This is Python 3, you'll have to modify for 2:
for token in tokens:
    print(token, "\t", token.encode('utf-8'))

    tokenized = re.findall(token_regex, token)
    tokenized_codepoints = [str(t.encode('utf-8')) for t in tokenized]
    print(" ".join(tokenized), "\t", " ".join(tokenized_codepoints), '\n')

मे 	 b'\xe0\xa4\xae\xe0\xa5\x87'
म 	 b'\xe0\xa4\xae'

है 	 b'\xe0\xa4\xb9\xe0\xa5\x88'
ह 	 b'\xe0\xa4\xb9'

नही 	 b'\xe0\xa4\xa8\xe0\xa4\xb9\xe0\xa5\x80'
नह 	 b'\xe0\xa4\xa8\xe0\xa4\xb9'

होत 	 b'\xe0\xa4\xb9\xe0\xa5\x8b\xe0\xa4\xa4'
ह त 	 b'\xe0\xa4\xb9' b'\xe0\xa4\xa4'

जात 	 b'\xe0\xa4\x9c\xe0\xa4\xbe\xe0\xa4\xa4'
ज त 	 b'\xe0\xa4\x9c' b'\xe0\xa4\xa4'

किय 	 b'\xe0\xa4\x95\xe0\xa4\xbf\xe0\xa4\xaf'
क य 	 b'\xe0\xa4\x95' b'\xe0\xa4\xaf'

लिए 	 b'\xe0\xa4\xb2\xe0\xa4\xbf\xe0\xa4\x8f'
ल ए 	 b'\xe0\xa4\xb2' b'\xe0\xa4\x8f'

मै 	 b'\xe0\xa4\xae\xe0\xa5\x88'
म 	 b'\xe0\xa4\xae'

हुए 	 b'\xe0\xa4\xb9\xe0\xa5\x81\xe0\xa4\x8f'
ह ए 	 b'\xe0\xa4\xb9' b'\xe0\xa4\x8f'

कुछ 	 b'\xe0\xa4\x95\xe0\xa5\x81\xe0\xa4\x9b'
क छ 	 b'\xe0\xa4\x95' b'\xe0\xa4\x9b'

रूप 	 b'\xe0\xa4\xb0\xe0\xa5\x82\xe0\xa4\xaa'
र प 	 b'\xe0\xa4\xb0' b'\xe0\xa4\xaa'

हुआ 	 b'\xe0\xa4\xb9\xe0\xa5\x81\xe0\xa4\x86'
ह आ 	 b'\xe0\xa4\xb9' b'\xe0\xa4\x86'

And so on. Note that it is matching three codepoints at a time. For example, for b'\xe0\xa4\xb9\xe0\xa5\x81\xe0\xa4\x86', it looked at b'\xe0\xa4\xb9', b'\xe0\xa5\x81', b'\xe0\xa4\x86', and the middle one didn't match.
for seq in [b'\xe0\xa4\xb9', b'\xe0\xa5\x81', b'\xe0\xa4\x86']:
    char = seq.decode('utf-8')
    print(char, "matched" if re.match(token_regex, char) else "didn't match")

ह matched
ु didn't match
आ matched

The middle character simply adds that little tail to the first character, and is being stripped out. Try pasting ु once or twice after ह and you'll get हु then हुु.
This seems like a problem that English speakers shouldn't be trying to figure out: surely it's been discussed ad nauseum somewhere?

Here's a more to-the-point demonstration: just split on each individual character. It seems \w in particular was fixed in regex, but not \p{L}. Using original module names:
regex.findall("\\p{L}", 'नही')
>> ['न', 'ह']

regex.findall("\w", 'नही')`
>>  ['न', 'ह', 'ी']

re.findall("\w", 'नही')`
>> ['न', 'ह']

Thanks @organisciak, very helpful info.
Agree that this is mostly fumbling in the dark without informed input. But it seems there would be real benefits to just switching to \\w instead of \\p{L}, on Arabic as well. I'll give it a shot.
Also noting this appears to be a moving target. According to the regex page, as of 2016-10 "The definition of a ‘word’ character has been expanded for Unicode. It now conforms to the Unicode specification at http://www.unicode.org/reports/tr29/. This applies to \w, \W, \b and \B."
So everyone should be sure to update their version of regex.
verse = u"بِسْمِ اللَّـهِ الرَّحْمَـٰنِ الرَّحِيمِ"

for token in regex.findall(token_regex, verse):
    print token

بِسْمِ
اللَّـهِ
الرَّحْمَـٰنِ
الرَّحِيمِ

OK, the switch to r"\w" is pushing Arabic as four tokens and should do Sanskrit the way Peter lays out.
I'm sure our unicode compatibility is not perfect, and I'd welcome expansions of the base regex for other languages.
But I'm going to close this issue for now.


No way to know when a request times out or which one did

It'd be great to get visibility into when requests are timing out and which ones are being timed out for operators to debug and fix timeouts. They're not supposed to happen, and give a really crap experience for a user, so providing devs with the ability to investigate why a timeout happened I think is pretty important.
The http.Server just emits a timeout event when a timeout happen, but this library nor anything else seem to listen to it by default, so there's no log entry or hook point for developers to do stuff when a request does time out. See https://nodejs.org/api/http.html#http_server_settimeout_msecs_callback .
Happy to PR something to add this, but it's a little tricky to know what to do exactly. Ideally, there'd be a log message routed through the normal fastify request logger reporting that the request timed out with all the logger's properties (like the request ID and any other stuff that the developer added). I am not really sure how to get a handle on the fastify request that did time out though, because this library implements the timeout at the node socket level, which passes the plain old socket object to the event handlers, not the fastify request or anything. Is there a way to get the request from the socket in order to get a handle on that async context for logging?
Or, is there a better way to implement this timeout closer to fastify instead of at the http level that would allow us to keep that context around? That would also fix the encapsulation issue where this plugin could be registered multiple times in different encapsulation contexts to allow different timeouts for different contexts, which would be nice.

Have you looked into onTimeout hook? https://www.fastify.io/docs/latest/Hooks/#ontimeout

Oh, no I didn't see that! Does that mean this plugin doesn't need to exist anymore?

We should probably deprecate this.

Ah, ok, my apologies for the extra noise, the fastify core way preserves the context explicitly and seems to solve this better so I'd agree!

+1 for deprecation.


Develop upstream sync 230508

Retest Ubuntu-CPU please

Retest Ubuntu-sanity please

Retest Ubuntu-GPU-multi

Retest Ubuntu-GPU-multi please


Feature/figure popup

This is an enhancement or feature.
Summary
Added a popup=true argument to figure helper to allow image to be opened in light box (like gallery images)
Context
Not related to any GitHub issue(s).

I think this is helpful and a good idea to be able to let users open the image in a lightbox to see a bigger image if needed.


res.attachment or res.download with long filename makes browser fail the download

Hi,
when giving a filename long enough (254 chars long, for example), some browsers fail the download with an error.
Express should warn, or even truncate the filename (but keep the extensions).


how can i contact you?

Hello , i have some offer to make please give me your contact info discord or telegram or gmail...

Won't accept


using kiln's struct definition instead?

the package github.com/pivotal-cf/kiln/proofing has everything defined to work with tile metadata, the struct defined in this project is basically a subset of the definitaion. What about just switch to kiln's package directly?
This for example, already decode a lot metadata information.
There are several benefits I could think of, mainly:

Reduce maintance, don't need to keep the structs up-to-date with metadata schema, as kiln should do that.
Easy to extend, kiln's definition should be the most feature-rich one for golang, as it is the tool for making tiles
Delete some code

That's good feedback.  Didn't even know that library existed...

I can do a PR if you want, since I haven’t played with both tools much, might be interesting to dig into the code a little bit

That would be great!

Nah... Actually I am going to close this issue without any substantial work done. It's not as straightforward as I thought it would be. Main difficulty is that kiln/proofing does not provide any method on those structs.
In order to use their struct and provide additional method, there are basically two ways:

inheritance
eg. type NewStruct { OriginalStruct }. It is doable if the number of structs is small, but when there are a lot of structs (~10 in this case), it becomes a mess.

use function instead of method
This is straightforward, but requires basically rewrite(restyle) the entire project, doesn't seems like worth doing.

Either way doesn't seem like a good enough solution...


Fix random user test (was: Remove NSSWRAPPER code)

Seems that this is not needed (anymore?) to have PostgreSQL working with random UIDs.
(We have a test that runs docker with UID 12345)
@csrwng @bparees PTAL

it's still needed because postgres still needs to be able to look up the current uid and be told "postgres", otherwise it gets unhappy.  at least that is my recollection of why it is here.

@bparees how can we test that? I'm proposing this based on a conversation with @csrwng.

@rhcarvalho @bparees testing it now on rhel7

if this change doesn't break our tests then we may need more tests :)

It passed RHEL tests https://ci.openshift.redhat.com/jenkins/job/postgresql/106/console

hmm..  well I defer to @csrwng but i was pretty sure we needed this to make postgres happy.

@rhcarvalho @bparees - looks like we're overwriting the DOCKER_ARGS variable that specifies the random user:
+ DOCKER_ARGS='-u 12345'
+ USER=user2
+ PASS=pass
+ run_tests no_admin_altuid
+ local name=no_admin_altuid
+ shift
+ envs='-e POSTGRESQL_USER=user2 -e POSTGRESQL_PASSWORD=pass -e POSTGRESQL_DATABASE=db'
+ '[' -v ADMIN_PASS ']'
+ DOCKER_ARGS='-e POSTGRESQL_USER=user2 -e POSTGRESQL_PASSWORD=pass -e POSTGRESQL_DATABASE=db'
+ create_container no_admin_altuid
+ local name=no_admin_altuid
+ shift
+ local 'cargs=-e POSTGRESQL_USER=user2 -e POSTGRESQL_PASSWORD=pass -e POSTGRESQL_DATABASE=db'
+ cidfile=/tmp/tmp.KdjGKKqLC0postgresql_test_cidfiles/no_admin_altuid
+ docker run -e POSTGRESQL_USER=user2 -e POSTGRESQL_PASSWORD=pass -e POSTGRESQL_DATABASE=db --cidfile /tmp/tmp.KdjGKKqLC0postgresql_test_cidfiles/no_admin_altuid -d openshift/postgresql-92-rhel7-candidate

@csrwng good catch :) we might run into the same issue in other images as well.
I made a small change to the test runner, let's see it it fails as it should now.

Fixed the test, the nsswrapper code stays and now tested.

LGTM

LGTM (do we need to fix it anywhere else?)

@mfojtik possibly yes, I'll look at other repos and open PRs as needed.

@mfojtik not a problem in mysql, and apparently not used anywhere else.


Receive pipeline

Not a lot of important stuff in this PR. Main change was to properly hook up the receivers in Props to the Context. Also removed the duplicate Receive delegate

Looks good!


Please add support for BLE Bonding

BLE is fairly common in home automation. Devices like locks needs to be bonded to the controller. Because of the limited distance of BLE, I would like multiple ESP32s around the house, and set them up as BLE clients.

Right now, I'm swamped with work that pays the bills.  Also, https://github.com/MrSurly/micropython-esp32/issues/8 is higher priority.


Replace the Tag class by a POJO to make the RenderableTreeNode serializable

What happened?
I understand my use case is pretty specific and there are workarounds but that would be lovely it this could be fixed.
I'm trying to use Markdoc in an SSR app where the Markdoc.parse and Markdoc.transform are done on the server. The renderable node is then sent to the client and rendered there. With Remix, everything's fine since they use JSON.stringify and JSON.parse under the hood.
However if you try to use Markdoc with sveltekit, you won't be able to do that as they use devalue to get the data around. It can be argued that sveltekit should add an option to either add custom reducers for devalue or an option to opt out but at the same time, looking at the implementation of the Tag class, it seems to be a simple wrapper around some values and could easily be replaced by a POJO. I'd be up to create a PR for that.
What to you guys thing?
To reproduce
Just forked the devalue svelte repl and included some Markdoc example:
https://svelte.dev/repl/f9f7c58c7a9d440f84f11ef697ee8bdb?version=3.49.0
Version
0.4.0
Additional context
No response

Hey @CapitaineToinon,
Using the class is actually completely optional. You can write your transform functions in your project so that they return a POJO with the shape {$$mdtype: 'Tag', name: 'foo', attributes: {}, children: []} and it should work exactly the same way. The Tag.isTag function checks the $$mdtype property rather than looking to see if a value is a tag instance, so it will still work as expected.
I'm not really familiar with svelte or devalue, but I would assume you would be able to manually serialize the renderable tree to JSON somewhere before it hits the framework. You can just wrap JSON.stringify around the output of Markdoc.transform and then call JSON.parse on it before rendering.
At Stripe, we also serialize our renderable tree to JSON on the server side and render it on the client — Markdoc was actually designed specifically to support that use case. I'm not sure I understand why that wouldn't work with Svelte, maybe there's something I'm missing about how it does SSR.
In any event, I don't think we can accept the PR as written because eliminating the Tag class would be a significant non-backwards-compatible change that would require everyone to rewrite their custom tag definitions.

Oh I wasn't aware I could do that. A bit of a shame you have to add a transform for all my node and tags though but for this use case it's just what I needed, thanks. I understand about the PR, I mostly did it to open the discussion and get familiar with the code base. I have to say, implementing a renderer for svelte and doing the PR made me realize how simple and straight forward markdoc is. Love using it.

Glad it's working out for you! I'm going to close the PR and move this issue to a discussion thread so that it's discoverable for anybody else who may run into this.


Can not install when casbin >= 3.0

Hi
my compose.json already has "casbin/casbin": "^3.3" , so I can't install yii-permission
$ composer require casbin/yii-permission
Using version ^1.0 for casbin/yii-permission
./composer.json has been updated
Running composer update casbin/yii-permission
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - casbin/yii-permission[v1.0.0, ..., v1.0.1] require casbin/casbin ^2.2 -> found casbin/casbin[v2.2.0, v2.3.0, v2.4.0] but it conflicts with your root composer.json require (^3.3).
    - Root composer.json requires casbin/yii-permission ^1.0 -> satisfiable by casbin/yii-permission[v1.0.0, v1.0.1].

Use the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.

@basakest