Spacer SWE and dev-ops.

This ticket is to do SWE and dev-ops for spacer. This includes:

[ ] Create a Dockerfile for better container building process. Support both caffe and pytorch inside the container.
[ ] Test new container to ensure backwards compatibility.
[ ] Setup Travis CI on GitHub.
[ ] Isolate Caffe such that the rest of repo functions even when Caffe is not available.
[ ] Make and publish PIP package. This would only suppor the PyTorch. Make sure the PIP package supports both python 2.7 and 3.7.
[ ] Better Modularization of codebase. Abstract away the specifics of the classifiers and feature extractors to allow implementation of new versions.
[ ] Document messages and "API".
[ ] Implement a simple PyTorch version, e.g. EfficientNet B0 trained on ImageNet. This would be mostly for testing purposes.

@StephenChan @qiminchen : FYI: the more I looked at the spacer codebase, the more I realized I need to do some dev-ops and SWE before we add the new feature extractors. This ticket lists the main todos related to this. My goal is that we continue supporting the current feature extractor.

Okay. If you feel that you're short on time to get all these tasks done, let us know if there's something we can help with.
By SWE you mean software engineering, right?

Hey @StephenChan .  Yeah, Software engineering. I think I got this one. I have done enough thinking on this that it should be relatively smooth sailing.
The one thing that would help is if I don't have to provide support for both python 2.7 and 3.{6, 7} in the pip package. I haven't looked at it yet, so maybe it is fine, but I do anticipate a bit of headache to get that going. So if you feel ready to move CoralNet to python 3.6 or 3.7, and that you can do it soon, then that would help. If not - and it turns out to be tricky to support both - I'll start with 2.7 only.

The one thing that would help is if I don't have to provide support for both python 2.7 and 3.{6, 7} in the pip package. I haven't looked at it yet, so maybe it is fine, but I do anticipate a bit of headache to get that going. So if you feel ready to move CoralNet to python 3.6 or 3.7, and that you can do it soon, then that would help. If not - and it turns out to be tricky to support both - I'll start with 2.7 only.

Well, the thing is, I had the opposite sequence of actions in mind: I figured it'd be a little safer to move to Python 3 once everything is easily testable, including the vision backend. That would mean an issue sequence of #284 -> #264 -> #59. But I guess you're saying #59 -> #284 -> #264 would make your work on this issue easier.
By 'safer', I mean there's a lower chance that some vision backend functionality will break without us noticing when I make code changes for Py3 compatibility. I think I know enough about the Python 2 and 3 differences to make relatively safe changes, but it's easy to miss something.
Still, you know the backend better than I do, so I can leave that risk-tradeoff decision to you. Are you OK with me going ahead with #59 before #264?

Oh, right. I do remember that. And I agree that #284 -> #264 -> #59 would have been the most robust way. But I wonder if we can do it concurrently? Something like this.

You cut a "Python 3" feature branch of CoralNet master and make Python 3 migration PRs into that. Test the backend part best you can; it's ok if parts are untested.
I simultaneously work on #284 on Spacer.
Once both are done, I do #264 and merge that onto the "Python 3" feature branch. This includes adding spacer to requirements and adding tests.

Once step 3 is complete we should have a well-tested system that we can merge to ,aster. The only issue is that we are doing changes without having full test cover. Which I agree is not best practice. But I think it's OK in this case.

Right, you can work on Spacer concurrently with my Py3 stuff. That branch plan makes sense; let's do that then.

Awesome.

Nice plan!
On Tue, Feb 4, 2020 at 10:41 AM Oscar Beijbom<EMAIL_ADDRESS>wrote:

Awesome.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=ABKA5ALT6XPWYF3SDBKA6TTRBGZD5A5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKYXH2I#issuecomment-582054889,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKA5AKEDUGYIZNOJ5HC4JLRBGZD5ANCNFSM4KPAKHHQ
.

For visibility, working on this in https://github.com/beijbom/ecs_spacer/pull/5.
It is more than a PR, really. It is basically a new repo. I'll ping everyone when its time to review.

Ok guys, the new spacer-repo is ready for review. As you will see, there are pretty massive changes. The algorithms are the same, but the whole repo has been reworked as defined in this ticket. Since the changes were so large I didn't put this in a PR, but simply merged it all to master. So please visit https://github.com/beijbom/pyspacer and try installing, testing, and playing around with it.

The repo is made public and renamed to pyspacer.
CI enabled through travis.
Test cover is at 100%
Release through pip install spacer at https://pypi.org/project/pyspacer/0.0.2/.
Support for both caffe and pytorch through Docker with a clean Dockerfile build-file.
Caffe imports isolated such that the rest of the repo can run without caffe (obviously without support for the caffe-based tasks).
Data-classes defined for all messages and data types.
Support for python 3.5 and 3.6 (more details below).
Storage backends provided for S3, filesystem and memory.
Scripts  for re-training classifiers on current extracted CoralNet features. These act as regression tests for the train_classifier task. https://github.com/beijbom/pyspacer/blob/master/scripts/classifier_training_regression.py (CC @qiminchen: I checked a few sources, and they all look fine, but please do try it out.)

What happens next? A few things:

Please review the code carefully, and test all three build concepts (docker, pip install, and git clone) on your local machines. Please let me know if you have thoughts and suggestions. There is a ton of new code so I'm sure there is plenty of room for improvements.
We have a bit of a problem with scikit-learn. We have been using 0.17.1 to train the current classifiers on CoralNet. This is OK in itself, but python3.7 and 3.8 needs a higher version of scikit-learn, so we'd need to go to 0.22.x if we want to use those python versions. But here is the problem: scikit-learn doesn't provide backwards compatibility for their serialized (pickled) models (confer: https://github.com/scikit-learn/scikit-learn/issues/10319). Since we want to make spacer a dependency for CoralNet we have three options (1) Stick with scikit-learn 0.17.1 and use python3.5 or 3.6 for CoralNet, (2) Upgrade to scikit-learn 0.22.x and use the newest python, but then we have to throw away all old robots and recompute them. (3) Use option 1 for now and then figure out a way to migrate the model files in the future.
Implement the new pytorch feature extractor.

Ouch. Bitten by pickle.
FWIW: We weren't allowed to use pickle at Dropbox. Mostly because of the
security issues of an untrusted pickle file, but also because of these
compatibility issues.
one possibility is if there are alternative serialization methods for the
scikitlearn that are compatible across versions. Read pickle in w/ 0.17.1,
dump in compatible format, ...
D
On Fri, Feb 28, 2020 at 11:04 PM Oscar Beijbom<EMAIL_ADDRESS>wrote:

Ok guys, the new spacer-repo is ready for review. As you will see, there
are pretty massive changes. The algorithms are the same, but the whole repo
has been reworked as defined in this ticket. Since the changes were so
large I didn't put this in a PR, but simply merged it all to master. So
please visit https://github.com/beijbom/pyspacer and try installing,
testing, and playing around with it.

The repo is made public and renamed to pyspacer.
CI enabled through travis.
Test cover is at 100%
Release through pip install spacer at
https://pypi.org/project/pyspacer/0.0.2/.
Support for both caffe and pytorch through Docker with a clean
Dockerfile build-file.
Caffe imports isolated such that the rest of the repo can run
without caffe (obviously without support for the caffe-based tasks).
Data-classes defined for all messages and data types.
Support for python 3.5 and 3.6 (more details below).
Storage backends provided for S3, filesystem and memory.
Scripts for re-training classifiers on current extracted CoralNet
features. These act as regression tests for the train_classifier task.
https://github.com/beijbom/pyspacer/blob/master/scripts/classifier_training_regression.py
(CC @qiminchen https://github.com/qiminchen: I checked a few
sources, and they all look fine, but please do try it out.)

What happens next? A few things:

Please review the code carefully, and test all three build concepts
(docker, pip install, and git clone) on your local machines. Please let me
know if you have thoughts and suggestions. There is a ton of new code so
I'm sure there is plenty of room for improvements.
We have a bit of a problem with scikit-learn. We have been using
0.17.1 to train the current classifiers on CoralNet. This is OK in
itself, but python3.7 and 3.8 needs a higher version of scikit-learn, so
we'd need to go to 0.22.x if we want to use those python versions. But
here is the problem: scikit-learn doesn't provide backwards compatibility
for their serialized (pickled) models (confer:
scikit-learn/scikit-learn#10319
https://github.com/scikit-learn/scikit-learn/issues/10319). Since we
want to make spacer a dependency for CoralNet we have three options (1)
Stick with scikit-learn 0.17.1 and use python3.5 or 3.6 for CoralNet, (2)
Upgrade to scikit-learn 0.22.x and use the newest python, but then we
have to throw away all old robots and recompute them. (3) Use option
1 for now and then figure out a way to migrate the model files in the
future.
Implement the new pytorch feature extractor.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=ABKA5ALSOPSY4U5DFR2CRQ3RFICHZA5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENLR3PQ#issuecomment-592911806,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKA5AN7Q3JLDFGFVQDQNELRFICHZANCNFSM4KPAKHHQ
.

Great work, awesome to see all the progress! I'll see how much of the code I can review/test this weekend, but most likely I'll need to finish doing so next week, since I've been putting off primary election voting...
Regarding the scikit-learn problem:

What about a stance somewhere between (2) and (3)?: Upgrade to scikit-learn 0.22.x, and continue storing the old robots, but hide them for the time being. Thus, we force sources to train a new robot which can be used in 0.22.x, but we don't throw away the old robots yet. If we later figure out a way to migrate the pre-0.22 robots to the new format, we can migrate them and then un-hide them for historical interest.

Would the situation look any better if I can finish the Python 3 upgrade before we roll out the new spacer system? I was already hoping to focus on Python 3 next, since I can't think of other API specification cleanups to make at the moment, and the other API stuff might not be immediately urgent. Realistically though, I'll need at least 3 solid weeks to complete the Py3 upgrade (including getting the other 3rd party packages up to speed as needed).

FWIW: Logistic regression models are pretty simple with one set of weights
(coefficients) and offset (intercept), and savable as .json without
really needing banfide pickle serialization.
It would be a pain, but you could convert the old models fro pick to json
in old version of scikitlearn and then use json for saving LR models rather
than pickle in the future.
Not sure how much bigger/slower json reading would be or if it matters.
here's a relevant blog post on saving LR as json.
https://medium.com/datadriveninvestor/machine-learning-how-to-save-and-load-scikit-learn-models-d7b99bc32c27
Feel free to ignore suggestion ....
On Sat, Feb 29, 2020 at 12:08 AM StephenChan<EMAIL_ADDRESS>wrote:

Great work, awesome to see all the progress! I'll see how much of the code
I can review/test this weekend, but most likely I'll need to finish doing
so next week, since I've been putting off primary election voting...
Regarding the scikit-learn problem:

What about a stance somewhere between (2) and (3)?: Upgrade to
scikit-learn 0.22.x, and continue storing the old robots, but hide them for
the time being. Thus, we force sources to train a new robot which can be
used in 0.22.x, but we don't throw away the old robots yet. If we later
figure out a way to migrate the pre-0.22 robots to the new format, we can
migrate them and then un-hide them for historical interest.
Would the situation look any better if I can finish the Python 3
upgrade before we roll out the new spacer system? I was already hoping to
focus on Python 3 next, since I can't think of other API specification
cleanups to make at the moment, and the other API stuff might not be
immediately urgent. Realistically though, I'll need at least 3 solid weeks
to complete the Py3 upgrade (including getting the other 3rd party packages
up to speed as needed).
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=ABKA5AMA2C4BHANRQJZPIRLRFDA6XA5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENLTMDY#issuecomment-592918031,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKA5ALDNF2CC2XAAGUTQILRFDA6XANCNFSM4KPAKHHQ
.

Great! I will try as much as I can to review and test the new repo this
weekend.
In terms of re-training classifiers on current extracted CoralNet features,
I will definitely try this out to verify the current baseline. And as far
as the newly trained CoralNet evaluation goes, we've seen the accuracy
improvement vary from sources, the comparison and confusion matrices are
available in the research plan
https://docs.google.com/document/d/1mL2n6KFi2CAcTpMI3OXBnHFIkkfVWBDEtDmGGLRU4hs/edit?usp=sharing,
these results will be kept updating as soon as I re-train the LR classifier
using the same train/test split (already have some sources re-trained).
The attachment is a sample of the original image where each annotation is
marked with its ground truth label and label predicted by the classifier
(formatted as x: (gt: xxx, est: xxx), correct prediction in yellow and
wrong in white, might change the color later, the classification accuracy
of this particular image is shown in the upper left corner). This might be
a good way to make the coral classification more visible and interpretable
and help us find out whether those misclassified labels are reasonable or
not. (And probably can be added to the backend as an API in the future,
feel free to ignore this)
cheers
i2227.png
https://drive.google.com/file/d/1_L0rGA8RUGsBarDh49y2fT8fUu2p-2NM/view?usp=drive_web
On Sat, Feb 29, 2020 at 12:15 AM kriegman<EMAIL_ADDRESS>wrote:

FWIW: Logistic regression models are pretty simple with one set of weights
(coefficients) and offset (intercept), and savable as .json without
really needing banfide pickle serialization.
It would be a pain, but you could convert the old models fro pick to json
in old version of scikitlearn and then use json for saving LR models rather
than pickle in the future.
Not sure how much bigger/slower json reading would be or if it matters.
here's a relevant blog post on saving LR as json.
https://medium.com/datadriveninvestor/machine-learning-how-to-save-and-load-scikit-learn-models-d7b99bc32c27
Feel free to ignore suggestion ....
On Sat, Feb 29, 2020 at 12:08 AM StephenChan<EMAIL_ADDRESS>wrote:

Great work, awesome to see all the progress! I'll see how much of the
code
I can review/test this weekend, but most likely I'll need to finish doing
so next week, since I've been putting off primary election voting...
Regarding the scikit-learn problem:

What about a stance somewhere between (2) and (3)?: Upgrade to
scikit-learn 0.22.x, and continue storing the old robots, but hide them
for
the time being. Thus, we force sources to train a new robot which can be
used in 0.22.x, but we don't throw away the old robots yet. If we later
figure out a way to migrate the pre-0.22 robots to the new format, we can
migrate them and then un-hide them for historical interest.
Would the situation look any better if I can finish the Python 3
upgrade before we roll out the new spacer system? I was already hoping to
focus on Python 3 next, since I can't think of other API specification
cleanups to make at the moment, and the other API stuff might not be
immediately urgent. Realistically though, I'll need at least 3 solid
weeks
to complete the Py3 upgrade (including getting the other 3rd party
packages
up to speed as needed).
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<
https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=ABKA5AMA2C4BHANRQJZPIRLRFDA6XA5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENLTMDY#issuecomment-592918031
,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/ABKA5ALDNF2CC2XAAGUTQILRFDA6XANCNFSM4KPAKHHQ
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=AK54LCWLH77WKVUGEM3RTKLRFDBZRA5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENLTQWI#issuecomment-592918617,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AK54LCXU5BONGOD5HBKW243RFDBZRANCNFSM4KPAKHHQ
.

@StephenChan : regarding rolling this out. We can't roll this out until CoralNet is on python3.x, so let's follow the work-order we agreed on above. Once you create a python3.x.base-branch I can start pushing the changes to the vision backend to that, but I'll let you get it to a state where it at least compiles and runs first.
What I can do is deploy this on a new elastic container service & queue such that we can do shakeouts on our dev systems (again, after the 3.x branch is up). I'll do that after we have ironed out the first fews round of bugs, and also agreed on what to do about the scikit-learn. Speaking of which: what python version had you envisioned we go to? The last LRS Ubuntu release (18.04.4) comes with Python 3.6 as standard. So it doesn't seem totally crazy to stay on python 3.6, in which case we can punt the scikit-learn problem down the road a year or two.
@kriegman : I think I understand you, but just to be clear. You are suggesting we extract the learned parameters from the skikit-learn model, store them as JSON and then write the inference step as native python & numpy (independent of scikit-learn)? I like that suggestion a lot. The only complication is that I'm not just doing logistic regression, but I'm training a calibrated logreg. Specifically this one https://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html. Calibration makes inference more complicated, as it involves some sort of weighted sigmoid or logistic function across the classes. I'll take a look at their code to see if I can figure out what they do exactly during inference, and see if I can replicate it. I'll create a new ticket to track that.
@qiminchen . Thanks. Do take your time. We can spend 2-3 weeks on shakeouts and reviews. I'd be neat if you could pip install and call spacer inside your code-base so that we know for sure we run the same LR training code.

I'm thinking of something simpler.
I haven't looked at your code, but I presume that the issue is simply the
pickling, and the same weights could be used for either version of
sckitlearn. So, my suggestion is to replace pickle with you own serializing
that saves to json. A 2 minute look at the documentation shows
get_params, set_params.
I know, easier said than done...
On Sun, Mar 1, 2020 at 7:20 PM Oscar Beijbom<EMAIL_ADDRESS>wrote:

@StephenChan https://github.com/StephenChan : regarding rolling this
out. We can't roll this out until CoralNet is on python3.x, so let's follow
the work-order we agreed on above. Once you create a python3.x.base-branch
I can start pushing the changes to the vision backend to that, but I'll let
you get it to a state where it at least compiles and runs first.
What I can do is deploy this on a new elastic container service & queue
such that we can do shakeouts on our dev systems (again, after the 3.x
branch is up). I'll do that after we have ironed out the first fews round
of bugs, and also agreed on what to do about the scikit-learn. Speaking of
which: what python version had you envisioned we go to? The last LRS Ubuntu
release (18.04.4) comes with Python 3.6 as standard. So it doesn't seem
totally crazy to stay on python 3.6, in which case we can punt the
scikit-learn problem down the road a year or two.
@kriegman https://github.com/kriegman : I think I understand you, but
just to be clear. You are suggesting we extract the learned parameters from
the skikit-learn model, store them as JSON and then write the inference
step as native python & numpy (independent of scikit-learn)? I like that
suggestion a lot. The only complication is that I'm not just doing logistic
regression, but I'm training a calibrated logreg. Specifically this one
https://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html.
Calibration makes inference more complicated, as it involves some sort of
weighted sigmoid or logistic function across the classes. I'll take a look
at their code to see if I can figure out what they do exactly during
inference, and see if I can replicate it. I'll create a new ticket to track
that.
@qiminchen https://github.com/qiminchen . Thanks. Do take your time. We
can spend 2-3 weeks on shakeouts and reviews. I'd be neat if you could pip
install and call spacer inside your code-base so that we know for sure we
run the same LR training code.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=ABKA5ANCDSZBZZB32KQH5QLRFMQXJA5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENNYK6Y#issuecomment-593200507,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKA5AKYYS7WSQWLP76WFHLRFMQXJANCNFSM4KPAKHHQ
.

Ok, yeah, I was considering that option as well. The problem is that the
object definition itself that has changed, not the pickle protocol, so it
may not even have the same parameter set. Or indeed implement the same
algorithm. :/
On Sun, Mar 1, 2020 at 7:31 PM kriegman<EMAIL_ADDRESS>wrote:

I'm thinking of something simpler.
I haven't looked at your code, but I presume that the issue is simply the
pickling, and the same weights could be used for either version of
sckitlearn. So, my suggestion is to replace pickle with you own serializing
that saves to json. A 2 minute look at the documentation shows
get_params, set_params.
I know, easier said than done...
On Sun, Mar 1, 2020 at 7:20 PM Oscar Beijbom<EMAIL_ADDRESS>wrote:

@StephenChan https://github.com/StephenChan : regarding rolling this
out. We can't roll this out until CoralNet is on python3.x, so let's
follow
the work-order we agreed on above. Once you create a
python3.x.base-branch
I can start pushing the changes to the vision backend to that, but I'll
let
you get it to a state where it at least compiles and runs first.
What I can do is deploy this on a new elastic container service & queue
such that we can do shakeouts on our dev systems (again, after the 3.x
branch is up). I'll do that after we have ironed out the first fews round
of bugs, and also agreed on what to do about the scikit-learn. Speaking
of
which: what python version had you envisioned we go to? The last LRS
Ubuntu
release (18.04.4) comes with Python 3.6 as standard. So it doesn't seem
totally crazy to stay on python 3.6, in which case we can punt the
scikit-learn problem down the road a year or two.
@kriegman https://github.com/kriegman : I think I understand you, but
just to be clear. You are suggesting we extract the learned parameters
from
the skikit-learn model, store them as JSON and then write the inference
step as native python & numpy (independent of scikit-learn)? I like that
suggestion a lot. The only complication is that I'm not just doing
logistic
regression, but I'm training a calibrated logreg. Specifically this one

https://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html
.

Calibration makes inference more complicated, as it involves some sort of
weighted sigmoid or logistic function across the classes. I'll take a
look
at their code to see if I can figure out what they do exactly during
inference, and see if I can replicate it. I'll create a new ticket to
track
that.
@qiminchen https://github.com/qiminchen . Thanks. Do take your time.
We
can spend 2-3 weeks on shakeouts and reviews. I'd be neat if you could
pip
install and call spacer inside your code-base so that we know for sure we
run the same LR training code.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<
https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=ABKA5ANCDSZBZZB32KQH5QLRFMQXJA5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENNYK6Y#issuecomment-593200507
,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/ABKA5AKYYS7WSQWLP76WFHLRFMQXJANCNFSM4KPAKHHQ
.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=AAITTF5PBIBIBY5Z6A7PJNTRFMSAJA5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENNY3XY#issuecomment-593202655,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAITTF3YNUNKYLMMBPOZVQLRFMSAJANCNFSM4KPAKHHQ
.

Ouch.
On Sun, Mar 1, 2020 at 8:40 PM Oscar Beijbom<EMAIL_ADDRESS>wrote:

Ok, yeah, I was considering that option as well. The problem is that the
object definition itself that has changed, not the pickle protocol, so it
may not even have the same parameter set. Or indeed implement the same
algorithm. :/
On Sun, Mar 1, 2020 at 7:31 PM kriegman<EMAIL_ADDRESS>wrote:

I'm thinking of something simpler.
I haven't looked at your code, but I presume that the issue is simply the
pickling, and the same weights could be used for either version of
sckitlearn. So, my suggestion is to replace pickle with you own
serializing
that saves to json. A 2 minute look at the documentation shows
get_params, set_params.
I know, easier said than done...
On Sun, Mar 1, 2020 at 7:20 PM Oscar Beijbom<EMAIL_ADDRESS>wrote:

@StephenChan https://github.com/StephenChan : regarding rolling this
out. We can't roll this out until CoralNet is on python3.x, so let's
follow
the work-order we agreed on above. Once you create a
python3.x.base-branch
I can start pushing the changes to the vision backend to that, but I'll
let
you get it to a state where it at least compiles and runs first.
What I can do is deploy this on a new elastic container service &
queue
such that we can do shakeouts on our dev systems (again, after the 3.x
branch is up). I'll do that after we have ironed out the first fews
round
of bugs, and also agreed on what to do about the scikit-learn. Speaking
of
which: what python version had you envisioned we go to? The last LRS
Ubuntu
release (18.04.4) comes with Python 3.6 as standard. So it doesn't seem
totally crazy to stay on python 3.6, in which case we can punt the
scikit-learn problem down the road a year or two.
@kriegman https://github.com/kriegman : I think I understand you,
but
just to be clear. You are suggesting we extract the learned parameters
from
the skikit-learn model, store them as JSON and then write the inference
step as native python & numpy (independent of scikit-learn)? I like
that
suggestion a lot. The only complication is that I'm not just doing
logistic
regression, but I'm training a calibrated logreg. Specifically this
one

https://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html

.

Calibration makes inference more complicated, as it involves some sort
of
weighted sigmoid or logistic function across the classes. I'll take a
look
at their code to see if I can figure out what they do exactly during
inference, and see if I can replicate it. I'll create a new ticket to
track
that.
@qiminchen https://github.com/qiminchen . Thanks. Do take your time.
We
can spend 2-3 weeks on shakeouts and reviews. I'd be neat if you could
pip
install and call spacer inside your code-base so that we know for sure
we
run the same LR training code.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<

https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=ABKA5ANCDSZBZZB32KQH5QLRFMQXJA5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENNYK6Y#issuecomment-593200507

,
or unsubscribe
<

https://github.com/notifications/unsubscribe-auth/ABKA5AKYYS7WSQWLP76WFHLRFMQXJANCNFSM4KPAKHHQ

.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<
https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=AAITTF5PBIBIBY5Z6A7PJNTRFMSAJA5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENNY3XY#issuecomment-593202655
,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AAITTF3YNUNKYLMMBPOZVQLRFMSAJANCNFSM4KPAKHHQ
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=ABKA5AOJCIESLMRX66BWCLTRFM2FDA5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENN4AZY#issuecomment-593215591,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKA5AINOCI6NURMWXTXJQTRFM2FDANCNFSM4KPAKHHQ
.

regarding rolling this out. We can't roll this out until CoralNet is on python3.x, so let's follow the work-order we agreed on above. Once you create a python3.x.base-branch I can start pushing the changes to the vision backend to that, but I'll let you get it to a state where it at least compiles and runs first.

Ah, okay, I get it now. Guess I wasn't paying enough attention there.
I'll start by adding a few more unit tests, and we can merge those in separately first. Then once I'm really working on the Python 3.x compatibility, I'll work in that branch as agreed previously.

What I can do is deploy this on a new elastic container service & queue such that we can do shakeouts on our dev systems (again, after the 3.x branch is up). I'll do that after we have ironed out the first fews round of bugs, and also agreed on what to do about the scikit-learn.

That sounds like a good idea.

Speaking of which: what python version had you envisioned we go to? The last LRS Ubuntu release (18.04.4) comes with Python 3.6 as standard. So it doesn't seem totally crazy to stay on python 3.6, in which case we can punt the scikit-learn problem down the road a year or two.

Hmm... that's a good point. In terms of Python features, I think 3.6 will already feel pretty great in comparison to 2.7. So my only concern would be the Python 3.6 end of life date, which appears to be the start of 2022 (https://www.python.org/dev/peps/pep-0494/). Presumably, most third-party packages would support 3.6 until then as well. So yes, unless any packages jump the gun on dropping 3.6, I think we'd indeed have a year or two to work out the serialization issue. That plan would be fine with me.

I saw that https://github.com/beijbom/pyspacer/pull/5/ requires boto 2.49.0 and numpy 1.17.0. If coralnet requires spacer, then we'll need to sync coralnet's boto and numpy versions accordingly, meaning we should check that coralnet works fine with those versions. So that's another thing to upgrade on top of Py3.

Numpy 1.17.5 actually. Yeah. But I’m not so worried about that one, it is a
very recent release. The boto 2.49.0 release is the last one on boto2, so
should be a good  stable one. You could use boto3 for CoralNet if we want
since it’s a different package. I just stayed on boto2 since I was familiar
with the interface.
On Mon, Mar 2, 2020 at 16:54 StephenChan<EMAIL_ADDRESS>wrote:

I saw that beijbom/pyspacer#5 https://github.com/beijbom/pyspacer/pull/5
requires boto 2.49.0 and numpy 1.17.0. If coralnet requires spacer, then
we'll need to sync coralnet's boto and numpy versions accordingly, meaning
we should check that coralnet works fine with those versions. So that's
another thing to upgrade on top of Py3.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=AAITTFYIDPE2CHPT55XIYL3RFRIMJA5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENRUV5I#issuecomment-593709813,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAITTF763TTT54FFBDNUL73RFRIMJANCNFSM4KPAKHHQ
.

Yeah, I just realized https://github.com/beijbom/pyspacer/pull/5 didn't have the most up to date requirements. I'll refer to the master branch from now on.

@beijbom I'm wondering, did you get some particular error when trying to install scikit-learn 0.17.1 on Python 3.7/3.8, or did you just read the compatibility notes? I'm noticing that the Python versions for packages might be more lenient than I thought.
For example, I tried installing Django 1.9 on Python 3.7 and it seemed to work, despite the release notes stating that Django 1.9 would work with Python 2.7, 3.4 or 3.5.
I didn't try with scikit-learn itself yet, since I seem to have some issue installing it in general right now (seems Windows related).

Yeah. I got errors. Both in the Travis CI and on my dev system. I don’t
recall the exact error, but it was resolved when moving to 0.22.
On Mon, Mar 2, 2020 at 22:09 StephenChan<EMAIL_ADDRESS>wrote:

@beijbom https://github.com/beijbom I'm wondering, did you get some
particular error when trying to install scikit-learn 0.17.1 on Python
3.7/3.8, or did you just read the compatibility notes? I'm noticing that
the Python versions for packages might be more lenient than I thought.
For example, I tried installing Django 1.9 on Python 3.7 and it seemed to
work, despite the release notes stating that Django 1.9 would work with
Python 2.7, 3.4 or 3.5.
I didn't try with scikit-learn itself yet, since I seem to have some issue
installing it in general right now (seems Windows related).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=AAITTF7GDSI722QJXNLJNRLRFSNLFA5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENSHDTQ#issuecomment-593785294,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAITTFY3UKU4BQ4NBXAE4D3RFSNLFANCNFSM4KPAKHHQ
.

Okay, just figured I'd double check.
I also triple checked by getting further with Windows installations myself. Results:

Python 3.6 + scikit-learn 0.17.1: works
Python 3.7 + scikit-learn 0.17.1: errors such as sklearn\svm\libsvm.c(10699): error C2039: 'exc_type': is not a member of '_ts'
Python 3.7 + scikit-learn 0.18.2: errors like the above
Python 3.7 + scikit-learn 0.19.2: works
Python 3.7 + scikit-learn 0.20.4: works
Python 3.7 + scikit-learn 0.21.3: works, after doing pip install Cython first
Python 3.7 + scikit-learn 0.22.1: works, after doing pip install Cython first

I used the --no-binary :all: option to ensure all of those were source installations, not binary.
So, one more check:

Do you think going from sklearn 0.17.1 to 0.19.2 would have the same model portability problem?
If not, does Py 3.7 + sklearn 0.19.2 install for you?

On Tue, Mar 3, 2020 at 01:47 StephenChan<EMAIL_ADDRESS>wrote:

Okay, just figured I'd double check.

great.

I also triple checked by getting further with Windows installations
myself. Results:

Python 3.6 + scikit-learn 0.17.1: works
Python 3.7 + scikit-learn 0.17.1: errors such as sklearn\svm\libsvm.c(10699):
error C2039: 'exc_type': is not a member of '_ts'
Python 3.7 + scikit-learn 0.18.2: errors like the above
Python 3.7 + scikit-learn 0.19.2: works
Python 3.7 + scikit-learn 0.20.4: works
Python 3.7 + scikit-learn 0.21.3: works, after doing pip install
Cython first
Python 3.7 + scikit-learn 0.22.1: works, after doing pip install
Cython first

I used the --no-binary :all: option to ensure all of those were source
installations, not binary.
So, one more check:

Do you think going from sklearn 0.17.1 to 0.19.2 would have the same
model portability problem?

Did the tests run? If they did we should be good to go.

If not, does Py 3.7 + sklearn 0.19.2 install for you?

You can create a branch, change the .jenkins (just add a new line for 3.7)
and the requirements files to see if it works there. If it does we are good
to go.

Thanks for double checking all of this.
Btw., IMHO ,if to go beyond 3.6 we should probably to go 3.8.
—

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=AAITTF5RDSSEC6SH3B7ERFLRFTGZPA5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENSZLZI#issuecomment-593860069,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAITTF6WKWN2TAIO3BQLAVTRFTGZPANCNFSM4KPAKHHQ
.

On Tue, Mar 3, 2020 at 06:52 Oscar Beijbom<EMAIL_ADDRESS>wrote:

On Tue, Mar 3, 2020 at 01:47 StephenChan<EMAIL_ADDRESS>wrote:

Okay, just figured I'd double check.

great.

I also triple checked by getting further with Windows installations
myself. Results:

Python 3.6 + scikit-learn 0.17.1: works
Python 3.7 + scikit-learn 0.17.1: errors such as sklearn\svm\libsvm.c(10699):
error C2039: 'exc_type': is not a member of '_ts'
Python 3.7 + scikit-learn 0.18.2: errors like the above
Python 3.7 + scikit-learn 0.19.2: works
Python 3.7 + scikit-learn 0.20.4: works
Python 3.7 + scikit-learn 0.21.3: works, after doing pip install
Cython first
Python 3.7 + scikit-learn 0.22.1: works, after doing pip install
Cython first

I used the --no-binary :all: option to ensure all of those were source
installations, not binary.
So, one more check:

Do you think going from sklearn 0.17.1 to 0.19.2 would have the
same model portability problem?

Did the tests run? If they did we should be good to go.

If not, does Py 3.7 + sklearn 0.19.2 install for you?

You can create a branch, change the .jenkins (just add a new line for
3.7) and the requirements files to see if it works there. If it does we are
good to go.

*and push to GitHub obviously.

Thanks for double checking all of this.
Btw., IMHO ,if to go beyond 3.6 we should probably to go 3.8.
—

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/284?email_source=notifications&email_token=AAITTF5RDSSEC6SH3B7ERFLRFTGZPA5CNFSM4KPAKHH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENSZLZI#issuecomment-593860069,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAITTF6WKWN2TAIO3BQLAVTRFTGZPANCNFSM4KPAKHHQ
.

Ah, sorry, when I said 'works' I just meant that I was able to install the packages. I had not tried running the spacer tests.
Unfortunately, after trying the spacer tests (changing requirements/setup files as needed), it looks like scikit-learn 0.19.2 won't load legacy models:
======================================================================
ERROR: test_load_legacy_model (test_storage.TestLocalStorage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\<...>\virtualenv_pyspacer_py37_sklearn0192\Lib\site-packages\spac
er\tests\test_storage.py", line 142, in test_load_legacy_model
    config.LOCAL_FIXTURE_DIR, 'legacy.model'))
  File "D:\<...>\virtualenv_pyspacer_py37_sklearn0192\lib\site-packages\spac
er\storage.py", line 106, in load_classifier
    return pickle.load(f, encoding='latin1')
ModuleNotFoundError: No module named 'copy_reg\r'

I've also tried several combinations including Python 3.8 now. The overall conclusion is that we do have to stay on Python 3.6 to load legacy models. (Again, I'm fine with this, but wanted to make sure.) scikit-learn 0.18.2 might be okay, if we're lucky, but that version's still Py3.6 only so it wouldn't matter much. Here are all of my results:
Python 3.6

scikit-learn 0.17.1: Installs. Spacer tests on legacy models pass.

Overall test results: no fails, no errors.

scikit-learn 0.18.2: Installs. Spacer tests on legacy models pass, but get warnings.

Overall test results: no fails, 1 error. The error is on TestDeploy.test_deploy_simple().
The warnings are of the form UserWarning: Trying to unpickle estimator SGDClassifier from version pre-0.18 when using version 0.18.2. This might lead to breaking code or invalid results. Use at your own risk.

Python 3.7

scikit-learn 0.17.1, 0.18.2: Can't install.

Gets errors of the form sklearn\svm\libsvm.c(10699): error C2039: 'exc_type': is not a member of '_ts'

scikit-learn 0.19.2, 0.20.4, 0.21.3, 0.22.1: Installs. Spacer tests on legacy models get an error.

The legacy model error is of the form ModuleNotFoundError: No module named 'copy_reg\r'. Overall test results: 5 fails, 2 errors.

Python 3.8

scikit-learn 0.17.1, 0.18.2, 0.19.2, 0.21.3, 0.22.1: Same results as Python 3.7.
scikit-learn 0.20.4: Installs. But only 33 Spacer tests run (instead of 93) and there are 7 errors of the form TypeError: an integer is required (got type bytes).

Not sure what's up with the Py3.8 + 0.20.4 case, but I guess it's not likely to matter.
On the bright side, I've confirmed that pyspacer can be installed on Windows. I only tried the pip install method so far. I should note that I had to copy the tests/fixtures subdirectory manually into my installation, because pip-install didn't include that directory automatically. I think pip install excludes non-.py files by default, in which case we might fix that by using distutils' data_files option: https://stackoverflow.com/questions/12191339/

Thanks for double checking this @StephenChan. For now, let's assume we stay on python 3.6 and those other libraries I found. If I can figure out a better way to do the serialization we can revisit this again.
Re. missing fixture files in the pip package. I actually think that is expected behavior. Indeed, I was meaning to exclude the test code itself in the package. One does usually not develop and run tests on a pip-installed package, right?  One would just assume it is packaged from a stable branch/build. So for running tests I'd say clone and use a virtualenv. or the docker build. Correct me if I'm wrong here, but at least I have never thought of trying to run tests on a pip-installed package.
Have you tried using Docker yet? If you haven't-- give it a go. The tooling is pretty amazing actually. The first time you build the image it will take a while, but subsequent builds are super fast.

Re. missing fixture files in the pip package. I actually think that is expected behavior. Indeed, I was meaning to exclude the test code itself in the package. One does usually not develop and run tests on a pip-installed package, right? One would just assume it is packaged from a stable branch/build. So for running tests I'd say clone and use a virtualenv. or the docker build. Correct me if I'm wrong here, but at least I have never thought of trying to run tests on a pip-installed package.

Oh yeah, you're right. I forgot about that.

Have you tried using Docker yet? If you haven't-- give it a go. The tooling is pretty amazing actually. The first time you build the image it will take a while, but subsequent builds are super fast.

Haven't tried it yet, that'll be the next step. Sounds pretty promising.

I tried Docker. Apparently Docker is a pain to install on Windows 10 Home edition, so I tried it out in Ubuntu server. Got all the tests passing. Notes:

Installed docker.io from apt on Ubuntu 18.04.
Cloned the pyspacer repo 3-4 hours ago, so it's the current master branch.
At first I didn't realize that SPACER_LOCAL_MODEL_PATH in secrets.json had to be /workspace/models if using Docker. I initially used the path of the folder I created in Ubuntu, which gave me some FileNotFoundErrors when running tests. @beijbom Maybe the README's installation instructions could clarify this part better?
The initial Docker build took 10 minutes, but the next one (after editing secrets.json) took 3 seconds.

Other than the model path thing, it was quite smooth to set up, so nice job on that!

Thanks for trying it @StephenChan , I'm glad it worked!
Re that path, I have already fixed in the last open PR by setting the env. variable inside the docker. That has priority over the content of secrets.json. (Line 80 of Dockerfile in https://github.com/beijbom/pyspacer/pull/10/files)

Oh, good to know you already fixed that! I'll keep an eye on your PR updates and at least glance over them. But let me know if there's any part you particularly want reviewed.

This can be closed, right?

Yeah, thanks.


Add a periodic timer flavor

There has been already a couple of requests for an easy way to create a pausable periodic timer in #19.
People seems to expect a PausableTimer.periodic() constructor, even when it's trivial to build on top of PausableTimer bacause it trivial to reset it and start it again.
Example:
timer = PausableTimer(
    Duration(seconds: 1),
    () {
      // This is really what your callback do.
      print('A period has passed');

      // we know the callback won't be called before the constructor ends, so
      // it is safe to use !
      timer
        ..reset()
        ..start();
    },
  )..start();

Once you want your timer to go away, you can just .pause() it and that's it. There is the caveat that if the actual work your callback does takes too long, the period will stretch. If you reset the timer before doing the actual work, it can be called without any further wait if it took longer than the period. Finally the time it takes for the work to be done can be calculated and the duration of the timer adjusted accordingly, which gets much less trivial, so maybe it is a worthy addition anyway :thinking:

@llucax
I am currently implementing this feature, and initially I was trying to address the stretching issue. However, I did encounter this in the documentation for Timer.periodic:

The exact timing depends on the underlying timer implementation. No more than n callbacks will be made in duration * n time, but the time between two consecutive callbacks can be shorter and longer than duration.

Indeed, I tested with Timer.periodic and the stretching issue also occurs with it.
I have decided not to solve the problem. Instead, I am using Zone.createPeriodicTimer internally so that the behavior of the PausableTimer will be the same as that of the Timer.
This means that in platforms where time does not stretch, it won't stretch; in platforms where time stretches, it will stretch.
This way, we have consistent behavior between Timer and PausableTimer.


Bug report

Describe the bug
can't install apk file when sending any apk file from the PC to the phone and clicking on it through the app (on android, version 13)
To reproduce

Send apk file from the PC to the phone with localSend
Click on the received file
Choose to install it
See that nothing happens

Expected behavior
When I click the apk file and choose to install it, it should let me install it
Screenshots
No response
Desktop (please complete the following information)
No response
Smartphone (please complete the following information)
Device: Poco X3
Android version: 13
OS: arrow OS 12
LocalSend version: 1.7.0 (26)
Additional context
No response

This should be already fixed in the coming v1.8.0 release. Hopefully, this will be released in the coming weekend.


why stop update?what happened?

Do you want to request a feature or report a bug?

What is the current behavior?
If the current behavior is a bug, please provide the steps to reproduce.

What is the expected behavior?
If this is a feature request, what is motivation or use case for changing the behavior?
Please mention other relevant information such as the browser version, Operating System and Device Name

Recently this project has became more open to collaborators, as me :) So maybe from now on we hope to see more maintenance, and more features to come.


Error initializing theme and mode

After registring it as a plugin in main.ts, I can't figure out how to use this component.
When i try to change mode/theme it search for it in wrong folder (http://localhost:8080/src/ace/theme-monokai.js,  http://localhost:8080/src/ace/mode-javascript.js). How can i solve that?
Furthermore, the plugin doesnt render in safari browser. How can i fix that? Thanx in advance

Hi @deliberaligiacomo,
Could you post your jspm config ?

map: { 	...,     "abalmus/aurelia-ace-editor": "github:abalmus/aurelia-ace-editor@master",     "github:abalmus/aurelia-ace-editor@master": {       "ace"<EMAIL_ADDRESS>   }, 	... }

@deliberaligiacomo,
It looks good, I didn't try it with TS project, maybe this is the problem. Let me check it and get back to you.

Ok i'll wait, thanx

Hi @deliberaligiacomo,
I've just checked it with fresh TS skeleton app, it works pretty well even in safari.
please make sure you included styles into your project.
`

`

Can you give me a working piece of code? Because from the examples that i've seen I don't know where I'm wrong.

main.ts: aurelia.use.plugin('abalmus/aurelia-ace-editor');

ace-test.html:

<style>      .ace_editor {          display: block;          border: 1px solid lightgray;          margin: auto;          height: 300px;          width: 100%;      }  </style>
<ace config-theme="ace/theme/monokai" config-mode="ace/mode/javascript" class="ace_editor"> console.log("content"); </ace>

Furthermore when i start typing, the content doesn't change, and the console signlas me an error (the last red line).

Yes sure.

Download the skeleton
npm install
jspm install
jspm install github:abalmus/aurelia-ace-editor
insert following line in main.ts aurelia.use.plugin('abalmus/aurelia-ace-editor');

And here is how I inserting editor component:
 <style>
    .ace_editor {
        display: block;
        border: 1px solid lightgray;
        margin: auto;
        height: 300px;
        width: 100%;
    }
  </style>
    <ace config-theme="ace/theme/eclipse"
         config-mode="ace/mode/json">
        {
            "menu": {
                "id": "file",
                "value": "File",
                "popup": {
                    "menuitem": [
                        {"value": "New", "onclick": "CreateNewDoc()"},
                        {"value": "Open", "onclick": "OpenDoc()"},
                        {"value": "Close", "onclick": "CloseDoc()"}
                    ]
                }
            }
        }
    </ace>
</template>```

That's seems equal to mine. Can't figure out what the problem is

I compressed my app, please run npm install && jspm install -y
skeleton-typescript.zip

after that I'm running gulp watch

It always giving me the same errors of above. Even your skeleton project. Ideas?

Yes after clear cache, I'm getting this errors in the console. Loos like some changes in new version of System.JS. I'll take a look today, it is something with System normalize url.

Seems fixed, I had installed ace-builds with npm. Where did you find that jspm package?

github:ajaxorg/ace-builds

OK, seems good the plugin, but still can't render in safari browser. Am I missing any tricks?

In Safari 9.0.2 works well for me :(

Do you see nay errors in Safari ?

Ok fixed safari issue.
In my component i used your <style> before <template>. After moving that class inside global .less it's all fixed. Seems that safari ignores my <style> inside the component <html>. Thanks for the support, all closed.


Update uuid to version 3.0.0

Version 3.0.0 of uuid just got published.
Hi there,
This week a new version of the uuid module got released. The old module which was available by the name node-uuid got deprecated and will show error message upon every install of the module.
To get rid of those install errors I've created this automated pull request. I've run some checks against your code base to test whether you're using one of the deprecated apis, but this isn't the case.
Please test the changes against your code. I didn't run any tests and therefore can't guarantee that it isn't breaking. You can also just close this pr if you don't want to update your module.
In case there's already another pr open to upgrade uuid, I'm sorry for the effort I'm causing.

I was having problems installing, now I see this is why - it'd be great if this could be implemented! Good job @marcbachmann let's hope this gets in

Sorry this took solong -- thanks guys!


failed to execute 'setrequestheader' on 'xmlhttprequest'

Hi,
In installed the last version of Visual Studio Code (x64 1.16.1) and I have this error.

I try to install Extensions.
Can you help to fix this problem?
Thank you

#34098
Fixed in master, will come with the next release. 👍


How to visualize the results?

Hi,
I am running on Ubuntu 22.04 with the south-building dataset.
after this cmd completed
~/dev/glomap/south-building$ glomap mapper --database_path ./database.db --image_path ./images/ --output_path ./sparse
I see a folder sparse/0/ with cameras.bin  images.bin  points3D.bin
how do i visualize them

Hi, one simple option is to use colmap gui,  then choose import model to visualize things

colmap gui is the answer. It also compatible with glomap output.

This is now also documented in the README.

@ahojnnes  Can you please update the README to show the full shell command to run the COLMAP visualization
The reconstruction can be visualized using the COLMAP GUI, for example:
colmap gui --import_path ./output/south-building/sparse/0 --database_path data/south-building/database.db --image_path data/south-building/images/

Thanks for your great work, I noticed that only sparse point clouds can be generated via glomap, can I further generate dense point clouds via colmap? If so how should I complete this step? Looking forward to your reply, thanks a lot!

Hi, The output of GLOMAP is what used to be provided by the COLMAP mapper. So you should be able to proceed to dense mapping in a similar as what is done with COLMAP before


when remote.py is ran it just shows this

Hello @godpow
Try run test1.py or TrainPPO.py
after running remote.py and see if it works.

@BabakAkbari this is what happens when I try running it I have to install the pip modules manually for some reason repeatedly
apps@4109d8750504:~$ python3 TrainPPO.py
WARNING: CPU random generator seem to be failing, disabling hardware random number generation
WARNING: RDRND generated: 0xffffffff 0xffffffff 0xffffffff 0xffffffff
Traceback (most recent call last):
File "TrainPPO.py", line 11, in
from baselines.common.cmd_util import atari_arg_parser
File "/home/apps/.local/lib/python3.8/site-packages/baselines/common/cmd_util.py", line 7, in
from gym.wrappers import FlattenDictWrapper
ImportError: cannot import name 'FlattenDictWrapper' from 'gym.wrappers' (/home/apps/.local/lib/python3.8/site-packages/gym/wrappers/init.py)

I had to edit the docker file to get this far since the download was dead and other stuff FROM ubuntu
RUN apt-get update && apt-get clean && apt-get install -y
x11vnc
xvfb
fluxbox
wmctrl
wget
gnupg
gnupg2
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
&& echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
&& apt-get update && apt-get -y install google-chrome-stable
RUN apt-get update && apt-get install -y
python3
python3-setuptools
python3-pip
unzip
curl
RUN wget -O /tmp/chromedriver.zip http://chromedriver.storage.googleapis.com/curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE/chromedriver_linux64.zip
RUN unzip /tmp/chromedriver.zip chromedriver -d /usr/local/bin/
RUN pip3 install selenium
RUN cd /home
RUN wget https://store2.gofile.io/download/28350cc4-13cf-40c2-90c9-9c5e3e90afc8/tigervnc-1.10.1.x86_64.tar.gz
RUN ls -all
RUN tar xzf tigervnc-1.10.1.x86_64.tar.gz
RUN cd /tigervnc-1.10.1.x86_64 && chown -R root:root usr && ls && mv usr local && tar czf local.tgz local && tar xzf local.tgz -C /usr/ && update-icon-caches /usr/local/share/icons/*
RUN useradd apps
&& mkdir -p /home/apps
&& chown -v -R apps:apps /home/apps
RUN chmod 777 /home/apps
#RUN export DISPLAY=:0
#COPY bootstrap.sh /
#RUN chmod 777 /bootstrap.sh
COPY . /home/apps
RUN chmod 777 /home/apps
#CMD '/bootstrap.sh'
RUN apt-get install libopenmpi-dev -y
RUN apt-get update && apt-get install cmake libopenmpi-dev python3-dev zlib1g-dev -y
RUN apt-get install golang -y
RUN pip install mpi4py
RUN pip install gym
RUN pip install mujoco-py
RUN pip install opencv-python

Would it be possible for you to export a working copy of the image of the ai? @BabakAkbari

@godpow
Sure, I can share the built image. I have tested my image and there wasn't any problem.
It seems there is no problem with yours as well. You just need to do as follows after running remote.py in your docker container:

Open your VNC client and connect to <IP_ADDRESS>:5900 (which you have already done)
On your local machine (not the docker container) try to run test1.py. You may be asked to install  go-vncdriver which you can install by runningpip install go-vncdriver.

I should definitely update the readme it is so confusing. Let me know if there is any problem.

When I try to install go-vncdriver more errors apps@f3db13689e89:~$ dir
Dockerfile  README.md TrainPPO.py  google615d65718871c1f2.html  remote.py
LICENSE     RemoteEnv.py pycache  gym_slitherio                slither.io
apps@f3db13689e89:~$ cd gym_slitherio/
apps@f3db13689e89:~/gym_slitherio$ cd envs/
apps@f3db13689e89:~/gym_slitherio/envs$ dir
init.py  init.pyc  pycache  slitherio_env.py  slitherio_env.pyc  slitherio_extrahard_env.py  test1.py
apps@f3db13689e89:~/gym_slitherio/envs$ python3 test1.py
Traceback (most recent call last):
File "test1.py", line 1, in
from slitherio_env import SlitherioEnv
File "/home/apps/gym_slitherio/envs/slitherio_env.py", line 3, in
import go_vncdriver
ModuleNotFoundError: No module named 'go_vncdriver'
apps@f3db13689e89:~/gym_slitherio/envs$ pip install go-vncdriver
Collecting go-vncdriver
Downloading go_vncdriver-0.4.19.tar.gz (638 kB)
|████████████████████████████████| 638 kB 1.6 MB/s
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from go-vncdriver) (1.21.4)
Building wheels for collected packages: go-vncdriver
Building wheel for go-vncdriver (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-p2d78mvk/go-vncdriver/setup.py'"'"'; file='"'"'/tmp/pip-install-p2d78mvk/go-vncdriver/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-9r0ymr26
cwd: /tmp/pip-install-p2d78mvk/go-vncdriver/
Complete output (40 lines):
running bdist_wheel
running build
Running new build
Could not build go_vncdriver: Could not find libjpeg. HINT: try 'sudo apt-get install libjpeg-turbo8-dev' on Ubuntu or 'brew install libjpeg-turbo' on OSX
Traceback (most recent call last):
File "/tmp/pip-install-p2d78mvk/go-vncdriver/build.py", line 54, in build
output = subprocess.check_output(['ld', '-ljpeg', '--trace-symbol', 'jpeg_CreateDecompress', '-e', '0'], stderr=subprocess.STDOUT)
File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ld', '-ljpeg', '--trace-symbol', 'jpeg_CreateDecompress', '-e', '0']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-p2d78mvk/go-vncdriver/setup.py", line 67, in
setup(name='go_vncdriver',
File "/usr/lib/python3/dist-packages/setuptools/init.py", line 144, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3/dist-packages/wheel/bdist_wheel.py", line 223, in run
self.run_command('build')
File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/tmp/pip-install-p2d78mvk/go-vncdriver/setup.py", line 23, in run
self.build()
File "/tmp/pip-install-p2d78mvk/go-vncdriver/setup.py", line 61, in build
build.build()
File "/tmp/pip-install-p2d78mvk/go-vncdriver/build.py", line 57, in build
raise BuildException("Could not find libjpeg. HINT: try 'sudo apt-get install libjpeg-turbo8-dev' on Ubuntu or 'brew install libjpeg-turbo' on OSX")
build.BuildException: Could not find libjpeg. HINT: try 'sudo apt-get install libjpeg-turbo8-dev' on Ubuntu or 'brew install libjpeg-turbo' on OSX
ERROR: Failed building wheel for go-vncdriver
Running setup.py clean for go-vncdriver
Failed to build go-vncdriver
Installing collected packages: go-vncdriver
Running setup.py install for go-vncdriver ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-p2d78mvk/go-vncdriver/setup.py'"'"'; file='"'"'/tmp/pip-install-p2d78mvk/go-vncdriver/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-fzpw66ro/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/apps/.local/include/python3.8/go-vncdriver
cwd: /tmp/pip-install-p2d78mvk/go-vncdriver/
Complete output (42 lines):
running install
running build
Running new build
Could not build go_vncdriver: Could not find libjpeg. HINT: try 'sudo apt-get install libjpeg-turbo8-dev' on Ubuntu or 'brew install libjpeg-turbo' on OSX
Traceback (most recent call last):
File "/tmp/pip-install-p2d78mvk/go-vncdriver/build.py", line 54, in build
output = subprocess.check_output(['ld', '-ljpeg', '--trace-symbol', 'jpeg_CreateDecompress', '-e', '0'], stderr=subprocess.STDOUT)
File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ld', '-ljpeg', '--trace-symbol', 'jpeg_CreateDecompress', '-e', '0']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-p2d78mvk/go-vncdriver/setup.py", line 67, in <module>
    setup(name='go_vncdriver',
  File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3/dist-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/usr/lib/python3.8/distutils/command/install.py", line 589, in run
    self.run_command('build')
  File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/tmp/pip-install-p2d78mvk/go-vncdriver/setup.py", line 23, in run
    self.build()
  File "/tmp/pip-install-p2d78mvk/go-vncdriver/setup.py", line 61, in build
    build.build()
  File "/tmp/pip-install-p2d78mvk/go-vncdriver/build.py", line 57, in build
    raise BuildException("Could not find libjpeg. HINT: try 'sudo apt-get install libjpeg-turbo8-dev' on Ubuntu or 'brew install libjpeg-turbo' on OSX")
build.BuildException: Could not find libjpeg. HINT: try 'sudo apt-get install libjpeg-turbo8-dev' on Ubuntu or 'brew install libjpeg-turbo' on OSX
----------------------------------------

ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-p2d78mvk/go-vncdriver/setup.py'"'"'; file='"'"'/tmp/pip-install-p2d78mvk/go-vncdriver/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-fzpw66ro/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/apps/.local/include/python3.8/go-vncdriver Check the logs for full command output.

nvm ignore the error above I was blind

It worked for a second! but then this happened apps@9a226a481524:~/gym_slitherio/envs$ python3 test1.py
WARNING: CPU random generator seem to be failing, disabling hardware random number generation
WARNING: RDRND generated: 0xffffffff 0xffffffff 0xffffffff 0xffffffff
2021/11/12 13:57:34 I1112 13:57:34.787526 2328 gymvnc.go:417] [conn1:localhost:5900] opening connection to VNC server
reset
Traceback (most recent call last):
File "test1.py", line 3, in
obs = env.reset()
File "/home/apps/gym_slitherio/envs/slitherio_env.py", line 106, in reset
observation = np.fromstring(observations['conn1'], dtype=np.uint8, sep='')
TypeError: a bytes-like object is required, not 'NoneType'
apps@9a226a481524:~/gym_slitherio/envs$

@godpow
First of all, why are you trying to run test1.py on a docker container?
The only thing you need to run on the container is remote.py. After running remote.py on your container, try to run 'test1.py' on your local machine.

It popped up entered the name and clicked play moved for a second then broke.

Not sure why you get that.

@godpow
Not sure why you get that. Also, you may want to do a pull request if you think it needs improvements. I guess you got a feel for how it works by now.


Teams fixes merged

@v-hearya,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

patterns-practices/reference/allactiveregion-deactivate-method-mspp-regions.md
:white_check_mark:Succeeded

patterns-practices/reference/allactiveregion-properties-mspp-regions.md
:white_check_mark:Succeeded

patterns-practices/reference/autopopulateregionbehavior-class-mspp-regions-behaviors.md
:white_check_mark:Succeeded

patterns-practices/reference/autopopulateregionbehavior-fields-mspp-regions-behaviors.md
:white_check_mark:Succeeded

patterns-practices/reference/autopopulateregionbehavior-methods-mspp-regions-behaviors.md
:white_check_mark:Succeeded

patterns-practices/reference/autopopulateregionbehavior-properties-mspp-regions-behaviors.md
:white_check_mark:Succeeded

patterns-practices/reference/backgroundeventsubscription-tpayload-class-mspp-pubsubevents.md
:white_check_mark:Succeeded

patterns-practices/reference/backgroundeventsubscription-tpayload-invokeaction-method-mspp-pubsubevents.md
:white_check_mark:Succeeded

patterns-practices/reference/bindregioncontexttodependencyobjectbehavior-methods-mspp-regions-behaviors.md
:white_check_mark:Succeeded

patterns-practices/reference/bootstrapper-class-mspp.md
:white_check_mark:Succeeded

patterns-practices/reference/bootstrapper-createshell-method-mspp.md
:white_check_mark:Succeeded

patterns-practices/reference/bootstrapper-members-mspp.md
:white_check_mark:Succeeded

patterns-practices/reference/bootstrapper-run-method-mspp.md
:white_check_mark:Succeeded

patterns-practices/reference/bootstrapper-shell-property-mspp.md
:white_check_mark:Succeeded

patterns-practices/reference/canexecute-mthd.md
:white_check_mark:Succeeded

patterns-practices/reference/collectionextensions-class-mspp.md
:white_check_mark:Succeeded

patterns-practices/reference/collectionextensions-members-mspp.md
:white_check_mark:Succeeded

patterns-practices/reference/compositecommand-class-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/compositecommand-constructor-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/compositecommand-ctor.md
:white_check_mark:Succeeded

patterns-practices/reference/compositecommand-members-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/compositepresentationevent-tpayload-class-mspp-events.md
:white_check_mark:Succeeded

patterns-practices/reference/compositepresentationevent-tpayload-contains-method-mspp-events.md
:white_check_mark:Succeeded

patterns-practices/reference/compositepresentationevent-tpayload-members-mspp-events.md
:white_check_mark:Succeeded

patterns-practices/reference/compositepresentationevent-tpayload-subscribe-method-mspp-events.md
:white_check_mark:Succeeded

patterns-practices/reference/compositepresentationevent-tpayload-unsubscribe-method-mspp-events.md
:white_check_mark:Succeeded

patterns-practices/reference/confirmation-class-mspp-interactivity-interactionrequest.md
:white_check_mark:Succeeded

patterns-practices/reference/contentcontrolregionadapter-class-mspp-regions.md
:white_check_mark:Succeeded

patterns-practices/reference/contentcontrolregionadapter-members-mspp-regions.md
:white_check_mark:Succeeded

patterns-practices/reference/contentcontrolregionadapter-methods-mspp-regions.md
:white_check_mark:Succeeded

patterns-practices/reference/createnewelement-mthd.md
:white_check_mark:Succeeded

patterns-practices/reference/cyclicdependencyfoundexception-constructor-mspp-modularity.md
:white_check_mark:Succeeded

patterns-practices/reference/dataeventargs-tdata-class-mspp-pubsubevents.md
:white_check_mark:Succeeded

patterns-practices/reference/defaultdispatcher-class-mspp-events.md
:white_check_mark:Succeeded

patterns-practices/reference/defaultdispatcher-members-mspp-events.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommand-class-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommand-constructor-action-func-boolean-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommand-constructor-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommand-fromasynchandler-method-func-task-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommand-members-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommand-t-canexecute-method-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommand-t-class-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommand-t-constructor-action-t-func-t-boolean-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommand-t-constructor-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommand-t-execute-method-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommand-t-members-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommandbase-class-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommandbase-constructor-action-object-func-object-boolean-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommandbase-constructor-func-object-task-func-object-boolean-mspp-commands.md
:white_check_mark:Succeeded

patterns-practices/reference/delegatecommandbase-members-mspp-commands.md
:white_check_mark:Succeeded

This comment lists only the first 50 files in the pull request.
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Portfolio Page (Mobile), click image once, one project per line, alt text

Portfolio Page (Mobile):
The Portfolio page is too small and cluttered on mobile.  I'd change it to display one project per line.
On mobile, the interaction is Click on Portfolio Image -> Get Mouse-Over -> Click Again -> Go to GitHub Page.  I'd change this so you only need to click on the image once.
To do this: Image -> Text Caption -> Text Description/Bullet Points.  Now you no longer need your placeholder image.  You'll probably have to change your css, so that it differentiates between hove and focus/active.  Otherwise, switch your mouse-over to be Javascript, and that should solve the problem.
You can set that text to display: hidden; for desktop, and then have it show up on mobile.  This will also be good for users with screen readers, as they're not going to be able to see your images, and your images don't have any alt text (which I'd recommend putting in descriptive alt text when you have a chance.  I'll even write some for you if you'd like :)

@lunacodes Could to talk about the differences between the JS and hove vs focus/active possibilities?  I know very little JS, but happy to investigate.
I am also have trouble visualizing your recommendation for image/text caption/bullet points.
Will try to catch you soon - just wanted to make a note so I didn't forget/gave you a heads up.

@vzhz yes, definitely ... and my comment should have been hover, as in a:hover vs. a:focus and a:active
But yeah, let's chat about it together soon :)


[Bug Fix] Translate box3d origin before lidar to camera coordinate transform for evaluation

To compute kitti eval result, the 3d boxes on the lidar coordinate need to be transformed to the one on the kitti camera coodinate. On the kitti camera coordinate, the box origin should be [0.5, 1., 0.5].
If a predicted lidar 3d box origin is [0.5, 0.5, 0.5], its origin should be translate to [0.5, 0.5, 0] before the rotation to kitti camera coordinate. We wrongly translates the origin to [0.5, 1., 0.5] after the rotation to kitti camera coordinate, which causes Car@AP_R40(moderate) drop 1.36% and Car@AP_R11(moderate) drop 4.48% for pv_rcnn . So now we adopt the right implementation.

会影响其他模型的精度


Mod not reporting cause of improper configuration

Minecraft 1.12.2, Forge <IP_ADDRESS>47, Block Replacer 2.2.0.
Summary: Mod says "see the logs" but doesn't put any content in.
Trying to replace regular minecraft:water with terrafirmacraft waters.
Currently all logs only say "See the logs to find out what wasn't configured properly."... but none of the logs say anything else.
I suspect that the biome names might not be right but I don't know.

Discovered circumstance: Mod does not tell user that selected replacement block is risky and that the operation has been cancelled.

Version 2.3.0 should fix this.


[Package Request]: Extreme Tux Racer

Package Requested

[X] I would like someone else to build the manifest.
[ ] I would like help so I can submit the manifest.
[X] I have performed a search and couldn't find this package.
[X] I have checked that the installer for this package is not a script file
[ ] I think there is a new version available and I have provided the URL.

Please provide the following information
Publisher: Extreme Tux Racer
Publisher Url: https://extremetuxracer.sourceforge.io/
Package Name: Extreme Tux Racer
Description: Extreme Tux Racer is a game where you help Tux the penguin race down the mountain.
Package Version: 0.8.2
Installer URL: https://sourceforge.net/projects/extremetuxracer/files/latest/download
Current Package Identifier (for new versions): N/A

Sorry, already added


Release/v0.1.1 beta.4

Maybe we havn't finished the milestone https://github.com/PaddlePaddle/cloud/milestone/1, so maybe this is released by v0.1.1-beta.4rc1 ?

@gongweibao can you please close this and open up a new release branch for latest updates?

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 7 committers have signed the CLA.:x: wuyi05:x: helinwang:x: typhoonzero:x: gongweibao:x: Yancey1989:x: tizhou86:x: wanghaoshuangwuyi05 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.2 out of 7 committers have signed the CLA.:white_check_mark: gongweibao:white_check_mark: wanghaoshuang:x: wuyi05:x: Yancey1989:x: typhoonzero:x: tizhou86:x: helinwangwuyi05 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Add security

This PR aims at adding security to the webserver with OAuth2 specification in order to make it safe (only the fleet of pi camera and central pi will be able to use routes)
I has been implemented with the help of the dedicated Fastapi documentation.
Features
Below are the main steps :

Add security using oauth2 with jwt tokens and password hashing
Add login route to retrieve token
Add auth requirements to inference and metrics routes
Updates unit test using dependency_overrides. (test to properly check security flow has not been implemented, maybe to add later )
Add new schemas related to not new workflow, and updates metricIn removing Id as long as device will be detected by auth

Usage

To date, registered devices/users are stored in a json file (fleet_accesses) which (a bit as an env file)

The file looks like:
{
    "camera_1": {
        "username": "camera_1",
        "hashed_password": "$2b$12$EixZaYVK1fsbw1ZfbX3OXePaWxn96p36WQoeG6Lruj3vjPGga31lW",
        "disabled": false
    },
    "michel": {
        "username": "michel",
        "hashed_password": "$2b$12$EixZaYVK1fsbw1ZfbX3OXePaWxn96p36WQoeG6Lruj3vjPGga31lW",
        "disabled": false
    }
}

(to generate the hash of the password, us get_password_hash from security.py  )

Hence, to use the webserver with ease, a pi camera/device need to retrieve its token using login route and then add it to headers of routes that require auth (all others)

Any feedback is welcome :)

Thanks a lot for the PR!
Generally speaking, I'm not getting why we take the current active user?
Considering the usage of this repo and this webserver, we are only dealing with devices :) I think the PR should be updated accordingly before moving on to review

Indeed, users of the webserver, are here, devices. I will change typo accordingly to make it clearer!

Hello @frgfm thanks for the feedbacks !
Isn’t just a webserver based on fastapi handling security using jwt token ? Looks like pyro-api indeed, a similar architecture male it easier to maintain ?
also, I agree with issues before implementations but first it has been discussed orally, and secondly seeing the success of others issue I directly dive into implementation.
However I would be happy to discuss the suitable solution for this webserver .

Secret key has been removed, and the secret key will be loaded from env file.
Will see soon to handle it via Github Secrets

@fe51 I can add the github secret if you need, contact me via slack :)

All good, thanks :)


Callout styling

Related to issue: https://devtopia.esri.com/runtime/kotlin/issues/3937
Description:
PR to add the CalloutTheme.kt and DefaultThemeTokens.kt to enable custom styling on the Callout.
Test Code:
Screenshot:

import androidx.compose.foundation.layout.PaddingValues
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.padding
import androidx.compose.material3.Scaffold
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.remember
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.unit.DpSize
import androidx.compose.ui.unit.dp
import com.arcgismaps.toolkit.geoviewcompose.MapView
import com.arcgismaps.toolkit.geoviewcompose.theme.CalloutDefaults

@Composable
fun GraphicScreen(mapViewModel: MapViewModel) {

    val tappedPoint = mapViewModel.mapPoint.collectAsState().value
    val calloutTheme = CalloutDefaults.colors(
        backgroundColor = Color.LightGray,
        borderColor = Color.Red
    )

    val calloutShape = CalloutDefaults.shapes(
        cornerRadius = 0.dp,
        leaderSize = DpSize(25.dp, 20.dp),
        borderWidth = 5.dp,
        calloutContentPadding = PaddingValues(25.dp),
        minSize = DpSize(100.dp, 100.dp)
    )

    Scaffold { contentPadding ->
        MapView(
            modifier = Modifier.fillMaxSize().padding(contentPadding),
            arcGISMap = mapViewModel.arcGISMap,
            graphicsOverlays = remember { listOf(mapViewModel.tapLocationGraphicsOverlay) },
            onSingleTapConfirmed = mapViewModel::setMapPoint
        ) {
            if (tappedPoint != null) {
                Callout(
                    location = tappedPoint,
                    colorScheme = calloutTheme,
                    shape = calloutShape
                ) {
                    Text(text = "HELLO WORLD")
                }
            }
        }
    }
}

Pre-merge Checklist

a vTest Job for this PR has been run

[ ] link:

Unit and/or integration tests have been added to exercise this PR's logic, and the tests are passing:

[ ] Yes
[ ] No

@sorenoid Thanks for the feedback, I've updated the PR with the chang requests and the description test code. Ready for your review.

@kaushikrw Ready for your re-review. I added the missing doc, removed the use of CalloutTheme & composition local, and changed the shape param to "shapes".

Ah sorry, thanks for pointing it out. Ready for your review @kaushikrw


Transpose does not work properly

The transpose function throws an error if there are more rows than columns in the matrix.

Fixed by using the first row of the matrix as an argument for mapIndex rather than the matrix itself.


added comments for csv file paths

Added the comments for csv file paths

Codecov Report

Merging #452 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #452   +/-   ##
=======================================
  Coverage   87.01%   87.01%
=======================================
  Files          12       12
  Lines         562      562
  Branches      136      136
=======================================
  Hits          489      489
  Misses         73       73

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7997c3c...b766f3b. Read the comment docs.


refactor: save duplicate resources with suffix

Changes

No longer replace duplicate resources, but save to files with suffixes.

Screenshots

Checklist

[x] tested locally
[ ] added new dependencies
[ ] updated the docs
[ ] added a test

wondering if we could suggest different filenames based on some other algorithm - for example add the resource-kind - before resorting to numbers!? i.e. if Monokle detects multiple objects with the same name it automatically attempts to generate filenames  with  name-kind.yaml !?

@olensmar - great suggestion. We can definitely try to do that.

@olensmar - We are now generating filenames including the kind. PR is ready to be reviewed.


Mwpw 154534 general fixes

Describe your specific features or fixes
Resolves: MWPW-NUMBER
Test URLs:

Pre Migration Link: https://adobe.com/express/
Before: https://main--express-milo--adobecom.hlx.page/express/
After: https://MWPW-154534-general-fixes--express-milo--adobecom.hlx.page/express/

Codecov Report
Attention: Patch coverage is 64.10256% with 14 lines in your changes missing coverage. Please review.

Project coverage is 66.30%. Comparing base (6b41515) to head (e67b6b5).
Report is 8 commits behind head on main.

Files with missing lines
Patch %
Lines

express/scripts/utils.js
53.33%
14 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #81      +/-   ##
==========================================
+ Coverage   65.58%   66.30%   +0.71%
==========================================
  Files          56       57       +1
  Lines        9163     9241      +78
==========================================
+ Hits         6010     6127     +117
+ Misses       3153     3114      -39

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Call Naxam Place Picker by Xamarin.Forms DI

Is there any example to show how to call Naxam Place Picker by Xamarin.Forms Dependency Injection?

Just added to the latest commit. Plz take a look on Demo project for Xamarin :). Enjoy!


Typo

Not sure if this is the right place for this, but there is a typo in the Readme.
npm install --save statistcal-js is missing the second i in statistical.
npm install --save statistical-js fixes it. Only really commenting as it prevents copy/pasting the command.

Thanks for this comment i'll change it really quick ;)


Proxy setup.

Hello!
When connecting to a proxy, my IP does not change. I am using ProxyMesh. When I try on my machine via OS Setting, connection by HTTPS are working fine. Does Crawly support HTTPS? Can it be a problem of the issue?
Here my config file:
use Mix.Config
# in config.exs
config :crawly,
  proxy: "us-ca.proxymesh.com:31280",
  closespider_timeout: 10,
  concurrent_requests_per_domain: 7,
  closespider_itemcount: 1000,
  middlewares: [
    Crawly.Middlewares.DomainFilter,
    Crawly.Middlewares.UniqueRequest,
    Crawly.Middlewares.UserAgent
  ],
  pipelines: [
    {Crawly.Pipelines.Validate, fields: [:url]},
    # {Crawly.Pipelines.DuplicatesFilter, item_id: :title},
    Crawly.Pipelines.JSONEncoder,
    {Crawly.Pipelines.WriteToFile, extension: "jl", folder: "/tmp"} # NEW IN 0.7.0
   ],
   port: 4001

To check an IPs of processes, I used this small module:
defmodule Spider.Proxy do
  @behaviour Crawly.Spider

  require Logger

  @impl Crawly.Spider
  def base_url(), do: "https://whatismyipaddress.com/"

  @impl Crawly.Spider
  def init() do
    [
      start_urls: [
        "https://whatismyipaddress.com/"
      ]
    ]
  end

  @impl Crawly.Spider
  def parse_item(response) do
    item = %{
      url:
        Floki.find(
          response.body,
          "div#section_left > div:nth-of-type(2) > div:nth-of-type(1) > a"
        )
        |> Floki.text()
    }

    %Crawly.ParsedItem{:items => [item], :requests => []}
  end
end

You will need to add in request options for HTTPoison through the RequestOptions request module. The format needs to match the option format in the HTTPoison docs
https://hexdocs.pm/crawly/Crawly.Middlewares.RequestOptions.html
https://hexdocs.pm/httpoison/HTTPoison.Request.html#content

@Unumus did you search through the docs before opening this issue? If you did but didnt' find the RequestOptions middleware, we might need to update the docs for it

@Ziinc I was also thinking about making an explicit example somewhere in docs...

@Ziinc, I searched. But not found. I think it will be helpful to mention here: https://hexdocs.pm/crawly/configuration.html#proxy-binary the middlewares to complete proxy configuration.

@Unumus ah I see, the docs got out of sync, the proxy setting on :config is from an older version that is a breaking change. We'll need to update that.
@oltarasenko I think perhaps a few common use case examples can be shown in the middleware module doc?  Besides proxy, what others would be good?

[ ] remove proxy setting from config.

@oltarasenko You might need to do a patch release to update hexdocs


Adjust camera height

Camera height is too low, at the level of the mouth or chin.
Let's align it with the avatars' eyes.

Duplicate


tasks 1331-1333

@aarunku5 thanks for the review. working on the changes. Also, thanks for the suggestion - will add these tasks as well.

@Ravsehajsinghpuri, can you add 3 new fields as the error message says so that tests pass?

@swarooprm which fields? the error message says something about the in input language.


Need to export more controls for panels/UI components

Hello! I have a somewhat generic question with specific examples. Apologies if it is too obvious, since I am new to UI development.
Background:
In several places it seems that the only non-hacky way to update a certain style option for a UI component would be to make a child class of that component, create a new renderer with new CSS and then use that new class. Would it be possible expose a way to more easily override style? Or maybe I am missing something - what is the recommended way for overriding the CSS styling of a container within a Scenes component?
Examples:

I would like to remove the header padding of a stat/gauge when there is no title set. Otherwise it seems to take up white space. I found the PanelTimeRange class mimics the "Hide Time Info" setting in Grafana. Could such functionality be exposed as part of a package?

A div element with class="css-<....>-page-content" is generated for my main SceneAppPage. I want to change the background-color of this div element in particular. This is what the style for my main page looks like:

const getStyles = (theme: GrafanaTheme2) => ({  container: css    display: flex;    flex-grow: 1;    height: 100%;    , });

I have a VariableValueSelector dropdown. I want the dropdown list's boxes to grow around the text instead of cutting off long strings. I believe this component is using the react-select-2-listbox. How could I override the styling to set the field size to stretch?

@kmaharshi213

A div element with class="css-<....>-page-content" is generated for my main SceneAppPage. I want to change the background-color of this div element in particular. This is what the style for my main page looks like:

this would break the design system so we do not allow that.

I have a VariableValueSelector dropdown. I want the dropdown list's boxes to grow around the text instead of cutting off long strings

This Select component is in the main Grafana repo, we are happy to take contributions.
You could try to disable virtualized list, it could help with the auto sizing but not sure

@kmaharshi213

I would like to remove the header padding of a stat/gauge when there is no title set

set hoverHeader to true, then there will not be any header space

this would break the design system so we do not allow that.

Thanks @torkelo! Those comments really help. And fair, that makes sense.
Do you happen to then have any tips or tricks for how I could fix this issue below? The background-color issue is small but really breaks the visual experience:
Basically, we have a long page with a bunch of panels. However, it looks like the main SceneAppPage has a fixed height for the background and a fixed color. So it shows up like so behind some of the panels.

How can make the "Scene App Page Background" blend in OR make it flexibly adapt and grow to the overall panels' height?
Thanks a lot again!

Looking for a resolution for this same issue

@torkelo if you could take another look at this I would be appreciative


POL-118 Update region datasources to use github data list

Updated region datasources to use github data list
Description
Identified AWS policies that used  hard coded list of regions and replaced it with the region list file in github.
Issues Resolved
Referring Regions from github and removed datasource which we were not using internally.
Contribution Check List

[ ] All tests pass.
[ ] New functionality includes testing.
[ ] New functionality has been documented in the README if applicable
[ ] New functionality has been documented in CHANGELOG.MD

          1 Warning

:warning:
Should this include readme changes

  Generated by :no_entry_sign: Danger


Render data as blog post

Hi,
Maybe it's only me but I couldn't find any example of rendering the data stored with Sir Trevor.
Once you have finish writing your post and want to publish it, do you need to do a custom method or is there something already implemented to render the data ?

@CptMaumau are you submitting the data with a form submission or javascript?

@raffij I submit the data with a form submission and store it in a mongo DB.

@CptMaumau sir-trevor will bind to the form submit function. Place the json in the textarea and then submit the form.
Are you seeing something different?

@raffij storing the data is fine. What I want is render the data in a view so it looks like a blog post.
Maybe there's a read-only version of Sir Trevor?
Sorry if my question was not understandable.

@CptMaumau Sir Trevor does not handle rendering, only an editing interface.

In my opinion Sir Trevor already handles rendering since it transforms the data object into html tags for editing.
A method that only does that without the editing part would be useful.

@CptMaumau you are partially correct. A little bit of history for you.
Sir Trevor is built to be display agnostic. Attempting to totally detach the editing / storage and display of the data. However we need a language to store text formatting.
Originally we used markdown as it's easy to parse and widely adopted. However working with markdown became troublesome so we moved to simple formatted HTML as you find today. So you should think of the HTML as a storage solution rather than necessarily the display language.
For rendering sirtrevor content you'll want to write your own logic. We have a rails gem which you can take inspiration from. There will be others you can find on GitHub to help you in your language of choice. For instance I once did a Wordpress renderer, not public. Basically what you'll be doing is parsing the Json and outputing each block to a template file.
The rendering is not part of sir-Trevor and never will be, but we will endeavour to add further rendering plugins as links to the repo as and when we create them and would encourage others to do the same.
If you find any then let us know and we'll add them.
If you have problems please create a new issue as I think this is a good one to archive for others.

Thank you for taking the time to answer me. I will follow your advice and create my own rendering logic.

What language are you using?

@CptMaumau: The implementations links in the readme might be a good place to seek inspiration/solutions. There are implementations for Rails, Umbraco, Django and several PHP CMSs.

@raffij Server side is Node JS/Express with Vash for view engine

It's not that hard to understand the rendering logic, you just need to have something to digest the JSON and format it as you please.
For me, once I receive the JSON I convert it directly into HTML (for use with bootstrap) so that I don't have to compile the HTML each time the page is loaded.


[Event Request] Codeunit 1201 Process Data Exch.

The following events are required for this object:
    [IntegrationEvent(false, false)]
    local procedure OnDataExchFieldMappingSetFiltersBeforeSetCurrentKey(var DataExchFieldMapping: Record "Data Exch. Field Mapping"; var IsHandled: Boolean)
    begin
    end;

    procedure ProcessColumnMapping(DataExch: Record "Data Exch."; DataExchLineDef: Record "Data Exch. Line Def"; RecRefTemplate: RecordRef)
    var
        DataExchField: Record "Data Exch. Field";
        DataExchFieldGroupByLineNo: Record "Data Exch. Field";
        DataExchMapping: Record "Data Exch. Mapping";
        DataExchFieldMapping: Record "Data Exch. Field Mapping";
        TempFieldIdsToNegate: Record "Integer" temporary;
        RecRef: RecordRef;
        LastKeyFieldId: Integer;
        DataExchLineNoFieldId: Integer;
        LineNoOffset: Integer;
        CurrLineNo: Integer;
        IsHandled: Boolean;
    begin
        LastKeyFieldId := GetLastIntegerKeyField(RecRefTemplate);
        LineNoOffset := GetLastKeyValueInRange(RecRefTemplate, LastKeyFieldId);
        DataExchLineNoFieldId := GetDataExchLineNoFieldId(RecRefTemplate.Number);

        DataExchMapping.Get(DataExch."Data Exch. Def Code", DataExchLineDef.Code, RecRefTemplate.Number);

        OnBeforeDataExchFieldMappingSetFilters(DataExchFieldMapping, DataExch);

        DataExchFieldMapping.SetRange("Data Exch. Def Code", DataExch."Data Exch. Def Code");
        DataExchFieldMapping.SetRange("Data Exch. Line Def Code", DataExchLineDef.Code);
        DataExchFieldMapping.SetRange("Table ID", RecRefTemplate.Number);
        DataExchFieldMapping.SetFilter(Priority, '<>%1', 0);
        //EVENT REQUEST
        IsHandled := false;
        OnDataExchFieldMappingSetFiltersBeforeSetCurrentKey(DataExchFieldMapping, IsHandled);
        if not IsHandled then
            if not DataExchFieldMapping.IsEmpty() then begin
                DataExchFieldMapping.SetCurrentKey("Data Exch. Def Code", "Data Exch. Line Def Code", "Table ID", Priority);
                DataExchFieldMapping.Ascending(false);
            end;
        //EVENT REQUEST

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 12399.